{"id":1831,"date":"2025-10-17T13:55:42","date_gmt":"2025-10-17T10:55:42","guid":{"rendered":"https:\/\/upcloud.com\/global\/us\/resources\/tutorials\/monitoring-with-prometheus-part-4\/"},"modified":"2025-10-17T13:55:42","modified_gmt":"2025-10-17T10:55:42","slug":"monitoring-with-prometheus-part-4","status":"publish","type":"tutorial","link":"https:\/\/upcloud.com\/global\/resources\/tutorials\/monitoring-with-prometheus-part-4\/","title":{"rendered":"Monitoring on UpCloud with Prometheus: Part 4"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">After setting up the foundation with infrastructure-level monitoring in earlier parts of this series (tracking Kubernetes workloads, VM metrics, and visualizing them in Grafana), you have a solid understanding of system health and resource usage monitoring. But infrastructure metrics only tell part of the story. To truly understand your applications&#8217; behavior, you need to observe them from the inside.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This is where application-level monitoring comes in. You can track key business and operational metrics like request latency, error rates, memory consumption, and custom KPIs unique to your domain by instrumenting your applications with Prometheus client libraries.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">In this article, you\u2019ll learn how to use Prometheus client libraries in <a href=\"https:\/\/upcloud.com\/global\/resources\/tutorials\/deploying-a-node-js-application-to-upcloud-from-gitlab-ci-cd\/\">Node.js<\/a> and Python applications, expose \/metrics endpoints, and configure Prometheus to scrape them. You\u2019ll also explore how to validate the setup, write useful PromQL queries, and create dashboards in Grafana to visualize your app\u2019s real-time performance.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">To see the previous parts of the series you can access here: <\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><a href=\"https:\/\/upcloud.com\/global\/resources\/tutorials\/monitoring-upcloud-prometheus-part-1\/\">Monitoring in UpCloud with Prometheus: Part 1<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/upcloud.com\/global\/resources\/tutorials\/monitoring-upcloud-prometheus-part-2\/\">Monitoring in UpCloud with Prometheus: Part 2<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/upcloud.com\/global\/resources\/tutorials\/monitoring-upcloud-prometheus-part-3\/\">Monitoring in UpCloud with Prometheus: Part 3<\/a><\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Prometheus Client Libraries<\/strong><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Prometheus offers a list of official and community-backed <a href=\"https:\/\/prometheus.io\/docs\/instrumenting\/clientlibs\/\" target=\"_blank\" rel=\"noopener\">client libraries<\/a> that allow you to instrument your applications by exposing internal metrics in a format that Prometheus understands. These libraries are available for most major languages, including:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Python<\/strong>: <a href=\"https:\/\/github.com\/prometheus\/client_python\" target=\"_blank\" rel=\"noopener\">client_python<\/a><\/li>\n\n\n\n<li><strong>Go<\/strong>: <a href=\"https:\/\/github.com\/prometheus\/client_golang\" target=\"_blank\" rel=\"noopener\">client_golang<\/a><\/li>\n\n\n\n<li><strong>Java<\/strong>: <a href=\"https:\/\/github.com\/prometheus\/client_java\" target=\"_blank\" rel=\"noopener\">client_java<\/a><\/li>\n\n\n\n<li><strong>Rust<\/strong>: <a href=\"https:\/\/github.com\/prometheus\/client_rust\" target=\"_blank\" rel=\"noopener\">client_rust<\/a><\/li>\n\n\n\n<li><strong>Ruby<\/strong>: <a href=\"https:\/\/github.com\/prometheus\/client_ruby\" target=\"_blank\" rel=\"noopener\">client_ruby<\/a><\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">A longer list of libraries is available via third-party or community support, which includes<br><a href=\"https:\/\/github.com\/siimon\/prom-client\" target=\"_blank\" rel=\"noopener\">Node.js<\/a>, <a href=\"https:\/\/github.com\/jupp0r\/prometheus-cpp\" target=\"_blank\" rel=\"noopener\">C++<\/a>, and more.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">These libraries expose an HTTP endpoint (usually \/metrics) that Prometheus scrapes at regular intervals.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Built-in Metrics Provided by Most Libraries<\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Prometheus client libraries offer several default metric types, and many include common process-level collectors by default. These include:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>process_cpu_seconds_total: total user and system CPU time<\/li>\n\n\n\n<li>process_resident_memory_bytes: memory used by the process<\/li>\n\n\n\n<li>http_requests_total: number of HTTP requests (with labels like status code and method)<\/li>\n\n\n\n<li>http_request_duration_seconds: request latency histogram<\/li>\n\n\n\n<li>Garbage collection stats (for languages with managed memory like Java, Node.js, Python)<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">On top of these, you can also define <strong>custom metrics<\/strong> specific to your business or application. Some common custom metrics include:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Orders processed<\/li>\n\n\n\n<li>Users signed up<\/li>\n\n\n\n<li>Background jobs queued or completed, etc.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">Next up, you will learn how to instrument two basic apps with the Prometheus client libraries, deploy them onto your UpCloud cluster, and view their metrics in your Grafana dashboards!<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Prerequisites<\/strong><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Before you can go ahead with the tutorial, you will need to ensure that you have the following:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>An existing Prometheus + Grafana setup on an <a href=\"https:\/\/upcloud.com\/global\/products\/managed-kubernetes\/\">UpCloud Managed Kubernetes<\/a> cluster (from earlier parts of this series).<\/li>\n\n\n\n<li>Basic knowledge of Node.js and Python.<\/li>\n\n\n\n<li>kubectl and helm installed and configured correctly<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Instrumenting a Node.js Application<\/strong><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">To expose application-level metrics from a Node.js service, you\u2019ll use the third-party Prometheus client library: <a href=\"https:\/\/github.com\/siimon\/prom-client\" target=\"_blank\" rel=\"noopener\">prom-client<\/a>. It supports a variety of metric types: counters, gauges, histograms, and summaries. And, it provides built-in process and garbage collection metrics.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Step 1: Install <\/strong><strong>prom-client<\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">In your Node.js project, install the package:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">npm install prom-client<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Step 2: Create a <\/strong><strong>\/metrics<\/strong><strong> Endpoint<\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">In your Express (or other HTTP framework) application, set up a \/metrics route that exposes Prometheus-compatible metrics:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">const express = require('express');\nconst client = require('prom-client');\n\nconst app = express();\nconst port = 3000;\n\n\/\/ Enable default metrics collection\nclient.collectDefaultMetrics();\n\n\/\/ Example custom metric\nconst httpRequestCounter = new client.Counter({\n  name: 'http_requests_total',\n  help: 'Total number of HTTP requests',\n  labelNames: ['method', 'route', 'status']\n});\n\n\/\/ Middleware to increment counter on each request\napp.use((req, res, next) =&gt; {\n  res.on('finish', () =&gt; {\n    httpRequestCounter.labels(req.method, req.path, res.statusCode).inc();\n  });\n  next();\n});\n\n\/\/ Your application routes here\napp.get('\/', (req, res) =&gt; {\n  res.send('Hello, world!');\n});\n\n\/\/ Metrics endpoint\napp.get('\/metrics', async (req, res) =&gt; {\n  res.set('Content-Type', client.register.contentType);\n  res.end(await client.register.metrics());\n});\n\napp.listen(port, () =&gt; {\n  console.log(`App listening at http:\/\/localhost:${port}`);\n});\n<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Step 3: Run the App Locally<\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Start your application:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">node app.js<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Navigate to <code>http:\/\/localhost:3000\/metrics<\/code>, and you should see Prometheus-formatted metrics including memory usage, CPU time, and your custom http_requests_total counter.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Next, we\u2019ll walk through the same process for a Python application using the prometheus_client library.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Instrumenting a Python Application<\/strong><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">In Python, the official Prometheus client library is <a href=\"https:\/\/github.com\/prometheus\/client_python\" target=\"_blank\" rel=\"noopener\">prometheus_client<\/a>. It makes it easy to expose standard and custom metrics via an HTTP endpoint that Prometheus can scrape.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Step 1: Install <\/strong><strong>prometheus_client<\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Install the library using pip:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">pip install prometheus_client<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Step 2: Create a Basic App With Metrics<\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Here\u2019s a simple Flask app that exposes a \/metrics endpoint and tracks request counts and latencies:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">Step 2: Create a Basic App With Metrics\nHere\u2019s a simple Flask app that exposes a \/metrics endpoint and tracks request counts and latencies:\nfrom flask import Flask, Response, request\nfrom prometheus_client import Counter, Histogram, generate_latest, CONTENT_TYPE_LATEST\nimport time\n\napp = Flask(__name__)\n\n# Define metrics\nREQUEST_COUNT = Counter('http_requests_total', 'Total HTTP requests', ['method', 'endpoint', 'status_code'])\nREQUEST_LATENCY = Histogram('http_request_duration_seconds', 'Request latency', ['endpoint'])\n@app.before_request\ndef start_timer():\n    request.start_time = time.time()\n\n@app.after_request\ndef record_metrics(response):\n    latency = time.time() - request.start_time\n    REQUEST_COUNT.labels(request.method, request.path, response.status_code).inc()\n    REQUEST_LATENCY.labels(request.path).observe(latency)\n    return response\n\n@app.route('\/')\ndef index():\n    return 'Hello, world!'\n\n@app.route('\/metrics')\ndef metrics():\n    return Response(generate_latest(), mimetype=CONTENT_TYPE_LATEST)\n\nif __name__ == '__main__':\n    app.run(host='0.0.0.0', port=8000)\n\n<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Step 3: Run the App Locally<\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Start the Flask app:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">python app.py<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Now visit <code>http:\/\/localhost:8000\/metrics<\/code> to see the exposed Prometheus metrics. These will include:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Default process metrics (memory usage, CPU time, etc.)<\/li>\n\n\n\n<li>Your custom metrics:\n<ul class=\"wp-block-list\">\n<li>http_requests_total<\/li>\n\n\n\n<li>http_request_duration_seconds<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">In the next section, we\u2019ll configure Prometheus to scrape these application endpoints so that you can analyze and visualize the metrics.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Deploying the Apps<\/strong><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">To collect metrics from your instrumented applications in a Kubernetes environment, you first need to containerize the apps and deploy them to your cluster.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Step 1: Dockerize the Applications<\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Create a Dockerfile in the root of your Node.js project:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\"># Use official Node.js image\nFROM node:24-alpine\n\nWORKDIR \/app\n\nCOPY package*.json .\/\nRUN npm install\n\nCOPY . .\n\nEXPOSE 3000\nCMD [\"node\", \"app.js\"]\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Similarly, create a Dockerfile for the Python Flask app:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">FROM python:3.13-slim\n\nWORKDIR \/app\n\nCOPY requirements.txt .\/\nRUN pip install -r requirements.txt\n\nCOPY . .\n\nEXPOSE 8000\nCMD [\"python\", \"app.py\"]\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\"><em>Make sure you create a requirements.txt file that includes flask and prometheus_client.<\/em><\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Step 2: Deploy Applications to Kubernetes<\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">With the Dockerfiles in place, build and push the images to a container registry (like Docker Hub or GitHub Container Registry). To do that, run the following commands in the Node.js project directory, replacing &lt;your-dockerhub-username&gt; with your actual Docker Hub username:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">docker build -t &lt;your-dockerhub-username&gt;\/nodejs-app:latest .\/nodejs-app\ndocker push &lt;your-dockerhub-username&gt;\/nodejs-app:latest<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Similarly, run the following commands in the Python project directory, replacing &lt;your-dockerhub-username&gt; with your actual Docker Hub username:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">docker build -t &lt;your-dockerhub-username&gt;\/python-app:latest .\/python-app\ndocker push &lt;your-dockerhub-username&gt;\/python-app:latest<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\"><em>If you are using an ARM architecture machine (such as a Mac with an Apple Silicon chip), you will need to use Docker Buildx to ensure that the Docker image is built for linux\/amd64, not ARM:<\/em><br><\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">docker buildx build --platform linux\/amd64 -t &lt;your-dockerhub-username&gt;\/nodejs-app:latest . --push<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Once the images are pushed to Dockerhub, you need to define Kubernetes manifests for both applications.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Here\u2019s what the manifest for the Node.js app will look like:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\"># nodejs-app.yaml\napiVersion: apps\/v1\nkind: Deployment\nmetadata:\n  name: nodejs-app\nspec:\n  replicas: 1\n  selector:\n    matchLabels:\n      app: nodejs-app\n  template:\n    metadata:\n      labels:\n        app: nodejs-app\n    spec:\n      containers:\n        - name: nodejs-app\n          image: &lt;your-dockerhub-username&gt;\/nodejs-app:latest\n          ports:\n            - containerPort: 3000\n          readinessProbe:\n            httpGet:\n              path: \/metrics\n              port: 3000\n\n---\napiVersion: v1\nkind: Service\nmetadata:\n  name: nodejs-app\n  labels:\n    app: nodejs-app\nspec:\n  selector:\n    app: nodejs-app\n  ports:\n    - name: http\n      port: 3000\n      targetPort: 3000<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">This manifest deploys a single-replica Node.js application to Kubernetes and exposes its \/metrics endpoint on port 3000 using a readiness probe. The associated Service makes the app accessible internally under the name nodejs-app, with a named port http, which is needed for the Prometheus ServiceMonitor to correctly identify and scrape it.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Here\u2019s what the manifest for the Python app will look like:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\"># python-app.yaml\napiVersion: apps\/v1\nkind: Deployment\nmetadata:\n  name: python-app\nspec:\n  replicas: 1\n  selector:\n    matchLabels:\n      app: python-app\n  template:\n    metadata:\n      labels:\n        app: python-app\n    spec:\n      containers:\n        - name: python-app\n          image: &lt;your-dockerhub-username&gt;\/python-app:latest\n          ports:\n            - containerPort: 8000\n          readinessProbe:\n            httpGet:\n              path: \/metrics\n              port: 8000\n\n---\napiVersion: v1\nkind: Service\nmetadata:\n  name: python-app\n  labels:\n    app: python-app\nspec:\n  selector:\n    app: python-app\n  ports:\n    - name: http\n      port: 8000\n      targetPort: 8000<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">This manifest deploys a single-replica Python application exposing Prometheus-compatible metrics on port 8000. Like the Node.js setup, it defines a readiness probe targeting \/metrics and exposes the app through a Kubernetes Service.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><em>Make sure to replace <\/em><em>&lt;your-dockerhub-username&gt;<\/em><em> with the correct value before saving both of the files.<\/em><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Once you have created both of these files, apply them to your UpCloud cluster by running the following commands:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">kubectl apply -f nodejs-app.yaml\nkubectl apply -f python-app.yaml<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">You can now run kubectl get svc to see if both of these apps have been deployed correctly on your cluster:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">$ p3 kubectl get svc                                           \nNAME         TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE\nkubernetes   ClusterIP   10.96.0.1      &lt;none&gt;        443\/TCP    22m\nnodejs-app   ClusterIP   10.96.203.53   &lt;none&gt;        3000\/TCP   5m12s\npython-app   ClusterIP   10.96.73.42    &lt;none&gt;        8000\/TCP   116s<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">This means that the apps are deployed correctly on your cluster! You could also try running kubectl port-forward on the services to access and verify that the \/metrics endpoint is up and running on both of the apps.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Configuring Prometheus to Scrape Application Endpoints<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Once your instrumented Node.js and Python applications are deployed in your Kubernetes cluster, all you need to do is instruct Prometheus to discover and scrape their \/metrics endpoints. In a standalone Prometheus setup, you might use a static list of targets. But in Kubernetes, especially when using the kube-prometheus-stack, the preferred and scalable way to do this is via ServiceMonitor resources.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">A ServiceMonitor is a custom resource provided by the Prometheus Operator. It defines:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>What services to scrape,<\/li>\n\n\n\n<li>On which ports and paths, and<\/li>\n\n\n\n<li>At what intervals.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">The Prometheus instance deployed by kube-prometheus-stack does not scrape arbitrary Kubernetes services by default. Instead, it watches for ServiceMonitor resources that match its configured label selectors. This ensures Prometheus only scrapes explicitly defined and authorized targets, improving both security and maintainability.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">You can use the ServiceMonitor configurations given below for both of your apps:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">For the Node.js app, save the following in a file called <strong>nodejs-servicemonitor.yaml<\/strong>:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">apiVersion: monitoring.coreos.com\/v1\nkind: ServiceMonitor\nmetadata:\n  name: nodejs-app\n  namespace: monitoring\n  labels:\n    release: prometheus-stack\nspec:\n  selector:\n    matchLabels:\n      app: nodejs-app\n  namespaceSelector:\n    matchNames:\n      - default\n  endpoints:\n    - port: http\n      path: \/metrics\n      interval: 15s<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">For the Python app, save the following in a file called <strong>python-servicemonitor.yaml<\/strong>:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">apiVersion: monitoring.coreos.com\/v1\nkind: ServiceMonitor\nmetadata:\n  name: python-app\n  namespace: monitoring\n  labels:\n    release: prometheus-stack\nspec:\n  selector:\n    matchLabels:\n      app: python-app\n  namespaceSelector:\n    matchNames:\n      - default\n  endpoints:\n    - port: http\n      path: \/metrics\n      interval: 15s<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Here is a quick summary of the important nodes in each ServiceMonitor resource, and what they control:<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>metadata.namespace:<\/strong> monitoring: This must match the namespace where your Prometheus instance is deployed. The Prometheus Operator only watches for ServiceMonitor resources in its own namespace (unless explicitly configured otherwise).<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>metadata.labels.release: prometheus-stack:<\/strong> This label must match the label selector used by your Prometheus resource. The kube-prometheus-stack chart defaults to release: prometheus-stack, but you can confirm this by inspecting your Prometheus custom resource:<br>kubectl get prometheus -n monitoring -o yaml | grep selector<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>spec.selector.matchLabels.app: nodejs-app \/ python-app: <\/strong>This must match the labels on the <strong>Kubernetes Service<\/strong>, not the pod or deployment. For example:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">kind: Service<br>metadata:<br>&nbsp; labels:<br>&nbsp; &nbsp; app: nodejs-app<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>spec.namespaceSelector.matchNames: [default]:<\/strong> Specifies the namespace where the target Service lives. In this setup, both apps run in the default namespace, while Prometheus and the ServiceMonitor live in monitoring.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>spec.endpoints.port: http:<\/strong> This must match the <strong>named<\/strong> port in your Service definition. If your service defines:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">ports:<br>&nbsp; &#8211; name: http<br>&nbsp; &nbsp; port: 3000<br>then port: http is correct. If the port name is different (or missing), Prometheus won\u2019t know which port to scrape.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>spec.endpoints.path:<\/strong> \/metrics: Defines the HTTP path where your application exposes metrics. This should match your app\u2019s implementation (e.g., Flask or Express route).<\/li>\n\n\n\n<li>spec.endpoints.interval: 15s: Controls how frequently Prometheus scrapes the endpoint. Use a smaller interval (e.g., 15s) for low-latency, high-resolution metrics, and increase it if scrape load becomes an issue.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">Once both ServiceMonitor resources are deployed, Prometheus will automatically discover and begin scraping your app metrics.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Verifying and Querying Application Metrics<\/strong><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Now, it\u2019s time to validate that Prometheus is successfully scraping the apps\u2019 metrics. To do that, port-forward the Prometheus service so you can access it locally:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">kubectl port-forward svc\/prometheus-stack-kube-prom-prometheus 9090 -n monitoring<\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Then open your browser to <code>http:\/\/localhost:9090\/query<\/code>. On the query page, run the following query:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">up{container=~\"nodejs-app|python-app\"}<\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">This should bring up the health status of the two service containers:<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" src=\"https:\/\/upcloud.com\/media\/image-254-1024x504.png\" alt=\"-\" class=\"wp-image-66594\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">If you don\u2019t see the targets, confirm that the ServiceMonitor label selectors match your services. Also, ensure the target port name matches what\u2019s defined in the ServiceMonitor.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">If the query runs fine, switch to the <strong>Graph<\/strong> tab and try a few more PromQL expressions to verify that metrics are coming in:<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Basic Up Check<\/strong><\/h3>\n\n\n\n<pre class=\"wp-block-preformatted\">up{job=\"nodejs-app\"}<\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">or<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">up{job=\"python-app\"}<\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">You should see a value of 1 plotted on the graph if the application is being scraped successfully:<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" src=\"https:\/\/upcloud.com\/media\/image-255-1024x731.png\" alt=\"-\" class=\"wp-image-66597\" \/><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>HTTP Request Count<\/strong><\/h3>\n\n\n\n<pre class=\"wp-block-preformatted\">http_requests_total{job=\"nodejs-app\"}<\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">This will return the number of HTTP requests handled by your Node.js app:<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" src=\"https:\/\/upcloud.com\/media\/image-255-1024x731.png\" alt=\"-\" class=\"wp-image-66598\" \/><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Request Latency Histogram (Python)<\/strong><\/h3>\n\n\n\n<pre class=\"wp-block-preformatted\">rate(http_request_duration_seconds_bucket{job=\"python-app\"}[5m])<\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">This will show the rate of requests falling into different latency buckets:<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" src=\"https:\/\/upcloud.com\/media\/image-256-1024x731.png\" alt=\"-\" class=\"wp-image-66599\" \/><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Memory Usage (Default Metric)<\/strong><\/h3>\n\n\n\n<pre class=\"wp-block-preformatted\">process_resident_memory_bytes{job=\"python-app\"}<\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">This will report the memory used by the Python process:<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" src=\"https:\/\/upcloud.com\/media\/image-257-1024x520.png\" alt=\"-\" class=\"wp-image-66600\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">Since you\u2019ve already set up Grafana as part of the previous part of this tutorial, you can access it by running the following port-forward command:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">kubectl port-forward svc\/prometheus-stack-grafana 3000:80 -n monitoring<\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Next, visit <code>http:\/\/localhost:3000<\/code> and log in. You\u2019re now ready to create app-specific dashboards!<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Create a Basic Dashboard<\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">To get started:<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Go to <strong>Dashboards &gt; New &gt; New Dashboard<\/strong><\/li>\n\n\n\n<li>Click <strong>Add visualization<\/strong> and select <strong>Prometheus<\/strong> as the data source, if asked.<\/li>\n<\/ol>\n\n\n\n<p class=\"wp-block-paragraph\">In the query box, enter:<br>rate(http_requests_total{job=&#8221;nodejs-app&#8221;}[1m])<\/p>\n\n\n\n<ol start=\"3\" class=\"wp-block-list\">\n<li><\/li>\n\n\n\n<li>Set visualization type to <strong>Time series<\/strong><\/li>\n\n\n\n<li>Give it a title like <em>Node.js Request Rate<\/em><\/li>\n\n\n\n<li>Repeat for<br>a. Node.js Memory Usage: (nodejs_heap_size_used_bytes \/ nodejs_heap_size_total_bytes) * 100<br>b. Python Request Rate: rate(http_requests_total{container=&#8221;python-app&#8221;}[$__rate_interval])<br>c. Python Memory Usage: process_resident_memory_bytes{container=&#8221;python-app&#8221;}<br>latency (histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[1m])) by (le))) or memory usage queries<\/li>\n<\/ol>\n\n\n\n<p class=\"wp-block-paragraph\">Here\u2019s what the dashboard should look like when done:<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" src=\"https:\/\/upcloud.com\/media\/image-257-1024x520.png\" alt=\"-\" class=\"wp-image-66601\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">You can also import community dashboards from <a href=\"https:\/\/grafana.com\/grafana\/dashboards\/\" target=\"_blank\" rel=\"noopener\">Grafana\u2019s dashboard directory<\/a> by searching for <a href=\"https:\/\/grafana.com\/grafana\/dashboards\/16111-flask-monitoring\/\" target=\"_blank\" rel=\"noopener\"><em>Flask Monitoring<\/em><\/a> or <a href=\"https:\/\/grafana.com\/grafana\/dashboards\/12230-node-js-dashboard\/\" target=\"_blank\" rel=\"noopener\"><em>Node.js Dashboard<\/em><\/a>.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Once dashboards are set up, you\u2019ll have real-time visibility into application-level metrics alongside your infrastructure observability, all within the same stack!<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Conclusion<\/strong><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">With application-level instrumentation in place, your observability stack is now complete! While Prometheus and Grafana already gave you deep insights into infrastructure health, adding custom metrics from your application logic, like request counts, response times, and memory usage, unlocks a new layer of visibility. This enables your team to monitor performance trends, troubleshoot issues faster, and track real business KPIs that infrastructure metrics alone can\u2019t provide.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">As your observability needs grow, defining domain-specific metrics becomes increasingly valuable. Think beyond technical health: track things like order volume, user registrations, job queue depth, or payment failures. These metrics bring observability closer to what actually matters to your business.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">In the final part of this series, we\u2019ll show you how to scale your monitoring setup with Thanos, enabling long-term metrics storage, high availability, and global querying, all running on UpCloud!<\/p>\n","protected":false},"author":19,"featured_media":0,"comment_status":"open","ping_status":"closed","template":"","community-category":[223,238],"class_list":["post-1831","tutorial","type-tutorial","status-publish","hentry"],"acf":[],"_links":{"self":[{"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/tutorial\/1831","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/tutorial"}],"about":[{"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/types\/tutorial"}],"author":[{"embeddable":true,"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/users\/19"}],"replies":[{"embeddable":true,"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/comments?post=1831"}],"version-history":[{"count":0,"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/tutorial\/1831\/revisions"}],"wp:attachment":[{"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/media?parent=1831"}],"wp:term":[{"taxonomy":"community-category","embeddable":true,"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/community-category?post=1831"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}