{"id":1843,"date":"2025-07-17T08:01:00","date_gmt":"2025-07-17T05:01:00","guid":{"rendered":"https:\/\/upcloud.com\/global\/us\/resources\/tutorials\/monitoring-upcloud-prometheus-part-1\/"},"modified":"2025-07-17T08:01:00","modified_gmt":"2025-07-17T05:01:00","slug":"monitoring-upcloud-prometheus-part-1","status":"publish","type":"tutorial","link":"https:\/\/upcloud.com\/global\/resources\/tutorials\/monitoring-upcloud-prometheus-part-1\/","title":{"rendered":"Monitoring on UpCloud with Prometheus: Part 1"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">In any new-age cloud setup, centralized monitoring is essential for gaining visibility across distributed systems if you intend to improve performance and respond to incidents quickly. Whether you\u2019re deploying workloads in containers or directly on virtual machines, a unified view of your infrastructure is a non-negotiable when it comes to maintaining reliability. That\u2019s where <a href=\"https:\/\/prometheus.io\/\" target=\"_blank\" rel=\"noopener\">Prometheus<\/a>, a widely adopted open-source tool for metrics collection and alerting, comes in. Known for its <a href=\"https:\/\/prometheus.io\/docs\/prometheus\/latest\/querying\/basics\/\" target=\"_blank\" rel=\"noopener\">powerful query language<\/a>, flexible integrations, and active ecosystem, Prometheus has grown to become the standard for observability in Kubernetes environments.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This series will walk you through setting up centralized monitoring using Prometheus on UpCloud\u2019s Managed Kubernetes. The combination is particularly powerful for teams running hybrid environments, where containerized workloads coexist with traditional VM-based services.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">In this first part of the series, you\u2019ll learn how to provision a dedicated monitoring cluster, install Prometheus using the <a href=\"https:\/\/github.com\/prometheus-community\/helm-charts\/tree\/main\/charts\/kube-prometheus-stack\" target=\"_blank\" rel=\"noopener\">kube-prometheus-stack<\/a>, secure your setup, and extend monitoring to include external VMs. By the end, you\u2019ll have a production-ready monitoring setup that can scale across your entire UpCloud infrastructure.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Prerequisites<\/strong><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">To follow along with this series, you will need the following:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><code><a href=\"https:\/\/kubernetes.io\/docs\/reference\/kubectl\/\" target=\"_blank\" rel=\"noopener\">kubectl<\/a><\/code> installed locally for interacting with your Kubernetes cluster.<\/li>\n\n\n\n<li><code><a href=\"https:\/\/helm.sh\/docs\/intro\/install\/\" target=\"_blank\" rel=\"noopener\">helm<\/a> <\/code>installed locally for installing the Prometheus stack.<\/li>\n\n\n\n<li>An UpCloud account. You can <a href=\"https:\/\/signup.upcloud.com\/\">create one here<\/a> if needed.<\/li>\n\n\n\n<li><code><a href=\"https:\/\/upcloud.com\/global\/docs\/tooling\/cli\/\">upctl<\/a> <\/code>installed locally for managing UpCloud resources from your CLI.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">You will also need a Kubernetes cluster to set up the Prometheus stack on, but you\u2019ll see step-by-step instructions below on how to set it up on UpCloud.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Provisioning a Dedicated Monitoring Cluster on UpCloud<\/strong><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">To get started with centralized monitoring, the first step is to provision a dedicated UpCloud Managed Kubernetes cluster that will run your Prometheus stack. Hosting your monitoring tools in a separate cluster makes sure that observability workloads remain separate from application workloads, preventing <a href=\"https:\/\/learn.microsoft.com\/en-us\/azure\/architecture\/antipatterns\/noisy-neighbor\/noisy-neighbor\" target=\"_blank\" rel=\"noopener\">noisy neighbor issues<\/a> and simplifying resource planning.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">You can create a cluster either through the UpCloud Control Panel or using the <a href=\"https:\/\/github.com\/UpCloudLtd\/upcloud-cli\" target=\"_blank\" rel=\"noopener\">upctl<\/a> CLI tool. For production-grade monitoring, consider a cluster with at least three nodes to ensure high availability. Each node should have a minimum of 4 vCPUs and 8GB of RAM to handle the load from Prometheus\u2019s time-series data, Alertmanager, and optional components like Grafana or exporters.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Before you can create a new Kubernetes cluster in UpCloud, you will need to create a private network for it. The network needs to have DHCP enabled, so you can use the following command to create it:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">upctl network create --name \"monitoring-net\" --zone de-fra1  --ip-network address=10.0.1.0\/24,dhcp=true<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Now, here\u2019s the command you need to run to provision the cluster:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">upctl k8s create \\\n  --name monitoring-cluster \\\n  --network monitoring-net \\\n  --zone de-fra1 \\\n  --plan \"production-small\" \\\n  --version 1.30 \\\n  --node-group name=monitoring-nodes,count=3,plan=4xCPU-8GB<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">This command provisions a three-node cluster in Frankfurt, with each node sized for moderate workloads. If you expect to scrape metrics from many Kubernetes services or external VMs, you may want to choose a larger plan (e.g., 6xCPU-16GB) to accommodate Prometheus\u2019s memory-intensive operations.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Once the cluster is ready, download its kubeconfig to your local machine:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">upctl k8s config monitoring-cluster --output yaml --write ~\/.kube\/config<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">With your monitoring cluster provisioned and configured, you\u2019re ready to install Prometheus using the <code>kube-prometheus-stack<\/code> Helm chart in the next section.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Installing Prometheus with <\/strong><strong>kube-prometheus-stack<\/strong><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The simplest way to deploy Prometheus on Kubernetes is by using the <code>kube-prometheus-stack<\/code> Helm chart. Maintained by the Prometheus Community, this chart bootstraps a full monitoring setup that includes Prometheus, Alertmanager, node exporters, and a set of preconfigured rules and dashboards. It\u2019s the recommended starting point for production monitoring environments.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Step 1: Add the Helm Repository<\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">If you haven\u2019t already, add the Prometheus Community Helm repository:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">helm repo add prometheus-community https:\/\/prometheus-community.github.io\/helm-charts\nhelm repo update<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Step 2: Create a Minimal <\/strong><strong>values.yaml<\/strong><strong> Configuration<\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Before installation, prepare a <code>values.yam<\/code>l file to customize the deployment. Here\u2019s a minimal example:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">prometheus:\n  prometheusSpec:\n    serviceMonitorSelectorNilUsesHelmValues: false\n    ruleSelectorNilUsesHelmValues: false\n\n\nalertmanager:\n  enabled: true\n\n\ngrafana:\n  enabled: false  # You can enable this later if you want built-in dashboards\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">This configuration ensures that Prometheus will respect custom service monitors and alerting rules you add in the future, while keeping the setup minimal to start.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Step 3: Install the Chart<\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Now deploy the chart into a dedicated namespace (e.g., monitoring):<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">kubectl create namespace monitoring\n\n\nhelm install prometheus-stack prometheus-community\/kube-prometheus-stack \\\n  --namespace monitoring \\\n  -f values.yaml<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Helm will deploy all components and set up the necessary service accounts, role bindings, and default alerting rules. You can monitor the progress of the installation with:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">kubectl get pods -n monitoring -w<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Once all pods are running, you\u2019ll have a working Prometheus setup collecting metrics from the cluster, ready to be extended for more advanced use cases like VM monitoring!<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Next, we\u2019ll secure this setup by configuring access controls and firewall rules.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Securing the Monitoring Setup<\/strong><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Once Prometheus is deployed, it\u2019s important to secure access to your monitoring infrastructure, both at the network and cluster levels. This helps prevent unauthorized access to sensitive metrics, ensures Prometheus is scraping only intended targets, and keeps your monitoring environment compliant with best practices.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Enable Role-Based Access Control (RBAC)<\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">The kube-prometheus-stack Helm chart automatically sets up service accounts and roles, but it\u2019s good practice to review them to ensure least privilege is enforced. For example, Prometheus should have permission to read metrics and service discovery data, but not to modify Kubernetes objects.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">You can inspect the bindings with:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">kubectl get clusterrolebinding -n monitoring<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">If you need to restrict access further, you should consider creating custom <code>ClusterRole<\/code> and <code>ClusterRoleBinding<\/code> resources scoped to Prometheus\u2019s namespace and service account.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Service Account Security<\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">You should also verify that each Prometheus component (Prometheus server, Alertmanager, exporters) runs with the correct service account and minimal privileges. This can help contain damage in case of a container escape or misconfiguration.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">kubectl get serviceaccount -n monitoring<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">You can use Kubernetes\u2019 built-in audit tools or integrations like <a href=\"https:\/\/github.com\/open-policy-agent\/gatekeeper\" target=\"_blank\" rel=\"noopener\">OPA Gatekeeper<\/a> for enforcing policy compliance.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">With your monitoring stack now secured, in the next section, you\u2019ll learn how to configure an external monitoring target.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Test out your Prometheus Operator with an External VM<\/strong><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">While Prometheus is optimized for Kubernetes environments, it can also monitor services running outside the cluster, including virtual machines. This is especially useful for hybrid deployments where legacy apps or backend jobs still run on UpCloud VMs.<\/p>\n\n\n\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\">\n<p class=\"wp-block-paragraph\"><em>If you don\u2019t already have a VM with the Prometheus node exporter set up on it, you will learn in detail how to set up one in the next part. For now, you can run <\/em><a href=\"https:\/\/prometheus.io\/docs\/guides\/node-exporter\/#installing-and-running-the-node-exporter\" target=\"_blank\" rel=\"noopener\"><em>the following commands<\/em><\/a><em> on any Linux-based VM to set up Node exporter on it. Make sure to note the public IP of the VM after you\u2019re done.<\/em><\/p>\n<\/blockquote>\n\n\n\n<p class=\"wp-block-paragraph\">To configure your Prometheus operator to scrape external services, you need to update the Prometheus Helm release (via the <code>values.yaml<\/code> file) to specify the host and port of the target VM:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">prometheus:\n  prometheusSpec:\n    additionalScrapeConfigs:\n    - job_name: \"upcloud-vm\"\n      static_configs:\n      - targets: [\"&lt;public ip&gt;:9100\"]<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">This will tell Prometheus to look for new targets from the given array and begin scraping them automatically.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Use Cases for VM Monitoring<\/strong><\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Legacy Applications<\/strong>: Track CPU, memory, disk, and network stats via Node Exporter installed on the VM.<\/li>\n\n\n\n<li><strong>Cron Jobs or Services<\/strong>: Expose custom metrics from standalone applications (e.g., via \/metrics HTTP endpoints).<\/li>\n\n\n\n<li><strong>Hybrid Environments<\/strong>: Combine Kubernetes and VM observability in a single Prometheus instance for a complete view.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">With this setup, you can monitor any VM alongside your Kubernetes workloads, using the same Prometheus queries and alerting rules. In the next section, we\u2019ll look at how to validate that your configuration is working and that all targets are being scraped correctly.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Testing the Setup<\/strong><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Once Prometheus is configured to monitor both Kubernetes services and external VMs, it\u2019s important to verify that everything is working as expected. Prometheus provides built-in ways to help you confirm that targets are being discovered, scraped, and stored properly.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Access the Prometheus Web UI<\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">First, port-forward the Prometheus service so you can access its web interface locally:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">kubectl port-forward svc\/prometheus-stack-kube-prom-prometheus 9090 -n monitoring<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Then open your browser and go to http:\/\/localhost:9090.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Check Target Discovery<\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Navigate to <strong>Status<\/strong> &gt; <strong>Target health<\/strong> in the Prometheus UI. You should see a list of all configured scrape targets, grouped by job name. Each target will show a status (e.g., UP) and scrape metrics such as latency and data size.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Make sure your external VM targets (e.g., <code>upcloud-vm<\/code>) are listed and in the UP state:<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" src=\"https:\/\/upcloud.com\/media\/1-prometheus-resource-health-1024x520.png\" alt=\"-\" class=\"wp-image-60204\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">If not, double-check:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>The IP and port are correct.<\/li>\n\n\n\n<li>The VM\u2019s firewall allows inbound traffic on the scrape port.<\/li>\n\n\n\n<li>The target file is mounted and referenced correctly in the scrape config.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Explore Metrics<\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Click <strong>Query<\/strong> and use PromQL queries to explore collected data. Try a few examples:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">up<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">This basic query returns a list of all targets and their status.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" src=\"https:\/\/upcloud.com\/media\/2-prometheus-promql-query-1024x520.png\" alt=\"-\" class=\"wp-image-60205\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">Next, try the following query:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">node_cpu_seconds_total{job=\"upcloud-vm\"}<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">If Node Exporter is running on your VM, this query should return CPU metrics segmented by mode (idle, user, system, etc.). You can view it as a chart by switching to the <strong>Graph<\/strong> tab:<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" src=\"https:\/\/upcloud.com\/media\/3-prometheus-query-cpu-metrics-1024x520.png\" alt=\"-\" class=\"wp-image-60207\" \/><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Up Next<\/strong><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">By setting up a dedicated monitoring cluster on UpCloud\u2019s Managed Kubernetes and deploying Prometheus using the <code>kube-prometheus-stack<\/code>, you\u2019ve laid a solid foundation for centralized observability across your infrastructure. This setup ensures your monitoring workloads are isolated, scalable, and production-ready, capable of handling both containerized and traditional VM-based services.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">You\u2019ve also extended Prometheus beyond Kubernetes using static service discovery to bring external VMs into the fold. This flexibility is key for hybrid environments where legacy services, custom jobs, or standalone applications still play an important role. With a single Prometheus instance scraping both types of targets, you gain unified metrics, streamlined alerting, and a consistent query interface.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">In <a href=\"https:\/\/upcloud.com\/global\/resources\/tutorials\/monitoring-upcloud-prometheus-part-2\/\">Part 2, we\u2019ll build on this setup by installing Node Exporter on your UpCloud VMs<\/a>. You\u2019ll learn how to collect key metrics like CPU usage, memory pressure, and disk I\/O from each virtual machine and visualize them alongside your Kubernetes workloads, further closing the gap between cloud-native and legacy monitoring.<\/p>\n","protected":false},"author":82,"featured_media":0,"comment_status":"open","ping_status":"closed","template":"","community-category":[238],"class_list":["post-1843","tutorial","type-tutorial","status-publish","hentry"],"acf":[],"_links":{"self":[{"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/tutorial\/1843","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/tutorial"}],"about":[{"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/types\/tutorial"}],"author":[{"embeddable":true,"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/users\/82"}],"replies":[{"embeddable":true,"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/comments?post=1843"}],"version-history":[{"count":0,"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/tutorial\/1843\/revisions"}],"wp:attachment":[{"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/media?parent=1843"}],"wp:term":[{"taxonomy":"community-category","embeddable":true,"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/community-category?post=1843"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}