{"id":1828,"date":"2025-10-19T22:55:55","date_gmt":"2025-10-19T19:55:55","guid":{"rendered":"https:\/\/upcloud.com\/global\/us\/resources\/tutorials\/monitoring-on-upcloud-with-prometheus-part-5\/"},"modified":"2025-10-19T22:55:55","modified_gmt":"2025-10-19T19:55:55","slug":"monitoring-on-upcloud-with-prometheus-part-5","status":"publish","type":"tutorial","link":"https:\/\/upcloud.com\/global\/resources\/tutorials\/monitoring-on-upcloud-with-prometheus-part-5\/","title":{"rendered":"Monitoring on UpCloud with Prometheus: Part 5"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">So far in this series, you\u2019ve built a complete observability foundation on UpCloud. You\u2019ve set up Prometheus and Grafana to monitor infrastructure-level metrics from Kubernetes and VMs, and you\u2019ve added application-level instrumentation using Prometheus client libraries for real-time insights into your workloads. With dashboards and alerts in place, your system is observant. The only aspect you haven\u2019t considered yet is the storage of all this data.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Prometheus, by default, only retains data for a limited time and stores it locally, making it unsuitable for historical analysis or disaster recovery purposes. If a pod is deleted or a cluster is wiped, all metrics vanish. For production environments, this is a major risk.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This is where Thanos comes in.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Thanos is a <a href=\"https:\/\/cncf.io\/projects\/thanos\" target=\"_blank\" rel=\"noopener\">CNCF project<\/a> that extends Prometheus by solving three major limitations: short retention periods, siloed data from individual Prometheus instances, and inefficient historical queries. It does this by:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Uploading metrics to cloud object storage for long-term retention<\/li>\n\n\n\n<li>Allowing federated querying across multiple Prometheus instances<\/li>\n\n\n\n<li>Supporting downsampling for faster queries over large time ranges<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">In this final part of the series, you\u2019ll integrate Thanos into your existing UpCloud observability stack. You\u2019ll install the Thanos Sidecar alongside Prometheus to enable metrics shipping to <a href=\"https:\/\/upcloud.com\/global\/products\/object-storage\/\">UpCloud Object Storage<\/a>. Then, you\u2019ll deploy key Thanos components\u2014Query and Store Gateway\u2014and wire the setup into Grafana for easy historical metrics analysis.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">By the end of this guide, your monitoring system will be ready for production at scale, capable of answering queries that span weeks or months, and resilient to Prometheus restarts or cluster failures.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">To see the previous parts of the series you can access them here:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><a href=\"https:\/\/upcloud.com\/global\/resources\/tutorials\/monitoring-upcloud-prometheus-part-1\/\">Monitoring in UpCloud with Prometheus: Part 1<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/upcloud.com\/global\/resources\/tutorials\/monitoring-upcloud-prometheus-part-2\/\">Monitoring in UpCloud with Prometheus: Part 2<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/upcloud.com\/global\/resources\/tutorials\/monitoring-upcloud-prometheus-part-3\/\">Monitoring in UpCloud with Prometheus: Part 3<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/upcloud.com\/global\/resources\/tutorials\/monitoring-with-prometheus-part-4\/\">Monitoring in UpCloud with Prometheus: Part 4<\/a><\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Prerequisites<\/strong><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Before you can go ahead with the tutorial, you will need to ensure that you have the following:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>An existing Prometheus + Grafana setup on an <a href=\"https:\/\/upcloud.com\/global\/products\/managed-kubernetes\/\">UpCloud Managed Kubernetes<\/a> cluster (from earlier parts of this series).<\/li>\n\n\n\n<li>Basic knowledge of Node.js and Python.<\/li>\n\n\n\n<li>kubectl and helm installed and configured correctly<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Installing Thanos Sidecar with Prometheus<\/strong><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">To start extending your observability stack with long-term storage and global querying, you\u2019ll deploy the <a href=\"https:\/\/thanos.io\/v0.4\/components\/sidecar\/\" target=\"_blank\" rel=\"noopener\">Thanos Sidecar<\/a> alongside your Prometheus instance. The Sidecar enables:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Uploading TSDB blocks from Prometheus to UpCloud Object Storage for long-term durability.<\/li>\n\n\n\n<li>Exposing a Store API endpoint, making Prometheus data available to other Thanos components like Query and Store Gateway.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">The kube-prometheus-stack chart simplifies this integration with built-in support for Thanos. Here\u2019s how to enable it:<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Step 1: Create an UpCloud Object Storage Bucket<\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Before you add the sidecar, you will need to set up an S3-like object storage bucket where Thanos will store your historic metrics data. To do that, head to your UpCloud dashboard and click <strong>Object Storage<\/strong> from the left navigation pane. This is where you need to:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Create a new object storage instance.<\/li>\n\n\n\n<li>Create an object bucket inside your storage instance.<\/li>\n\n\n\n<li>Create a new User object in the storage instance.<\/li>\n\n\n\n<li>Attach the <strong>ECSS3FullAccess<\/strong> policy to the user.<\/li>\n\n\n\n<li>Generate and download the access key and the secret key for the user.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">You will find detailed instructions on how to do that <a href=\"https:\/\/upcloud.com\/global\/docs\/guides\/get-started-managed-object-storage\/\">in this guide<\/a>.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Make sure to retrieve the bucket name, object storage endpoint, access key, and secret key from your UpCloud storage instance before moving ahead.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Step 2: Create a Config Secret to Store the Bucket Details<\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Thanos needs a config file (objstore.yml) with the credentials of the object storage bucket to connect to it.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Create a file named objstore.yml and save the following in it, replacing the placeholders with the values you retrieved from UpCloud in the last step:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">type: S3\nconfig:\n  bucket: \"&lt;your-bucket-name&gt;\"\n  endpoint: \"your-region-specific-endpoint\"\n  access_key: \"&lt;your-access-key&gt;\"\n  secret_key: \"&lt;your-secret-key&gt;\"\n  insecure: false<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Using this file, create the secret in Kubernetes:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">kubectl create secret generic thanos-objstore-config \\<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">&nbsp;&nbsp;&#8211;from-file=objstore.yml \\<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">&nbsp;&nbsp;-n monitoring<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">You will reference this secret in your Thanos configuration to enable it to connect to your bucket.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Step 3: Enable Thanos Sidecar via Helm Values<\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Update your values.yaml for the kube-prometheus-stack chart to include:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">prometheus:\n  prometheusSpec:\n    disableCompaction: true\n    thanos:\n      image: quay.io\/thanos\/thanos:v0.39.2\n      objectStorageConfig:\n        existingSecret:\n          name: thanos-objstore-config\n          key: objstore.yml\n\nalertmanager:\n  enabled: true\n\ngrafana:\n  enabled: true<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Here\u2019s what each setting does:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>disableCompaction: true: disables Prometheus\u2019s built-in compaction to prevent conflicts with Thanos, which handles compaction centrally via its own components.<\/li>\n\n\n\n<li>thanos.image: explicitly sets the Thanos version to ensure consistency across environments.<\/li>\n\n\n\n<li>objectStorageConfig.existingSecret: tells Prometheus where to find the object storage credentials, using the secret created in Step 2.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Step 4: Upgrade the Stack<\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Apply your changes by upgrading the Helm release:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">helm upgrade prometheus-stack prometheus-community\/kube-prometheus-stack \\<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">&nbsp;&nbsp;-n monitoring \\<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">&nbsp;&nbsp;-f values.yaml<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Once the deployment finishes, verify that the Prometheus pod now includes the Thanos Sidecar (as thanos-sidecar) by running the following command:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">$ kubectl get pod prometheus-prometheus-stack-kube-prom-prometheus-0 -n monitoring -o jsonpath=&#8221;{.spec.containers[*].name}&#8221;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">prometheus config-reloader thanos-sidecar<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">To confirm the Sidecar is running properly, check its logs:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">kubectl logs pods\/prometheus-prometheus-stack-kube-prom-prometheus-0 -c thanos-sidecar -n monitoring<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">If you see something like this:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">level=warn ts=2025-08-15T09:43:26.285420195Z caller=sidecar.go:345 err=&#8221;check exists: stat s3 object: Access Denied.&#8221; uploaded=0<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">It might indicate that either the credentials of your UpCloud storage instance user are incorrect, or you have not attached the right policy to the user.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">If you see something like this:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">level=info ts=2025-08-15T09:47:56.28366975Z caller=shipper.go:334 msg=&#8221;upload new block&#8221; id=01K2N83AK6V82AAY94FC58NBZD<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">It means that the sidecar has been set up correctly and is storing data in your UpCloud bucket!<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">You can also check out the stored blocks in the bucket:<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" src=\"https:\/\/upcloud.com\/media\/image-258-1024x506.png\" alt=\"-\" class=\"wp-image-66670\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">However, the setup isn\u2019t complete at this point. Your stack can only upload TSDB data blocks for long-term storage. To be able to retrieve and query that data, you need to set up some more Thanos components: <a href=\"https:\/\/thanos.io\/tip\/components\/store.md\/\" target=\"_blank\" rel=\"noopener\">Store Gateway<\/a> and <a href=\"https:\/\/thanos.io\/tip\/components\/query.md\/\" target=\"_blank\" rel=\"noopener\">Query<\/a>.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Setting Up Other Thanos Components: Query and Store Gateway<\/strong><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Now, you\u2019ll set up Thanos Store Gateway, Query, and Compactor. Here\u2019s a quick summary of what each one of those does:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><a href=\"https:\/\/thanos.io\/tip\/components\/query.md\" target=\"_blank\" rel=\"noopener\"><strong>Thanos Query<\/strong><\/a> acts as a unified PromQL query layer that can aggregate metrics from multiple sources.<\/li>\n\n\n\n<li><a href=\"https:\/\/thanos.io\/tip\/components\/store.md\/\" target=\"_blank\" rel=\"noopener\"><strong>Thanos Store Gateway<\/strong><\/a> serves historical TSDB blocks stored in UpCloud Object Storage.<\/li>\n\n\n\n<li><a href=\"https:\/\/thanos.io\/tip\/components\/compact.md\/\" target=\"_blank\" rel=\"noopener\"><strong>Thanos Compactor<\/strong><\/a> deduplicates, downsamples, and cleans up stored blocks, making queries over weeks or months efficient and reliable.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">All three can be installed using the <a href=\"https:\/\/github.com\/bitnami\/charts\/tree\/main\/bitnami\/thanos\" target=\"_blank\" rel=\"noopener\">Bitnami Thanos Helm chart<\/a>, which works well alongside your existing kube-prometheus-stack installation.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">To set these up, follow these steps:<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Step 1: Add the Bitnami Chart Repository<\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Add the Bitnami repo (if you haven\u2019t already):<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">helm repo add bitnami https:\/\/charts.bitnami.com\/bitnami<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">helm repo update<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Step 2: Set up the <\/strong><strong>thanos-values.yaml<\/strong><strong> file<\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Save the following configuration for the chart in a file named thanos-values.yaml:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">existingObjstoreSecret: thanos-objstore-config\n\nquery:\n  enabled: true\n  replicaCount: 1\n  stores:\n    - dnssrv+_grpc._tcp.prometheus-operated.monitoring.svc.cluster.local  # Prometheus Sidecar\n    - thanos-storegateway.monitoring.svc.cluster.local:10901              # Store Gateway\n\nstoregateway:\n  enabled: true\n  replicaCount: 1\n\ncompactor:\n  enabled: true\n  replicaCount: 1\n  retentionResolutionRaw: 30d\n  retentionResolution5m: 180d\n  retentionResolution1h: 2y\n  consistencyDelay: 30m\n  persistence:\n    enabled: true\n    storageClass: standard\n    size: 10Gi<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Let\u2019s break down what each section does:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>existingObjstoreSecret: Tells all Thanos components to use the existing Kubernetes secret named thanos-objstore-config, which you created earlier<\/li>\n\n\n\n<li>query: Turns on the Thanos Query component, sets the replica count to one, and defines a list of stores to pull data from. The first entry on this list will set the query component to dynamically discover all Prometheus Sidecars in the monitoring namespace via DNS SRV records. The second entry is the internal service address of the Store Gateway, which serves historical blocks from object storage<\/li>\n\n\n\n<li>storegateway: Deploys the Store Gateway and sets the replica count to one.<\/li>\n\n\n\n<li>compactor: Enables the Thanos Compactor, and sets the replica count to 1 (which must not be changed, as the compactor is not meant to be run in parallel). Here\u2019s what the other keys do:\n<ul class=\"wp-block-list\">\n<li>retentionResolutionRaw: Keep raw (high-resolution) metrics for 30 days.<\/li>\n\n\n\n<li>retentionResolution5m: Keep 5-minute downsampled data for 180 days.<\/li>\n\n\n\n<li>retentionResolution1h: Keep 1-hour downsampled data for 2 years.<\/li>\n\n\n\n<li>consistencyDelay: 30m: Adds a buffer time to ensure blocks are fully uploaded before compaction kicks in, avoiding corruption or race conditions.<\/li>\n\n\n\n<li>persistence: Enables durable disk storage for temporary files during compaction.\n<ul class=\"wp-block-list\">\n<li>enabled: true: Activates persistent volume usage.<\/li>\n\n\n\n<li>storageClass: standard: Specifies the storage class to use\u2014change this to match your UpCloud setup if needed.<\/li>\n\n\n\n<li>size: 10Gi: Allocates 10Gi of disk space for compactor temp files.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">Without persistence, the compactor state would be lost on pod restart, which could lead to redundant or failed compaction cycles.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Step 3: Install the Thanos Release<\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Install all components in one go with this command:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">helm upgrade &#8211;install thanos bitnami\/thanos \\<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">&nbsp;&nbsp;&#8211;namespace monitoring \\<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">&nbsp;&nbsp;-f thanos-values.yaml<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This deploys all three components into your monitoring namespace and wires them up to your Prometheus Sidecar and object storage backend.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Step 4: Verify the Deployment<\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Check that all pods are running:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">$ kubectl get pods -n monitoring -l app.kubernetes.io\/instance=thanos\nNAME                                     READY   STATUS    RESTARTS   AGE\nthanos-compactor-844d8c8d76-9z8n5        1\/1     Running   0          5h21m\nthanos-query-c89bf5d95-ktmv6             1\/1     Running   0          5h21m\nthanos-query-frontend-59564bf878-qctvz   1\/1     Running   0          5h21m\nthanos-storegateway-0                    1\/1     Running   0          5h13m<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Once all components are healthy, your Thanos stack is fully functional: it can store, optimize, and serve months or even years of Prometheus metrics at scale!<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Exploring the New Setup<\/strong><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Now that Thanos is set up, let\u2019s explore a few things.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Accessing the Thanos Query UI<\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Start by port-forwarding the Thanos Query service:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">kubectl port-forward svc\/thanos-query -n monitoring 9090:9090<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Visit <code>http:\/\/localhost:9090<\/code> in your browser. You should see the Thanos Query UI, which looks similar to the Prometheus expression browser.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">On the <strong>Endpoints<\/strong> tab, verify that two endpoints are listed:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>The Prometheus Sidecar (prometheus-stack-kube-prometheus-prometheus)<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">The Thanos Store Gateway (thanos-store)<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" src=\"https:\/\/upcloud.com\/media\/image-259-1024x506.png\" alt=\"-\" class=\"wp-image-66672\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">If both are present and healthy, it means your federated query setup is functional.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Checking for Successful Block Uploads<\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">The Thanos Sidecar uploads TSDB blocks to object storage at regular intervals (usually 2 hours by default). You can check its logs to confirm successful uploads:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">kubectl logs deploy\/prometheus-stack-kube-prometheus-prometheus -c thanos-sidecar -n monitoring<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Look for log lines like:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">level=info ts=2025-08-15T09:47:57.422230553Z caller=shipper.go:334 msg=&#8221;upload new block&#8221; id=01K2NTQ05XNEN5TCNJ6VCGFE4K<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This indicates that metrics blocks are being written to your UpCloud Object Storage bucket.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Check Store Gateway Access to Object Storage<\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Next, confirm that the Thanos Store Gateway can access those uploaded blocks. Run:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">kubectl logs deploy\/prometheus-stack-thanos-store -n monitoring<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">You should see logs that mention discovered blocks and bucket syncing:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">ts=2025-08-15T09:48:22.131311006Z caller=fetcher.go:628 level=info component=block.BaseFetcher msg=&#8221;successfully synchronized block metadata&#8221; duration=302.762563ms duration_ms=302 cached=7 returned=7 partial=0<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">ts=2025-08-15T09:48:22.133060752Z caller=bucket.go:872 level=info msg=&#8221;loaded new block&#8221; elapsed=1.684564ms id=01K2P1JQEAYSP50GRJXKXAQSB9<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">If you see repeated bucket access errors like this:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Errors in credentiaLS: level=warn ts=2025-08-15T02:08:26.288428734Z caller=sidecar.go:345 err=&#8221;check exists: stat s3 object: Access Denied.&#8221; uploaded=0 in prom operator<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Double-check your objstore.yml configuration and Kubernetes secret.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Query Historical Metrics<\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">With Thanos fully deployed, you can now query metrics that span weeks or even months, far beyond what Prometheus stores locally. Start by port-forwarding the Thanos Query service and visit <code>http:\/\/localhost:9090\/<\/code><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Here, set the engine to <strong>Thanos<\/strong> and run this long-range query:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">sum by (instance) (rate(node_cpu_seconds_total{mode=&#8221;system&#8221;}[30d]))<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This query shows how much time each node\u2019s CPU spent in system mode (running kernel-level tasks) over the last 30 days, averaged per second. It\u2019s useful for:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Detecting nodes with consistently high system overhead<\/li>\n\n\n\n<li>Investigating noisy workloads (e.g., logging, I\/O-heavy containers)<\/li>\n\n\n\n<li>Identifying infrastructure-level regressions over time<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">Here\u2019s what the result graph should look like:<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" src=\"https:\/\/upcloud.com\/media\/image-260-1024x602.png\" alt=\"-\" class=\"wp-image-66675\" \/><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Integrating Thanos With Grafana<\/strong><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">To be able to view historic data in Grafana and build dashboards with it, you\u2019ll need to add Thanos Query as a new data source. This allows Grafana to access metrics stored in UpCloud Object Storage via the Thanos Store Gateway, while still querying recent data from Prometheus.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Add Thanos Query as a Grafana Data Source<\/strong><\/h3>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Visit your Grafana UI (usually via the prometheus-stack-grafana service or Ingress).<\/li>\n\n\n\n<li>Navigate to <strong>Settings \u2192 Data Sources \u2192 Add Data Source<\/strong>.<\/li>\n\n\n\n<li>Choose <strong>Prometheus<\/strong> as the type.<\/li>\n<\/ol>\n\n\n\n<p class=\"wp-block-paragraph\">Set the <strong>URL<\/strong> to the internal service name for Thanos Query:<br>http:\/\/thanos-query.monitoring.svc.cluster.local:9090<\/p>\n\n\n\n<ol start=\"4\" class=\"wp-block-list\">\n<li><\/li>\n\n\n\n<li>Set a name like Thanos (Long-Term) to distinguish it from your existing Prometheus source.<\/li>\n\n\n\n<li>Click <strong>Save &amp; Test<\/strong> to validate connectivity.<\/li>\n<\/ol>\n\n\n\n<p class=\"wp-block-paragraph\">You\u2019ll receive a success message:<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" src=\"https:\/\/upcloud.com\/media\/image4-1024x504.png\" alt=\"-\" class=\"wp-image-71827\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">Grafana now treats Thanos Query like any other Prometheus-compatible source, but with access to downsampled metrics, block storage, and query fan-out to multiple stores.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">When building dashboards, try keeping your original Prometheus data source for high-resolution, recent metrics (e.g., [5m], [1h] windows), and using the Thanos source for long time ranges ([7d], [30d], etc.) in dashboards and analysis. This lets you build dashboards that span from real-time diagnostics to historical trends without worrying about Prometheus\u2019s native retention limits.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Conclusion<\/strong><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">By integrating Thanos into your kube-prometheus-stack deployment on UpCloud, you\u2019ve extended Prometheus from a short-term, single-cluster monitoring system into a resilient, long-term observability platform. Thanos unlocks durable metrics storage in UpCloud Object Storage, enabling queries across weeks or months of data; even after Prometheus has purged local blocks. It also introduces a unified query layer that can aggregate metrics from multiple Prometheus instances and storage backends, paving the way for multi-cluster observability at scale.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">With this setup, your UpCloud observability stack is now complete, from infrastructure and workload metrics to custom application instrumentation, visualized in Grafana and retained for long-term analysis. You\u2019re well-positioned to explore additional capabilities like alerting on historical trends, recording rules for efficiency, or even cross-region federation. Observability is about unlocking insights, and now you have all the tools to do just that.<\/p>\n","protected":false},"author":19,"featured_media":0,"comment_status":"open","ping_status":"closed","template":"","community-category":[223,238],"class_list":["post-1828","tutorial","type-tutorial","status-publish","hentry"],"acf":[],"_links":{"self":[{"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/tutorial\/1828","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/tutorial"}],"about":[{"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/types\/tutorial"}],"author":[{"embeddable":true,"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/users\/19"}],"replies":[{"embeddable":true,"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/comments?post=1828"}],"version-history":[{"count":0,"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/tutorial\/1828\/revisions"}],"wp:attachment":[{"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/media?parent=1828"}],"wp:term":[{"taxonomy":"community-category","embeddable":true,"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/community-category?post=1828"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}