{"id":1822,"date":"2025-10-20T00:53:44","date_gmt":"2025-10-19T21:53:44","guid":{"rendered":"https:\/\/upcloud.com\/global\/us\/resources\/tutorials\/extending-opentelemetry-on-upcloud-with-thanos-for-scalable-metrics-storage\/"},"modified":"2025-10-20T00:53:44","modified_gmt":"2025-10-19T21:53:44","slug":"extending-opentelemetry-on-upcloud-with-thanos-for-scalable-metrics-storage","status":"publish","type":"tutorial","link":"https:\/\/upcloud.com\/global\/resources\/tutorials\/extending-opentelemetry-on-upcloud-with-thanos-for-scalable-metrics-storage\/","title":{"rendered":"Extending OpenTelemetry on UpCloud with Thanos for Scalable Metrics Storage"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">In our earlier guide, <em><a href=\"https:\/\/upcloud.com\/global\/blog\/what-is-opentelemetry-understanding-the-standard-for-cloud-native-observability\/\">What is OpenTelemetry? Understanding the Standard for Cloud-Native Observability<\/a><\/em>, we explained how OpenTelemetry standardizes the collection of metrics, traces, and logs across distributed systems. Then, in the follow-up tutorial, <em><a href=\"https:\/\/upcloud.com\/global\/resources\/tutorials\/how-to-deploy-opentelemetry-on-upcloud-managed-kubernetes\/\">How to Deploy OpenTelemetry on UpCloud Managed Kubernetes<\/a><\/em>, we showed how to put those concepts into practice by deploying the OpenTelemetry Collector and instrumenting a Python application on UpCloud\u2019s infrastructure.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">That setup was intentionally lean. It used the Collector\u2019s <strong>debug exporter<\/strong> (no persistence) and, optionally, an external backend such as <strong>Grafana Cloud<\/strong>, which provides short-term retention (~3 days for traces and ~14 days for metrics on the free tier). This is perfect for smoke tests and immediate feedback, but not enough for trend analysis, capacity planning, compliance, or post-incident reviews.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This tutorial closes that gap by adding <strong>Thanos<\/strong> for cost-efficient, long-term metrics storage and scalable querying. You\u2019ll configure the <strong>Thanos Receive<\/strong> component to ingest OTLP metrics directly from your OpenTelemetry Collector, persist them to <strong>UpCloud Object Storage<\/strong>, and explore historical data through <strong>Thanos Query<\/strong>.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">With <strong>compaction<\/strong> and <strong>optional downsampling<\/strong>, you\u2019ll keep storage predictable while unlocking fast queries over months of history. This will enable SLO tracking, seasonality insights, regression detection, and auditability without overburdening your cluster.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Prerequisites<\/strong><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Before you can go ahead with this tutorial, you will need to ensure that you have the following:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>An existing OpenTelemetry Collector + sample Python app setup on an UpCloud Managed Kubernetes cluster (from Part 1 of this series).<\/li>\n\n\n\n<li>An <a href=\"https:\/\/upcloud.com\/global\/products\/object-storage\/\">UpCloud Object Storage<\/a> instance ready for long-term metric retention.<\/li>\n\n\n\n<li>kubectl, helm, and <a href=\"https:\/\/github.com\/UpCloudLtd\/upcloud-cli\" target=\"_blank\" rel=\"noopener\">upctl<\/a> installed and configured correctly.<\/li>\n\n\n\n<li>Basic knowledge of Kubernetes and OpenTelemetry concepts.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">If you successfully completed Part 1, your environment already satisfies most of these requirements. The only additional step you\u2019ll need before deploying Thanos is to prepare object storage access for it.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Step 1: Create an UpCloud Object Storage Bucket<\/strong><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Thanos Receiver writes ingested metrics into object storage for durability and long-term querying. Head to your <a href=\"https:\/\/hub.upcloud.com\/\">UpCloud Control Panel<\/a> and navigate to <strong>Object Storage<\/strong>. From here:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Create a new object storage instance.<\/li>\n\n\n\n<li>Add a bucket inside that instance to store metric blocks.<\/li>\n\n\n\n<li>Create a new <strong>User<\/strong> for this storage instance.<\/li>\n\n\n\n<li>Attach the <strong>ECSS3FullAccess<\/strong> policy to that user.<\/li>\n\n\n\n<li>Generate and download the access key and secret key.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">You will find detailed instructions <a href=\"https:\/\/upcloud.com\/global\/docs\/guides\/get-started-managed-object-storage\/\">in this guide<\/a>.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Keep the following details handy before moving ahead:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Bucket name<\/li>\n\n\n\n<li>Object storage endpoint (region-specific)<\/li>\n\n\n\n<li>Access key<\/li>\n\n\n\n<li>Secret key<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Step 2: Create a Config Secret to Store the Bucket Details<\/strong><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Thanos needs an objstore.yml file with your object storage credentials to connect to the bucket. Create a file named objstore.yml with the following content (replace placeholders with your own values):<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">type: S3\nconfig:\n  bucket: \"&lt;your-bucket-name&gt;\"\n  endpoint: \"&lt;your-endpoint&gt;.upcloudobjects.com\"\n  access_key: \"&lt;your-access-key&gt;\"\n  secret_key: \"&lt;your-secret-key&gt;\"\n  insecure: false\n\nNow create a Kubernetes secret from this file:\nkubectl create secret generic thanos-objstore-config \\\n  --from-file=objstore.yml \\\n  -n observability<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">This secret will be referenced by Thanos components to persist and read back metric blocks.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Step 3: Prepare a Hashring ConfigMap<\/strong><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The Thanos Receive Router requires a <a href=\"https:\/\/thanos.io\/tip\/components\/receive.md\/#hashring-management-and-autoscaling-in-kubernetes\" target=\"_blank\" rel=\"noopener\">hashring configuration<\/a> to decide how to forward writes to the store, especially in multi-tenant environments. Even in a single-replica setup, this configuration must exist. Create a file called hashrings.json with the following:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">[\n  {\n    \"tenant\": \"default\",\n    \"endpoints\": [\n      \"thanos-receive-ingestor-0.thanos-receive-ingestor.observability.svc.cluster.local:10901\"\n    ]\n  }\n]<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Apply it to the cluster:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">kubectl create configmap hashring-config \\\n  --from-file=hashrings.json \\\n  -n observability<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">With your cluster from Part 1, an UpCloud Object Storage bucket, and these Kubernetes objects prepared, you\u2019re ready to deploy Thanos Receiver and extend your observability stack with long-term, cost-efficient metric retention.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Deploy Thanos with <\/strong><strong>kube-thanos<\/strong><strong> and Jsonnet<\/strong><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">We\u2019ll use <a href=\"https:\/\/github.com\/thanos-io\/kube-thanos\" target=\"_blank\" rel=\"noopener\">kube-thanos\u2019 Jsonnet library<\/a> to generate lean Kubernetes manifests for all Thanos components you need in this part: Receive (router + ingestor), Store Gateway, Query, and Compactor. All of these will be scoped to the observability namespace and connected to your thanos-objstore-config Secret and hashring-config ConfigMap.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Here\u2019s what the setup will look like at a high level:<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" src=\"https:\/\/upcloud.com\/media\/image-266-1024x740.png\" alt=\"-\" class=\"wp-image-66715\" \/><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Step 1: Install Jsonnet tooling<\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">To start, first install jsonnet tooling on your system:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">go install github.com\/jsonnet-bundler\/jsonnet-bundler\/cmd\/jb@latest<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">go install github.com\/google\/go-jsonnet\/cmd\/jsonnet@latest<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Ensure $GOPATH\/bin (or your Go bin path) is on your PATH.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Step 2: Set up kube-thanos locally<\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Once jb and jsonnet are ready, run the following commands in a fresh working directory (containing your objstore.yml Secret and hashring-config already applied):<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">jb init<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">jb install github.com\/thanos-io\/kube-thanos\/jsonnet\/kube-thanos@v0.31.0<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This creates a vendor\/ folder with the kube-thanos Jsonnet library.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Step 3: Create the Jsonnet configuration<\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">To configure details like the Thanos image, object storage credentials secret mapping, hashring config mapping, and other specific details about the Thanos setup, you will need to save the following code in a new file named config.jsonnet:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">local t = import 'kube-thanos\/thanos.libsonnet';\n\n\/\/ Shared knobs for all components\nlocal commonConfig = {\n  config+:: {\n    local cfg = self,\n    namespace: 'observability',\n    version: 'v0.31.0',\n    image: 'quay.io\/thanos\/thanos:' + cfg.version,\n    imagePullPolicy: 'IfNotPresent',\n\n    \/\/ Point Thanos components (store\/receive\/compact) to your Secret:\n    \/\/   kubectl -n observability create secret generic thanos-objstore-config \\\n    \/\/     --from-file=objstore.yaml=\/path\/to\/thanos-s3.yaml\n    objectStorageConfig: {\n      name: 'thanos-objstore-config',\n      key:  'objstore.yml',\n    },\n\n    hashringConfigMapName: 'hashring-config', \/\/ required by Receive\n    volumeClaimTemplate: {\n      spec: {\n        accessModes: ['ReadWriteOnce'],\n        resources: { requests: { storage: '10Gi' } },\n      },\n    },\n  },\n};\n\n\/\/ -------------------- Thanos Receive (ingestors + router) --------------------\n\n\/\/ Ingestors: enable shipping to object storage by inheriting commonConfig.objectStorageConfig\n\/\/ and disable ServiceMonitor generation.\nlocal i = t.receiveIngestor(commonConfig.config {\n  replicas: 1,\n  replicaLabels: ['receive_replica'],\n  replicationFactor: 1,\n  serviceMonitor: false,\n});\n\n\/\/ Router: forwards Prometheus Remote Write to ingestors via the hashring.\n\/\/ (Router doesn't generate a ServiceMonitor in kube-thanos.)\nlocal r = t.receiveRouter(commonConfig.config {\n  replicas: 1,\n  replicaLabels: ['receive_replica'],\n  replicationFactor: 1,\n  endpoints: i.endpoints,\n});\n\n\/\/ ----------------------------- Store &amp; Query ---------------------------------\n\n\/\/ Store Gateway: reads blocks from S3 (uses objectStorageConfig); no ServiceMonitor.\nlocal s = t.store(commonConfig.config {\n  replicas: 1,\n  serviceMonitor: false,\n});\n\n\/\/ Query: points to Store + Receive store endpoints; no ServiceMonitor.\nlocal q = t.query(commonConfig.config {\n  replicas: 1,\n  replicaLabels: ['prometheus_replica', 'rule_replica'],\n  serviceMonitor: false,\n  stores: [s.storeEndpoint] + i.storeEndpoints,\n});\n\n\/\/ -------------------------------- Compactor ----------------------------------\n\n\/\/ Compactor: compacts\/downsampes S3 blocks; no ServiceMonitor.\nlocal c = t.compact(commonConfig.config {\n  replicas: 1,\n  serviceMonitor: false,\n});\n\n\/\/ --------------------------- Render to files ---------------------------------\n\n\/\/ Flatten objects to per-component YAMLs the build script can convert to *.yaml.\n{\n  ['thanos-store-' + name]: s[name]\n  for name in std.objectFields(s)\n} +\n{\n  ['thanos-query-' + name]: q[name]\n  for name in std.objectFields(q)\n} +\n{\n  ['thanos-receive-router-' + name]: r[name]\n  for name in std.objectFields(r)\n} +\n{\n  \/\/ receiveIngestor returns top-level bits + a nested \"ingestors\" map (per ring)\n  ['thanos-receive-ingestor-' + name]: i[name]\n  for name in std.objectFields(i)\n  if name != 'ingestors'\n} +\n{\n  ['thanos-receive-ingestor-' + ring + '-' + name]: i.ingestors[ring][name]\n  for ring in std.objectFields(i.ingestors)\n  for name in std.objectFields(i.ingestors[ring])\n  if i.ingestors[ring][name] != null\n} +\n{\n  ['thanos-compact-' + name]: c[name]\n  for name in std.objectFields(c)\n}<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Similar to the <a href=\"https:\/\/github.com\/thanos-io\/kube-thanos\/blob\/main\/example.jsonnet\" target=\"_blank\" rel=\"noopener\">example.jsonnet<\/a> file in the kube-thanos repo, this configuration file defines a Thanos setup that gives you:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Receive Router<\/strong> (fan-out) + <strong>Receive Ingestor<\/strong> (writes TSDB blocks &amp; ships to object storage)<\/li>\n\n\n\n<li><strong>Store Gateway<\/strong> (reads historical blocks from object storage)<\/li>\n\n\n\n<li><strong>Query<\/strong> (PromQL UI + federation across Receive\/Store)<\/li>\n\n\n\n<li><strong>Compactor<\/strong> (compaction &amp; optional downsampling for cost\/latency trade-offs)<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">All components target the observability namespace and reference the thanos-objstore-config secret (your objstore.yml) and the hashring-config configmap.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Step 4: Render manifests<\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">To be able to use the JSONNET file to generate Kubernetes manifests, you\u2019ll need to run the jsonnet and gojsontoyaml tools you installed earlier. Here\u2019s a handy little batch script to help configure the options for these tools and chain them together for your convenience:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">#!\/usr\/bin\/env bash\nset -euo pipefail\nJSONNET=${JSONNET:-jsonnet}\nGOJSONTOYAML=${GOJSONTOYAML:-gojsontoyaml}\nrm -rf manifests &amp;&amp; mkdir manifests\n$JSONNET -J vendor -m manifests \"${1-example.jsonnet}\" \\\n  | xargs -I{} sh -c \"cat {} | $GOJSONTOYAML &gt; {}.yaml; rm -f {}\" -- {}\nfind manifests -type f ! -name '*.yaml' -delete<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Save the script in a file named build.sh and run .\/build.sh config.jsonnet to generate your Thanos manifests. This will write one .yaml per generated manifest under the manifests\/ directory.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Step 5: Apply to the cluster<\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">To apply the generated manifests to your cluster and start up Thanos, run the following command:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">kubectl apply -f manifests\/<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">In a few minutes, you should see resources created for:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>thanos-receive-ingestor-* (StatefulSet + Service)<\/li>\n\n\n\n<li>thanos-receive-router (Deployment + Service)<\/li>\n\n\n\n<li>thanos-store (StatefulSet + Service)<\/li>\n\n\n\n<li>thanos-query (Deployment + Service)<\/li>\n\n\n\n<li>thanos-compact (StatefulSet)<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">You can verify it by running the following command:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">kubectl -n observability get pods<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Make sure to wait until all pods are Running\/Ready before moving ahead.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Optional: Faster feedback while testing<\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">The Thanos ingestor cuts, compacts, and uploads blocks on a 2-hour interval. This means that you might need to wait for two hours to see your first block get uploaded to your object storage after setting everything up. If you want to check it out sooner, you can temporarily shrink the receive ingestor\u2019s TSDB block duration so uploads to object storage happen frequently.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">To do that, run the following command:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">kubectl -n observability patch sts thanos-receive-ingestor-default \\\n  --type='json' \\\n  -p='[{\"op\":\"add\",\"path\":\"\/spec\/template\/spec\/containers\/0\/args\/-\",\"value\":\"--tsdb.min-block-duration=2m\"},\n       {\"op\":\"add\",\"path\":\"\/spec\/template\/spec\/containers\/0\/args\/-\",\"value\":\"--tsdb.max-block-duration=2m\"}]'\n\nkubectl -n observability rollout status sts\/thanos-receive-ingestor-default<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Later, bump back to something saner (or remove both to use ~2h defaults):<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\"># 15-minute blocks are a good middle ground for labs\nkubectl -n observability patch sts thanos-receive-ingestor-default \\\n  --type='json' \\\n  -p='[{\"op\":\"replace\",\"path\":\"\/spec\/template\/spec\/containers\/0\/args\",\"value\":[\"--tsdb.min-block-duration=15m\",\"--tsdb.max-block-duration=15m\"]}]'\n\nkubectl -n observability rollout status sts\/thanos-receive-ingestor-default<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">With Thanos now running, the next step is to point your OpenTelemetry Collector at the Receive router using the prometheusremotewrite exporter, then send requests to your Python app and watch metrics flow into Thanos and persist to object storage.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Configure the OpenTelemetry Collector to Send Metrics to Thanos<\/strong><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Now that Thanos is running, the next step is to connect your OpenTelemetry pipeline so metrics flow end-to-end. The idea is simple: your applications keep exporting OTLP signals, the Collector translates them into <em>Prometheus Remote Write<\/em>, and those metrics are pushed straight into the Thanos Receive router.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Why the extra step with the Collector in the middle? Thanos Receive only understands Prometheus Remote Write. By letting the Collector handle the OTLP to Prometheus conversion, your applications stay Thanos-agnostic while still benefiting from its long-term storage and query capabilities.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Step 1: Define the Collector configuration<\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Start by creating or updating a file named otel-values.yaml. Unless you\u2019ve customized service names, you can use the following as-is:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">mode: 'deployment'\n\nimage:\n  repository: \"otel\/opentelemetry-collector-contrib\"\n\nreplicaCount: 1\n\npresets:\n  clusterMetrics:\n    enabled: true\n  kubernetesEvents:\n    enabled: true\n\nconfig:\n  receivers:\n    otlp:\n      protocols:\n        grpc:\n          endpoint: 0.0.0.0:4317\n        http:\n          endpoint: 0.0.0.0:4318\n    hostmetrics:\n      collection_interval: 60s\n      scrapers: { cpu: {}, memory: {}, load: {} }\n\n  exporters:\n    debug:\n      verbosity: detailed\n    prometheusremotewrite:\n      endpoint: http:\/\/thanos-receive-router.observability.svc.cluster.local:19291\/api\/v1\/receive\n\n  extensions:\n    health_check:\n      endpoint: \"0.0.0.0:13133\"\n\n  service:\n    telemetry:\n      logs:\n        level: debug\n    extensions: [health_check]\n    pipelines:\n      traces:\n        receivers: [otlp]\n        exporters: [debug]\n      metrics:\n        receivers: [otlp, hostmetrics]\n        exporters: [debug, prometheusremotewrite]\n      logs:\n        receivers: [otlp]\n        exporters: [debug]<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Here are the important nodes to keep in mind:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Receivers<\/strong>: You had already defined otlp (HTTP + gRPC) for your apps. You can optionally add hostmetrics for node basics.<\/li>\n\n\n\n<li><strong>Exporter<\/strong>: Add the prometheusremotewrite exporter targeting the Thanos Receive router service on port <strong>19291<\/strong>. Make sure to add to config.service.pipelines.metrics.exporters as well.<\/li>\n\n\n\n<li><strong>Pipelines<\/strong>: Metrics flow to both debug (easy troubleshooting) and prometheusremotewrite (Thanos ingest).<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Step 2: Install\/upgrade the Collector (Helm)<\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">With this updated values.yaml file, upgrade your OpenTelemetry Collector Helm installation by running the following command:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">helm upgrade --install otel-collector-cluster \\\n  open-telemetry\/opentelemetry-collector \\\n  -n observability \\\n  -f otel-values.yaml<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Step 3: Confirm that Collector to Thanos Receive Exports are Working<\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Confirm the Collector is successfully remote-writing by looking at the logs:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">kubectl -n observability logs deploy\/otel-collector-cluster-opentelemetry-collector \\\n  | grep -i prometheusremotewrite | tail -n +1\n# Expect periodic 200 OK \/ \"remote write request successful\" lines<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Optionally, you can also peek at the router\u2019s \/metrics to see it working:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">kubectl -n observability port-forward svc\/thanos-receive-router 10902:10902\ncurl -s localhost:10902\/metrics<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">At this point, your setup is complete! Your Python app is generating OTLP metrics, traces, and logs, and is sending them to the OTel Collector. The Collector is sending metrics off to the Thanos Receive Router, which is routing them to the Thanos Receive Ingestor. The Ingestor is uploading the TSDB blocks to the configured UpCloud Object Storage bucket.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Thanos Store is then querying those data blocks so that you can view them in Thanos Query UI. Compaction runs on the storage bucket to keep storage usage low over the long term. You\u2019ll see all this in action next!<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><em>If you temporarily set <\/em><strong><em>short TSDB blocks<\/em><\/strong><em> on the ingestor (earlier), uploads to object storage will happen faster. This can be very useful while validating the pipeline. However, as mentioned before, you\u2019ll want to revert to longer blocks later for cost\/performance balance.<\/em><\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Generate Some Traffic on Your App<\/strong><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">You\u2019ll reuse the OTLP-instrumented Flask app from Part 1 to generate real traffic and metrics. To do that, port-forward the service and hit the \/ route a bunch of times to emit some spans, logs, and metrics:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">kubectl port-forward svc\/python-app 8080:80\n# in a separate shell:\nfor i in $(seq 1 200); do curl -s http:\/\/localhost:8080\/ &gt;\/dev\/null; done<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>What metrics should you expect?<\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">With opentelemetry-instrumentation-flask and the OTel metrics pipeline enabled, you\u2019ll typically see HTTP server metrics exposed using Prometheus-compatible names derived from OTel semantic conventions, for example:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>http_server_active_requests<\/li>\n\n\n\n<li>http_server_duration_milliseconds_* (histogram family)<\/li>\n\n\n\n<li>process_* and python_* runtime metrics (if present in your env)<\/li>\n\n\n\n<li>Host\/node metrics via Collector\u2019s hostmetrics receiver (e.g., system_cpu_time_*, system_memory_*)<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\"><em>Note: OTel metric names map from OTel instruments (e.g., http.server.active_requests) to Prometheus-style names (e.g., http_server_active_requests). You\u2019ll see how to query these via Thanos Query up next.<\/em><\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Exploring Thanos After Setup<\/strong><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Now that your pipeline is complete, it\u2019s time to explore the setup.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Step 1: Verify Receive and Ingestor Health<\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Check that the Thanos Receive router and ingestors are alive and responding:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">$  otel-p2 kubectl get pods -n observability \nNAME                                                             READY   STATUS    RESTARTS   AGE\notel-collector-cluster-opentelemetry-collector-96ffd4f6b-b67b8   1\/1     Running   0          5h52m\nthanos-compact-0                                                 1\/1     Running   0          7h48m\nthanos-query-744cc4688b-xbdbp                                    1\/1     Running   0          7h48m\nthanos-receive-ingestor-default-0                                1\/1     Running   0          4h51m\nthanos-receive-router-867bb49bf9-9ghl8                           1\/1     Running   0          7h48m\nthanos-store-0                                                   1\/1     Running   0          7h48m<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Then confirm they are serving metrics:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">kubectl -n observability port-forward svc\/thanos-receive-router 10902:10902\ncurl -s localhost:10902\/metrics<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">You should see counters like thanos_receive_write_timeseries that should denote the amount of data received in incoming write requests.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Step 2: Confirm Block Uploads to Object Storage<\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Logs from the ingestor will tell you if TSDB blocks are making it into your UpCloud Object Storage bucket:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">kubectl -n observability logs statefulset\/thanos-receive-ingestor-default<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Look for lines like:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">level=info ts=2025-09-15T10:27:57Z caller=shipper.go:334 msg=\"upload new block\" id=01JZ0NY0MQCY5PTN9T6P8J7HQM<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">If you see <strong>Access Denied<\/strong> or missing credentials errors, revisit your objstore.yml secret.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Step 3: Explore the Thanos Query UI<\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Port-forward the Thanos Query service:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">kubectl -n observability port-forward svc\/thanos-query 9090:9090<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Open <code>http:\/\/localhost:9090<\/code> in your browser. Check the <strong>Stores<\/strong> tab. You should see:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>One or more Receive ingestors<\/li>\n\n\n\n<li>The Store Gateway<\/li>\n<\/ul>\n\n\n\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" src=\"https:\/\/upcloud.com\/media\/image-267-1024x511.png\" alt=\"-\" class=\"wp-image-66718\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">If both appear healthy, the setup is working.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Step 4: Run a PromQL Query<\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Try out some queries that confirm both recent and historical data:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>App-level metric (from your Flask app):<\/strong><\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">rate(http_server_duration_milliseconds_count[12h])<\/code><\/pre>\n\n\n\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" src=\"https:\/\/upcloud.com\/media\/image-268-1024x681.png\" alt=\"-\" class=\"wp-image-66721\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Node\/system metric (via hostmetrics):<\/strong><\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">avg_over_time(system_memory_usage_bytes{state=\"used\"}[6h])<\/code><\/pre>\n\n\n\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" src=\"https:\/\/upcloud.com\/media\/image-268-1024x681.png\" alt=\"-\" class=\"wp-image-66722\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Total requests over 7 days (per service):<\/strong><\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">sum by (service_name) (\n\n&nbsp;&nbsp;increase(http_server_request_duration_seconds_count[7d])\n\n)<\/code><\/pre>\n\n\n\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" src=\"https:\/\/upcloud.com\/media\/image-269-1024x681.png\" alt=\"-\" class=\"wp-image-66723\" \/><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Step 5: Check Store Gateway Sync<\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Confirm that Store Gateway is fetching blocks from your bucket:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">kubectl -n observability logs statefulset\/thanos-store<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Healthy logs look like:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">msg=\"successfully synchronized block metadata\" cached=10 returned=10 partial=0<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>What You\u2019ve Achieved<\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">At this point:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Short-term metrics<\/strong> flow directly through Prometheus-like semantics via Receive.<\/li>\n\n\n\n<li><strong>Long-term blocks<\/strong> are stored in UpCloud Object Storage via the ingestors.<\/li>\n\n\n\n<li><strong>Query federation<\/strong> lets you ask questions across both live and historical data.<\/li>\n\n\n\n<li><strong>Grafana dashboards<\/strong> can now cover everything from real-time incidents to multi-month trends.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">This completes your production-grade observability stack on UpCloud, combining OpenTelemetry with Thanos for cost-efficient, scalable, and durable metrics analysis.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Conclusion<\/strong><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">At this stage, you\u2019ve moved from a short-term metrics pipeline into something much more powerful. Your applications now send telemetry over OTLP to the OpenTelemetry Collector, which forwards it directly into Thanos Receive. From there, metrics are shipped into UpCloud Object Storage for long-term retention, and Thanos Query stitches everything together so you can explore both live and historical data through a single PromQL interface. The end result is an observability stack that doesn\u2019t just tell you what\u2019s happening right now, but also lets you look back weeks or months to spot trends, review incidents, and make informed decisions about capacity and performance.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">As you continue working with this setup, it\u2019s worth thinking about how to refine it for your needs. Thanos\u2019s multi-tenancy features give you a way to isolate metrics between teams or environments, making governance cleaner and scaling easier. Compaction and downsampling settings are another lever: by tuning them, you can find the right balance between keeping costs manageable and ensuring queries remain fast even as your dataset grows.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">With those considerations in mind, you\u2019ve built a system that\u2019s not only functional but also efficient and scalable. It\u2019s a solid foundation for long-term observability on UpCloud, and one that will grow with your workloads.<\/p>\n","protected":false},"author":19,"featured_media":0,"comment_status":"open","ping_status":"closed","template":"","community-category":[229,238,232],"class_list":["post-1822","tutorial","type-tutorial","status-publish","hentry"],"acf":[],"_links":{"self":[{"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/tutorial\/1822","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/tutorial"}],"about":[{"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/types\/tutorial"}],"author":[{"embeddable":true,"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/users\/19"}],"replies":[{"embeddable":true,"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/comments?post=1822"}],"version-history":[{"count":0,"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/tutorial\/1822\/revisions"}],"wp:attachment":[{"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/media?parent=1822"}],"wp:term":[{"taxonomy":"community-category","embeddable":true,"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/community-category?post=1822"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}