{"id":1891,"date":"2025-04-22T09:00:00","date_gmt":"2025-04-22T06:00:00","guid":{"rendered":"https:\/\/upcloud.com\/global\/us\/resources\/tutorials\/deploy-lightning-fast-github-actions-runners-on-upclouds-managed-kubernetes-part-3\/"},"modified":"2025-04-22T09:00:00","modified_gmt":"2025-04-22T06:00:00","slug":"deploy-lightning-fast-github-actions-runners-on-upclouds-managed-kubernetes-part-3","status":"publish","type":"tutorial","link":"https:\/\/upcloud.com\/global\/resources\/tutorials\/deploy-lightning-fast-github-actions-runners-on-upclouds-managed-kubernetes-part-3\/","title":{"rendered":"Deploy Lightning-Fast GitHub Actions Runners on UpCloud\u2019s Managed Kubernetes: Part 3"},"content":{"rendered":"\n\n\n

Welcome back to the four-part series on supercharging your CI\/CD pipelines by deploying lightning-fast GitHub Actions Runners on UpCloud\u2019s Managed Kubernetes! In Part 2<\/a>, we looked at advanced configurations, including customizing runner deployments, implementing network policies for enhanced security, and setting up autoscaling to optimize costs.<\/p>\n\n\n\n

In this installment, Part 3, we turn our attention to maintaining and troubleshooting your self-hosted runner environment. From monitoring runner health to diagnosing and resolving common issues like registration errors, resource constraints, and job execution failures, we will look at the tools and techniques needed to ensure smooth operations. Let\u2019s dive right in!<\/p>\n\n\n\n

Monitoring Runners<\/strong><\/h2>\n\n\n\n

Monitoring self-hosted GitHub Actions runners is important to ensure the reliability, performance, and scalability of your CI\/CD pipelines. By tracking a few key metrics, you can identify bottlenecks, optimize resource utilization, and promptly address issues before they impact your development workflow.<\/p>\n\n\n\n

Luckily, GitHub Actions Runner ships with out-of-the-box support for key performance and usage metrics. In this section, you will learn how to enable them and which ones to track.<\/p>\n\n\n\n

Exposing Metrics From Your GitHub Actions Runner Controller<\/strong><\/h3>\n\n\n\n

To enable Prometheus to collect metrics from the Actions Runner Controller, first update the controller\u2019s values file to enable the metrics:<\/p>\n\n\n\n

metrics:\n  controllerManagerAddr: \":8080\"\n  listenerAddr: \":8080\"\n  listenerEndpoint: \"\/metrics\"<\/code><\/pre>\n\n\n\n
You will need to do a fresh install of the controller to apply these changes.<\/p>\n\n\n\n
Next, you will also need to enable the default listener metrics in the scalesets you deploy in your cluster. To do that, create a values file for the scaleset and save the following contents in it:<\/p>\n\n\n\n
listenerMetrics:\n  counters:\n    gha_started_jobs_total:\n      labels:\n        [\"repository\", \"organization\", \"enterprise\", \"job_name\", \"event_name\"]\n    gha_completed_jobs_total:\n      labels:\n        [\n          \"repository\",\n          \"organization\",\n          \"enterprise\",\n          \"job_name\",\n          \"event_name\",\n          \"job_result\",\n        ]\n  gauges:\n    gha_assigned_jobs:\n      labels: [\"name\", \"namespace\", \"repository\", \"organization\", \"enterprise\"]\n    gha_running_jobs:\n      labels: [\"name\", \"namespace\", \"repository\", \"organization\", \"enterprise\"]\n    gha_registered_runners:\n      labels: [\"name\", \"namespace\", \"repository\", \"organization\", \"enterprise\"]\n    gha_busy_runners:\n      labels: [\"name\", \"namespace\", \"repository\", \"organization\", \"enterprise\"]\n    gha_min_runners:\n      labels: [\"name\", \"namespace\", \"repository\", \"organization\", \"enterprise\"]\n    gha_max_runners:\n      labels: [\"name\", \"namespace\", \"repository\", \"organization\", \"enterprise\"]\n    gha_desired_runners:\n      labels: [\"name\", \"namespace\", \"repository\", \"organization\", \"enterprise\"]\n    gha_idle_runners:\n      labels: [\"name\", \"namespace\", \"repository\", \"organization\", \"enterprise\"]\n  histograms:\n    gha_job_startup_duration_seconds:\n      labels:\n        [\"repository\", \"organization\", \"enterprise\", \"job_name\", \"event_name\"]\n      buckets:\n        [\n          0.01,\n          0.05,\n          0.1,\n          0.5,\n          1.0,\n          2.0,\n          3.0,\n          4.0,\n          5.0,\n          6.0,\n          7.0,\n          8.0,\n          9.0,\n          10.0,\n          12.0,\n          15.0,\n          18.0,\n          20.0,\n          25.0,\n          30.0,\n          40.0,\n          50.0,\n          60.0,\n          70.0,\n          80.0,\n          90.0,\n          100.0,\n          110.0,\n          120.0,\n          150.0,\n          180.0,\n          210.0,\n          240.0,\n          300.0,\n          360.0,\n          420.0,\n          480.0,\n          540.0,\n          600.0,\n          900.0,\n          1200.0,\n          1800.0,\n          2400.0,\n          3000.0,\n          3600.0,\n        ]\n    gha_job_execution_duration_seconds:\n      labels:\n        [\n          \"repository\",\n          \"organization\",\n          \"enterprise\",\n          \"job_name\",\n          \"event_name\",\n          \"job_result\",\n        ]\n      buckets:\n        [\n          0.01,\n          0.05,\n          0.1,\n          0.5,\n          1.0,\n          2.0,\n          3.0,\n          4.0,\n          5.0,\n          6.0,\n          7.0,\n          8.0,\n          9.0,\n          10.0,\n          12.0,\n          15.0,\n          18.0,\n          20.0,\n          25.0,\n          30.0,\n          40.0,\n          50.0,\n          60.0,\n          70.0,\n          80.0,\n          90.0,\n          100.0,\n          110.0,\n          120.0,\n          150.0,\n          180.0,\n          210.0,\n          240.0,\n          300.0,\n          360.0,\n          420.0,\n          480.0,\n          540.0,\n          600.0,\n          900.0,\n          1200.0,\n          1800.0,\n          2400.0,\n          3000.0,\n          3600.0,\n        ]\n\ntemplate:\n    spec:\n      containers:\n      - name: runner\n        image: ghcr.io\/actions\/actions-runner:latest\n        command: [\"\/home\/runner\/run.sh\"]<\/code><\/pre>\n\n\n\n
Now, you can verify that the metrics are being exposed by setting up a port forward and accessing them in your browser:<\/p>\n\n\n\n
kubectl port-forward pod\/<your-controller-pod-name> 8080:8080 -n arc-systems<\/code><\/pre>\n\n\n\n
You can get your controller pod\u2019s name by running kubectl get pods -n arc-systems<\/code> and looking for a pod that has the word \u201ccontroller\u201d in its name:<\/p>\n\n\n\n
\u279c  gh-actions-runner-p3 kubectl get pods -n arc-systems\nNAME                                     READY   STATUS    RESTARTS   AGE\narc-gha-rs-controller-5d96468674-5hctf   1\/1     Running   0          157m\narc-runner-set-754b578d-listener         1\/1     Running   0          154m\n<\/code><\/pre>\n\n\n\n
Once you have set up the port forwarding, try accessing the controller metrics now at http:\/\/localhost:8080\/metrics<\/code>. You should see something like this:<\/p>\n\n\n\n
\"-\"<\/figure>\n\n\n\n
Similarly, you can try port-forwarding the listener pod and viewing the metrics that it exposes:
<\/p>\n\n\n\n
\"-\"<\/figure>\n\n\n\n
Understanding GitHub Actions Runner metrics<\/strong><\/h3>\n\n\n\n
The configuration you have used above enabled and exposed all metrics supported by the Actions Runner Controller. The GitHub docs describe these metrics here<\/a>.<\/p>\n\n\n\n
If you were planning to build a performance monitoring dashboard, here are some metrics that would come in handy:<\/p>\n\n\n\n
Metric<\/strong><\/td>Description<\/strong><\/td>Purpose<\/strong><\/td><\/tr>
gha_running_jobs<\/td>Number of jobs currently running or queued<\/td>Monitors workload and queue depth<\/td><\/tr>
gha_assigned_jobs<\/td>Number of jobs assigned to the runner scale set<\/td>Measures how many jobs were accepted for processing<\/td><\/tr>
gha_started_jobs_total<\/td>Cumulative number of jobs started<\/td>Tracks overall job volume<\/td><\/tr>
gha_completed_jobs_total<\/td>Cumulative number of jobs completed<\/td>Ensures all jobs complete successfully<\/td><\/tr>
gha_job_startup_duration_seconds<\/td>Time to start a job (histogram)<\/td>Highlights scheduling\/startup latency<\/td><\/tr>
gha_job_execution_duration_seconds<\/td>Job execution time (histogram)<\/td>Helps monitor execution duration trends and outliers<\/td><\/tr>
gha_busy_runners<\/td>Number of runners currently executing a job<\/td>Shows how much capacity is being used<\/td><\/tr>
gha_idle_runners<\/td>Number of idle runners<\/td>Identifies underutilized capacity<\/td><\/tr>
gha_desired_runners<\/td>Target number of runners to scale to<\/td>Verifies if autoscaling is working correctly<\/td><\/tr>
gha_min_runners, gha_max_runners<\/td>Scale bounds set for the scale set<\/td>Helps correlate with autoscaling behaviors<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
For a health monitoring dashboard, the following metrics would be useful:<\/p>\n\n\n\n
Metric<\/strong><\/td>Description<\/strong><\/td>Purpose<\/strong><\/td><\/tr>
gha_controller_pending_ephemeral_runners<\/td>Number of runners stuck in pending state<\/td>Detects issues in provisioning runners<\/td><\/tr>
gha_controller_running_ephemeral_runners<\/td>Runners that are active<\/td>Baseline to compare expected runner load<\/td><\/tr>
gha_controller_failed_ephemeral_runners<\/td>Runners that failed to start<\/td>Alerts on provisioning or execution issues<\/td><\/tr>
gha_controller_running_listeners<\/td>Number of active listener pods<\/td>Verifies if listener components are healthy<\/td><\/tr>
gha_registered_runners<\/td>Runners registered with the scale set<\/td>Tracks overall runner availability<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
Some other insights you can draw from these metrics include:<\/p>\n\n\n\n
Metric<\/strong><\/td>Description<\/strong><\/td>Purpose<\/strong><\/td><\/tr>
Job Success Rate<\/td>gha_completed_jobs_total \/ gha_started_jobs_total<\/td>Helps monitor CI stability and test flakiness<\/td><\/tr>
Runner Utilization Rate<\/td>gha_busy_runners \/ gha_registered_runners<\/td>Determines scaling efficiency<\/td><\/tr>
Startup Latency Trends<\/td>Percentiles from gha_job_startup_duration_seconds<\/td>Identify if jobs are waiting too long to start (e.g., due to image pull or resource contention)<\/td><\/tr>
Execution Time Trends<\/td>Histogram from gha_job_execution_duration_seconds<\/td>Benchmark CI job complexity and highlight anomalies<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
Also, all counter metrics (like gha_started_jobs_total) reset when the pod restarts. So you must ensure you\u2019re using Prometheus with a persistent backend (like Thanos<\/a> or Cortex<\/a>) or reset-aware alerting. And you can also use histogram metrics (like durations) to create SLOs (e.g., \u201c90% of jobs start within 10s\u201d).<\/p>\n\n\n\n
Understanding Runner Logs<\/strong><\/h2>\n\n\n\n
Now that you have an active monitoring setup in place, it is time to look at the tools to use when you need to troubleshoot your controllers. Analyzing runner logs can come in quite handy for identifying and diagnosing issues in your self-hosted runners. Logs provide valuable insights into runner operations, errors, and warnings, allowing you to resolve problems quickly and maintain seamless CI\/CD workflows.<\/p>\n\n\n\n
Below are step-by-step instructions for accessing and interpreting runner logs within your Kubernetes environment:<\/p>\n\n\n\n
Identify Your Runner Pods<\/strong><\/h3>\n\n\n\n
First, identify the runner pods deployed by the Actions Runner Controller (ARC). Use the following command to list runner pods in your ARC namespace (typically arc-runners):<\/p>\n\n\n\n
kubectl get pods -n arc-runners<\/code><\/pre>\n\n\n\n
This command lists all runner pods along with their current status:<\/p>\n\n\n\n
NAME                                           READY   STATUS    RESTARTS   AGE\nrunner-deployment-example-runner-abcde-12345   2\/2     Running   0          3h<\/code><\/pre>\n\n\n\n
\n
It is possible to receive an empty (\u201cno resources found\u201d) response for this command. This can happen if the runner has finished executing a job and has been terminated by the scaleset. In such situations, you might need to add a temporary sleep 3600 or similar step in your GitHub Actions workflow file and trigger the workflow again to make the runner pod stay alive for longer while you can try identifying it and accessing its logs.<\/em><\/p>\n<\/blockquote>\n\n\n\n
Access Runner Logs Using kubectl<\/strong><\/h3>\n\n\n\n
Once you have retrieved the name of the runner pod, run the following command to access its logs:<\/p>\n\n\n\n
kubectl logs <runner-pod-name> -n arc-runners<\/code><\/pre>\n\n\n\n
Analyze Runner Logs for Common Errors<\/strong><\/h3>\n\n\n\n
Now, review the logs carefully, looking specifically for patterns like:<\/p>\n\n\n\n