Supercharge Your CI/CD: Deploy Lightning-Fast GitHub Actions Runners on UpCloud’s Managed Kubernetes: Part 4

UpCloud Kubernetes tutorial series:
Part 1 – Setting up a functional self-hosted runner
Part 2 – Advanced configurations for security and scalability
Part 3 – Maintenance and troubleshooting with monitoring and logging
Part 4 – Managing Kubernetes secrets and credentials

Welcome back to this series on supercharging your CI/CD pipelines by deploying lightning-fast GitHub Actions Runners on UpCloud’s Managed Kubernetes! In Part 3 of this series, you looked at monitoring, logging, and some common troubleshooting scenarios, including diagnosing runner registration issues, debugging scale-set controllers, and effectively examining logs to resolve operational hiccups. Now, in this final installment, the focus shifts to fortifying your GitHub Actions runners with solid security practices to ensure your CI/CD pipeline remains safe, reliable, and compliant.

This part will begin by discussing how to securely manage Kubernetes secrets and credentials, highlighting best practices for safeguarding GitHub tokens and sensitive configuration data. Next, it’ll dive into Kubernetes Role-Based Access Control (RBAC) to demonstrate how you can effectively control permissions and define roles to secure your runners and other resources in your cluster.

Then, we’ll cover essential runner maintenance strategies, highlighting methods to set up regular security updates and patching. Wrapping up, you’ll get a handy security checklist to ensure you’re consistently following best practices, helping you maintain a secure and stable CI/CD environment long-term.

Managing Secrets and Credentials

Securely handling secrets and sensitive credentials, such as GitHub tokens and configuration data, is critical for maintaining a secure and reliable CI/CD pipeline. Kubernetes Secrets offer an effective way to store this sensitive information securely by encoding it and limiting access to only authorized pods.

When storing GitHub tokens or credentials within Kubernetes, you should always ensure they’re managed through Secrets rather than plain-text configuration files or environment variables. Needless to say, it is extremely important to avoid directly embedding tokens into manifests or Docker images. This is because manifests and images are very easy for anyone who has (or unlawfully gains) access to your cluster to access.

Here are some other best practices to keep in mind:

• Use namespaces: Store secrets within dedicated namespaces, restricting access through RBAC to ensure only relevant services can access them. You might recall using the arc-runners namespace when deploying the runners. Make sure all secrets and credentials are limited to only this namespace.
• Limit Secret access: Utilize RBAC roles to restrict secret access strictly to pods and services that require them. You’ll learn more about RBAC in the next section.
• Use fine-grained GitHub access tokens: Prefer fine-grained access tokens over classic access tokens wherever possible. Remember, you only need the repo (full control) and workflow (full control) scopes on your target GitHub repository or organization to set up your runner, as mentioned in part 1.
• Regularly rotate tokens: You should frequently update GitHub tokens and credentials to mitigate potential risks from leaks or breaches. Make sure to set an expiration date on the tokens when creating them to ensure that it necessitates updating them at set intervals.

It is important to note that ARC also supports authenticating via a GitHub App to avoid token maintenance. However, this is only available for non-enterprise use cases. Here are the docs to help you get going with this.

Understanding Role-Based Access Control (RBAC) in GitHub Actions Runner Controller

The GitHub Actions Runner Controller (ARC) comes preconfigured with Role-Based Access Control (RBAC) resources, ensuring secure operations out of the box. When you deploy ARC, several key RBAC resources are automatically created using predefined Helm templates, specifically tailored to managing GitHub Actions runner scale sets and their controllers.

Key RBAC Roles in ARC

The main RBAC roles created during deployment include:

• manager_cluster_role.yaml for the controller: Defines a cluster-wide ClusterRole with broader permissions necessary for the controller when managing runners across multiple namespaces. This is used by default for every deployment, and it allows the runner controller to access all resources across all namespaces in the cluster. A common security practice would be to disable that by setting flags.watchSingleNamespace to the name of the namespace when deploying the runner controller.
• manager_single_namespace_controller_role.yaml for the controller: Used specifically when ARC is restricted to a single namespace (flags.watchSingleNamespace=<controller namespace>). It securely binds the single-namespace controller to its ClusterRole.
• manager_role.yaml for the scaleset: Defines a Kubernetes Role granting permissions such as managing runner pods and runner scale set custom resources within a specific namespace.

The helm charts you use to install the controller and the runner scaleset automatically generate roles and rolebindings using these templates. You can customize the generated roles to further fine-tune the role-based access control setup in your cluster.

Restricting Controller to a single namespace

A common customization with these is to limit the controller to only access resources within its namespace. To do that, you will need to create a values file for it that sets the value of flags.watchSingleNamespace to “arc-systems” (or the namespace in which the controller will be deployed):

image:
  repository: "ghcr.io/actions/gha-runner-scale-set-controller"
  pullPolicy: IfNotPresent
  tag: ""

serviceAccount:
  create: true
  annotations: {}
  name: ""

# ..other values

flags:
  watchSingleNamespace: "arc-systems" # set this to the namespace where the controller is deployed

You need to install the controller using this values file.

Next, you also need to set the value of the controller’s service account (namespace and name) in the runner scaleset deployment so that it is able to bind the relevant role to the controller’s service account:

# values.yaml

template:
    spec:
      containers:
      - name: runner
        image: ghcr.io/actions/actions-runner:latest
        command: ["/home/runner/run.sh"]
        
controllerServiceAccount:
  namespace: arc-system
  name: arc-gha-rs-controller # This must be set as <controller installation name>-gha-rs-controller

Now, once you install the runner scale set using this values file and run kubectl get rolebinding -n arc-systems, you will notice the following output:

➜  gh-actions-runner-p4 kubectl get rolebinding -n arc-systems
NAME                                           ROLE                                                AGE
arc-gha-rs-controller-listener                 Role/arc-gha-rs-controller-listener                 8m16s
arc-gha-rs-controller-single-namespace         Role/arc-gha-rs-controller-single-namespace         8m16s
arc-gha-rs-controller-single-namespace-watch   Role/arc-gha-rs-controller-single-namespace-watch   8m16s

This shows that the roles and rolebindings for the single namespace mode have been set up for the controller. The controller will now not be able to access resources outside its own namespace arc-systems.

Other RBAC tips

Here are a few other tips you can use to get the most out of RBAC with GitHub Actions Runner Controller:

• Implement Principle of least privilege: Make sure to grant the minimum required access to each user or service. Feel free to update the default roles and rolebindings created by the controller to do so.
• Implement namespace isolation: Particularly when you are hosting multiple runner scale sets on the same cluster, make sure to isolate runner operations into specific namespaces to limit cross-namespace access. Also, implement strict resource limits as well.
• Audit regularly: Regularly review and refine RBAC rules to adapt to changing requirements or security policies.

Regular Updates and Runner Maintenance

Keeping your GitHub Actions runners regularly updated is vital to maintaining a secure and high-performance CI/CD environment. Outdated runner versions or unpatched vulnerabilities can expose your infrastructure to risks, compromise sensitive data, or cause build instability.

To mitigate these risks, you should regularly update the runner software and apply security patches.

Updating Your Runner Deployment

While you can update your runner scalesets using Helm, it is not possible to update the controller using Helm. Therefore, to completely update your runner deployment, you need to uninstall all instances of the runner scalesets, then uninstall the controller, remove all CRDs associated with the actions.github.com API group, and then reinstall everything, as explained in part 1. Here’s the official guide on how to go about doing that.

A Few Tips for Runner Maintenance

• Automate security scanning: Use container scanning tools (e.g., Trivy or Clair) to identify vulnerabilities in your runner images regularly.
• Stay informed: Subscribe to runner release notifications to promptly address critical updates or security fixes.
• Backup and recovery: Ensure regular backups of configurations, enabling quick recovery in the event of an issue.
• Scheduled maintenance windows: Plan for routine, scheduled maintenance periods to apply critical patches and updates.

Security Checklist and Best Practices

You can use this checklist as a quick reference to maintain a secure CI/CD environment when managing GitHub Actions runners on UpCloud’s Managed Kubernetes:

• Manage Secrets Securely
  • Store sensitive credentials in Kubernetes Secrets.
  • Avoid embedding tokens directly in manifests or Docker images.
  • Regularly rotate GitHub tokens and secrets.
  • Enable encryption at rest for Kubernetes secrets.
• Implement Robust RBAC
  • Make sure to limit permissions by using dedicated namespaces for isolation.
  • Consider modifying the default roles offered by Actions Runner Controller to further meet your security requirements.
• Regularly Update and Patch
  • Make sure to keep your Actions Runner Controller and your runner images up to date.
  • If you use custom images, regularly scan them for vulnerabilities.
  • Schedule periodic maintenance windows to apply critical patches to your runner controller and scalesets as need arises.

Feel free to add or remove points to this checklist based on your specific needs. Following a checklist such as this one ensures your GitHub Actions runners remain secure, resilient, and compliant, significantly reducing your organization’s exposure to potential security threats.

Conclusion

And that brings this series to an end! Throughout this series, we have explored self-hosting GitHub Actions Runner Controller on a UpCloud Managed Kubernetes cluster in detail. Starting from creating the right type of cluster to ensuring all roles and security configurations are in place, we have looked at everything that you need to know to get the most out of GitHub Actions Runner Controller. For further reading, feel free to take a look at the GitHub docs. Make sure to try out the GitHub Actions Runner on an UpCloud Managed Kubernetes cluster today!