Best practices for UpCloud Managed Kubernetes

UpCloud Managed Kubernetes gives you a production-ready cluster out of the box, with the control plane fully managed for you. The worker nodes and everything running on them are yours to configure, and a few small adjustments can make a real difference to how well your cluster handles node failures, upgrades, and busy periods.

This guide covers practical recommendations based on common patterns we see across customer clusters. You don't need to read it end to end - jump to the sections most relevant to your setup.

Make CoreDNS resilient to node failures

Every DNS lookup in your cluster goes through CoreDNS, running as a deployment in the kube-system namespace. If CoreDNS becomes unavailable, name resolution fails for every workload at once, including lookups for external services and managed database hostnames. It is worth a few minutes of hardening.

CoreDNS is part of your cluster's data plane, so you are free to customise it, and your changes persist across cluster version upgrades. See the shared responsibility model for how this split works.

Check what your cluster ships

Newly created clusters ship with two CoreDNS replicas and a rule that spreads them across different nodes. Clusters created earlier ship with a single replica and no scheduling rules, and existing clusters are not changed retroactively - so check what your cluster actually runs:

kubectl get deployment coredns -n kube-system

kubectl -n kube-system get deploy coredns \
  -o jsonpath='{.spec.template.spec.topologySpreadConstraints}'

A single replica is a single point of failure: if the node hosting it fails, DNS is down for the whole cluster until the pod is rescheduled or the node recovers. And two replicas without a spread rule are not much better - nothing stops both from landing on the same node, so one node failure can still take out cluster DNS.

If the first command shows two replicas and the second returns a constraint with topologyKey: kubernetes.io/hostname, your cluster already has the important part of the setup below - skip ahead and add the PodDisruptionBudget. If either is missing, apply the full patch.

Spread the replicas across nodes

The fix is to run at least two replicas and add a topology spread constraint so they always sit on different nodes. You can do both with a single patch:

kubectl -n kube-system patch deploy coredns --type=strategic -p '
spec:
  replicas: 2
  template:
    spec:
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              k8s-app: coredns'

Note that CoreDNS pods on Managed Kubernetes carry the label k8s-app=coredns, which is what the constraint above selects on.

Avoid a bare kubectl scale --replicas=2 on its own. It usually looks fine, because the scheduler tends to spread pods anyway, but nothing guarantees it. Both replicas can end up on one node and you are back to a single point of failure without knowing it.

Then add a PodDisruptionBudget so that node drains, including the ones that happen during cluster version upgrades, never take both replicas down at once:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: coredns-pdb
  namespace: kube-system
spec:
  minAvailable: 1
  selector:
    matchLabels:
      k8s-app: coredns

A PodDisruptionBudget cuts both ways. It protects the workload during drains, but it also blocks any drain it cannot satisfy - including the ones the platform performs when you scale a node group down. A budget whose minAvailable equals the replica count can never be satisfied, so always keep minAvailable below the replica count, as the manifest above does. If a node group scale-down ever stops progressing, an unsatisfiable budget is the first thing to check - see How to scale your Managed Kubernetes cluster for the details.

Once the rollout finishes, check that the replicas really are on different nodes:

kubectl get pods -n kube-system -l k8s-app=coredns \
  -o custom-columns=NAME:.metadata.name,STATUS:.status.phase,NODE:.spec.nodeName

NAME                       STATUS    NODE
coredns-56d6976dc7-prcrc   Running   default-g2ldr-jxtw6
coredns-56d6976dc7-qzzdk   Running   default-g2ldr-jdg9r

Two pods, two different values in the NODE column - that is what you want to see.

If you prefer pod anti-affinity over topology spread constraints, that works too. Spread constraints behave a little better during node replacement, which is how Managed Kubernetes upgrades work, but there is no need to switch an existing anti-affinity setup.

A few things to watch out for:

Right after patching, both replicas can briefly land on the same node. The old terminating pod still counts toward the spread calculation while the rollout is in progress. If you see both new pods on one node, delete one of them - the replacement is forced onto a different node:

kubectl -n kube-system delete pod <one-of-the-coredns-pods>

A rollout restart is not a reliable fix for this on small clusters, so use the pod delete instead.

You need at least as many worker nodes as replicas. With DoNotSchedule, an extra replica sits Pending until a node is free - this applies to the shipped defaults too, so a brand-new single-node cluster runs one CoreDNS replica with the second Pending until a second node exists. Scaling down to one node does not break DNS - one replica keeps serving - but the redundancy is gone until another node joins.

This is not zero-downtime against frozen nodes. When a node freezes rather than cleanly failing, its pod stays in the Service endpoints until the node is marked NotReady, which takes roughly 40 seconds plus eviction timings. During that window some queries still hit the dead pod and time out. Two spread replicas turn a total outage into a partial one, and clients that retry ride it out. This is standard Kubernetes behaviour, not specific to Managed Kubernetes.

This matters for upgrades, not just failures. A Managed Kubernetes version upgrade replaces worker nodes one at a time, draining pods as it goes. With a single CoreDNS replica, that means a brief DNS gap during every routine upgrade, not just during rare node failures. With two spread replicas and the PodDisruptionBudget in place, at least one replica keeps serving throughout.

Keep the configuration in your manifests

Your CoreDNS changes persist and survive cluster version upgrades. Even so, keep the patched configuration in your own manifests or GitOps repository, alongside the rest of your cluster configuration. That way it re-applies itself no matter what, and it is visible to anyone else working on the cluster.

Set resource requests and limits on your workloads

The single most common cause of node trouble is a workload running without memory limits. A memory leak or an unexpected traffic spike lets the pod grow until it exhausts the node, and a node under memory pressure can slow down, evict other pods, or freeze entirely. If that node also happens to host your only CoreDNS replica, one misbehaving application takes down DNS for the whole cluster - which is exactly the failure mode the previous section defends against.

Worker nodes and everything scheduled on them are your responsibility, and resource requests and limits are the main tool for keeping one workload from harming the rest.

Requests are what Kubernetes reserves for the pod. The scheduler uses them to pick a node with enough free capacity, so a pod with honest requests lands somewhere it can actually run.

Limits are the ceiling. The two resources behave differently when the ceiling is hit, and the difference matters:

• A pod exceeding its memory limit is terminated (OOMKilled) and restarted. That sounds harsh, but it is protective behaviour: better to restart one pod than to crash the whole node.
• A pod hitting its CPU limit is throttled, not killed. It keeps running, just slower. Throttling shows up as latency, so an over-tight CPU limit can be hard to spot - the pod looks healthy while its clients time out.

For most workloads: always set memory requests and limits, always set a CPU request, and treat CPU limits as optional. If you do set a CPU limit, leave real headroom above typical usage.

Check what your workloads have set

kubectl get deployments --all-namespaces

Then for each of your deployments:

kubectl describe deployment <deployment-name> -n <namespace>

Look for the Limits and Requests under each container in the pod template. If they are missing, empty, or set to values so high they limit nothing, the workload is unprotected.

Measure before you choose values

To see what a pod actually uses:

kubectl top pod <pod-name> -n <namespace>

Note that kubectl top needs the Metrics Server, which Managed Kubernetes does not ship by default - error: Metrics API not available on a fresh cluster is expected, not a fault. You can install it in two commands - see our Metrics Server installation guide - or use your own monitoring stack if you already run one.

One thing to be aware of once metrics are in place: a node whose kubelet has stopped reporting does not show zero usage - its metrics disappear, and some tools render that absence as 0. A NotReady node showing zeros is a node that is not reporting, not a node that is idle. Check its conditions with kubectl describe node before drawing any conclusions about resource pressure; when the kubelet is not reporting, every condition reads Unknown with reason NodeStatusUnknown. The scaling guide covers how to read those conditions.

A reasonable starting point once you know typical usage: set the memory request to typical usage, the memory limit 30 to 50 percent above it, and the CPU request to typical usage or 100m, whichever is higher - startup and traffic spikes need more than an idle measurement suggests. If the pod is mostly idle when you measure, current usage is not a good guide; start from a sensible round number for the type of application instead and tune from there.

Apply the values

The quickest way:

kubectl set resources deployment <deployment-name> -n <namespace> \
  --requests=memory=1000Mi,cpu=100m \
  --limits=memory=1300Mi

Or set the resources block directly in your manifests, which is the better long-term home for it:

resources:
  requests:
    cpu: 100m
    memory: 1000Mi
  limits:
    memory: 1300Mi

The change rolls the deployment. Watch it complete with kubectl rollout status, and if something goes wrong, kubectl rollout undo puts the previous configuration back.

Expect to tune. If a pod starts getting OOMKilled after you add limits (check with kubectl describe pod - look for Last State: Terminated, Reason: OOMKilled), your limit was below the application's real peak. Raise it with more headroom and go again. Setting limits is an iterative process, not a one-shot calculation - and a pod that keeps growing past any reasonable limit is usually telling you about a memory leak, not a wrong limit.

Set namespace defaults so nothing slips through

Managed Kubernetes does not apply resource defaults to your workloads - a pod deployed without a resources block runs unbounded. A LimitRange closes that gap by giving every new pod in a namespace sensible defaults automatically:

kubectl apply -n <namespace> -f - <<EOF
apiVersion: v1
kind: LimitRange
metadata:
  name: default-resource-limits
spec:
  limits:
    - type: Container
      default:
        memory: 512Mi
      defaultRequest:
        memory: 256Mi
        cpu: 100m
EOF

Adjust the values to suit the namespace, and treat them as a safety net rather than a substitute for setting resources deliberately on each workload.

Remove worker nodes through the platform, not kubectl

Worker nodes belong to a node group, and the node group is managed by the platform - the UpCloud service that provisions, joins, drains, and removes worker nodes on your behalf. When you need a node gone, change the node group - scale it down, or remove the specific node through the API - and let the platform handle the drain and the server removal.

What you should not do is kubectl delete node. It looks like the obvious command, and on a cluster you run yourself it often is. On Managed Kubernetes it only removes the Node object from the Kubernetes API. The Cloud Server behind it keeps running and keeps being billed, the node group keeps counting it, and the platform can no longer manage it. The kubelet on that server does not re-register itself either - it registers once at startup and only sends status updates after that, so it sits there logging node not found indefinitely. The node group moves to a scaling-up state and stays there until the server is dealt with.

To remove a specific node, use the per-node endpoint:

curl -X DELETE -H "Authorization: Bearer $UPCLOUD_TOKEN" \
  https://api.upcloud.com/1.3/kubernetes/<cluster-uuid>/node-groups/<group-name>/<node-name>

This drains the node, deletes the server, and reduces the node group count by one. The node name is the same one kubectl get nodes shows. If you use it on a node that is missing from kubectl get nodes, the platform may keep the count and build a replacement instead.

If you have already deleted a Node object, there are two ways back. Restarting the kubelet on that server (sudo systemctl restart kubelet, logged in as debian) makes it register again within seconds - this needs SSH access, which means an SSH key configured on the node group. Restarting the whole server from the control panel or the server API does the same, at the cost of a reboot for the pods on it. Or call the per-node delete endpoint for it, which removes the server cleanly and lets the platform build a replacement.

Don't stop or delete a worker's Cloud Server through the server API or the server view either. A worker that has been shut down just sits NotReady until you start it again; a deleted one leaves a phantom entry that parks the node group in scaling-up until you remove it with the per-node delete endpoint, and the platform does not build a replacement.

The scaling guide covers how to tell whether you have an orphaned server (the node group count is higher than kubectl get nodes shows) and what the cluster autoscaler does with one.

Configure the cluster autoscaler deliberately

The cluster autoscaler treats its own view of the cluster as the source of truth, and it has a few behaviours that catch people out.

Pause it before manual changes. If you scale a node group by hand while the autoscaler is running, it undoes the change once it decides the extra nodes are unneeded - typically after about ten minutes, which is its default --scale-down-unneeded-time. Before troubleshooting, scaling, or removing nodes manually, scale the autoscaler to zero and re-enable it when you are done:

kubectl -n kube-system scale deploy/cluster-autoscaler --replicas=0

Give every node group an explicit range. The --nodes=min:max:group flag sets a range for the named group, but it does not limit the autoscaler to that group. It manages every node group in the cluster, and any group without an explicit range gets a minimum of 1 and a maximum inherited from your cluster plan - 30 nodes on dev-md, 120 on prod-md-ha. Set a range for each group so nothing can grow further than you intend.

Make sure no pod can be permanently unschedulable. The autoscaler responds to a Pending pod by adding a node. If the pod can never be placed - a node selector no node will ever match, for example - it keeps adding nodes until the group hits its maximum, and it suppresses scale-down across the whole cluster while it does. A single pod like this can take a group from 2 nodes to 12 in a quarter of an hour. Check kubectl get pods -A --field-selector=status.phase=Pending from time to time, and fix or delete anything that has been Pending for a long time.

Don't leave a node group at zero nodes. The autoscaler needs at least one node in every group to build its picture of the cluster. A group at count 0 makes it abort every loop with No node info for, and scale-down stops cluster-wide until the group is deleted or has a node again. Delete groups you have emptied.

Keep some headroom above your normal peak. If a node group sits at its maximum and some nodes become unusable, the autoscaler cannot add capacity - it logs max size reached and waits. A maximum a few nodes above your usual peak gives it room to work around a bad node.

Minimum size only limits scale-down. The autoscaler will not add nodes to bring a group up to its minimum. If a group falls below it for any reason, you scale it back up yourself.

Watch for longUnregistered in the logs. It means a server the platform lists in the node group has never registered with Kubernetes - usually an orphaned node. The autoscaler reports it but will not remove it if doing so would breach the group's minimum size, so it can sit there indefinitely, and a group can end up consisting of nothing but the orphan. Remove it yourself with the per-node delete endpoint described above. The scaling guide has the full troubleshooting steps.