How to scale your Managed Kubernetes cluster

A Managed Kubernetes cluster has two things you can scale: the worker nodes, which are yours to size, and the workloads running on them. This guide is about the nodes. It covers how node group scaling works, how to do it from the control panel, the API, and Terraform, how to remove one specific node, how the cluster autoscaler behaves, and what to do when a scale-down stops part way.

For scaling workloads - replica counts, the Horizontal Pod Autoscaler, the Vertical Pod Autoscaler - see the Kubernetes autoscaling documentation. Those work on Managed Kubernetes exactly as they do anywhere else.

The control plane is sized by the cluster plan and is not something you scale. See Managed Kubernetes control plane for the node limit each plan sets.

How node group scaling works

Worker nodes belong to node groups, and node groups are managed by the platform. Throughout this guide, "the platform" means the UpCloud service that provisions, joins, drains, and removes worker nodes on your behalf. It is separate from your cluster: it keeps its own list of which servers belong to each node group, and that list is what the node group API returns. Your cluster keeps its own list of nodes, which is what kubectl get nodes returns. Normally the two match, and a lot of what can go wrong is the two drifting apart. When you change a node group's count, the platform does the work: it creates or removes Cloud Servers, and the Kubernetes side follows.

Scaling up creates new Cloud Servers from the node group's plan, boots them, and joins them to the cluster. New nodes appear in kubectl get nodes as NotReady and turn Ready once the kubelet and networking are up. Expect one to two minutes for a node group change; a new cluster takes longer.

Scaling down removes the surplus nodes together, not one after another. For each node the platform:

1. Picks a node from the group. You don't choose which one, and the choice takes no account of what is running on it - if you need a specific node gone, use the per-node delete endpoint described below.
2. Cordons it, so nothing new is scheduled there.
3. Drains it, evicting pods and honouring any PodDisruptionBudgets.
4. Deletes the Cloud Server.

For a node with no workload this typically takes under a minute. For a node running a few ordinary deployments it typically takes one to two minutes; the drain honours termination grace periods and PodDisruptionBudgets, so a slow-stopping workload or a tight budget stretches it.

While this is happening the node group reports a state of scaling-up or scaling-down, and returns to running when it's done. You can read the state in the control panel or from the API:

curl -s -H "Authorization: Bearer $UPCLOUD_TOKEN" \
  https://api.upcloud.com/1.3/kubernetes/<cluster-uuid>/node-groups/<group-name> \
  | jq '{state, count, nodes: [.nodes[] | {name, state}]}'

The nodes list is the platform's view of the group. It should match kubectl get nodes for that group. When it doesn't, something has gone wrong - see the troubleshooting section. Note that the list endpoint for all node groups returns an empty nodes list for each group; only the single-group request fills it in.

Scaling a node group

In the control panel

Open your cluster, go to the node groups tab, and click Scale on the group you want to change.

Enter the new node count and click Scale.

With the API

Send the new count with a PATCH to the node group:

curl -X PATCH -H "Authorization: Bearer $UPCLOUD_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"count": 5}' \
  https://api.upcloud.com/1.3/kubernetes/<cluster-uuid>/node-groups/<group-name>

The call returns immediately with 200; the group state tells you when the change has completed. See the Managed Kubernetes API reference for the full request and response.

With Terraform

Change node_count on the upcloud_kubernetes_node_group resource and apply. Terraform waits for the group to return to running before it reports success.

If you also run the cluster autoscaler, add a lifecycle block so Terraform stops trying to put the count back to whatever your configuration says:

resource "upcloud_kubernetes_node_group" "workers" {
  # ...
  node_count = 3

  lifecycle {
    ignore_changes = [node_count]
  }
}

Without it, every terraform apply after the autoscaler has changed the size will propose reverting it.

Removing a specific node

A normal scale-down lets the platform choose which nodes to remove. When you need a particular node gone - it has gone NotReady and isn't coming back, or it's a server the node group lists but kubectl get nodes doesn't show - use the per-node delete endpoint instead:

curl -X DELETE -H "Authorization: Bearer $UPCLOUD_TOKEN" \
  https://api.upcloud.com/1.3/kubernetes/<cluster-uuid>/node-groups/<group-name>/<node-name>

The node name is the one kubectl get nodes shows. The platform cordons and drains that node, deletes its server, and reduces the group count by one. The response is 202 with a message that deletion has started. On a node running a few deployments this typically takes a minute or two, with every pod rescheduled elsewhere.

If the node you name is missing from kubectl get nodes - see the troubleshooting section for how that happens - the platform may instead keep the count and build a replacement.

Don't use kubectl delete node

On a cluster you run yourself, kubectl delete node is a normal way to remove a node. On Managed Kubernetes it is the wrong tool, and the result is worse than doing nothing.

The command only removes the Node object from the Kubernetes API. The Cloud Server behind it keeps running and keeps being billed, the node group keeps counting it, and the platform can no longer manage it. The kubelet on that server doesn't recover by itself either: it registers its Node object once, at startup, and only sends status updates after that. With the object gone it logs node not found on every update, indefinitely. The node group moves to scaling-up and stays there.

Nothing on the Kubernetes side needs doing before a scale-down. The platform drains the node itself. If you have already deleted a Node object, there are two ways back:

• Restart the kubelet on that server. It re-registers within seconds and the node reappears. This needs SSH access, which means an SSH key on the node group; log in as debian and run sudo systemctl restart kubelet. If you can't SSH in, restarting the server from the control panel or the server API does the same job - the kubelet re-registers on boot - at the cost of a reboot for the pods still running on that node.
• Call the per-node delete endpoint for the node. The platform removes the server cleanly and, if the group is short, builds a replacement.

Don't stop or delete the Cloud Server

A worker's Cloud Server is also visible in the server list, and the server API and control panel will let you act on it directly. Don't, unless you mean to.

Shutting the server down - the Stop action in the control panel or the server API - leaves the node in the group as NotReady. The platform neither removes nor replaces it; it waits. When you start the server again, the same node rejoins within a minute or two.

Deleting the server is worse. The node group keeps listing the vanished server as running, moves to scaling-up, and stays there - and the platform does not build a replacement. To clear it, call the per-node delete endpoint with the missing node's name; the group returns to running with the count reduced by one.

Don't drain nodes manually

The same applies to kubectl drain. It's tempting to drain the node you want removed and then scale the group down by one, but the platform doesn't know which node you drained and picks its own. The drained node is unlikely to be the one removed: it stays in the cluster, cordoned and empty, while a busy node is taken instead. If you want a specific node removed, use the per-node delete endpoint; it does the drain for you.

Changing the node plan

Node groups have a fixed plan. To move workloads onto bigger or smaller nodes, create a new node group with the plan you want and migrate across.

1. Create the new node group in the control panel or with the API, with the plan, labels, taints, and SSH keys you need. Give it enough nodes for the workloads you're moving.
2. Wait for the new nodes to show Ready in kubectl get nodes.
3. Move workloads across. Node group labels are forwarded to the Kubernetes Node objects, so a nodeSelector or node affinity on your deployments is the cleanest way to steer them. Rolling the deployments after adding the selector moves the pods.
4. Once your workloads are running on the new nodes, delete the old node group. The platform drains anything still on it: pods without a selector are rescheduled onto the new nodes, and pods that still select the old group are left Pending until you fix their selector. Deleting the group is better than leaving it at zero nodes - see the autoscaler section for why.

In Terraform, changing plan on a node group resource replaces the whole group. Use a new resource for the new group and remove the old one once the migration is done, so you control the order.

Don't resize the Cloud Servers in a node group individually. The platform accepts the change and then ignores it: the node keeps its place in the group, Kubernetes keeps scheduling against the old capacity until the kubelet restarts, and the next node the platform builds uses the group's plan. You end up with a group of mixed sizes that nothing reconciles.

Using the cluster autoscaler

The cluster autoscaler for UpCloud adds nodes when pods can't be scheduled and removes nodes that have been underused for a while. Setup is covered in the cluster autoscaler guide. This section is about how it behaves once it's running, because a few of those behaviours catch people out.

It scales nodes through the same operations described above. Scale-up is a change to the node group count; scale-down is a per-node delete of the node it has chosen and drained. So everything in this guide about how those operations behave applies to the autoscaler too, including that it honours PodDisruptionBudgets - a node it can't drain is logged as cannot be removed: not enough pod disruption budget and left alone.

Pause it before making manual changes. The autoscaler treats its own view as authoritative. If you scale a node group up by hand while it's running, it will remove the extra nodes as soon as it decides they're unneeded - after ten minutes by default (--scale-down-unneeded-time). Before scaling manually, removing nodes, or troubleshooting, scale it to zero, and bring it back when you're done:

kubectl -n kube-system scale deploy/cluster-autoscaler --replicas=0

Give every node group an explicit range. The --nodes=min:max:group flag sets the range for the group you name, but it doesn't restrict the autoscaler to that group. It manages every node group in the cluster. Groups you haven't named get a minimum of 1 and a maximum inherited from the cluster plan - 30 on Development plans, 120 on Production plans. Set a --nodes range for each group so nothing can grow further than you intend.

Make sure no pod can be permanently unschedulable. The autoscaler's response to a Pending pod is to add a node. If the pod can never be placed - a node selector no node matches, a taint nothing tolerates - it keeps adding nodes until the group hits its maximum. While it's doing that, it also holds off scale-down for the whole cluster. A single such pod can take a group from 2 nodes to 12 in a quarter of an hour. Check for long-lived Pending pods now and then:

kubectl get pods -A --field-selector=status.phase=Pending

Don't leave a node group at zero nodes. The autoscaler needs a live node to build its picture of each group. A group at count 0 makes it abort every loop with could not compute total resources: No node info for, and scale-down stops for the whole cluster until the group is deleted or has a node again. Delete groups you have emptied.

Keep headroom above your normal peak. If a node group is at its maximum and some of its nodes become unusable, the autoscaler can't add capacity. It logs max size reached and waits. A maximum a few nodes above your usual peak gives it room to work around a bad node.

Minimum size only limits scale-down. The autoscaler will not add nodes to bring a group up to its minimum. If a group is below it for any reason, scale it up yourself.

A server the cluster doesn't know about gets logged, not fixed. If the node group contains a server that never registered as a Kubernetes node, the autoscaler reports it after about fifteen minutes as longUnregistered. It counts the server toward the group's size, so if removing it would take the group below its minimum, it logs min size reached, skipping removal and leaves it. A group can end up as one dead server and nothing else. Remove it yourself with the per-node delete endpoint.

PodDisruptionBudgets

A PodDisruptionBudget protects a workload during drains by capping how many of its pods can be evicted at once. The platform honours budgets when it drains a node for a scale-down, and so does the autoscaler. That's what you want - but a budget that can't be satisfied blocks the drain for as long as it stays that way, and nothing reports an error while it waits.

The clearest case is a budget that can never be satisfied: minAvailable set equal to the replica count, or a single-replica workload with any budget at all. Either gives the budget zero allowed disruptions from the start.

A budget that is normally fine can also block a drain when the evicted pods have nowhere to go. If the remaining nodes lack capacity, or a node selector, affinity, or topology spread rule excludes them, the evicted pod stays Pending, the budget's allowed disruptions drop to zero, and the drain waits before evicting the next pod. kubectl get pods -A --field-selector=status.phase=Pending shows the stuck pod.

The scale-down itself looks normal at first. The API call returns 200, the node group moves to scaling-down, and the chosen node is cordoned. Then nothing happens. An unblocked scale-down of an empty node completes in under a minute; behind a blocking budget the same operation is still waiting half an hour later, and completes within seconds of the budget being relaxed.

Check budgets with:

kubectl get pdb -A

Anything showing ALLOWED DISRUPTIONS 0 can block a drain of whichever node its pods are on. For most workloads a maxUnavailable: 1 budget is the safe choice; if you use minAvailable, keep it below the replica count.

Troubleshooting

Each heading below is a symptom. Start with the one that matches what you see.

Scale-down not progressing

The node group shows scaling-down, one or more nodes are cordoned (SchedulingDisabled in kubectl get nodes), and nothing changes. Check for a PodDisruptionBudget with zero allowed disruptions covering a pod on a cordoned node, and for evicted pods stuck Pending:

kubectl get pdb -A
kubectl get pods -A --field-selector=spec.nodeName=<node-name>
kubectl get pods -A --field-selector=status.phase=Pending

Relax the budget, give the evicted pods somewhere to land, or scale the protected workload up so the budget can be met, and the scale-down resumes on its own. Don't uncordon the node or delete the pods by hand - let the drain finish.

If no budget is involved and the node has been cordoned for more than a few minutes, contact the support team with the cluster UUID and node group name.

Count higher than kubectl get nodes

The node group lists a server that has no Kubernetes Node object. The usual cause is a Node object having been deleted with kubectl delete node; a node that failed to join can produce the same picture. Compare the two lists to find the server:

curl -s -H "Authorization: Bearer $UPCLOUD_TOKEN" \
  https://api.upcloud.com/1.3/kubernetes/<cluster-uuid>/node-groups/<group-name> \
  | jq -r '.nodes[].name' | sort > platform.txt
kubectl get nodes -o name | sed 's|node/||' | sort > kubernetes.txt
comm -23 platform.txt kubernetes.txt

Any name printed is in the node group but not in the cluster. Remove each one with the per-node delete endpoint, or if the server is healthy and its Node object was deleted, restart the kubelet on it so it re-registers. Scaling the group down instead won't reliably remove these - the platform picks its own node, and it may be a working one.

Stuck in scaling-up

The node group shows scaling-up and never returns to running. Almost always the same cause as the previous symptom: the node group lists a server that has no Kubernetes Node object, and the platform is waiting for it. That includes a server that was deleted through the server API - the group still lists it. Run the same comparison and remove the name with the per-node delete endpoint. The group returns to running once the platform's list matches the cluster's.

longUnregistered and nothing happens

The autoscaler logs Found longUnregistered Nodes but the node is never removed. It has found a server in the node group that never registered with Kubernetes and has decided not to remove it, usually because doing so would take the group below its minimum size. Look for min size reached, skipping removal in its logs to confirm. Remove the server with the per-node delete endpoint. The autoscaler won't do it for you however long you wait.

No node info for and no scaling

The autoscaler logs Failed to scale up: could not compute total resources: No node info for on every loop and no scaling happens. A node group in the cluster has zero nodes. The autoscaler can't build its picture of the cluster without at least one node per group, so it aborts every loop. Delete the empty group, or give it a node.

409 "already marked for deletion"

A per-node delete - from the autoscaler or from you - returns:

Node <group>/<node-name> already marked for deletion, state conflict.
(type=https://developers.upcloud.com/api/1.3/getting-started#failed-requests, status=409)

The node is Ready, the group state is running, and the deletion never happens. This is a deletion that the platform accepted and did not complete, and the state that causes it isn't visible through the API. Retrying doesn't help - the autoscaler will retry every few seconds indefinitely and get the same answer each time.

Don't delete the Node object, don't stop the server, and don't leave the autoscaler hammering the endpoint. Scale the autoscaler to zero, then contact the support team with the cluster UUID, the node group name, the node names, and a few correlation_id values from the 409 responses. The support team can get the node unstuck; the correlation IDs are what let them find the request on our side.

NotReady with all conditions Unknown

kubectl describe node shows Ready, MemoryPressure, DiskPressure and PIDPressure all as Unknown with reason NodeStatusUnknown. The kubelet has stopped reporting. Common causes are the kubelet having crashed or been stopped, the server having been shut down, and the node having exhausted its memory so badly that the kubelet itself can't run. Note that a node under memory pressure that is still reporting shows MemoryPressure=True, not Unknown - Unknown means the kubelet is silent, whatever the reason.

If you have metrics installed, kubectl top node returns a NotFound error for the node rather than zeros. Tools that show 0 are displaying a missing metric, not a measurement. Managed Kubernetes doesn't install a Metrics Server by default; see the Metrics Server guide if you want kubectl top.

To recover, restart the server from the control panel or the server API - the kubelet comes back on boot and the node returns to Ready if the underlying problem has cleared. If you have SSH access, restarting just the kubelet (sudo systemctl restart kubelet as debian) is the lighter option. If neither brings the node back, remove it with the per-node delete endpoint and let the platform replace it. Don't use kubectl delete node - see above for why.

Something else

Contact the support team with the cluster UUID, the node group name, the names of the nodes involved, what you were trying to do, and any correlation_id values from API error responses. The correlation ID is in the body of every error the API returns and lets us find the exact request.