Deploying Managed Kubernetes cluster with private node groups and NAT Gateway using Terraform
Managed Kubernetes easily automates the deployment, scaling, and management of container workloads but takes a couple of steps to deploy by itself. In Deploying Managed Kubernetes cluster with Terraform, we saw how to simplify the process with Terraform.
In this guide, we’ll explore how to use Terraform to deploy a Managed Kubernetes cluster with private node groups and NAT Gateway.
Installing Terraform
First things first. If you haven’t used Terraform before, you will need to start by installing it. You can follow How to get started with Terraform to accomplish this and then come back and continue here.
You can also check out the official UpCloud Terraform provider documentation to learn more.
Creating the Terraform configuration
Once you have Terraform installed and ready, start by creating a new directory on your own computer to house your Terraform configuration files. The actual location of this folder doesn’t matter, as the only requirement is that only one Terraform configuration is saved in any specific directory.
The configuration is organized into four files. Create each of them in your configuration directory with the contents below, and feel free to modify them as needed.
provider.tf sets the provider versions and configures the UpCloud provider.
terraform {
required_providers {
upcloud = {
source = "UpCloudLtd/upcloud"
version = "~> 5.47"
}
local = {
source = "hashicorp/local"
version = "~> 2.5"
}
}
}
# The UpCloud provider reads your API token from the UPCLOUD_TOKEN environment variable
provider "upcloud" {}The UpCloud provider reads your API token from the UPCLOUD_TOKEN environment variable, so set it in the terminal you run Terraform from. If you don't have a token yet, see How to create and use UpCloud API Tokens.
export UPCLOUD_TOKEN=ucat_Your_API_Tokenvariables.tf defines the input variables used in the configuration, such as the location (zone) and the network's IP range.
variable "basename" {
default = "uks-cluster-example"
description = "Basename to use when naming resources created by this configuration."
type = string
}
variable "zone" {
default = "de-fra1"
description = "UpCloud zone for resource provisioning."
type = string
}
variable "store_kubeconfig" {
default = true
description = "If set to `true`, store kubeconfig as a file with `local_file` to module path."
type = bool
}
variable "ip_network_range" {
default = "172.16.2.0/24"
description = "CIDR range used by the cluster SDN network."
type = string
}main.tf defines the managed resources: the router, the network, the NAT gateway, the cluster and its node group, and a local_file that saves the cluster's kubeconfig. Private node groups need a network whose traffic is routed through a NAT gateway, so the network hands out a default route through the router (dhcp_default_route = true), and the NAT gateway is attached to that router.
# Create a router for your network
resource "upcloud_router" "example" {
name = "${var.basename}-router"
}
# Create a network for your cluster
resource "upcloud_network" "example" {
name = "${var.basename}-net"
zone = var.zone
ip_network {
address = var.ip_network_range
dhcp = true
dhcp_default_route = true
family = "IPv4"
}
router = upcloud_router.example.id
}
# Create a Managed NAT GW for Internet connectivity from the SDN network
resource "upcloud_gateway" "example" {
name = "${var.basename}-gw"
zone = var.zone
features = ["nat"]
router {
id = upcloud_router.example.id
}
}
# Create a cluster
resource "upcloud_kubernetes_cluster" "example" {
name = "${var.basename}-cluster"
network = upcloud_network.example.id
zone = var.zone
plan = "dev-md"
private_node_groups = true
control_plane_ip_filter = ["0.0.0.0/0"]
depends_on = [upcloud_gateway.example]
}
# Create a node group for your cluster
# Node group is a group of worker nodes that are created based on the same template
# You can have multiple node groups with different configurations in your cluster
resource "upcloud_kubernetes_node_group" "group" {
name = "medium"
// All nodes in this group will be joined to this cluster
cluster = upcloud_kubernetes_cluster.example.id
// The amount of created nodes (servers)
node_count = 2
// Plan for each node; you can list the available plans with the UpCloud CLI (`upctl server plans`)
plan = "PREMIUM-2xCPU-4GB"
// With `anti_affinity` set to true, UKS will attempt to deploy nodes in this group to different compute hosts
anti_affinity = true
// Each node in this group will have the following labels
labels = {
managedBy = "terraform"
project = "uks-instructions"
}
// If uncommented, each node in this group will have this taint
# taint {
# effect = "NoExecute"
# key = "key"
# value = "value"
# }
// Each node in this group will have keys defined in this list configured as authorized keys (for "debian" user)
ssh_keys = []
}
data "upcloud_kubernetes_cluster" "example" {
id = upcloud_kubernetes_cluster.example.id
}
# With `hashicorp/local` Terraform provider one can output the kubeconfig to a file. The file can be easily
# used to configure `kubectl` or any other Kubernetes client.
resource "local_file" "kubeconfig" {
count = var.store_kubeconfig ? 1 : 0
content = data.upcloud_kubernetes_cluster.example.kubeconfig
filename = "${path.module}/kubeconfig.yml"
}The example allows connections to the cluster's API from any address (control_plane_ip_filter = ["0.0.0.0/0"]). Before you use it for anything beyond testing, replace that with your own IP addresses or ranges.
outputs.tf shows the cluster ID and the NAT gateway's public IP address when the deployment is ready.
output "cluster_id" {
value = upcloud_kubernetes_cluster.example.id
}
output "nat_gateway_ip" {
value = [for address in upcloud_gateway.example.addresses : address.address]
}Provisioning
When you have the configuration files all set, you can begin by initializing the directory.
terraform initTerraform will then download the required plugins. You should see an output similar to below.
Initializing provider plugins found in the configuration...
- Finding upcloudltd/upcloud versions matching "~> 5.47"...
- Finding hashicorp/local versions matching "~> 2.5"...
- Installing upcloudltd/upcloud v5.47.0...
- Installed upcloudltd/upcloud v5.47.0 (signed by a HashiCorp partner, key ID 6182C780EB46767E)
- Installing hashicorp/local v2.9.1...
- Installed hashicorp/local v2.9.1 (signed by HashiCorp)
Partner and community providers are signed by their developers.
If you'd like to know more about provider signing, you can read about it here:
https://developer.hashicorp.com/terraform/cli/plugins/signing
Initializing the backend...
Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.
Terraform has been successfully initialized!
You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.
If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.Next, test that the configuration is ready to deploy by running the terraform plan command.
terraform planTerraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
+ create
<= read (data resources)
Terraform will perform the following actions:
# data.upcloud_kubernetes_cluster.example will be read during apply
# (config refers to values not yet known)
...
Plan: 6 to add, 0 to change, 0 to destroy.
Changes to Outputs:
+ cluster_id = (known after apply)
+ nat_gateway_ip = (known after apply)The above example output has been truncated for brevity.
Lastly, if the plan looks all correct and Terraform didn’t report any issues, go ahead and deploy the setup.
terraform applyRunning apply creates a router, a network, a NAT gateway, a Managed Kubernetes cluster with a node group of two nodes, and a kubeconfig.yml file for connecting to the cluster. Terraform shows the plan again and asks you to confirm it. Answer yes, and Terraform then creates the resources.
...
Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.
Enter a value: yes
upcloud_router.example: Creating...
...
Apply complete! Resources: 6 added, 0 changed, 0 destroyed.At the end of the run, Terraform prints the configuration's outputs. You can show them again at any time.
terraform outputcluster_id = "0d540da7-f11d-450d-812c-db221015a6c1"
nat_gateway_ip = [
"87.58.152.234",
]Checking the cluster
Terraform saved the cluster's kubeconfig as kubeconfig.yml in your configuration directory. If you don't have kubectl installed yet, follow the instructions for your computer on the Kubernetes documentation's Install Tools page. Then point kubectl to the kubeconfig for the rest of this guide.
export KUBECONFIG=$PWD/kubeconfig.ymlThe kubeconfig file gives full access to your cluster, so keep it safe and out of version control. Terraform also stores the cluster's credentials in its state file, terraform.tfstate, so protect that file too.
List the cluster's nodes with their addresses.
kubectl get nodes -o wideNAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
medium-l8lrc-bdjxz Ready <none> 4m3s v1.36.4 172.16.2.2 <none> Debian GNU/Linux 13 (trixie) 6.12.101+deb13-amd64 (amd64) containerd://2.2.4
medium-l8lrc-jwgpv Ready <none> 3m54s v1.36.4 172.16.2.3 <none> Debian GNU/Linux 13 (trixie) 6.12.101+deb13-amd64 (amd64) containerd://2.2.4The nodes only have private addresses from the cluster's network, and EXTERNAL-IP shows <none>, because nodes in a private node group have no public IP addresses.
To check that the nodes reach the internet through the NAT gateway, run a pod that asks an external service for its public IP address. Then read the pod's output and delete the pod.
kubectl run egress-check --restart=Never --image=curlimages/curl -- curl -s -w '\n' https://ifconfig.me
kubectl wait --for=jsonpath='{.status.phase}'=Succeeded pod/egress-check --timeout=60s
kubectl logs egress-check
kubectl delete pod egress-checkpod/egress-check created
pod/egress-check condition met
87.58.152.234
pod "egress-check" deleted from default namespaceThe address is the NAT gateway's IP address from nat_gateway_ip, so the nodes' outbound traffic goes through the NAT gateway. Your Managed Kubernetes cluster is now running with a private node group and a NAT gateway, ready for your workloads.
Cleaning up
When you no longer need the cluster, delete everything the configuration created.
terraform destroyTerraform lists the resources it will delete and asks you to confirm. Answer yes to delete them.
...
Plan: 0 to add, 0 to change, 6 to destroy.
...
Do you really want to destroy all resources?
Terraform will destroy all your managed infrastructure, as shown above.
There is no undo. Only 'yes' will be accepted to confirm.
Enter a value: yes
...
Destroy complete! Resources: 6 destroyed.