Managed Kubernetes now available in Open Beta. Test out the new service yourself at your Control Panel.
Managed Kubernetes now available in Open Beta. Test out the new service yourself at your Control Panel.
Kubernetes is an open-source orchestration system that automates the process of deploying and maintaining containerized applications. It gives you the mechanism to schedule and run containers on clusters of several physical and/or virtual machines. For more information please read the official Kubernetes documentation.
Kubespray is a composition of Ansible playbooks, inventory, provisioning tools, and domain knowledge for generic OS/Kubernetes clusters configuration management tasks. It provides a highly available cluster, composable attributes and support for the most popular Linux distributions. It has become the de-facto production-ready Kubernetes installer that is trusted throughout the cloud-native community (10k stars on GitHub).
In this tutorial, we’ll show the steps required to deploy a Kubernetes cluster on UpCloud using Kubespray. The tutorial assumes that you have basic knowledge about Kubernetes and the different terminologies that comes with it but the steps are easy enough for beginners to follow along as well.
In principle, the steps in this guide can be divided into the following two main procedures which are required in order to set up a new Kubernetes cluster.
Before delving into the actual steps, clone Kubespray onto your own computer, for example by using the git command-line tool. If you do not already have git installed, you can use the command below to install git on Ubuntu or other Debian-based operating systems or check the git install guide for other OS options.
sudo apt install git-all
Then download the Kubespray package and change to the new directory.
git clone https://github.com/kubernetes-sigs/kubespray.git cd kubespray
You’ll also need to install Ansible and other dependencies. Luckily, Kubespray provides a handy list of the requirements which can be used to install all prerequisites with a single command. However, for this to work, you’ll first need to have Python’s package installer, pip, available.
sudo apt install python3-pip sudo pip3 install -r requirements.txt
If you face any issue while installing the prerequisites, please check the official Kubespray repository for troubleshooting steps.
Terraform is an infrastructure provisioning tool. It is used for building, changing, and versioning infrastructure safely and efficiently. Installing Terraform CLI on your computer provides you with all the tools you need to manage your infrastructure in the cloud.
To install Terraform, find the appropriate package for your system, download and install it.
For example, to install Terraform on most Linux systems, first, download the latest version.
wget https://releases.hashicorp.com/terraform/0.14.7/terraform_0.14.7_linux_amd64.zip
Then extract the binaries to a suitable location, such as /usr/local/bin and make sure it is included in your PATH environment variable. For example with the command below.
sudo unzip terraform_0.14.7_linux_amd64.zip -d /usr/local/bin
You can verify that the installation worked by listing Terraform’s available subcommands in a terminal.
terraform -help
Usage: terraform [-version] [-help] [args] The available commands for execution are listed below. The primary workflow commands are given first, followed by less common or more advanced commands. ...
For Terraform to be able to deploy cloud resources on your UpCloud account, you first need to grant it access. To do so, follow the steps below.
It is recommended to create a separate API user for Terraform to interact with UpCloud API during infrastructure deployment instead of using your main account. You can do this at the UpCloud Control Panel using the workspace member accounts. Your API account name and password are very much comparable to a traditional API ID and key pair with the added benefit of being able to set them freely yourself.
To create a new account, select People on the left side of the UpCloud Control Panel and follow the instructions. For more information, see this tutorial on how to get started with the UpCloud API.
Once you’ve created your API account, you will need to allow it to access the UpCloud API so that Terraform can create your infrastructure on UpCloud. For this purpose, make sure you select the Allow API connections on the new account and set it to All addresses to easily ensure later steps in this guide will work.
You can and for security reasons perhaps should restrict this later to your own IP.
Lastly, you’ll need to pass the new API account credentials to Terraform. Use the commands below to export your new API username and password as environment variables in your current shell session. The username and password will then be accessible to Terraform CLI when creating your cluster.
export TF_VAR_UPCLOUD_USERNAME= export TF_VAR_UPCLOUD_PASSWORD=
Note: The credentials above are in plaintext. It is advisable to store the username and password Base64 encoded for more permanent use.
A cluster in Kubernetes is composed of multiple control plane and worker nodes. Control plane nodes are nodes that control and manage a set of worker nodes (workloads runtime) and worker nodes are nodes that run containerized applications.
An example of a Kubernetes cluster setup could look like the following.
You need to decide the number of both control plane and worker nodes for your cluster. In this post, we will use one control plane node and three worker nodes.
To create the Kubernetes cluster, you need to make sure you have Terraform CLI installed on your system as well as the proper configuration for your cluster. Terraform helps us to define infrastructure as a code. Defining infrastructure as code brings many advantages such as simple editing, reviewing, and versioning, as well as easy sharing amongst team members.
Next, we’ll set up the configuration for our cluster. To avoid modifying the template files, let us copy the required files into a new directory and do the changes there.
Create a directory called my-upcloud-cluster as follows. The CLUSTER variable here is a shorthand for our directory name. If you want to name your directory differently, just change the next line and the rest of the configuration works the same.
CLUSTER=my-upcloud-cluster
mkdir inventory/$CLUSTER
Copy the sample inventory and the default cluster configuration to the new directory.
cp -r inventory/sample inventory/$CLUSTER cp -r contrib/terraform/upcloud/* inventory/$CLUSTER/
Change your working directory to the new configuration directory and edit the cluster-settings.tfvars file to match your requirements.
cd inventory/$CLUSTER
vim cluster-settings.tfvars
The following are the main Terraform variables that you can change in cluster-settings.tfvars file.
For example, to create a cluster with two control plane nodes, three worker nodes, and each node with 2 cores, 4GB memory, and 250GB disk size, replace the machines section in the variables with the following code snippet.
machines = { "master-0" : { "node_type" : "master", #number of cpu cores "cpu" : "2", #memory size in MB "mem" : "4096" # The size of the storage in GB "disk_size" : 250 }, "worker-0" : { "node_type" : "worker", #number of cpu cores "cpu" : "2", #memory size in MB "mem" : "4096" # The size of the storage in GB "disk_size" : 250 }, "worker-1" : { "node_type" : "worker", #number of cpu cores "cpu" : "2", #memory size in MB "mem" : "4096" # The size of the storage in GB "disk_size" : 250 }, "worker-2" : { "node_type" : "worker", #number of cpu cores "cpu" : "2", #memory size in MB "mem" : "4096" # The size of the storage in GB "disk_size" : 250 } }
Don’t forget to replace the value of the ssh_public_keys variable with your public SSH key as it will be used to ssh to each machine when installing Kubernetes using Kubespray.
Now that the configurations are done, you can start deploying your cluster.
The terraform init command is used to initialize a working directory containing Terraform configuration files. It will download and install the UpCloud Terraform provider plugin.
Run the following command to initialize Terraform.
terraform init
The terraform plan command verifies your configuration is syntactically correct and creates an execution plan. Terraform performs a refresh, unless explicitly disabled, and then determines what actions are necessary to achieve the desired state specified in the configuration files.
Run the following command to see your execution plan:
terraform plan --var-file cluster-settings.tfvars -state=tfstate-$CLUSTER.tfstate
The terraform apply command is used to apply the changes required to reach the desired state of the configuration or the predetermined set of actions generated by a Terraform execution plan. It creates new or makes changes to the existing infrastructure as defined in your configuration.
Run the following command to create your cluster and answer yes when asked to confirm:
terraform apply --var-file cluster-settings.tfvars -state=tfstate-$CLUSTER.tfstate
Once Terraform has finished deploying, you can go and check your cluster resources at your UpCloud Control Panel. The following figure shows the four servers (one control plane and three worker nodes) created by Terraform.
You should also get an inventory file named inventory.ini that you can use with Kubespray. We will use the inventory file to set up the Kubernetes cluster later.
ls
cluster-settings.tfvars inventory.ini sample
terraform.tfstate tfstate-my-upcloud-cluster.tfstate
Ansible is an open-source software provisioning, configuration management, and application-deployment tool enabling infrastructure as code.
Set ANSIBLE_CONFIG environment variables to Kubespray’s ansible configuration file as follows:
export ANSIBLE_CONFIG=../../ansible.cfg
Check that you have basic SSH connectivity to the nodes. You can do this by running the following ansible command.
ansible -i inventory.ini -m ping all
You should see similar to the following figure if all nodes are reachable.
master-0.example.com | SUCCESS => { "ansible_facts": { "discovered_interpreter_python": "/usr/bin/python3" }, "changed": false, "ping": "pong" } worker-0.example.com | SUCCESS => { "ansible_facts": { "discovered_interpreter_python": "/usr/bin/python3" }, "changed": false, "ping": "pong" } worker-1.example.com | SUCCESS => { "ansible_facts": { "discovered_interpreter_python": "/usr/bin/python3" }, "changed": false, "ping": "pong" } worker-2.example.com | SUCCESS => { "ansible_facts": { "discovered_interpreter_python": "/usr/bin/python3" }, "changed": false, "ping": "pong" }
You can now deploy Kubernetes with Kubespray using the inventory file generated during the Terraform apply step as follows.
Note: if you use a different user to access the nodes other than the default ubuntu, please replace ubuntu with the new user in the inventory.ini file.
ansible-playbook -i inventory.ini ../../cluster.yml -b -v
Then sit back and relax while Ansible and Kubespray do all the heavy lifting!
Once done, you will see a play recap of the deployment like in the screenshot below.
By default, Kubespray configures kube-master hosts with access to kube-apiserver via port 6443 as http://127.0.0.1:6443. You can connect to this from one of the master nodes.
Get the IP address of one of the master nodes and SSH to it.
For example, the following script retrieves the IP of master-0 node from the inventory file and opens an SSH connection to it using the default username.
# get the IP address of master-0 ip=$(grep -m 1 "master-0" inventory.ini | grep -o '[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}' | head -n 1) # ssh to master-0 node ssh [email protected]$ip
Once you are logged into one of the master nodes, you can run any of the kubectl commands. For example, use the command below shows the list of nodes in the cluster.
sudo kubectl get nodes
NAME STATUS ROLES AGE VERSION master-0.example.com Ready control-plane,master 10m v1.20.5 worker-0.example.com Ready 10m v1.20.5 worker-1.example.com Ready 10m v1.20.5 worker-2.example.com Ready 10m v1.20.5
While in the example above, we logged into one of the nodes in the cluster, it’s also possible to command Kubernetes right from your own workstation. To make this work, simply copy the /etc/kubernetes/admin.conf from a master node to your workstation and use it with kubectl.
The following script does the trick.
# get the IP address of master-0 ip=$(grep -m 1 "master-0" inventory.ini | grep -o '[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}' | head -n 1) # copy /etc/kubernetes/admin.conf file on the local system ssh [email protected]$ip 'sudo cat /etc/kubernetes/admin.conf' > admin.conf
Make sure you have installed the kubectl tool on your local machine.
If you haven’t done so already please follow the instructions here. Once you have installed the tool you can interact with your cluster.
To access your Kubernetes cluster remotely, you need to tell kubectl where your configuration file is. One way to do that is to point the KUBECONFIG environment variable to your cluster kubeconfig file as shown below:
export KUBECONFIG=admin.conf
One last thing you need to do before you run any of the kubectl commands is to replace the Kubernetes API IP address in the admin.conf with one of the control plane node IP addresses.
Open admin.conf file with your favourite text editor and replace 127.0.0.1 with the IP address you get from above.
vim admin.conf
Alternatively, if you already have the IP address of the first master node save in the $ip variable. You can swap the 127.0.0.1 IP with the IP saved in the variable using the following command.
sed -i "s/127.0.0.1/$ip/g" admin.conf
With the master node IP set in the admin file, you are ready to start playing with your cluster from your local machine!
For example, use the following command to show a list of namespaces in your cluster.
kubectl get namespace
NAME STATUS AGE default Active 12m kube-node-lease Active 12m kube-public Active 12m kube-system Active 12m
Congratulations, you now have a fully functional production-ready Kubernetes cluster up and running!
Once you are done testing the cluster and no longer need it, you can use Terraform to tear down the deployed infrastructure.
The terraform destroy command is used to destroy the Terraform-managed infrastructure. It terminates resources defined in your Terraform configuration and performs the reverse of what the terraform apply does.
You can tear down your infrastructure using the following Terraform command:
terraform destroy --var-file cluster-settings.tfvars -state=tfstate-$CLUSTER.tfstate ../../contrib/terraform/upcloud/
After deletion, you can always use the same configuration files to teak and modify your cluster and deploy it again with a moments notice!
Summer
Can’t SSH into nodes with the following command:
* ansible -i inventory.ini -m ping all
The error is quite lengthy, but here is a gist of it:
1. [WARNING]: Unhandled error in Python interpreter discovery for host kubespray-worker-0: Failed to connect to the host.
2. via ssh: Warning: Permanently added ‘IP address’ (ECDSA) to the list of known hosts. [email protected]: Permission denied (publickey).
/*Same is the error for every other nodes/
After that:
3. kubespray-worker-0 | UNREACHABLE! => {
“changed”: false,
4. “msg”: “Data could not be sent to remote host “IP address”. Make sure this host can be reached over ssh: Warning: Permanently added ‘IP address (ECDSA) to the list of known [email protected] address: Permission denied (publickey).rn”,
“unreachable”: true
}
*Please note: list 1 and 2 are continuation and list 3 and 4 are also the same.
Summer
There are two SSH keys enabled in settings.tfvars file and both are passphrase protected. Does that mean we need to generate passphrase-less SSH keys?
Summer
Hi Janne, I’m not clear as to which SSH key you’re referring to. Here are my doubts:
1. Which Public Key(s) from should be added to the cluster-settings.tfvars file?
2. Is it the one that is being generated to SSH to the main server deployed in the cloud?
3. Or am I supposed to generate two separate SSH keys from the sub-user account which is granted with API access?
BTW, when I used the SSH key which is used for connecting to the main server in the cloud, one of the nodes (master-0) actually gets pinged and the rest don’t.
Thanks
Dev
There’s no /etc/kubernetes/admin.conf file in the master node. I have set-up everything perfectly till that step but can’t run `sudo kubectl get nodes`. I tried to `ls` into the master node and there’s no such file as `admin.conf`. Any suggestions?
Dev
Hi Jane, thanks for the suggestion. It seems to work. I wanted to clarify if it is necessary to follow the last step `Accessing Kubernetes Cluster from a workstation` if the control plane node (worker -0) is already accessible from the steps just before that one.
Dev
Hi mate. Thanks for all the info. Now, how can I deploy my Docker Image [which is available in Docker Desktop for windows to the K8s cluster I just created in UpCloud using the above tutorial? I’m using WSL (Ubuntu) and it seems like a daunting tasks with no further info on that.
Dev
When will UpCloud provide Load Balancer as a Service type in Kubernetes?
wvh
Is there any officially sanctioned Kubernetes CSI driver for storage volumes so we can reliably use Upcloud volumes in Kubernetes?
Sanjeev
Hi,
The k8s 1.24 is with Ansible Kubespray containerd.
The snapshots subfolder got deleted hence Control plane master KubeScheduler is not coming up.
Any suggestion how to recover.
kube-scheduler: failed to create containerd container: failed to create prepare snapshot dir : stat /data/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots: no such file or directory Warning BackOff kubelet Back-off restarting failed container.