How to deploy Kubernetes using Kubespray

Kubernetes is an open-source orchestration system that automates the process of deploying and maintaining containerized applications. It gives you the mechanism to schedule and run containers on clusters of several physical and/or virtual machines. For more information please read the official Kubernetes documentation.

Kubespray is a composition of Ansible playbooks, inventory, provisioning tools, and domain knowledge for generic OS/Kubernetes clusters configuration management tasks. It provides a highly available cluster, composable attributes and support for the most popular Linux distributions. It has become the de-facto production-ready Kubernetes installer that is trusted throughout the cloud-native community (10k stars on GitHub).

In this tutorial, we’ll show the steps required to deploy a Kubernetes cluster on UpCloud using Kubespray. The tutorial assumes that you have basic knowledge about Kubernetes and the different terminologies that comes with it but the steps are easy enough for beginners to follow along as well.

Setting up prerequisites

In principle, the steps in this guide can be divided into the following two main procedures which are required in order to set up a new Kubernetes cluster.

1. Create the infrastructure
2. Deploy Kubernetes

Before delving into the actual steps, clone Kubespray onto your own computer, for example by using the git command-line tool. If you do not already have git installed, you can use the command below to install git on Ubuntu or other Debian-based operating systems or check the git install guide for other OS options.

sudo apt install git-all

Then download the Kubespray package and change to the new directory.

git clone https://github.com/kubernetes-sigs/kubespray.git
cd kubespray

You’ll also need to install Ansible and other dependencies. Luckily, Kubespray provides a handy list of the requirements which can be used to install all prerequisites with a single command. However, for this to work, you’ll first need to have Python’s package installer, pip, available.

sudo apt install python3-pip
sudo pip3 install -r requirements.txt

If you face any issue while installing the prerequisites, please check the official Kubespray repository for troubleshooting steps.

Installing Terraform

Terraform is an infrastructure provisioning tool. It is used for building, changing, and versioning infrastructure safely and efficiently. Installing Terraform CLI on your computer provides you with all the tools you need to manage your infrastructure in the cloud.

To install Terraform, find the appropriate package for your system, download and install it.

For example, to install Terraform on most Linux systems, first, download the latest version.

wget https://releases.hashicorp.com/terraform/0.14.7/terraform_0.14.7_linux_amd64.zip

Then extract the binaries to a suitable location, such as /usr/local/bin and make sure it is included in your PATH environment variable. For example with the command below.

sudo unzip terraform_0.14.7_linux_amd64.zip -d /usr/local/bin

You can verify that the installation worked by listing Terraform’s available subcommands in a terminal.

terraform -help

Usage: terraform [-version] [-help]  [args]
The available commands for execution are listed below.
The primary workflow commands are given first, followed by
less common or more advanced commands.
...

Setting up API access

For Terraform to be able to deploy cloud resources on your UpCloud account, you first need to grant it access. To do so, follow the steps below.

Create a separate API account in your UpCloud Control Panel

It is recommended to create a separate API user for Terraform to interact with UpCloud API during infrastructure deployment instead of using your main account. You can do this at the UpCloud Control Panel using the workspace member accounts. Your API account name and password are very much comparable to a traditional API ID and key pair with the added benefit of being able to set them freely yourself.

To create a new account, select People on the left side of the UpCloud Control Panel and follow the instructions. For more information, see this tutorial on how to get started with the UpCloud API.

Allow API Access to your UpCloud Account

Once you’ve created your API account, you will need to allow it to access the UpCloud API so that Terraform can create your infrastructure on UpCloud. For this purpose, make sure you select the Allow API connections on the new account and set it to All addresses to easily ensure later steps in this guide will work.

You can and for security reasons perhaps should restrict this later to your own IP.

Set up UpCloud user credentials

Lastly, you’ll need to pass the new API account credentials to Terraform. Use the commands below to export your new API username and password as environment variables in your current shell session. The username and password will then be accessible to Terraform CLI when creating your cluster.

export TF_VAR_UPCLOUD_USERNAME=
export TF_VAR_UPCLOUD_PASSWORD=

Note: The credentials above are in plaintext. It is advisable to store the username and password Base64 encoded for more permanent use.

Overview of the infrastructure

A cluster in Kubernetes is composed of multiple control plane and worker nodes.  Control plane nodes are nodes that control and manage a set of worker nodes (workloads runtime) and worker nodes are nodes that run containerized applications.

An example of a Kubernetes cluster setup could look like the following.

You need to decide the number of both control plane and worker nodes for your cluster. In this post, we will use one control plane node and three worker nodes.

To create the Kubernetes cluster, you need to make sure you have Terraform CLI installed on your system as well as the proper configuration for your cluster. Terraform helps us to define infrastructure as a code. Defining infrastructure as code brings many advantages such as simple editing, reviewing, and versioning, as well as easy sharing amongst team members.

Configuring the cluster

Next, we’ll set up the configuration for our cluster. To avoid modifying the template files, let us copy the required files into a new directory and do the changes there.

Create a directory called my-upcloud-cluster as follows. The CLUSTER variable here is a shorthand for our directory name. If you want to name your directory differently, just change the next line and the rest of the configuration works the same.

CLUSTER=my-upcloud-cluster

mkdir inventory/$CLUSTER

Copy the sample inventory and the default cluster configuration to the new directory.

cp -r inventory/sample inventory/$CLUSTER
cp -r contrib/terraform/upcloud/* inventory/$CLUSTER/

Change your working directory to the new configuration directory and edit the cluster-settings.tfvars file to match your requirements.

cd inventory/$CLUSTER

vim cluster-settings.tfvars

The following are the main Terraform variables that you can change in cluster-settings.tfvars  file.

• hostname: A valid domain name, e.g. example.com. The maximum length is 128 characters.
• template_name: The name or UUID of a base image.
• username: A user to access the nodes e.g., via SSH. Note that the username kube is reserved by kubernetes.
• ssh_public_keys: One or more public SSH keys is required to be able to access and provision the machines after deployment.
• zone: The zone where the cluster will be created. Check the available zones for reference.
• machines: The Cloud Servers that will be provisioned. Contain the list of machines composing the cluster. The key of this object will be used as the name of the machine.
  • node_type: The role of this node (master|worker, in Kubespray and hence in this guide called “master” due to legacy naming of the control plane — is likely to change in the future).
  • cpu: Number of CPU cores.
  • mem: Memory size in MB.
  • disk_size: The size of the storage in GB.

For example, to create a cluster with two control plane nodes, three worker nodes, and each node with 2 cores, 4GB memory, and 250GB disk size, replace the machines section in the variables with the following code snippet.

machines = {
  "master-0" : {
    "node_type" : "master",
    #number of cpu cores
    "cpu" : "2",
    #memory size in MB
    "mem" : "4096"
    # The size of the storage in GB
    "disk_size" : 250
  },
  "worker-0" : {
    "node_type" : "worker",
    #number of cpu cores
    "cpu" : "2",
    #memory size in MB
    "mem" : "4096"
    # The size of the storage in GB
    "disk_size" : 250
  },
  "worker-1" : {
    "node_type" : "worker",
    #number of cpu cores
    "cpu" : "2",
    #memory size in MB
    "mem" : "4096"
    # The size of the storage in GB
    "disk_size" : 250
  },
  "worker-2" : {
    "node_type" : "worker",
    #number of cpu cores
    "cpu" : "2",
    #memory size in MB
    "mem" : "4096"
    # The size of the storage in GB
    "disk_size" : 250
  }
}

Don’t forget to replace the value of the ssh_public_keys variable with your public SSH key as it will be used to ssh to each machine when installing Kubernetes using Kubespray.

Deploying the cluster

Now that the configurations are done, you can start deploying your cluster.

Initialise your configuration directory

The terraform init command is used to initialize a working directory containing Terraform configuration files. It will download and install the UpCloud Terraform provider plugin.

Run the following command to initialize Terraform.

terraform init

Verify your infrastructure

The terraform plan command verifies your configuration is syntactically correct and creates an execution plan. Terraform performs a refresh, unless explicitly disabled, and then determines what actions are necessary to achieve the desired state specified in the configuration files.

Run the following command to see your execution plan:

terraform plan --var-file cluster-settings.tfvars 
-state=tfstate-$CLUSTER.tfstate

Deploy the infrastructure

The terraform apply command is used to apply the changes required to reach the desired state of the configuration or the predetermined set of actions generated by a Terraform execution plan. It creates new or makes changes to the existing infrastructure as defined in your configuration.

Run the following command to create your cluster and answer yes when asked to confirm:

terraform apply --var-file cluster-settings.tfvars 
-state=tfstate-$CLUSTER.tfstate

Once Terraform has finished deploying, you can go and check your cluster resources at your UpCloud Control Panel. The following figure shows the four servers (one control plane and three worker nodes) created by Terraform.

You should also get an inventory file named inventory.ini that you can use with Kubespray. We will use the inventory file to set up the Kubernetes cluster later.

ls

cluster-settings.tfvars  inventory.ini  sample
terraform.tfstate tfstate-my-upcloud-cluster.tfstate

Setting up Kubernetes with Kubespray

Ansible is an open-source software provisioning, configuration management, and application-deployment tool enabling infrastructure as code.

Configure Ansible

Set  ANSIBLE_CONFIG environment variables to Kubespray’s ansible configuration file as follows:

export ANSIBLE_CONFIG=../../ansible.cfg

Check that you have basic SSH connectivity to the nodes. You can do this by running the following ansible command.

ansible -i inventory.ini -m ping all

You should see similar to the following figure if all nodes are reachable.

master-0.example.com | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python3"
    },
    "changed": false,
    "ping": "pong"
}
worker-0.example.com | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python3"
    },
    "changed": false,
    "ping": "pong"
}
worker-1.example.com | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python3"
    },
    "changed": false,
    "ping": "pong"
}
worker-2.example.com | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python3"
    },
    "changed": false,
    "ping": "pong"
}

Deploy Kubernetes

You can now deploy Kubernetes with Kubespray using the inventory file generated during the Terraform apply step as follows.

Note: if you use a different user to access the nodes other than the default ubuntu, please replace ubuntu with the new user in the inventory.ini file.

ansible-playbook -i inventory.ini ../../cluster.yml -b -v

Then sit back and relax while Ansible and Kubespray do all the heavy lifting!

Once done, you will see a play recap of the deployment like in the screenshot below.

Accessing your new Kubernetes cluster

By default, Kubespray configures kube-master hosts with access to kube-apiserver via port 6443 as http://127.0.0.1:6443. You can connect to this from one of the master nodes.

Get the IP address of one of the master nodes and SSH to it.

For example, the following script retrieves the IP of master-0 node from the inventory file and opens an SSH connection to it using the default username.

# get the  IP address of  master-0
ip=$(grep -m 1  "master-0" inventory.ini | grep -o '[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}' | head -n 1)
# ssh to master-0 node
ssh [email protected]$ip

Once you are logged into one of the master nodes, you can run any of the kubectl commands. For example, use the command below shows the list of nodes in the cluster.

sudo kubectl get nodes

NAME                   STATUS   ROLES                  AGE    VERSION
master-0.example.com   Ready    control-plane,master   10m   v1.20.5
worker-0.example.com   Ready                           10m   v1.20.5
worker-1.example.com   Ready                           10m   v1.20.5
worker-2.example.com   Ready                           10m   v1.20.5

Accessing Kubernetes Cluster from a workstation

While in the example above, we logged into one of the nodes in the cluster, it’s also possible to command Kubernetes right from your own workstation. To make this work, simply copy the /etc/kubernetes/admin.conf from a master node to your workstation and use it with kubectl.

The following script does the trick.

# get the  IP address of  master-0
ip=$(grep -m 1  "master-0" inventory.ini | grep -o '[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}' | head -n 1)
# copy /etc/kubernetes/admin.conf file on the local system
ssh [email protected]$ip 'sudo cat /etc/kubernetes/admin.conf' > admin.conf

Make sure you have installed the kubectl tool on your local machine.

If you haven’t done so already please follow the instructions here.  Once you have installed the tool you can interact with your cluster.

To access your Kubernetes cluster remotely, you need to tell kubectl where your configuration file is. One way to do that is to point the KUBECONFIG environment variable to your cluster kubeconfig file as shown below:

export KUBECONFIG=admin.conf

One last thing you need to do before you run any of the kubectl commands is to replace the Kubernetes API IP address in the admin.conf with one of the control plane node IP addresses.

Open admin.conf file with your favourite text editor and replace 127.0.0.1 with the IP address you get from above.

vim admin.conf

Alternatively, if you already have the IP address of the first master node save in the $ip variable. You can swap the 127.0.0.1 IP with the IP saved in the variable using the following command.

sed -i "s/127.0.0.1/$ip/g" admin.conf

With the master node IP set in the admin file, you are ready to start playing with your cluster from your local machine!

For example, use the following command to show a list of namespaces in your cluster.

kubectl get namespace

NAME             STATUS  AGE
default          Active  12m
kube-node-lease  Active  12m
kube-public      Active  12m
kube-system      Active  12m

Congratulations, you now have a fully functional production-ready Kubernetes cluster up and running!

Teardown

Once you are done testing the cluster and no longer need it, you can use Terraform to tear down the deployed infrastructure.

The terraform destroy command is used to destroy the Terraform-managed infrastructure. It terminates resources defined in your Terraform configuration and performs the reverse of what the terraform apply does.

You can tear down your infrastructure using the following Terraform command:

terraform destroy --var-file cluster-settings.tfvars 
-state=tfstate-$CLUSTER.tfstate ../../contrib/terraform/upcloud/

After deletion, you can always use the same configuration files to teak and modify your cluster and deploy it again with a moments notice!