How to deploy UpCloud Object Storage using Terraform

Terraform is a simple yet powerful open-source infrastructure management tool developed by HashiCorp. Terraform allows you to safely and predictably manage your cloud infrastructure from Cloud Servers to Object Storage by codifying APIs into declarative configuration files.

Building your cloud setup around Terraform is a great way to configure something once and be able to deploy it again and again with no effort at all. In this guide, we will show how to deploy UpCloud Object Storage using Terraform.

Setting up new deployment

Each Terraform deployment lives in its own directory. To get started, create a new directory for your Terraform project and move into the new directory.

mkdir -p ~/terraform/object-storage && cd ~/terraform/object-storage

Deploying onto UpCloud using Terraform works using our verified provider module. Using the UpCloud Terraform provider is as simple as declaring the required providers in a configuration file. A standard name for this file is version.tf.

Create the file with the below command then open it for editing with your preferred text editor.

touch version.tf

Then, add the following to the file.

terraform {
  required_providers {
    upcloud = {
      source = "UpCloudLtd/upcloud"
    }
  }
}

provider "upcloud" {
  # Your UpCloud credentials are read from the environment variables
  # export UPCLOUD_USERNAME="Username for Upcloud API user"
  # export UPCLOUD_PASSWORD="Password for Upcloud API user"
  # Optional configuration settings can be depclared here
}

Afterwards, save the file and exit the editor. With the required module declaration in place, you can continue with the next step.

Next, initialise the new Terraform configuration directory by running the command underneath.

terraform init

You should then see an output similar to the example below.

Initializing the backend...

Initializing provider plugins...
- Finding latest version of upcloudltd/upcloud...
- Installing upcloudltd/upcloud v2.1.0...
- Installed upcloudltd/upcloud v2.1.0 (self-signed, key ID 60B4E1988F222907)

Partner and community providers are signed by their developers.
If you'd like to know more about provider signing, you can read about it here:
https://www.terraform.io/docs/cli/plugins/signing.html

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

Also, as shown in the provider section of the version.tf file, to allow Terraform to access your UpCloud account, you need to set an account name and password in your environmental variables. If you haven’t done this already, check out our Terraform intro tutorial to set your API credentials before continuing with the rest of this tutorial.

Your Terraform configuration directory should then be ready. Continue in the next section to create your Object Storage configuration.

Configuring Object Storage

With Terraform, you can easily create Object Storage instances by simply declaring the resource in your Terraform config files. For this guide, we’ll configure an Object Storage instance with the starter storage size and some examples for buckets.

Create a new Terraform configuration file, e.g. object-storage.tf with the following command.

touch object-storage.tf

Then open the file for edit and add the following configuration to the file. Remember to set your access and secret keys highlighted in red.

resource "upcloud_object_storage" "my_object_storage" {
  # Allocate 250GB for this Object Storage
  size  = 250

  # Instance name which forms part of the URL used to access the storage
  name = "storage-name"

  # Location in which the Object Storage is deployed
  zone  = "nl-ams1"

  # The access and secret keys used to access the Object Storage instance
  access_key = "awodhouafsefrfhj"
  secret_key = "3piafn895a9fjpi5"
  
  description = "catalogue"

  # Create a bucket called "images"
  bucket {
    name = "images"
  }

  # Create a bucket called "products"
  bucket {
    name = "products"
  }
}

The configuration options include the following:

• Storage size in GB – can only be increased
• The name of the storage – change will cause a replacement
• Location the Object Storage is deployed to – change will cause replacement
• Access and secret keys for authentication
• Description of the storage
• Optional bucket definitions

Once your Object Storage configuration is all set, run the usual check and deploy commands.

terraform plan

terraform apply

You’ll then see an output similar to the example below.

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # upcloud_object_storage.my_object_storage will be created
  + resource "upcloud_object_storage" "my_object_storage" {
      + access_key  = "awodhouafsefrfhj"
      + created     = (known after apply)
      + description = "catalogue"
      + id          = (known after apply)
      + name        = "object-storage-terraform"
      + secret_key  = "3piafn895a9fjpi5"
      + size        = 250
      + state       = (known after apply)
      + url         = (known after apply)
      + used_space  = (known after apply)
      + zone        = "uk-lon1"

      + bucket {
          + name = "images"
        }
      + bucket {
          + name = "products"
        }
    }

Plan: 1 to add, 0 to change, 0 to destroy.
upcloud_object_storage.my_object_storage: Creating...
upcloud_object_storage.my_object_storage: Still creating... [10s elapsed]
upcloud_object_storage.my_object_storage: Still creating... [20s elapsed]
upcloud_object_storage.my_object_storage: Creation complete after 25s [id=06c73581-a9db-4e8b-ad53-4be1a437054a]

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Once deployed, your Object Storage is ready to use!

Making changes to Object Storage

Terraform also allows you to easily modify any Object Storage deployment by just making changes to the configuration file. Terraform will figure out the differences between the existing and desired state and then deploy the changes to the live services.

Note that not all configuration options can be changed without replacing the Object Storage instance. For example, storage size can only be increased to prevent data loss. Furthermore, changing the Object Storage name or location will require resources to be replaced.

Warning: operations that force replacement will cause all data in the Object Storage to be deleted.

In the example configuration below, we’ve increased the Object Storage size from 250 GB to 500 GB, updated the access and secret keys, changed the description and removed one of the buckets.

resource "upcloud_object_storage" "my_object_storage" {
  # Increasing storage size
  size  = 500

  # The access and secret keys as well as the description can be updated
  access_key = "iejdyugafsefsfhw"
  secret_key = "wad1e2evgh3olhym"
  description = "hosting"

  # (unchanged attributes hidden)
  # Buckets can be added and deleted
  bucket {
    name = "images"
  }
}

When you have made your changes to the configuration, use the plan and apply commands to update your Object Storage.

terraform plan

terraform apply

You’ll then see an output similar to the example below.

upcloud_object_storage.my_object_storage: Refreshing state... [id=06a6930f-d5ed-4e65-bdb6-28f73d562f12]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # upcloud_object_storage.my_object_storage will be updated in-place
  ~ resource "upcloud_object_storage" "my_object_storage" {
      ~ access_key  = "awodhouafsefrfhj" -> "iejdyugafsefsfhw"
      ~ description = "catalogue" -> "hosting"
        id          = "06a6930f-d5ed-4e65-bdb6-28f73d562f12"
      ~ name        = "obj-terraform-raiou" -> "storage-name"
      ~ secret_key  = "3piafn895a9fjpi5" -> "wad1e2evgh3olhym"
      ~ size        = 250 -> 500
        # (5 unchanged attributes hidden)

      - bucket {
          - name = "products" -> null
        }
        # (1 unchanged block hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.
upcloud_object_storage.my_object_storage: Modifying... [id=06a6930f-d5ed-4e65-bdb6-28f73d562f12]
upcloud_object_storage.my_object_storage: Modifications complete after 4s [id=06a6930f-d5ed-4e65-bdb6-28f73d562f12]

Apply complete! Resources: 0 added, 1 changed, 0 destroyed.

Afterwards, your Object Storage is up and running again with the new changes in place.

Connecting to Object Storage using S3cmd

When you’ve deployed your Object Storage using Terraform, you’ll next probably want to connect to it. S3cmd is a simple yet powerful command-line tool for managing S3-compatible object storage and works great with UpCloud Object Storage.

To make use of S3cmd, you’ll need to install and configure it for your Object Storage instance.

You’ll need to install Python, download and install pip, and then install S3cmd using pip.

# Ubuntu and Debian
sudo apt-get install python3 python3-distutils -y
wget https://bootstrap.pypa.io/get-pip.py -O ~/get-pip.py
sudo python3 ~/get-pip.py
sudo pip install s3cmd

Next, get the required connection details from the Terraform state file using the following command.

terraform show

# upcloud_object_storage.my_object_storage:
resource "upcloud_object_storage" "my_object_storage" {
    access_key  = "iejdyugafsefsfhw"
    created     = "2021-06-14T15:10:31Z"
    description = "hosting"
    id          = "068ea905-7120-4927-b9ef-acf34243aa2e"
    name        = "storage-name"
    secret_key  = "wad1e2evgh3olhym"
    size        = 500
    state       = "started"
    url         = "https://storage-name.nl-ams1.upcloudobjects.com/"
    used_space  = 0
    zone        = "nl-ams1"

    bucket {
        name = "images"
    }
}

Make note of your Object Storage’s access_key, sercret_key, and the endpoint url which are needed to be able to connect using an S3 client.

Next, use the command below to run the S3cmd configuration script.

s3cmd --configure

Then enter the required details in the order they are asked. Set your keys and endpoint URL as shown in the output of the terraform show command. Also, set a password for file transfer encryption. Other fields can be skipped by simply pressing Enter key to use the default value.

Then at the end, confirm to save settings.

Save settings? [y/N] y
Configuration saved to '/home/user/.s3cfg'

If you created a bucket in your Terraform configuration, you can confirm it by listing the buckets in your Object Storage.

s3cmd ls

2021-06-16 16:34  s3://images

A response showing your buckets confirms you’ve configured S3cmd correctly and are all set to start uploading files.

If you want to learn more about S3cmd, check out our getting started tutorial. It includes example commands on how to get started using your Object Storage straight from your command line.

How to use S3cmd to manage your Object Storage