How to migrate from another S3 provider to UpCloud's Object Storage

While AWS CLI is a very versatile and useful tool for managing s3 compatible Object Storages, it is generally designed to work within the AWS ecosystem.

For a quick and simple migration from another S3 provider to UpCloud’s new Object Storage, the best option is to use rclone, an open-source command-line tool for managing files in cloud storage.

A comprehensive guide for installing and using rclone is already available on our tutorials page, so we wont be covering that here. Instead, this guide will summarise those steps, focusing specifically on how to migrate data from one Object Storage instance to another. This could be from the older legacy Object Storage (EOL) to UpCloud’s new Object Storage, or from another S3 provider to UpCloud’s Object Storage.

Install rclone

You can install rclone quickly and easily using their handy install script that works with most Unix operating systems:

For Ubuntu or other Debian-based systems:

sudo apt update && sudo apt upgrade -y
sudo apt install -y curl unzip
sudo curl https://rclone.org/install.sh | sudo bash

For CentOS-based systems:

sudo yum update -y
sudo yum install -y curl unzip
sudo curl https://rclone.org/install.sh | sudo bash

Configure rclone

With rclone installed, the next step is to open its configuration file. This is where you’ll enter the details for both your old and new Object Storage devices. The default location for this file is /root/.config/rclone/rclone.conf. To confirm this, you can run the command below:

rclone config file

Once you know the file’s location, open it with a text editor. Keep in mind that your configuration file might be in a different place. Just refer to the output from the command you just ran to find out.

nano /root/.config/rclone/rclone.conf

[old-objectstorage]
type = s3
provider = Other
env_auth = false
access_key_id = {access_key_id}
secret_access_key = {secret_access_key}
region = {region}
endpoint = {domain_name}
location_constraint = {region}
acl = private

[new-objectstorage]
type = s3
provider = Other
env_auth = false
access_key_id = {access_key_id}
secret_access_key = {secret_access_key}
region = {region}
endpoint = {domain_name}
location_constraint = {region}
acl = private

• {access_key_id}: replace this with your “access_key” for this Object Storage
• {secret_access_key}: replace this with your “secret_key” for this Object Storage
• {region}: This is your region in lowercase, so for FI-HEL2 you’d use fi-hel2
• {domain_name}: Your Object Storage endpoint domain name. For legacy UpCloud Object storage this can be found from the “S3 API Access” link

Then open the new file in a text editor. Note that the location of your configuration file may differ. Check the output of the previous command.

To confirm that the two new Object Storage are available to rclone, run the configuration check using the command below. The output will display the name and type of the Object Storage:

rclone config

Now, migrating your files between Object Storage instances is as simple as running the command below:

rclone copy --progress old-objectstorage:/ new-objectstorage:/

This will result in all data from the old Object Storage instance to be copied to the new one. Depending on the amount of data being transferred, the migration may take some time.

Once you have confirmed that all your data has moved across successfully, you’re free to delete your old Object Storage instance.