Tutorials How to migrate Object Storage data using rclone

How to migrate Object Storage data using rclone

Migrating to a new cloud provider can be stressful due to all the little details that need your attention. Luckily, migrating data from one S3 compliant object storage instance to another can be made super simple with rclone!

Rclone is an open-source command-line tool for managing files in cloud storage. It is a feature-rich alternative to web file browsers offered by the cloud provider that makes quick work of even larger file operations. Rclone supports more than 40 cloud storage products and S3 object stores, business & consumer file storage services, as well as standard transfer protocols.

In this guide, we’ll show how to migrate data between two Object Storage instances on UpCloud. But the steps are easy to adapt for migrating from any S3 compliant object storage provider.

Installing rclone

To start with, you’ll need a server to facilitate the migration so deploy a new Cloud Server of your choice. Rclone runs great on just about any Unix based operating system but we’ll be focusing on the popular OS options available on UpCloud.

This guide assumes that you already have a new UpCloud Object Storage instance that you wish to migrate into. If not, check out our guide on how to get started with UpCloud Object Storage.

Let’s start by updating your package manager and ensuring we have the necessary software!

For Ubuntu or other Debian based systems:

sudo apt update && sudo apt upgrade -y
sudo apt install -y curl unzip

For CentOS based systems:

sudo yum update -y
sudo yum install -y curl unzip

Option 1. Installing via script

You can install rclone quickly and easily using their handy install script that works with most Unix operating systems:

sudo curl https://rclone.org/install.sh | sudo bash

Note that blindly executing scripts from the internet is not ideal and you need to be able to trust the script provider. If you’re not sure about what you’re running here, we recommend double-checking the install.sh file before executing it.

Option 2. Installing manually

Alternatively, you can install rclone manually by downloading the precompiled binary.

First, download the binary package and unzip it to a convenient location. Then change into the resulting new directory.

curl -O https://downloads.rclone.org/rclone-current-linux-amd64.zip
unzip rclone-current-linux-amd64.zip
cd rclone-*-linux-amd64

Next, copy the binary file to the relevant location and set the permissions accordingly.

sudo cp rclone /usr/bin/
sudo chown root:root /usr/bin/rclone
sudo chmod 755 /usr/bin/rclone

Optionally, also can also install the manpage.

sudo mkdir -p /usr/local/share/man/man1
sudo cp rclone.1 /usr/local/share/man/man1/
sudo mandb

Once you’ve got rclone installed, continue below with configuring the migration source and target.

Configuring object storage connections

Rclone offers a simple and easy-to-read configuration file which we’ll use to add our old and new Object Storage remotes, that rclone calls remote storage devices.

Next, check the rclone.conf file location with the following command:

rclone config file
Configuration file is stored at:
/root/.config/rclone/rclone.conf

Then open the new file in a text editor. Note that the location of your configuration file may differ. Check the output of the previous command.

nano /root/.config/rclone/rclone.conf

Enter the following example configuration and replace the details highlighted in red. For example, <REGION> would be fi-hel2 if your Object Storage is deployed in FI-HEL2.

[OLD-OBJECTSTORAGE]
type = s3
provider = Other
env_auth = false
access_key_id = <ACCESS_KEY>
secret_access_key = <SECRET_KEY>
region = <REGION>
endpoint = <ENDPOINT>
location_constraint = <REGION>
acl = private

[NEW-OBJECTSTORAGE]
type = s3
provider = Other
env_auth = false
access_key_id = <ACCESS_KEY>
secret_access_key = <SECRET_KEY>
region = <REGION>
endpoint = <ENDPOINT>
location_constraint = <REGION>
acl = private

You will need to set the following for both old and new Object Storage in the config file:

<ACCESS_KEY>: This should be your "access_key" for this Object Storage instance
<SECRET_KEY>: This should be your "secret_key" for this Object Storage instance
<REGION>: This is your region in lowercase, so for FI-HEL2 you'd use fi-hel2
<ENDPOINT>: Your Object Storage endpoint URL, available from the "S3 API Access" link

The Access and Secret keys would have been presented to you when you created the Object Storages.
S3 Object Storage keys

If you did not save the keys at the time when you created the Object Storage, you can regenerate them by clicking Settings button at your Object Storage details.
S3 Object Storage Settings

Then scrolling down to the bottom of the Settings page where you’ll find the Regenerate button. After new Access and Secret keys have been generated, download the key file and store it somewhere safe.

S3 Object Storage access key settings

Note that the format of the region and endpoint will differ depending on which provider you are migrating from. If you are unsure, check your provider’s documentation for S3cmd configuration instructions on where to find these details.

Confirm the 2 new remotes are available to rclone by running the configuration check.

rclone config

If you see the following, you’re all set:

Name                 Type
====                 ====
NEW-OBJECTSTORAGE    s3
OLD-OBJECTSTORAGE    s3

The last step then is to initiate the migration. Continue in the next section on how to run rclone.

Migrating object storage data

To migrate your files between Object Storage instances we need a simple, one-line command:

rclone copy --progress OLD-OBJECTSTORAGE:/ NEW-OBJECTSTORAGE:/

This will result in all data from the old Object Storage instance to be copied to the new one. You will see the progress thanks to --progress and once complete with no errors, you can confirm that all of your data has moved across successfully, leaving you free to delete your old object storage instance.

Note that depending on the amount of data and the network between the locations, the migration might take some time.

Running rclone in screen

Luckily, rclone is able to continue the operation if the copy process is interrupted. But you may wish to set up a screen to run the process which allows you to close any SSH connections to the copying server without cancelling the process.

First, check that screen is installed or install it if missing.

# Debian and Ubuntu
sudo apt install screen -y
# CentOS
sudo yum install screen -y

Next, open a new screen with the following command.

screen -U

Then run the migration in the new terminal screen.

rclone copy --progress OLD-OBJECTSTORAGE:/ NEW-OBJECTSTORAGE:/

This way, you can leave the terminal by detaching the screen with ctrl+D and the and the copy operation won’t be interrupted. Additionally, you can always resume a detached screen by using the following command.

screen -Urd

That’s it! You now have everything in place to migrate data between any S3 compliant object storages.

Editor-in-chief and Technical writer at UpCloud since 2015. Cloud enthusiast writing about server technology and software.

Leave a Reply

Your email address will not be published. Required fields are marked *

Locations

Helsinki (HQ)

In the capital city of Finland, you will find our headquarters, and our first data centre. This is where we handle most of our development and innovation.

London

London was our second office to open, and a important step in introducing UpCloud to the world. Here our amazing staff can help you with both sales and support, in addition to host tons of interesting meetups.

Singapore

Singapore was our 3rd office to be opened, and enjoys one of most engaged and fastest growing user bases we have ever seen.

Seattle

Seattle is our 4th and latest office to be opened, and our way to reach out across the pond to our many users in the Americas.