Increasing Object Storage instance size is quick and easy using your UpCloud Control Panel, but downsizing doesn’t work quite the same. Fortunately, it’s a simple task to copy data from one Object Storage instance to another using rclone. In essence, to downsize the Object Storage instance, we will be deploying a new smaller one and copying the data across.
This guide assumes that you have already created your new, smaller Object Storage instance that you wish to copy into. If you haven’t, this can be done from your UpCloud Control Panel.
Step 1: Installing rclone
To start with, you’ll need a server for running rclone to downsize Object Storage so deploy a new Cloud Server of your choice. Rclone runs great on just about any Unix-based operating system but we’ll be focusing on the popular OS options available on UpCloud.
Once your Cloud Server is up and running, log into it using SSH to get started.
First, update your package manager and ensure we have the necessary software.
For Debian/Ubuntu-based systems:
sudo apt update && sudo apt upgrade -y sudo apt install -y curl unzip
For CentOS-based systems:
sudo yum update -y sudo yum install -y curl unzip
Option 1. Installing via script
You can install rclone quickly and easily using their handy install script that works with most Unix operating systems:
sudo curl https://rclone.org/install.sh | sudo bash
Note that blindly executing scripts from the internet is not ideal and you need to be able to trust the script provider. If you’re not sure about what you’re running here, we recommend double-checking the install.sh file before executing it.
Option 2. Installing manually
Alternatively, you can install rclone manually by downloading the precompiled binary.
First, download the binary package and unzip it to a convenient location. Then change into the resulting new directory.
curl -O https://downloads.rclone.org/rclone-current-linux-amd64.zip unzip rclone-current-linux-amd64.zip cd rclone-*-linux-amd64
Next, copy the binary file to the relevant location and set the permissions accordingly.
sudo cp rclone /usr/bin/ sudo chown root:root /usr/bin/rclone sudo chmod 755 /usr/bin/rclone
Optionally, also can also install the manpage.
sudo mkdir -p /usr/local/share/man/man1 sudo cp rclone.1 /usr/local/share/man/man1/ sudo mandb
Once you’ve got rclone installed, continue below with configuring rclone to copy the data in order to downsize Object Storage.
Step 2: Configuring Object Storage remotes
Rclone offers a simple and easy-to-read configuration file which we’ll use to add our old and new Object Storage remotes, which rclone calls remote storage devices.
Next, check the rclone.conf file location with the following command:
rclone config file
Configuration file is stored at: /root/.config/rclone/rclone.conf
Then open the new file in a text editor. Note that the location of your configuration file may differ. Check the output of the previous command.
Enter the following example configuration and replace the details highlighted in red. For example, <REGION> would be fi-hel2 if your Object Storage is deployed in FI-HEL2.
[OLD-OBJECTSTORAGE] type = s3 provider = Other env_auth = false access_key_id = <ACCESS_KEY> secret_access_key = <SECRET_KEY> region = <REGION> endpoint = <ENDPOINT> location_constraint = <REGION> acl = private [NEW-OBJECTSTORAGE] type = s3 provider = Other env_auth = false access_key_id = <ACCESS_KEY> secret_access_key = <SECRET_KEY> region = <REGION> endpoint = <ENDPOINT> location_constraint = <REGION> acl = private
You will need to edit the following in the above file:
<ACCESS_KEY>: This should be your "access_key" for this Object Storage instance <SECRET_KEY>: This should be your "secret_key" for this Object Storage instance <REGION>: This is your region in lowercase, so for FI-HEL2 you'd use fi-hel2 <ENDPOINT>: Your Object Storage endpoint URL, available from the "S3 API Access" link
The Access and Secret keys would have been presented to you when you created the Object Storages.
If you did not save the keys at the time when you created the Object Storage, you can regenerate them by clicking the Settings button at your Object Storage details. Note that this will invalidate any previous keys which might have been in use on the old Object Storage instance.
Then scrolling down to the bottom of the Settings page where you’ll find the Regenerate button. After new Access and Secret keys have been generated, download the key file and store it somewhere safe.
Confirm the 2 new remotes are available to rclone:
If you see the following, you’re all set:
Name Type ==== ==== NEW-OBJECTSTORAGE s3 OLD-OBJECTSTORAGE s3
With the configuration done, the last step then is to actually copy the data. Continue in the next section on how to run rclone.
Step 3: Downsize Object Storage
To copy your files between Object Storage instances we need a simple, one-line command:
rclone copy --progress OLD-OBJECTSTORAGE:/ NEW-OBJECTSTORAGE:/
This will result in all data from the existing Object Storage instance being copied to the new one. You will see the progress thanks to --progress and once complete with no errors, you can confirm that all of your data has moved across successfully. When you are confident your data is intact on the new downsized Object Storage, you are free to delete your old, larger instance.
Note that depending on the amount of data and the network between the locations, the migration might take some time.
Running rclone in screen
Luckily, rclone is able to continue the operation in case the copy process is interrupted. But you may wish to set up a screen to run the process which allows you to close any SSH connections to the copying server without cancelling the process.
First, check that screen is installed or install it if missing.
# Debian and Ubuntu sudo apt install screen -y
# CentOS sudo yum install screen -y
Next, open a new screen with the following command.
Then run the copy in the new terminal screen.
rclone copy --progress OLD-OBJECTSTORAGE:/ NEW-OBJECTSTORAGE:/
This way, you can leave the terminal by detaching the screen with ctrl+D and the copy operation won’t be interrupted. Additionally, you can always resume a detached screen by using the following command.
That’s it! You now have everything in place to copy data between your Object Storage instances.