How to configure lifecycle policies

Some files are only useful for a while, such as log files, temporary exports, or the older versions that a bucket with versioning keeps. Lifecycle rules delete these files for you once they reach the age you set. You don't have to clean up the bucket yourself, and it doesn't keep growing. A bucket's lifecycle rules together make up its lifecycle policy.

This guide walks you through writing lifecycle rules, applying them to a bucket with s3cmd, checking them and removing them.

Before you start

You need:

• A Managed Object Storage service with a bucket. To create them, see How to get started with UpCloud Managed Object Storage. The examples in this guide use the bucket site-assets.
• s3cmd, set up with an access key of the service. To install and set it up, see Connecting to an UpCloud Object Storage instance using S3cmd.

Lifecycle rules are set with s3cmd, because the AWS CLI's put-bucket-lifecycle-configuration command fails on Managed Object Storage with this error:

aws: [ERROR]: An error occurred (InvalidRequest) when calling the PutBucketLifecycleConfiguration operation: Missing required header for this request: Content-MD5

Write the rules

Lifecycle rules for s3cmd are written in XML. This file has three rules:

<?xml version="1.0" encoding="UTF-8"?>
<LifecycleConfiguration>
  <Rule>
    <ID>delete-old-logs</ID>
    <Filter>
      <Prefix>logs/</Prefix>
    </Filter>
    <Status>Enabled</Status>
    <Expiration>
      <Days>30</Days>
    </Expiration>
  </Rule>
  <Rule>
    <ID>delete-old-versions</ID>
    <Filter/>
    <Status>Enabled</Status>
    <NoncurrentVersionExpiration>
      <NoncurrentDays>30</NoncurrentDays>
    </NoncurrentVersionExpiration>
  </Rule>
  <Rule>
    <ID>abort-unfinished-uploads</ID>
    <Filter/>
    <Status>Enabled</Status>
    <AbortIncompleteMultipartUpload>
      <DaysAfterInitiation>7</DaysAfterInitiation>
    </AbortIncompleteMultipartUpload>
  </Rule>
</LifecycleConfiguration>

Each rule has an ID, which is a name you choose, and a Filter that sets which files it applies to. A Prefix in the filter limits the rule to files whose names start with it, such as the files in the logs/ directory. An empty <Filter/> applies the rule to every file in the bucket.

The rules do the following:

• delete-old-logs deletes the files in logs/ 30 days after they were uploaded.
• delete-old-versions is for a bucket with versioning. It deletes a file's older versions 30 days after a newer version replaced them. The current version of each file stays.
• abort-unfinished-uploads cleans up large uploads that never finished. Tools such as the AWS CLI upload large files in parts. If such an upload is interrupted, it stays open in the bucket. This rule removes uploads that are still open 7 days after they started.

Change the number of days to suit your files, and keep only the rules you need. Save the file as lifecycle.xml.

In a bucket with versioning, deleting a file adds a delete marker that hides the file, and the file's versions are kept. When delete-old-versions has deleted all of a file's older versions, the delete marker is left on its own. To remove those delete markers too, add this rule to the file:

  <Rule>
    <ID>delete-expired-markers</ID>
    <Filter/>
    <Status>Enabled</Status>
    <Expiration>
      <ExpiredObjectDeleteMarker>true</ExpiredObjectDeleteMarker>
    </Expiration>
  </Rule>

Rules that move files to another storage class (Transition) aren't supported.

Apply the rules

s3cmd setlifecycle lifecycle.xml s3://<bucket-name>

s3://site-assets/: Lifecycle Policy updated

Applying a file replaces all of the bucket's rules, so keep every rule that you want in the one file.

Check the rules

s3cmd getlifecycle s3://<bucket-name>

<?xml version="1.0" ?>
<LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
	<Rule>
		<ID>delete-old-logs</ID>
		<Filter>
			<Prefix>logs/</Prefix>
		</Filter>
		<Status>Enabled</Status>
		<Expiration>
			<Days>30</Days>
		</Expiration>
	</Rule>
	<Rule>
		<ID>delete-old-versions</ID>
		<Filter/>
		<Status>Enabled</Status>
		<NoncurrentVersionExpiration>
			<NoncurrentDays>30</NoncurrentDays>
		</NoncurrentVersionExpiration>
	</Rule>
	<Rule>
		<ID>abort-unfinished-uploads</ID>
		<Filter/>
		<Status>Enabled</Status>
		<AbortIncompleteMultipartUpload>
			<DaysAfterInitiation>7</DaysAfterInitiation>
		</AbortIncompleteMultipartUpload>
	</Rule>
</LifecycleConfiguration>

Remove the rules

s3cmd dellifecycle s3://<bucket-name>

s3://site-assets/: Lifecycle Policy deleted

Removing the rules stops them from deleting anything more. Files that they haven't deleted yet stay in the bucket, and so do the older versions in a bucket with versioning. To see and delete older versions yourself, see How to enable and manage S3 object versioning.

Next steps

• How to enable and manage S3 object versioning explains how to keep older versions of the bucket's files.
• Connecting to an UpCloud Object Storage instance using S3cmd shows more s3cmd commands for working with buckets and files.