Object Storage - Smart Storage Management with Lifecycle Rules

Objective

Learn how to optimise your OVHcloud Object Storage costs by configuring lifecycle rules for automatic expiration, transition, and cleanup of objects.

Introduction

What is lifecycle?

OVHcloud Object Storage bucket lifecycle is a feature that allows you to optimize your storage costs by managing your objects throughout their lifecycle. By uploading a lifecycle configuration to a bucket, you define a set of rules that the object storage solution applies to the objects of the bucket to perform specific actions.

There are 2 types of actions that OVHcloud Object Storage performs on your objects:

• expiration: these actions determine when your objects expire. Expired objects are then automatically deleted.
• transition: these actions determine when your objects are transitioned to a lower cost storage tier. For instance, you might want to transition your objects stored in the High Performance to Standard tier after 30 days.

Examples of use cases

By leveraging the lifecycle configuration feature, you can tell OVHcloud Object Storage to:

• clean incomplete multi-parts uploads: suppose you have uploaded a large number of large (>5GB) objects using multi-part uploads, but for some reasons, for many objects, the multi-part upload did not complete successfully. In this scenario, even if you haven't fully uploaded all the parts of an object, you still have to pay for the storage cost of the uploaded parts. In that case, you might want to clean the parts of all the incomplete multi-parts uploads to save money.
• clean old unused data: suppose you have an application that stores its logs in a bucket. Your organization might define a log retention policy of 30 days. After that, the logs are no longer needed and you might want to delete them to save money.
• optimize storage costs by transitioning infrequently accessed data to a less expensive storage tier: suppose you have certain files which are often used for a brief duration before they are hardly used again. You may not need immediate access, but your organization or laws may require you to keep them for a certain period. After that, you can delete them to save money.

Special considerations

When an object reaches the end of its lifetime based on its lifecycle configuration, the result of transition or expiration actions performed by OVHcloud Object Storage varies according to the versioning state of the bucket:

• non-versioned: there is only one existing version of the object which is the current version and it is deleted permanently.
• versioned: a delete marker is created and becomes the current version. You can also choose how many old versions you want to keep. If the current object version is the only object version and it is also a delete marker, that delete marker will be removed.
• versioning-suspended: we currently do not allow for the suspension of versioning if you have a lifecycle configuration in effect and vice-versa. We currently do not allow for the upload of a lifecycle configuration if versioning is suspended on the bucket.

Expiration

Lifecycle rules are processed asynchronously and on a best-effort basis. Most rules are applied within 24 hours, but for very large buckets or when processing many objects, it might take longer. During this delay, you continue to be billed for the object's current storage tier, even if the rule (e.g., expiration or transition) has already been triggered but not yet completed. For example, if an object is set to be deleted on day 30 but is only processed on day 32, you may be billed for an additional two days.

Conflicting expiration dates

Typically, the lifecycle feature is designed to help you optimize your storage costs. For instance, if two expiration rules overlap i.e. they target the same set of objects but with different expiration dates, the rule with the shorter duration is applied, ensuring that data is not retained beyond the anticipated timeframe: OVHcloud Object Storage always tries to choose the most cost-effective path for you.

Generally speaking, when you have multiple rules in a bucket lifecycle configuration that apply to the same set of objects:

• Permanent deletion takes precedence over transition.
• Transition takes precedence over creation of delete markers.
• The shorter expiration/transition date takes precedence over the longer one.

Current vs noncurrent versions

In a versioning-enabled bucket, each object has one current version and zero or more noncurrent versions. When you delete an object:

• If you don't specify a version ID in your delete request, a delete marker is created instead of deleting the object. The current object version becomes non-current, and the delete marker becomes the current version.
• If you specify a version ID in your delete request, the version of the object that is specified is permanently deleted (a delete marker is not created).
• A delete marker with zero noncurrent versions is called an expired object delete marker.

Configuration

The following is the basic structure of a lifecycle configuration JSON containing expiration rules

{
  "Rules": [
    {
      "ID": "string",
      "Status": "Enabled"|"Disabled",
      "Filter": {
        "Prefix": "string",
        "Tag": {
          "Key": "string",
          "Value": "string"
        },
        "ObjectSizeGreaterThan": long,
        "ObjectSizeLessThan": long,
        "And": {
          "Prefix": "string",
          "Tags": [
            {
              "Key": "string",
              "Value": "string"
            }
          ],
          "ObjectSizeGreaterThan": long,
          "ObjectSizeLessThan": long
        }
      },
      "Expiration": {
        "Date": timestamp,
        "Days": integer,
        "ExpiredObjectDeleteMarker": "true"|"false"
      },
      "NoncurrentVersionExpiration": {
        "NoncurrentDays": integer,
        "NewerNoncurrentVersions": integer
      },
      "AbortIncompleteMultipartUpload": {
        "DaysAfterInitiation": integer
      }
    }
  ]
}

Understanding the NoncurrentDays parameter

The NoncurrentDays parameter defines the minimum number of days since a version is no longer current. This parameter must not be confused with the age of the object but rather indicates the minimum age of a non-current version.

Example 1:

Suppose you have an object A with 10 versions:

• A v10 (current version, creation date: 2024-10-23).
• A v9 (noncurrent version, creation date: 2024-10-22).
• A v8 (noncurrent version, creation date: 2024-10-21).
• A v7 (noncurrent version, creation date: 2024-10-20).
• A v6 (noncurrent version, creation date: 2024-10-19).
• A v5 (noncurrent version, creation date: 2024-10-18).
• A v4 (noncurrent version, creation date: 2024-10-17).
• A v3 (noncurrent version, creation date: 2024-10-16).
• A v2 (noncurrent version, creation date: 2024-10-15).
• A v1 (noncurrent version, creation date: 2024-10-14).

If current date is 2024-10-23 and NoncurrentDays=5, the lifecycle rule will delete the noncurrent versions older than 5 days i.e v1, v2, v3 and v4 because:

• A v1 is non-current since 2024-10-15 (when A v2 was created) i.e its age as a non-current version is 8 days.
• A v2 is non-current since 2024-10-16 (when A v3 was created) i.e its age as a non-current version is 7 days.
• A v3 is non-current since 2024-10-17 (when A v4 was created) i.e its age as a non-current version is 6 days.
• A v4 is non-current since 2024-10-18 (when A v5 was created) i.e its age as a non-current version is 5 days.

Example 2:

Suppose you have an object B with 5 versions:

• B v5 (current version, creation date: 2024-10-28).
• B v4 (noncurrent version, creation date: 2024-10-27).
• B v3 (noncurrent version, creation date: 2024-10-20).
• B v2 (noncurrent version, creation date: 2024-10-15).
• B v1 (noncurrent version, creation date: 2024-10-14).

If current date is 2024-10-29 and NoncurrentDays=5, the lifecycle rule will delete the noncurrent versions older than 5 days i.e only v1 and v2 because:

• B v1 is non-current since 2024-10-15 (when B v2 was created) i.e its age as a non-current version is 14 days.
• B v2 is non-current since 2024-10-20 (when B v3 was created) i.e its age as a non-current version is 9 days.

Get the scheduled expiration date

If an object is scheduled to be deleted, a HEAD-OBJECT call will return a special http response header x-amz-expiration that contains a timestamp indicating its expiry date and an id of the lifecycle rule that has been applied.

The header format is: x-amz-expiration: expiry-date=<timestamp>, rule-id=<rule_id>

• expire-date: is obtained by adding the creation date and the delay from expiration
• rule-id: the matched rule id triggering the deletion

Example: Get expiration date via direct http API call

HEAD /{key}
Host: {bucket}.s3.{region}.io.cloud.ovh.net
...

HTTP/1.1 200 OK
...
x-amz-expiration: expiry-date="Fri, 21 Dec 2024 00:00:00 GMT", rule-id="123456789"

Example: Get expiration date via the cli

aws s3api head-object --bucket <bucket_name> --key <object_key>
{
  ...
  "Expiration" : "expiry-date=\"Fri, 21 Dec 2024 00:00:00 GMT\", rule-id=\"123456789\"",
  ...
}

Examples of expiration configurations

{
  "Rules": [
    {
      "ID": "123456",
      "Status": "Enabled",
      "Filter": { },
      "Expiration": {
        "Days": 1
      }
    }
  ]
}

{
  "Rules": [
    {
      "ID": "123456",
      "Status": "Enabled",
      "Filter": {
        "Prefix": "/mpus"
      },
      "AbortIncompleteMultipartUpload": {
        "DaysAfterInitiation": 7
      }
    }
  ]
}

{
  "Rules": [
    {
      "ID": "123456",
      "Status": "Enabled",
      "Filter": {
        "Prefix": "/old"
      },
      "Expiration": {
        "Days": 30
      }
    },
    {
      "ID": "456789",
      "Status": "Enabled",
      "Filter": {
        "Prefix": "/old/logs"
      },
      "Expiration": {
        "Days": 65
      }
    }
  ]
}

{
  "Rules": [
    {
      "ID": "123456",
      "Status": "Enabled",
      "Filter": {
        "Tag": {
           "Key": "age",
           "Value": "old"
        }
      },
      "Expiration": {
        "Days": 30
      }
    },
    {
      "ID": "456789",
      "Status": "Enabled",
      "Filter": {        
        "Tag": {
           "Key": "type",
           "Value": "logs"
        }
      },
      "Expiration": {
        "Days": 65
      }
    }
  ]
}

{
  "Rules": [
    {
      "ID": "123456",
      "Status": "Enabled",
      "Filter": { },
      "Expiration": {
        "Days": 1
      }
    },
    {
      "ID": "78910",
      "Status": "Enabled",
      "Filter": { },
      "AbortIncompleteMultipartUpload": {
        "DaysAfterInitiation": 1
      }
    }
  ]
}

{
  "Rules": [
    {
      "ID": "123456",
      "Status": "Enabled",
      "Filter": { },
      "Expiration": {
        "Days": 1
      },
      "NoncurrentVersionExpiration": {
        "NoncurrentDays": 1
      }
    },
    {
      "ID": "654789",
      "Status": "Enabled",
      "Filter": { },
      "Expiration": {
        "ExpiredObjectDeleteMarker": true
      }
    },
    {
      "ID": "963852",
      "Status": "Enabled",
      "Filter": { },
      "AbortIncompleteMultipartUpload": {
        "DaysAfterInitiation": 1
      }
    }
  ]
}

Transition

Supported transitions

The following are the currently supported transitions:

Minimum object size

OVHcloud Object Storage will prevent any transition to any storage tier for objects smaller than 128KB. However, you can allow transitions and expirations for smaller objects by using ObjectSizeGreaterThan or ObjectSizeLessThan parameters to filter with a minimum size or maximum size. In the following example, objects smaller than 128KB are also allowed to transition to Standard class.

{
  "Rules": [
    {
      "ID": "123456",
      "Status": "Enabled",
      "Filter": {
        "And":{
            "ObjectSizeGreaterThan": 1
        }
      },      
      "Transitions": [
        {
          "Days": 30,
          "StorageClass": "STANDARD"
        }
      ]
     }
  ]
}

Minimum transition delay

The minimum duration for transition rules is 30 days i.e your lifecycle configuration will not be valid and you will get an error if the number of days for your transition rule is less than 30 days. In practice, it means that the lifecycle feature will only consider objects older than 30 days. For example, let's say you upload an object in the High Performance storage tier and you want to transition said object to Standard storage tier after a certain amount of time and then from Standard to Standard Infrequent Access storage tier after another amount of time. That means: - your object can only be transitioned from High Performance to Standard after 30 days - your object can only be transitioned from Standard to Standard Infrequent Access after 60 days (30 days after the previous transition)

Expiration vs transitions

As already mentioned before, when you have multiple rules in a bucket lifecycle configuration that apply to the same set of objects:

• Permanent deletion takes precedence over transition
• Transition takes precedence over creation of delete markers
• The shorter expiration/transition date takes precedence over the longer one

Configuration

The following is the basic structure of a lifecycle configuration JSON containing transition rules:

{
  "Rules": [
    {   
       ...

       "Transitions": [
           {
             "Date": timestamp,
             "Days": integer,
             "StorageClass": "STANDARD"
           }
        ],
       "NoncurrentVersionTransitions": [
          {
              "NoncurrentDays": integer,
              "StorageClass": "STANDARD",
              "NewerNoncurrentVersions": integer
          }
       ]
    }
  ]
}

Examples of transition configurations

{
  "Rules": [
    {
      "ID": "123456",
      "Status": "Enabled",
      "Filter": {
        "Prefix": "old/"
      },      
      "Transitions": [
        {
          "Days": 30,
          "StorageClass": "STANDARD"
        }
      ]
     }
  ]
}

{
  "Rules": [
    {
      "ID": "123456",
      "Status": "Enabled",
      "Filter": { },      
      "Transitions": [
        {
          "Days": 30,
          "StorageClass": "STANDARD"
        }
      ],
      "NoncurrentVersionTransitions": [
        {
          "NoncurrentDays": 30,
          "StorageClass": "STANDARD"
        }
       ]
     }
  ]
}

{
  "Rules": [
    {
      "ID": "123456",
      "Status": "Enabled",
      "Filter": {
        "And":{
            "Prefix": "old/",
            "Tag": {
              "Key": "type",
              "Value": "logs"
            }
        }
      },
      "Expiration": {
        "Days": 90,
      },      
      "Transitions": [
        {
          "Days": 30,
          "StorageClass": "STANDARD"
        }
      ]
    }
  ]
}

{
  "Rules": [
    {
      "ID": "123456",
      "Status": "Enabled",
      "Filter": {
        "And":{
            "Prefix": "old/",
            "Tag": {
              "Key": "type",
              "Value": "logs"
            }
        }
      },
      "Expiration": {
        "Days": 90,
      }
    },
    {
      "ID": "456789",
      "Status": "Enabled",
      "Filter": {
        "And":{
            "Prefix": "old/",
            "Tag": {
              "Key": "type",
              "Value": "logs"
            }
        }
      },
      "Transitions": [
        {
          "Days": 90,
          "StorageClass": "STANDARD"
        }
      ]
    }
  ]
}

Instructions

Using the CLI

As a prerequisite, you must have a bucket containing data on which you want to apply the lifecycle configuration and have the necessary permissions (by default the bucket owner or the s3:putLifecycleConfiguration permission given via a user access policy) to do so.

cat lifecycle.json
{
  "Rules": [
    {
      "ID": "123456",
      "Status": "Enabled",
      "Filter": {
        "Prefix": "old/"
      },
      "Expiration": {
        "Days": 45
      },
      "NoncurrentVersionExpiration": {
        "NoncurrentDays": 15,
        "NewerNoncurrentVersions": 3
      }
    }
  ]
}

aws s3api put-bucket-lifecycle-configuration --bucket my-bucket --lifecycle-configuration file://lifecycle.json

Using the OVHcloud Control Panel (coming soon)

FAQ

Why are my objects not expired yet although I set up lifecycle rules?

Lifecycle rules are processed asynchronously and on a best-effort basis. Most rules are applied within 24 hours, but for very large buckets or when processing many objects, it might take longer. During this delay, you continue to be billed for the object's current storage tier, even if the rule (e.g., expiration or transition) has already been triggered but not yet completed.

Why is the object count of my versioning-enabled bucket still increasing although I set up lifecycle rules?

When a delete object operation (i.e expiration) is performed on an object in a versioning-enabled bucket, it does not delete the object permanently but it creates a delete marker on the object. This delete marker becomes the latest and current version of the object with a new version ID thus increasing the object count of the bucket.

Extra lifecycle configuration is needed to remove objects permanently, including incomplete multipart uploads, expired delete markers, and previous versions of objects.

How can I empty my S3¹ bucket using Lifecycle rules?

To empty an S3 bucket, you will need to consider the following:

• expire current versions of objects
• remove previous versions of objects
• remove expired delete markers
• remove incomplete multipart uploads

How can I monitor actions taken by my lifecycle rules?

You can use the Server Access Logging feature to monitor actions taken by your lifecycle rules on your S3 bucket. Server Access Logging provides detailed records for the requests that are made to a bucket including expiration operations, transitions of objects to another storage tier,etc.

How can I recover objects deleted by my lifecycle rules?

Versioning is the only way to recover objects that have been expired by lifecycle rules. It must be activated on your bucket before you set up your lifecycle rules. However, objects that are permanently deleted by lifecycle rules cannot be recovered.

How can I exclude prefixes from my lifecycle rules?

Prefix exclusion is not supported.

I have set up replication rules between 2 S3 buckets: why are objects deleted in my source bucket by my lifecycle rules not deleted in the destination bucket?

Delete operations resulting from application of lifecycle rules are not replicated. See the documentation for asynchronous replication for further details on what is replicated and what is not.

Go further

If you need training or technical assistance to implement our solutions, contact your sales representative or click on this link to get a quote and ask our Professional Services experts for assisting you on your specific use case of your project.

Join our community of users.

¹: S3 is a trademark of Amazon Technologies, Inc. OVHcloud's service is not sponsored by, endorsed by, or otherwise affiliated with Amazon Technologies, Inc.