Using Retention Rules to Preserve Data

Retention rules provide immutable, WORM-compliant storage options for data written to Object Storage and Archive Storage for data governance, regulatory compliance, and legal hold requirements. Retention rules can also protect your data from accidental or malicious update, overwrite, or deletion. Retention rules can be locked to prevent rule modification and data deletion or modification even by administrators.

This topic describes Object Storage rule-based retention and provides details on how to create and manage these rules.

About Object Storage Data Retention

Retention rules are configured at the bucket level and are applied to all individual objects in the bucket. Object Storage provides a flexible approach to data retention that supports the following use cases.

REGULATORY COMPLIANCE
Your industry might require you to retain a certain class of data for a defined length of time. Your data retention regulations might also require that you lock the retention settings. If you lock the settings, the only change you can make is to increase the retention duration.
For Object Storage regulatory compliance, you create a time-bound retention rule and specify a duration. Object modification and deletion are prevented for the duration specified. Duration is applied to each object individually, and is based on the object's Last Modified timestamp. Lock the rule as required.
DATA GOVERNANCE
You might need to protect certain data sets as a part of internal business process requirements. While retaining the data for a defined length of time is necessary, that time period could change.
For Object Storage data governance, you create a time-bound retention rule and specify a duration. Object modification and deletion are prevented for the duration specified. Duration is applied to each object individually, and is based on the object's Last Modified timestamp. To be able to delete the rule and allow changes to the duration as required, do not lock the rule.
LEGAL HOLD
You might need to preserve certain business data in response to potential or on-going lawsuits. A legal hold does not have a defined retention period and remains in effect until removed.
For Object Storage legal holds, you create an indefinite retention rule. Object modification and deletion are prevented until you delete the rule. You cannot lock an indefinite retention rule because the rule has no duration.

It's important to understand retention duration for time-bound rules. Even though you are creating retention rules for a bucket, the duration of a rule is applied to each object in the bucket individually, and is based on the object's Last Modified timestamp. Let's say you have two objects in the bucket, ObjectX and ObjectY. ObjectX was last modified 14 months ago and ObjectY was last modified 3 months ago. You create a retention rule with a duration of 1 year. This rule prevents the modification or deletion of ObjectY for the next 9 months. The rule allows the modification or deletion of ObjectX because the retention rule duration (1 year) is less that the object's Last Modified timestamp (14 months). If ObjectX is overwritten some time in the coming year, modification and deletion would be prevented for the rule duration time remaining.

Locking a retention rule is an irreversible operation. Not even a tenancy administrator can delete a locked rule. There is a mandatory 14-day delay before a rule is locked. This delay lets you thoroughly test, modify, or delete the rule or the rule lock before the rule is permanently locked. A rule is active at the time of creation. The lock only controls whether the rule itself can be modified. After a rule is locked, only increases in the duration are allowed. Object modification is prevented and the rule can only be deleted by deleting the bucket. A bucket must be empty before it can be deleted.

Required IAM Policies

To use Oracle Cloud Infrastructure, you must be granted security access in a policy  by an administrator. This access is required whether you're using the Console or the REST API with an SDK, CLI, or other tool. If you get a message that you don’t have permission or are unauthorized, verify with your administrator what type of access you have and which compartment  you should work in.

If you're new to policies, see Getting Started with Policies and Common Policies.

For administrators:

  • You can create a rule that lets the specified IAM group manage Object Storage namespaces, buckets, and their associated objects in all compartments in the tenancy. For example, to let the IAM group StorageAdmins do everything in the tenancy:

    Allow group StorageAdmins to manage object-family in tenancy
  • Alternatively, you can create policies that reduce the scope of access. For example, you can create the policies to let the StorageAdmins group manage only buckets and objects in a compartment called ObjectStore in the tenancy:

    Allow group StorageAdmins to manage buckets in compartment ObjectStore
    Allow group StorageAdmins to manage objects in compartment ObjectStore
  • If you create more restrictive policies that grant individual permissions, BUCKET_UPDATE and RETENTION_RULE_MANAGE is required to create, edit, and delete retention rules. BUCKET_UPDATE, RETENTION_RULE_MANAGE, and RETENTION_RULE_LOCK is required to lock retention rules.

For more information about other alternatives for writing policies, see Details for Object Storage, Archive Storage, and Data Transfer.

Scope and Constraints

  • Retention rules can be applied to a bucket in the Standard (Object Storage) or Archive Storage tier.
  • The actions that you can perform on a bucket with active retention rules are limited. You cannot update, overwrite, or delete objects or object metadata until the retention rule is deleted (indefinite rule) or for the duration specified (time-bound rules). The duration for time-bound rules is applied to each object individually, and is based on the object's Last Modified timestamp.
  • You can create multiple retention rules for a bucket. Indefinite retention rule is applied before any time-bound rule is considered.
  • When a retention rule is locked, the rule can only be deleted by deleting the bucket. A bucket must be empty before it can be deleted.

Interaction Between Retention and Other Object Storage Features

Carefully review the policies and rules that you have in place for the other Object Storage features that you are using. Some of these policies and rules may no longer make sense with retention rules. This section describes some key things you need to know about the interaction between retention rules and other Object Storage features.

Bucket Re-Encryption

Retention rules do not block bucket re-encryption using either Oracle or your own Vault master encryption keys.

Multipart Uploads

Uncommitted (unfinished or failed) multipart uploads are not protected by retention rules and can be deleted at any time.

Lifecycle Management

  • Lifecycle policies can archive objects with retention rules.
  • Lifecycle Management cannot remove objects protected by active retention rules.

Replication

  • You can create retention rules on a replication source bucket.
  • You cannot create retention rules on a replication destination bucket.
  • You cannot enable replication on a bucket that has retention rules.

Versioning

  • You cannot add retention rules to a bucket that has versioning enabled.
  • You cannot enable versioning on a bucket with active retention rules.
  • You can add retention rules to bucket that has versioning suspended. However, you cannot resume versioning with active retention rules.

Troubleshooting Retention Rules

This topic provides troubleshooting solutions for issues you might encounter using retention rules.

Unable to create a retention rule

If creating a retention rule fails, the most likely cause is missing or incomplete IAM permissions. Rule creation requires:

  • User permissions that let you access the bucket and manage the objects in those buckets.
  • Minimally, BUCKET_UPDATE and RETENTION_RULE_MANAGE permissions.

Review the existing policies that grant user permissions. For more information, see Required IAM Policies.

Unable to lock a retention rule

If locking a retention rule fails, the most likely cause is missing or incomplete IAM permissions. Minimally, BUCKET_UPDATE, RETENTION_RULE_MANAGE, and RETENTION_RULE_LOCK permissions are required to lock retention rules.

Review the existing policies that grant user permissions. For more information, see Required IAM Policies.

Unable to delete a retention rule

You cannot delete a time-bound retention rule that is locked. When a retention rule is locked, the rule can only be deleted by deleting the bucket. A bucket must be empty before it can be deleted.

If deleting an indefinite retention rule fails, the most likely cause is missing or incomplete IAM permissions. Rule deletion requires:

  • User permissions that let you access the bucket and manage the objects in those buckets.
  • Minimally, BUCKET_UPDATE and RETENTION_RULE_MANAGE permissions.

Using the Console

To create a retention rule
  1. Open the navigation menu. Under Core Infrastructure, click Object Storage.
  2. Ensure that the correct region is selected from the regions menu (shown at the top of the Console).
  3. Choose the compartment that contains the bucket where you want to create a retention rule.

  4. Click the bucket name.

  5. Click Retention Rules under Resources to access the retention rule list.
  6. Click Create Rule.

  7. In the Create Retention Rule dialog, enter the required rule Name. The system generates a default rule name that reflects the current year, month, day, and time, for example retention-rule-20200229-1002. If you change this default to a different rule name, use letters, numbers, dashes, underscores, and periods. Avoid entering confidential information.

  8. Choose the retention rule type that you want to create:

    • Time-Bound rules have a user-defined duration. Object modification is prevented for the duration specified. Duration is applied to each object individually, and is based on the object's Last Modified timestamp.
    • Indefinite rules have no duration or expiration. Object modification is prevented until an indefinite rule is deleted.
  9. If you are creating a time Time-Bound rule, enter the retention rule duration attributes:

    • Retention Time Amount
    • Retention Time Unit
  10. Optionally, select Enable Retention Rule Lock if you want to lock the rule.
  11. Click Create.

The rule is displayed in the Retention Rules list.

To lock a time-bound retention rule
  1. Open the navigation menu. Under Core Infrastructure, click Object Storage.
  2. Ensure that the correct region is selected from the regions menu (shown at the top of the Console).
  3. In the List Scope section, select the compartment that contains the retention rule.
  4. Click the bucket name.
  5. Click Retention Rule under Resources to access the rule list.

    • If you are creating a retention rule, click Create Retention Rule. Specify the duration amount and unit, then select Enable Retention Rule Lock. Click Create.
    • If you are editing a retention rule, click the Actions icon (three dots) to the right of the rule name, and then click Edit. Select Enable Retention Rule Lock. Click Save Changes.
To view retention rule details
  1. Open the navigation menu. Under Core Infrastructure, click Object Storage.
  2. Ensure that the correct region is selected from the regions menu (shown at the top of the Console).
  3. In the List Scope section, select the compartment that contains the retention rule.
  4. Click the bucket name.
  5. Click Retention Rule under Resources to access the rule list.
  6. Click the Actions icon (three dots) to the right of the rule name, and then click View Details.
To edit a retention rule
  1. Open the navigation menu. Under Core Infrastructure, click Object Storage.
  2. Ensure that the correct region is selected from the regions menu (shown at the top of the Console).
  3. In the List Scope section, select the compartment that contains the retention rule.
  4. Click the bucket name.
  5. Click Retention Rule under Resources to access the rule list.
  6. Click the Actions icon (three dots) to the right of the rule name, and then click Edit.

    • If you are editing an indefinite retention rule, you can only edit the name of the rule.
    • If you are editing a time-bound retention rule, you can edit multiple attributes. You can edit the name, the duration time amount and time unit, and optionally select whether to lock the rule.
    • If you are removing a retention rule lock during the delay period, deselect Enable Retention Rule Lock. For more information about retention rule locking, see About Object Storage Data Retention.
  7. If you are enabling a retention rule lock, confirm the rule lock details by selecting Update the Retention Rule with a Scheduled Lock Time of <date and time>.
  8. Click Save Changes.
To delete a retention rule
  1. Open the navigation menu. Under Core Infrastructure, click Object Storage.
  2. Ensure that the correct region is selected from the regions menu (shown at the top of the Console).
  3. In the List Scope section, select the compartment that contains the bucket.
  4. Click the bucket name.
  5. Click Retention Rule under Resources to access the rule list.
  6. Click the Actions icon (three dots) to the right of the rule name, and then click Delete.
  7. Confirm deletion when prompted.

Using the Command Line Interface (CLI)

For information about using the CLI, see Command Line Interface (CLI). For a complete list of flags and options available for CLI commands, see the Command Line Reference.

To create an indefinite retention rule
oci os retention-rule create --namespace <object_storage_namespace> --bucket-name <bucket_name> --display-name <display_name>

For example:

oci os retention-rule create --namespace MyNamespace --bucket-name MyBucket --display-name LegalHold
{
  "data": {
    "display-name": "LegalHold",
    "duration": null,
    "etag": "7f51ef6c-3fca-48f7-9060-c129911c1a50",
    "id": "5772c87f-6723-4ecc-b44c-bef86643be92",
    "time-created": "2020-03-25T14:53:20.792000+00:00",
    "time-modified": "2020-03-25T14:53:20.792000+00:00",
    "time-rule-locked": null
  },
  "etag": "7f51ef6c-3fca-48f7-9060-c129911c1a50"
}
To create a time-bound, unlocked retention rule
oci os retention-rule create --namespace <object_storage_namespace> --bucket-name <bucket_name> --display-name <display_name> --time-amount <time_integer> --time-unit <days|years>

For example:

oci os retention-rule create --namespace MyNamespace --bucket-name MyBucket --display-name DataGovernance --time-amount 5 --time-unit days
{
  "data": {
    "display-name": "DataGovernance",
    "duration": {
      "time-amount": 5,
      "time-unit": "DAYS"
    },
    "etag": "efb9178f-4213-49f7-878d-7bbe57decc0b",
    "id": "89f4ca0c-4ad9-4fa5-8005-95e7741c531c",
    "time-created": "2020-03-25T15:08:01.601000+00:00",
    "time-modified": "2020-03-25T15:08:01.601000+00:00",
    "time-rule-locked": null
  },
  "etag": "efb9178f-4213-49f7-878d-7bbe57decc0b"
}
To create a time-bound, locked retention rule

See the Command Line Interface (CLI) or the Command Line Reference for supported date and time formats for --time-rule-locked.

oci os retention-rule create --namespace <object_storage_namespace> --bucket-name <bucket_name> --display-name <display_name> --time-amount <time_integer> --time-unit <days|years> --time-rule-locked <date and time>

For example:

oci os retention-rule create --namespace MyNamespace --bucket-name MyBucket --display-name RegulatoryCompliance --time-amount 5 --time-unit years --time-rule-locked "2020-04-28 00:00"
{
  "data": {
    "display-name": "RegulatoryCompliance",
    "duration": {
      "time-amount": 5,
      "time-unit": "YEARS"
    },
    "etag": "c05f02d3-d2b5-4378-9fcb-3a92ba0e018f",
    "id": "b1a6c84c-57c4-416c-b006-f864b0904c9e",
    "time-created": "2020-03-25T15:11:44.423000+00:00",
    "time-modified": "2020-03-25T15:11:44.423000+00:00",
    "time-rule-locked": "2020-04-28T00:00:00+00:00"
  },
  "etag": "c05f02d3-d2b5-4378-9fcb-3a92ba0e018f"
}
To list retention rules

oci os retention-rule list --namespace <object_storage_namespace> --bucket-name <bucket_name>

For example:


oci os retention-rule list --namespace MyNamespace --bucket-name MyBucket
{
  "data": {
    "items": [
      {
        "display-name": "RegulatoryCompliance",
        "duration": {
          "time-amount": 5,
          "time-unit": "YEARS"
        },
        "etag": "c05f02d3-d2b5-4378-9fcb-3a92ba0e018f",
        "id": "b1a6c84c-57c4-416c-b006-f864b0904c9e",
        "time-created": "2020-03-25T15:11:44.423000+00:00",
        "time-modified": "2020-03-25T15:11:44.423000+00:00",
        "time-rule-locked": "2020-04-28T00:00:00+00:00"
      },
      {
        "display-name": "DataGovernance",
        "duration": {
          "time-amount": 5,
          "time-unit": "DAYS"
        },
        "etag": "efb9178f-4213-49f7-878d-7bbe57decc0b",
        "id": "89f4ca0c-4ad9-4fa5-8005-95e7741c531c",
        "time-created": "2020-03-25T15:08:01.601000+00:00",
        "time-modified": "2020-03-25T15:08:01.601000+00:00",
        "time-rule-locked": null
      },
      {
        "display-name": "LegalHold",
        "duration": null,
        "etag": "7f51ef6c-3fca-48f7-9060-c129911c1a50",
        "id": "5772c87f-6723-4ecc-b44c-bef86643be92",
        "time-created": "2020-03-25T14:53:20.792000+00:00",
        "time-modified": "2020-03-25T14:53:20.792000+00:00",
        "time-rule-locked": null
      }
    ]
  }
}
To view retention rule details
oci os retention-rule get --namespace <object_storage_namespace> --bucket-name <bucket_name> --retention-rule-id <retention_rule_identifier> 

For example:

oci os retention-rule get --namespace MyNamespace --bucket-name MyBucket --retention-rule-id b1a6c84c-57c4-416c-b006-f864b0904c9e
{
  "data": {
    "display-name": "RegulatoryCompliance",
    "duration": {
      "time-amount": 5,
      "time-unit": "YEARS"
    },
    "etag": "c05f02d3-d2b5-4378-9fcb-3a92ba0e018f",
    "id": "b1a6c84c-57c4-416c-b006-f864b0904c9e",
    "time-created": "2020-03-25T15:11:44.423000+00:00",
    "time-modified": "2020-03-25T15:11:44.423000+00:00",
    "time-rule-locked": "2020-04-28T00:00:00+00:00"
  },
  "etag": "c05f02d3-d2b5-4378-9fcb-3a92ba0e018f"
}
To update a time-bound retention rule after creation
oci os retention-rule update --namespace <object_storage_namespace> --bucket-name <bucket_name> --retention-rule-id --display-name <display_name> --time-amount <time_integer> --time-unit <days|years> --time-rule-locked <date and time>

For example, to increase the duration of and change the date a retention rule is locked:

oci os retention-rule update --namespace MyNamespace --bucket-name MyBucket --retention-rule-id b1a6c84c-57c4-416c-b006-f864b0904c9e --time-amount 6 --time-unit years --time-rule-locked "2020-04-30 00:00"
{
  "data": {
    "display-name": "RegulatoryCompliance",
    "duration": {
      "time-amount": 6,
      "time-unit": "YEARS"
    },
    "etag": "700ada5c-6a2a-4c6c-acb6-4ebb173e0f8f",
    "id": "b1a6c84c-57c4-416c-b006-f864b0904c9e",
    "time-created": "2020-03-25T15:11:44.423000+00:00",
    "time-modified": "2020-03-25T15:46:28.724000+00:00",
    "time-rule-locked": "2020-04-30T00:00:00+00:00"
  },
  "etag": "700ada5c-6a2a-4c6c-acb6-4ebb173e0f8f"
}
To remove a retention rule lock during the delay period
oci os retention-rule update --namespace <object_storage_namespace> --bucket-name <bucket_name> --retention-rule-id --display-name <display_name> --time-rule-locked ""
oci os retention-rule update --namespace MyNamespace --bucket-name MyBucket --retention-rule-id b1a6c84c-57c4-416c-b006-f864b0904c9e --time-rule-locked ""
{
  "data": {
    "display-name": "RegulatoryCompliance",
    "duration": {
      "time-amount": 6,
      "time-unit": "YEARS"
    },
    "etag": "5b4fa526-faec-47d4-9162-4acdf1813ee0",
    "id": "b1a6c84c-57c4-416c-b006-f864b0904c9e",
    "time-created": "2020-03-25T15:11:44.423000+00:00",
    "time-modified": "2020-03-25T22:02:43.745000+00:00",
    "time-rule-locked": null
  },
  "etag": "5b4fa526-faec-47d4-9162-4acdf1813ee0"
}
To delete a retention rule
oci os retention-rule delete --namespace <object_storage_namespace> --bucket-name <bucket_name> --retention-rule-id <retention_rule_identifier>

For example:

oci os retention-rule delete --namespace MyNamespace --bucket-name MyBucket --retention-rule-id 4ed833b1-fb27-4a40-a8ab-14e09636a724

Are you sure you want to delete this resource? [y/N]: y