The Oracle Cloud Infrastructure Object Storage service supports multipart uploads for more efficient and resilient uploads, especially for large objects. You can perform multipart uploads using the API, the Software Development Kits and Command Line Interface, or the Command Line Interface (CLI). With multipart uploads, individual parts of an object can be uploaded in parallel to reduce the amount of time you spend uploading. Multipart uploads performed through the API can also minimize the impact of network failures by letting you retry a failed part upload instead of requiring you to retry an entire object upload.
Multipart uploads can accommodate objects that are too large for a single upload operation. Oracle recommends that you perform a multipart upload to upload objects larger than 100 MiB. The maximum size for an uploaded object is 10 TiB. Object parts must be no larger than 50 GiB. For very large uploads performed through the API, you have the flexibility of pausing between the uploads of individual parts, and resuming the upload as your schedule and resources allow.
Required IAM Policy
To use Oracle Cloud Infrastructure, you must be given the required type of access in a An IAM document that specifies who has what type of access to your resources. It is used in different ways: to mean an individual statement written in the policy language; to mean a collection of statements in a single, named "policy" document (which has an Oracle Cloud ID (OCID) assigned to it); and to mean the overall body of policies your organization uses to control access to resources. written by an administrator, whether you're using the Console or the REST API with an SDK, CLI, or other tool. If you try to perform an action and get a message that you don’t have permission or are unauthorized, confirm with your administrator the type of access you've been granted and which A collection of related resources that can be accessed only by certain groups that have been given permission by an administrator in your organization. you should work in.
For administrators: The policy in Let Object Storage admins manage buckets and objects lets the specified group do everything with buckets and objects.
If you are new to policies, see Getting Started with Policies and Common Policies. If you want to dig deeper into writing policies for Object Storage, see Details for Object Storage, Archive Storage, and Data Transfer.
You can monitor the health, capacity, and performance of your Oracle Cloud Infrastructure resources by using metrics, alarms, and notifications. For more information, see Monitoring Overview and Notifications Overview.
For more information about monitoring multipart uploads, see Object Storage Metrics.
Using the Multipart Upload API
A multipart upload performed using the API consists of the following steps:
- Initiating an upload
- Uploading object parts
- Committing the upload
Before you use the multipart upload API, you are responsible for creating the parts to upload. Object Storage Provides API operations for the remaining steps. The service also provides API operations for listing in-progress multipart uploads, listing the object parts in an in-progress multipart upload, and aborting in-progress multipart uploads initiated through the API.
Here we describe the API steps at a high level, but you can refer to the API Reference for specifics about supported API calls.
Creating Object Parts
With multipart upload, you split the object you want to upload into individual parts. Individual parts can be as large as 50 GiB or as small as 10 MB. (Object Storage waives the minimum part size restriction for the last uploaded part.) Decide what part number you want to use for each part. Part numbers can range from 1 to 10,000. You do not need to assign contiguous numbers, but Object Storage constructs the object by ordering part numbers in ascending order.
Initiating an Upload
After you finish creating object parts, initiate a multipart upload by making a CreateMultipartUpload REST API call. Provide the object name and any object metadata. Object Storage Responds with a unique upload ID that you must include in any requests related to this multipart upload. Object Storage also marks the upload as active. The upload remains active until you explicitly commit it or abort it.
Uploading Object Parts
Make an UploadPart request for each object part upload. In the request parameters, provide the Object Storage namespace, bucket name, upload ID, and part number. In the request body, include the object part. Object parts can be uploaded in parallel and in any order. When you commit the upload, Object Storage uses the part numbers to sequence object parts. Part numbers do not have to be contiguous. If multiple object parts are uploaded using the same upload ID and part number, the last upload overwrites the part and is committed when you call the CommitMultipartUpload API.
Object Storage returns an ETag (entity tag) value for each part uploaded. You need both the part number and corresponding ETag value for each part when you commit the upload.
If you have network issues, you can restart a failed upload for an individual part. You do not need to restart the entire upload. If, for some reason, you cannot perform an upload all at once, multipart upload lets you continue uploading parts at your own pace. While a multipart upload is still active, you can keep adding parts as long as the total number is less than 10,000.
You can check on an active multipart upload by listing all parts that have been uploaded. (You cannot list information for an individual object part in an active multipart upload.) The ListMultipartUploadParts operation requires the Object Storage namespace, bucket name, and upload ID. Object Storage responds with information about the parts associated with the specified upload ID. Parts information includes the part number, ETag value, MD5 hash, and part size (in bytes).
Similarly, if you have multiple multipart uploads occurring simultaneously, you can see what uploads are in-progress. Make an ListMultipartUploads API call to list active multipart uploads in the specified Object Storage namespace and bucket.
Charges for parts storage begin accruing when you upload data.
Committing the Upload
When you have uploaded all object parts, commit the upload. Use the CommitMultipartUpload request parameters to specify the Object Storage namespace, bucket name, and upload ID. Include the part number and corresponding ETag value for each part in the body of the request. When you commit the upload, Object Storage constructs the object from its constituent parts. The object is stored in the specified bucket and Object Storage namespace. You can treat it like you would any other object. Garbage collection releases storage space occupied by any part numbers you uploaded, but did not include in the CommitMultipartUpload request.
You cannot list or retrieve parts from a completed upload. You cannot append or remove parts from the completed upload either. If you want, you can replace the object by initiating a new upload.
If you decide to abort a multipart upload instead of committing it, wait for in-progress part uploads to complete and then use the AbortMultipartUpload operation. If you abort an upload while part uploads are still in progress anyway, Object Storage cleans up both completed and in-progress parts. Upload IDs from aborted multipart uploads cannot be reused.
For information about using the API and signing requests, see REST APIs and Security Credentials. For information about SDKs, see Software Development Kits and Command Line Interface.
Use the following operations to manage multipart uploads:
- UploadPart (see Special Instructions for Object Storage PUT for signing request requirements)
Using the CLI
When you perform a multipart upload using the CLI, you do not need to split the object into parts as you are required to do by the API. Instead, you specify the part size of your choice, and Object Storage splits the object into parts and performs the upload of all parts automatically. You can choose to set the maximum number of parts that can be uploaded in parallel. By default, the CLI limits the number of parts that can be uploaded in parallel to three. When using the CLI, you do not have to perform a commit when the upload is complete.
You can also use the CLI to list in-progress multipart uploads, and to abort multipart uploads initiated through the API.
To upload an object, open a command prompt and run
oci os object put with the
--part-size flag. The
--part-size value represents the size of each part in mebibytes (MiB). Object Storage waives the minimum part size restriction for the last uploaded part. The
--part-size value must be an integer.
Optionally, you can use the
--parallel-upload-count flag to set the maximum number of parallel uploads allowed.
oci os object put -ns <object_storage_namespace> -bn <bucket_name> --file <file_location> --name <object_name> --part-size <upload_part_size_in_MB> --parallel-upload-count <maximum_number_parallel_uploads>
For more information on the
oci os object put command, see To upload an object to a bucket.
Open a command prompt and run
oci os multipart list to list all in-progress multipart uploads for a bucket:
oci os multipart list -ns <object_storage_namespace> -bn <bucket_name>
See CLI Help for command options to control the pagination of the list output.
Open a command prompt and run
oci os multipart abort to abort an in-progress multipart upload initiated through the API:
oci os multipart abort -ns <object_storage_namespace> -bn <bucket_name> --object-name <object_name> --upload-id <upload_ID>
The CLI interface asks you to confirm the deletion request. To abort without the confirmation prompt, use the