1. Move data to object storage in the cloud by using rclone
  2. Learn About Moving Data to Cloud-based Object Storage by Using rclone

Learn About Moving Data to Cloud-based Object Storage by Using rclone

If you have a strategy to move your apps and app data to the cloud, but find the prospect of moving existing unstructured and object data to the cloud from multiple sources daunting, you can use rclone with this simple architecture to simplify this task.

rclone lets you transfer large amounts of unstructured data like videos, backups, and logs from cloud or on-premises sources into Oracle Cloud Infrastructure Object Storage.

Considerations for Moving Data to Cloud-based Object Storage

This solution is intended for technical audiences, including systems and network administrators, site reliability engineers, solutions architects, and engineers. Before attempting it, consider the following:

  • To successfully complete this solution, you should have:
    • Basic familiarity with Linux and command line (CLI) tools.
    • Non-administrator-level access to a workstation or server running Linux with uninterrupted and fast connectivity to the internet.
    • Valid credentials for Object Storage.
  • The methods outlined in this paper are suitable only if the amount of transferred data is less than 100 tebibytes (TiB). The maximum amount of data that can be transferred depends on available internet bandwidth and, to a lesser degree, the technical specifications of the workstation or server that is used. If the amount of data is higher than 100 TiB, this method might take a prohibitively long time.
  • You should use an Oracle Cloud Infrastructure Compute instance when transferring data from other cloud providers. All examples in this solution have been tested on a VM.DenseIO1.4 Compute instance.
  • Oracle does not charge additional fees for transferring data from other cloud providers. Data transferred to Object Storage is priced based on standard Object Storage pricing. If you use an Oracle Cloud Infrastructure Compute instance for transferring data, you will be charged separately for that usage.

About Oracle Cloud Infrastructure Object Storage

The Oracle Cloud Infrastructure Object Storage service provides unlimited scale for large amounts of unstructured data like videos, backups, and logs. Data in Object Storage is automatically replicated across multiple fault domains.

One of the first steps in using Object Storage is to transfer your existing data. You can choose from a variety of different methods, tools, and services, and your choices depend on your specific use cases, the amount of data that needs to be transferred, and available network bandwidth.

This solution shows how to transfer data to Object Storage from other cloud providers. It provides specific instructions and examples for transferring data from Oracle Cloud Infrastructure Object Storage Classic, Amazon S3, and Microsoft Azure Blob Storage. It also provides instructions on a secondary use case, transferring data to Object Storage from a local file system.

About the Steps for Using rclone to Move Data to Oracle Cloud Infrastructure Object Storage

This playbook leads you through the four steps required to move data to cloud storage:

  1. Install rclone. You can install rclone either for all users on a system or for a single user.
  2. Configure object storage to use the Amazon S3 Compatibility API. You will also need to test your connection to Object Storage.
  3. Configure your source. This simply that you export your source path as an environment variable and depends on the source you are using.
  4. Transfer the data and verify the transfer.