Oracle Cloud Infrastructure Documentation

Using Storage Gateway Cloud Sync

Use Cloud Sync to move on-premises datasets from a local file system to Storage Gateway, where the data is then moved asynchronously to Oracle Cloud Infrastructure Object Storage. You can also use Cloud Sync to synchronize Storage Gateway file system changes back to the local file system.

Cloud Sync generates the following for each sync job:

  • Sync status (Created, Running, Completed, Failed, or Canceled)
  • Number of files and directories to be copied from the source to the target
  • File and directory upload progress
  • Number of files and directories uploaded to the target
  • Time the job started, the time the job ended, and the run duration

You can use the Storage Gateway management console or the command line interface (CLI) to create, manage, and monitor Cloud Sync jobs.

About Cloud Sync

Cloud Sync is an enhanced wrapper around the Linux rsync command and relies on rsync to detect new and changed files using size and modification time. Cloud Sync also relies on rsync to verify the files once the data transfer is complete using checksums of the files. Cloud Sync calls the Storage Gateway diagnostic commands to provide detailed data movement progress and status between your local server, Storage Gateway, and Oracle Cloud Infrastructure.

Understanding Failure Behavior

Because the file system is mounted locally, the NFS client running on the host handles any issues resulting from file system availability or connectivity.

Cloud Sync does the following:

  • Reports and logs any failures to list or copy specific files (for example, resulting from permission issues)
  • Monitors and reports on the pending and completed uploads to Object Storage

Storage Gateway handles any connectivity and access issues to Object Storage and performs retry operations as needed.

Prerequisites for Cloud Sync

  • Create the Storage Gateway file system that serves as either the source or target destination for the sync operation. A file system on the Storage Gateway host maps to a bucket with an identical name in Oracle Cloud Infrastructure Object Storage. See Creating Your First File System or Adding a File System for details.
  • Obtain the proper credentials to mount the file system share from the local server.
  • The local server source must:

    • Have names services set up correctly so that UIDs and GIDs are preserved and are not remapped to nobody
    • Be exported with root permissions to read and traverse the entire source tree

Using the Management Console

You can use the Storage Gateway management console to create, manage, and monitor Cloud Sync jobs.

To create a Cloud Sync job that syncs all files and directories at the specified source location
To list Cloud Sync jobs
To run a Cloud Sync job
To get the status of a Cloud Sync job
To cancel a Cloud Sync job
To delete a Cloud Sync job

Using the CLI

You can use the ocisg command line interface (CLI) to create, manage, and monitor Cloud Sync jobs. Using ssh, log in to the host on which you installed Storage Gateway to use the CLI.

To create a Cloud Sync job that syncs all files and directories at the specified source location
To list Cloud Sync jobs
To run a Cloud Sync job
To get the status of a Cloud Sync job
To cancel a Cloud Sync job
To delete a Cloud Sync job