Using Storage Gateway Cloud Sync

Use Cloud Sync to move on-premises datasets from a local file system to Storage Gateway, where the data is then moved asynchronously to Oracle Cloud Infrastructure Object Storage. You can also use Cloud Sync to synchronize Storage Gateway file system changes back to the local file system.

Cloud Sync generates the following for each sync job:

  • Sync status (Created, Running, Completed, Failed, or Canceled).
  • Number of files and directories to be copied from the source to the target.
  • File and directory upload progress .
  • Number of files and directories uploaded to the target.
  • Time the job started, the time the job ended, and the run duration.
  • The number of files skipped. (Cloud Sync skips non-regular files, such as symlinks, in the source directory.)
  • A list of skipped files.

You can use the Storage Gateway management console or the command line interface (CLI) to create, manage, and monitor Cloud Sync jobs.

About Cloud Sync

Cloud Sync is an enhanced wrapper around the Linux rsync command and relies on rsync to detect new and changed files using size and modification time. Cloud Sync also relies on rsync to verify the files once the data transfer is complete using checksums of the files. Cloud Sync calls the Storage Gateway diagnostic commands to provide detailed data movement progress and status between your local server, Storage Gateway, and Oracle Cloud Infrastructure.

Understanding Failure Behavior

Because the file system is mounted locally, the NFS client running on the host handles any issues resulting from file system availability or connectivity.

Cloud Sync does the following:

  • Reports and logs any failures to list or copy specific files (for example, resulting from permission issues).
  • Monitors and reports on the pending and completed uploads to Object Storage.

Storage Gateway handles any connectivity and access issues to Object Storage and performs retry operations as needed.

Prerequisites for Cloud Sync

  • Create the Storage Gateway file system that serves as either the source or target destination for the sync operation. A file system on the Storage Gateway host maps to a bucket with an identical name in Oracle Cloud Infrastructure Object Storage. See Creating Your First File System or Adding a File System for details.
  • Obtain the proper credentials to mount the file system share from the local server.
  • The local server source must:

    • Have names services set up correctly so that UIDs and GIDs are preserved and are not remapped to nobody.
    • Be exported with root permissions to read and traverse the entire source tree.

Using the Management Console

You can use the Storage Gateway management console to create, manage, and monitor Cloud Sync jobs.

To create a Cloud Sync job that syncs all files and directories at the specified source location
  1. Log in to the management console.
  2. Click Cloud Sync in the upper-right corner of the management console.

    By default, a list of all the Cloud Sync jobs that have already been created is displayed.

  3. Click Create Cloud Sync Job.
  4. In Create Cloud Sync Job page, specify the attributes of the job:

    • Job Name: Required. A unique, user-friendly name for the job. Avoid entering confidential information.
    • Source Path: Path to the Cloud Sync source directory containing the files to sync.

      • If your are syncing a local file system to Oracle Cloud Infrastructure, specify the source path as:

        /cloudsync/mounts/<user_mount>[/<path_to_directory>]
      • If your are syncing Oracle Cloud Infrastructure to a local file system, specify the source path as:

        <storage_gateway_file_system>/<path_to_directory>]
    • Target Path: Path to the Cloud Sync target directory for the synced files.

      • If your are syncing a local file system to Oracle Cloud Infrastructure, specify the target path as:

        <storage_gateway_file_system>/<path_to_directory>]
      • If your are syncing Oracle Cloud Infrastructure to a local file system, specify the target path as:

        /cloudsync/mounts/<user_mount>[/<path_to_directory>]
    • Enable Auto-Deletion: Enable this option if you want Cloud Sync to automatically delete files from the target when:

      • Files are deleted from the source.
      • Source files have been renamed.

      By default, when a file is deleted on the source, Cloud Sync does not automatically delete the file on the target unless you enable Enable Auto-Deletion. Also, when a source file is renamed, the file with the old name is deleted and a file with the new name is created. By default, Cloud Sync does not delete the file with the old name on the target (retaining both a file with the old name and a file with the new name) unless you choose Enable Auto-Deletion.

      The names of all deleted files are stored in a job-specific log directory.

  5. Click Create Cloud Sync Job.

A Cloud Sync job is created and displayed in the list of jobs.

To list Cloud Sync jobs
  1. Log in to the management console.
  2. Click Cloud Sync in the upper-right corner of the management console.

    By default, a list of all the Cloud Sync jobs is displayed.

  3. Optionally, you can filter the job listing by status (Created, Running, Completed, Failed, or Canceled) and type of Cloud Sync job (upload or download) by clicking one of the Quick Filters.
To run a Cloud Sync job
  1. Log in to the management console.
  2. Click Cloud Sync in the upper-right corner of the management console.
  3. In the list of jobs, find the Cloud Sync job that you want to run.
  4. Click Run to the right of the job name.

    Cloud Sync runs the job. The management console displays the status of the job just below the job name.

To get the status of a Cloud Sync job
  1. Log in to the management console.
  2. Click Cloud Sync in the upper-right corner of the management console.
  3. In the list of jobs, find the Cloud Sync job for which you want status.

    The management console displays the status of the job (Created, Running, Completed, Failed, or Canceled) just below the job name.

To cancel a Cloud Sync job

You can only cancel a job if it is running.

  1. Log in to the management console.
  2. Click Cloud Sync in the upper-right corner of the management console.
  3. In the list of jobs, find the Cloud Sync job that you want to cancel.

    Tip

    Use Quick Filters to get a list of Running jobs.
  4. Click Cancel to the right of the job name.
To delete a Cloud Sync job

You cannot delete a running job.

Tip

Cancel a running job before you try to delete it.
  1. Log in to the management console.
  2. Click Cloud Sync in the upper-right corner of the management console.
  3. In the list of jobs, find the created, canceled, or failed Cloud Sync job that you want to delete.
  4. Click Delete to the right of the job name.

Using the CLI

You can use the ocisg command line interface (CLI) to create, manage, and monitor Cloud Sync jobs. Using ssh, log in to the host on which you installed Storage Gateway to use the CLI.

To create a Cloud Sync job that syncs all files and directories at the specified source location
Caution

Avoid entering confidential information in the job name.

Open a command prompt and run ocisg cloudsync create to create a job:

sudo ocisg cloudsync create [--schedule=<schedule>] [--auto-delete] [--parallel=<number>] [--files-from=<file>] [--exclude-from=<file>] --verify-contents <job_name> <source_path> <target_path>

--schedule is an option to automate the Cloud Sync job so it runs according to the specified schedule. Set the schedule value using the format used for cron jobs. For example, --schedule="*/5 * * * *" runs the job every five minutes.

--auto-delete is an option to direct Cloud Sync to automatically delete files from the target when files are deleted from the source, and old names of files that have been renamed. By default, Cloud Sync does not automatically delete the files on the target unless you specify this option. The names of all deleted files are stored in a job-specific log directory.

--parallel=<number> is an option to specify the number of processes for data synchronization. By default, the number of processes is set to one. With a single process using a hard disk drive, you can expect a sync rate of 60-70 MB/s. If your system has higher disk throughput, you can use the number of processes that is proportional to the available bandwidth. Oracle recommends from two to five processes for optimal performance. The maximum number of processes allowed is 10.

--files-from=<file> is an option to specify a set of files that you want to sync to the target. If you do not set this option, the service syncs all files. The <file> should be a file under the /clousdync/ directory. For example, --files-from="/cloudsync/files.list".

--exclude-from=<file> is an option to specify a set of files that you want to exclude from the Cloud Sync job. The <file> should be a file under the /clousdync/ directory. For example, --exclude-from="/cloudsync/exclude.list".

--verify-contents is an option to enable verification of the destination contents against the source. If you do not explicitly enable verification, new files added to Cloud Sync sources after the job has started are not reported as errors when they do not appear in the destination.

The Cloud Sync job is created and displayed in the list of jobs.

To list Cloud Sync jobs

Open a command prompt and run ocisg cloudsync list to list jobs:

sudo ocisg cloudsync list [-s <status>] [<job_name_or_type>]

Optionally, you can filter the list of jobs by specifying job status (Created, Running, Completed, Failed, or Canceled). You can also list a single job by specifying the name of that job.

For example:

sudo ocisg cloudsync list sync_to_os1
To run a Cloud Sync job

Open a command prompt and run ocisg cloudsync run to run a job:

sudo ocisg cloudsync run <job_name>

For example:

sudo ocisg cloudsync run sync_to_os1
To get the status of a Cloud Sync job

Open a command prompt and run ocisg cloudsync status to get the status of a job:

sudo ocisg cloudsync status <job_name>

For example:

sudo ocisg cloudsync status sync_to_os1
To cancel a Cloud Sync job

You can cancel a job only if the job is in progress.

Open a command prompt and run ocisg cloudsync cancel to cancel a job:

sudo ocisg cloudsync cancel <job_name>

For example:

sudo ocisg cloudsync cancel sync_to_os1
To delete a Cloud Sync job

Open a command prompt and run ocisg cloudsync delete to delete a job:

sudo ocisg cloudsync delete <job_name>

For example:

sudo ocisg cloudsync delete sync_to_os1