Use Cloud Sync to move on-premises datasets from a local file system to Storage Gateway, where the data is then moved asynchronously to Oracle Cloud Infrastructure Object Storage. You can also use Cloud Sync to synchronize Storage Gateway file system changes back to the local file system.
Cloud Sync generates the following for each sync job:
Sync status (Created, Running, Completed, Failed, or Canceled).
Number of files and directories to be copied from the source to the target.
File and directory upload progress .
Number of files and directories uploaded to the target.
Time the job started, the time the job ended, and the run duration.
The number of files skipped. (Cloud Sync skips non-regular files, such as symlinks, in the source directory.)
Cloud Sync is an enhanced wrapper around the Linux rsync command and relies on rsync to detect new and changed files using size and modification time. Cloud Sync also relies on rsync to verify the files once the data transfer is complete using checksums of the files. Cloud Sync calls the Storage Gateway diagnostic commands to provide detailed data movement progress and status between your local server, Storage Gateway, and Oracle Cloud Infrastructure.
Understanding Failure Behavior
Because the file system is mounted locally, the NFS client running on the host handles any issues resulting from file system availability or connectivity.
Cloud Sync does the following:
Reports and logs any failures to list or copy specific files (for example, resulting from permission issues).
Monitors and reports on the pending and completed uploads to Object Storage.
Storage Gateway handles any connectivity and access issues to Object Storage and performs retry operations as needed.
Prerequisites for Cloud Sync
Create the Storage Gateway file system that serves as either the source or target destination for the sync operation. A file system on the Storage Gateway host maps to a bucket with an identical name in Oracle Cloud Infrastructure Object Storage. See Creating Your First File System or Adding a File System for details.
Obtain the proper credentials to mount the file system share from the local server.
The local server source must:
Have names services set up correctly so that UIDs and GIDs are preserved and are not remapped to nobody.
Be exported with root permissions to read and traverse the entire source tree.
Using the Management Console
You can use the Storage Gateway management console to create, manage, and monitor Cloud Sync jobs.
Enable Auto-Deletion: Enable this option if you want Cloud Sync to automatically delete files from the target when:
Files are deleted from the source.
Source files have been renamed.
By default, when a file is deleted on the source, Cloud Sync does not automatically delete the file on the target unless you enable Enable Auto-Deletion. Also, when a source file is renamed, the file with the old name is deleted and a file with the new name is created. By default, Cloud Sync does not delete the file with the old name on the target (retaining both a file with the old name and a file with the new name) unless you choose Enable Auto-Deletion.
The names of all deleted files are stored in a job-specific log directory.
Click Create Cloud Sync Job.
A Cloud Sync job is created and displayed in the list of jobs.
--schedule is an option to automate the Cloud Sync job so it runs according to the specified schedule. Set the schedule value using the format used for cron jobs. For example, --schedule="*/5 * * * *" runs the job every five minutes.
--auto-delete is an option to direct Cloud Sync to automatically delete files from the target when files are deleted from the source, and old names of files that have been renamed. By default, Cloud Sync does not automatically delete the files on the target unless you specify this option. The names of all deleted files are stored in a job-specific log directory.
--parallel=<number> is an option to specify the number of processes for data synchronization. By default, the number of processes is set to one. With a single process using a hard disk drive, you can expect a sync rate of 60-70 MB/s. If your system has higher disk throughput, you can use the number of processes that is proportional to the available bandwidth. Oracle recommends from two to five processes for optimal performance. The maximum number of processes allowed is 10.
--files-from=<file> is an option to specify a set of files that you want to sync to the target. If you do not set this option, the service syncs all files. The <file> should be a file under the /clousdync/ directory. For example, --files-from="/cloudsync/files.list".
--exclude-from=<file> is an option to specify a set of files that you want to exclude from the Cloud Sync job. The <file> should be a file under the /clousdync/ directory. For example, --exclude-from="/cloudsync/exclude.list".
--verify-contents is an option to enable verification of the destination contents against the source. If you do not explicitly enable verification, new files added to Cloud Sync sources after the job has started are not reported as errors when they do not appear in the destination.
The Cloud Sync job is created and displayed in the list of jobs.