Oracle Cloud Infrastructure Documentation

Managing Disk Data Transfers

This topic describes in detail all the tasks required to transfer data to Oracle Cloud Infrastructure using the Data Transfer Disk service.

In preparation for data migration, an IAM user that has administrative authorization and permissions must perform the prerequisite tasks described in Preparing for Data Transfer.

The Data Transfer Utility is a command-line utility that you install on your host machine. Many data transfer tasks can be performed either using the Console or using the Data Transfer Utility. However, some data transfer tasks can only be performed using the Data Transfer Utility. This section guides you to the appropriate management interface to use for each task. See Data Transfer Utility for detailed installation and configuration instructions.

Important

The Data Transfer Utility must be run as the root user.

There are two ways you can specify Data Transfer Utility parameters:
--parameter <value>
or
--parameter=<value>

Creating a Transfer Job

Tip

You can use the Console or the Data Transfer Utility to create a transfer job.

A transfer job represents the collection of files that you want to transfer and signals the intention to upload those files to Oracle Cloud Infrastructure. A transfer job combines at least one transfer disk with a transfer package. Identify which compartment and Object Storage bucket that Oracle is to upload your data to. Create the transfer job in the same compartment as the upload bucket and supply a human-readable name for the transfer job. Avoid entering confidential information when providing transfer job names.

Creating a transfer job returns a job ID that you specify in other transfer tasks. For example:

ocid1.datatransferjob.region1.phx..<unique_ID>
To create a transfer job using the Console
To create a transfer job using the Data Transfer Utility

Performing Other Transfer Job Tasks

Tip

You can use the Console or the Data Transfer Utility to perform other transfer job-related tasks.

Displaying the List of Transfer Jobs

To display the list of transfer jobs using the Console
To display the list of transfer jobs using the Data Transfer Utility

Displaying the Details of a Transfer Job

To display the details of a transfer job using the Console
To display the details of a transfer job using the Data Transfer Utility

Editing the Name of a Transfer Job

To edit the name of a transfer job using the Console
To edit the name of a transfer job using the Data Transfer Utility

Deleting a Transfer Job

Typically, you would delete a transfer job early in the transfer process and before you create any transfer packages or disks. For example, you initiated the data transfer by creating a transfer job, but changed your mind. If you want to delete a transfer job later in the transfer process, you must first delete all transfer packages and disks associated with the transfer job.

To delete a transfer job using the Console
To delete a transfer job using the Data Transfer Utility

Attaching HDDs to the Host Machine

Before creating a transfer disk from an attached HDD, remove all partitions and any file systems. To prevent the accidental deletion of data, the Data Transfer Utility doesn't work with HDDs that already have partitions or file systems. HDDs are visible to the host as block devices and must provide a valid response to the hdparm -I <device> Linux command.

Creating a Transfer Disk

Tip

You can only use the Data Transfer Utility to create a transfer disk.

When you create a transfer disk, the Data Transfer Utility:

  • Sets up the HDD for encryption using the passphrase
  • Creates a file system on the HDD
  • Mounts the file system at /mnt/orcdts_<label>

For example:

 /mnt/orcdts_DJZNWK3ET

When you register a transfer disk, Oracle Cloud Infrastructure generates a strong encryption passphrase that is used to encrypt the transfer disk. The encryption passphrase is displayed to standard output to the data transfer administrator user and cannot be retrieved again. Create a local, secure copy of the encryption passphrase, so you can reference the passphrase again.

Creating a transfer disk requires the job ID returned from when you created the transfer job and the path to the attached HDD (for example, /dev/sdb).

To create a transfer disk using the Data Transfer Utility

Performing Other Transfer Disk Tasks

Tip

You can only use the Data Transfer Utility to delete or cancel a transfer disk.

Deleting a Transfer Disk

Typically, you would delete a transfer disk during the disk preparation process. You created, attached, and/or copied data to the transfer disk, but have changed your mind about shipping the disk. If you want to reuse the disk, remove all file systems and create the disk again.

To delete a transfer disk using the Data Transfer Utility

Canceling a Transfer Disk

If you shipped a disk to Oracle, but have changed your mind about uploading the files, you can cancel the transfer disk. You can cancel a disk in a transfer package, while allowing the file upload from other disks.

Oracle cannot process canceled transfer disks. Oracle returns canceled transfer disks to the sender.

To cancel a transfer disk using the Data Transfer Utility

Copying Data to the Transfer Disk

Attach the HDDs to the host machine and copy files to the mount point created by the Data Transfer Utility.

You can only copy regular files to transfer disks. Special files (links, sockets, pipes, and so forth) cannot be copied directly. To transfer special files, create a tar archive of the files and copy the tar archive to the transfer disk.

Note

Copy all Files Before Disconnecting a Transfer Disk

Do not disconnect a transfer disk until you copy all files from the host and generate the manifest file. If you accidentally disconnect the transfer disk before copying all files, you must unlock the disk using the encryption passphrase. The encryption passphrase was generated and displayed when you created the transfer disk. If the generated encryption passphrase is not available, you must delete the transfer disk from the transfer job and re-create the disk. All data previously copied to that transfer disk is lost.

Generating a Manifest File

Tip

You can only use the Data Transfer Utility to generate a manifest file.
The amount of time to generate the manifest file depends on the size of the upload files, disk speed, and available processing power.

After copying your data to a transfer disk, generate a manifest file using the Data Transfer Utility. The manifest contains an index of all of the copied files and generated data integrity hashes. The Data Transfer Utility copies the config_upload_user configuration file and referenced IAM credentials to the encrypted transfer disk. This configuration file describes the temporary IAM data transfer upload user. Oracle uses the credentials and entries defined in the config_upload_user file when processing the transfer disk and uploading files to Oracle Cloud Infrastructure Object Storage.

Note

Data Transfer Service Does Not Support Passphrases on Private Keys

While we recommend encrypting a private key with a passphrase when generating API signing keys, Data Transfer does not support passphrases on the key file required for the config_upload_user. If you use a passphrase, Oracle personnel cannot upload your data.

Oracle cannot upload data from a transfer disk without the correct credentials defined in this configuration file. See Data Transfer Utility for more information about the required configuration files.

To create a manifest file using the Data Transfer Utility
Note

Do You Need to Regenerate the Manifest File?

If you add, remove, or modify any files on the disk after generating the manifest file, you must regenerate the file. If the manifest file does not match the contents of the target bucket, Oracle cannot upload the data.

Generating a Dry-Run Report of the Transfer

Tip

You can only use the Data Transfer Utility to generate a dry-run report.

You can generate a dry-run report to review the transfer results before the actual data upload. The report compares the contents of the generated manifest file with the contents of the target bucket. This report can help determine if you have duplicate files or naming collision issues.

Note

Do You Need to Regenerate the Manifest File?

If you add, remove, or modify any files on the disk after generating the manifest file, you must regenerate the file. If the manifest file does not match the contents of the target bucket, Oracle cannot upload the data.

To generate a dry-run report

Locking a Transfer Disk

Tip

You can only use the Data Transfer Utility to lock a transfer disk.

Locking a transfer disk safely unmounts the disk and removes the encryption passphrase from the host. When you lock a transfer disk, you are prompted for the encryption passphrase that was generated when you created the transfer disk.

To lock a transfer disk using the Data Transfer Utility

If you need to unlock the transfer disk, you are prompted for the encryption passphrase that was generated when you created the transfer disk.

To unlock a transfer disk using the Data Transfer Utility

Creating a Transfer Package

Tip

You can use the Console or the Data Transfer Utility to create a transfer package.

A transfer package is the virtual representation of the physical package of disks that you are shipping to Oracle for upload to Oracle Cloud Infrastructure.

Creating a transfer package requires the job ID returned from when you created the transfer job. For example:

ocid1.datatransferjob.region1.phx..exampleuniqueID
To create a transfer package using the Console
To create a transfer package using the Data Transfer Utility

Performing Other Transfer Package Tasks

Tip

You can use the Console or the Data Transfer Utility to perform other transfer package-related tasks.

Displaying the Details of a Transfer Package

To display the details of a transfer package using the Console
To display the details of a transfer package using the Data Transfer Utility

Editing a Transfer Package

Edit the transfer package and supply the tracking information when you ship the package.

To edit a transfer package using the Console
To edit a transfer package using the Data Transfer Utility

Deleting a Transfer Package

Typically, you would delete a transfer package early in the transfer process and before you created any transfer disks. You initiated the transfer job and package, but have changed your mind. If you delete a transfer package later in the transfer process, you must first detach all associated transfer disks. You cannot delete a transfer package once the package has been shipped to Oracle.

To delete a transfer package using the Console
To delete a transfer package using the Data Transfer Utility

Canceling a Transfer Package

If you shipped a transfer package, but have changed your mind about uploading the data, you can cancel a transfer package. Before canceling a transfer package, you must first cancel all transfer disks associated with that transfer package. Oracle cannot process canceled transfer packages. Oracle returns canceled transfer packages to the sender.

To cancel a transfer package using the Console
To cancel a transfer package using the Data Transfer Utility

Attaching Transfer Disks to a Transfer Package

Tip

You can use the Console or the Data Transfer Utility to attach a transfer disk to a transfer package.

You attach a transfer disk to a transfer package after you have copied your data onto the disk, generated the required manifest file, run and reviewed the dry-run report, and then locked the transfer disk in preparation for shipment.

A disk can be attached to one package, detached, and then attached to another package.

To attach a transfer disk to a transfer package using the Console
To attach a transfer disk to a transfer package using the Data Transfer Utility

You have attached a transfer disk to a transfer package, but have changed your mind about shipping that disk with the transfer package. You can also detach a transfer disk from one transfer package and attach that disk to a different transfer package.

To detach a transfer disk to a transfer package using the Console
To detach a transfer disk to a transfer package using the Data Transfer Utility

Getting the Shipping Address for the Transfer Package

Tip

You can use the Console or the Data Transfer Utility to get the shipping address for a transfer package.

You can find the shipping address in the transfer package details.

To get the shipping address for a transfer package using the Console
To get the shipping address for a transfer package using the Data Transfer Utility

Packaging and Shipping Transfer Disks

General

Include the required return shipping label in the box when packaging transfer disks for shipment.

Note

Return Shipment Label Requirement

If you do not include the return shipping label inside the box, Oracle cannot process the transfer package.

Ensure that the transfer job and transfer package label are clearly readable on the outside of the box containing the transfer disks.

Important

If you are shipping transfer disks to eu-frankfurt-1, you must request that the shipping vendor requires a signature for delivery to ensure that the package is correctly delivered to Oracle.

Shipping Transfer Disks Internationally

You need to create a commercial invoice when shipping transfer disks internationally. To ensure that packages are not held up in customs, follow these guidelines when creating the commercial invoice:

  • Show a unique reference number.
  • Show the "bill-to party as follows:
    • For shipments to the European Union (Frankfurt) location:

      ORACLE Deutschland B.V. & Co. KG

      Riesstrasse 25

      Munich, 80992

      GERMANY

    • For shipments to the United States location:

      Oracle America, Inc.

      500 Oracle Parkway

      REDWOOD CITY CA 94065

      UNITED STATES

  • Show the "ship-to party" as the address provided in the transfer package details. See Getting the Shipping Address for the Transfer Package for details.
  • State that "The value shown includes the value of software and data recorded onto the hard drive unit."
  • State that the "Goods are free of charge - no payment required."
  • State that the type of export is "Temporary."
  • Ensure that the commodity code shows the correct HS code for a hard drive unit as specified in the source country's HS code list.
  • State the description as the manufacture's description of the hard drive unit and include the words "Hard Disk Drive."
  • Ensure the invoice is signed and includes the printed name of the signer.

Updating the Transfer Package With Tracking Information

Tip

You can use the Console or the Data Transfer Utility to update the transfer package with tracking information.

After delivering the transfer package to the shipping vendor, update the transfer package with the tracking information.

To update the transfer package with tracking information using the Console
To update the transfer package with tracking information using the Data Transfer Utility

Checking the Transfer Package Status

Tip

You can use the Console or the Data Transfer Utility to check the status of a transfer package.

When Oracle has processed the transfer disks associated with a transfer package, the status of the transfer package changes to Processed. When Oracle has shipped the transfer disks associated with a transfer package, the status of the transfer package changes to Returned.

To check the status of a transfer package using the Console
To check the status of a transfer package using the Data Transfer Utility

Reviewing the Log Files for Each Transfer Disk

Oracle creates log files for each transfer disk uploaded. The logs are placed in the bucket where the transfer disk data was uploaded. The log file compares the transfer disk's manifest file to the contents of the target Oracle Cloud Infrastructure Object Storage bucket after file upload.

The top of the log report summarizes the overall file processing status:

P - Present: The file is present in both the disk and the target bucket
M - Missing: The file is present in the disk but not the target bucket. It was likely uploaded and then deleted by another user before the summary was generated.
C - Name Collision: The file is present in the manifest but a file with the same name but different contents is present in the target bucket.
U - Unreadable: The file is not readable from the disk
N - Name Too Long: The file name on disk is too long and could not be uploaded

Complete file upload details follow the summary.

This screenshot shows an example of the data transfer log file.

Closing a Transfer Job

Tip

You can use the Console or the Data Transfer Utility to close a transfer job.

Typically, you would close a transfer job when no further transfer job activity is required or possible. Closing a transfer job requires that the status of all associated transfer packages be returned, canceled, or deleted. In addition, the status of all associated transfer disks must be complete, in error, missing, canceled, or deleted.

To close a transfer job using the Console
To close a transfer job using the Data Transfer Utility