Oracle Cloud Infrastructure Documentation

Managing Appliance Data Transfers

This topic describes in detail all the tasks required to transfer data to Oracle Cloud Infrastructure using the Data Transfer Appliance service.

In preparation for data migration, an IAM user that has administrative authorization and permissions must perform the prerequisite tasks described in Preparing for Data Transfer.

Preparing Your Host Machine

The Data Transfer Utility is a command-line utility that you install on a computer (host machine) at your site. Many data transfer tasks can be performed either using the Console or using the Data Transfer Utility. However, some data transfer tasks can only be performed using the Data Transfer Utility. Each topic in this section guides you to the appropriate management interface to use for each task.

The host machine also requires terminal emulator software to access the transfer appliance serial console.

Step 1: Download and install the Data Transfer Utility on the host computer

Download and install the Data Transfer Utility on your host computer. See Installing and Configuring the Data Transfer Utility for detailed installation and configuration instructions.

Important

The Data Transfer Utility must be run as the root user.

Step 2: Configure the Data Transfer Utility to Use the Required Local Utilities

To export the paths for the required local utilities

Step 3: Install and configure terminal emulator software on the host computer

Terminal emulator software provides access to the transfer appliance serial console. We recommend using the following terminal emulator software:

  • PuTTY for Windows
  • ZOC for OS X
  • PuTTY or Minicom for Linux

Configure the terminal emulator software settings as follows:

  • Baud Rate: 115200
  • Emulation: VT102
  • Handshaking: Disabled/off
  • RTS/DTS: Disabled/off

Creating a Transfer Job

Tip

You can use the Console or the Data Transfer Utility to create a transfer job.

A transfer job represents the collection of files that you want to transfer and signals the intention to upload those files to Oracle Cloud Infrastructure. Identify which compartment and Object Storage bucket that Oracle is to upload your data to. Create the transfer job in the same compartment as the upload bucket and supply a friendly name for the transfer job. Avoid entering confidential information when providing transfer job names.

Creating a transfer job returns a job ID that you specify in other transfer tasks. For example:

ocid1.datatransferjob.region1.phx..<unique_ID>
To create a transfer job using the Console
To create a transfer job using the Data Transfer Utility

Performing Other Transfer Job Tasks

Tip

You can use the Console or the Data Transfer Utility to perform other transfer job-related tasks.

Displaying the List of Transfer Jobs

To display the list of transfer jobs using the Console
To display the list of transfer jobs using the Data Transfer Utility

Displaying the Details of a Transfer Job

To display the details of a transfer job using the Console
To display the details of a transfer job using the Data Transfer Utility

Editing the name of a Transfer Job

To edit the name of a transfer job using the Console
To edit the name of a transfer job using the Data Transfer Utility

Deleting a Transfer Job

You can only delete a transfer job early in the transfer process. For example, you initiated the data transfer by creating a transfer job, but changed your mind. If you want to delete a transfer job after you requested transfer appliances, you must first delete those appliance requests.

To delete a transfer job using the Console
To delete a transfer job using the Data Transfer Utility

Requesting a Transfer Appliance

Tip

You can use the Console or the Data Transfer Utility to request a transfer appliance.

Oracle Cloud Infrastructure customers can use data transfer appliances to migrate data for free. You are only charged for Object Storage usage once the data is successfully transferred to your designated bucket. All appliance requests still require approval from Oracle.

Tip

We recommend that you identify the data you intend to upload and make data copy preparations before requesting the transfer appliance.

Creating a transfer appliance request returns an Oracle-assigned appliance label. For example:

XA8XM27EVH
To request a transfer appliance using the Console
To request a transfer appliance using the Data Transfer Utility

When you submit an appliance request, Oracle generates a unique label (name) to identify the transfer appliance and your request is sent to Oracle for approval and processing.

Monitoring the Status of Your Request for a Transfer Appliance

The time it takes to approve, prepare, and ship your transfer appliance request varies and depends on various factors, including current available inventory. Oracle provides status updates daily throughout the transfer appliance request and ship process.

To monitor the status of your transfer appliance request using the Console
To monitor the status of your transfer appliance request using the Data Transfer Utility

See Data Transfer Appliance Statuses for an explanation of all transfer appliance statuses.

Performing Other Transfer Appliance Tasks

Tip

You can use the Console or the Data Transfer Utility to perform other transfer appliance-related tasks.

Displaying the Details of a Transfer Appliance

To display the details of a transfer appliance using the Console
To display the details of a transfer appliance using the Data Transfer Utility

Editing the Appliance Request Shipping Information

You can only edit the shipping information when the status is Requested.

To edit the appliance request shipping information using the Console
To edit the appliance request shipping information using the Data Transfer Utility

Deleting a Transfer Appliance Request

You can delete a transfer appliance request before Oracle approves the request—the status must be Requested. For example, you initiated the transfer by creating a transfer job and requested a transfer appliance, but changed your mind.

To delete a transfer appliance request using the Console
To delete a transfer appliance request using the Data Transfer Utility

Unpacking and Preparing Your Transfer Appliance

When the shipping vendor delivers your transfer appliance, Oracle updates the status as Delivered and provides the date and time the appliance was received in the Transfer Appliance Details.

Important

Your transfer appliance arrives in a transit case with a telescoping handle and wheels. The case amenities allow for easy movement to the location where you intend to place the appliance to upload your data.

Retain all packaging materials! When shipping the transfer appliance back to Oracle, you must package the appliance in the same manner and packaging in which the appliance was received.

Here are the tasks involved in unpacking and getting your transfer appliance ready to configure.

  1. Inspect the tamper-evident security tie on the transit case.

    If the appliance was tampered with during transit, the tamper-evident security tie serves to alert you.

    Warning

    If the security tie is damaged or is missing, do not plug the appliance into your network! Immediately file a Service Request (SR).
  2. Remove and compare the number on the security tie with the number logged by Oracle.

    To see the security tie number logged by Oracle using the Console
    To see the security tie number logged by Oracle using the Data Transfer Utility
    Warning

    If the number on the physical security tie does not match the number logged by Oracle, do not plug the appliance into your network! Immediately file a Service Request (SR).
  3. Open the transit case and ensure that the case contains the following items:

    • Appliance unit and power cable (two types of power cables provided: C14 and C13 to 14)
    • USB to DB-9 serial cable
    • Return shipping instructions (retain these instructions)
    • Return shipping label, label sleeve, tie-on tag, and zip tie
    • Return shipment tamper-evident security tie (use this tie to ensure secure transit case back to Oracle)
  4. Compare the number on the return shipment security tie with the number logged by Oracle.

    To see the security tie number logged by Oracle using the Console
    To see the security tie number logged by Oracle using the Data Transfer Utility
    Warning

    If the number on the return security tie does not match the number logged by Oracle, file a Service Request (SR). These security tie numbers must match or Oracle cannot upload data from your returned transfer appliance.
  5. Remove the transfer appliance from the case and place the appliance on a solid surface or in a rack.

    Warning

    We recommend assistance lifting the transfer appliance out of the transit case and placing the appliance in a rack or on a desk top. The total shipping weight is about 64 lbs (29.0299 kg) and appliance weight is 38 lbs (17.2365 kg).
  6. Connect the appliance to your local network using one of the following:

    • 10GBase-T: Standard RJ-45
    • SFP+: The transceiver must be compatible with Intel X520 NICs.
  7. Attach one of the provided power cords to the appliance and plug the other end into a grounded power source.
  8. Turn on the appliance by flipping the power switch on the back of the appliance.
  9. Connect the appliance to your host computer using the provided USB to DB-9 serial cable.

    Note

    You might need to download the driver for this cable on your host computer:
    https://www.cablestogo.com/product/26887/5ft-usb-to-db9-male-serial-rs232-adapter-cable#support

Configuring Your Transfer Appliance Networking

When the appliance boots up, an appliance serial console configuration menu is displayed on the terminal emulator on the host machine. It can take up to 5 minutes for the serial console menu to display.

The transfer appliance supports a single active network interface on any of the 10-Gbps network ports. If only one interface is cabled and active, that interface is chosen automatically. If multiple interfaces are active, you are given the choice to select the interface to use.

To configure your transfer appliance networking

When you configure a network interface, the appliance software generates a new client access token and appliance X.509/SSL certificate. The access token is used to authorize your host machine to communicate with the Data Transfer Appliance's Management Service. The x.509/SSL certificate is used to encrypt communications with the Data Transfer Appliance's Management Service over the network. You will need provide the access token and SSL certificate fingerprint values displayed here when you use the Data Transfer Utility to initialize authentication on your host machine.

You can change the selected interface, network information, and reset the authentication material at any time by selecting Configure Networking again from the appliance serial console menu.

Setting Up Your Host Machine

This topic describes the tasks required to set up your host machine to communicate with the transfer appliance using the Data Transfer Appliance Management Service. This Management Service controls all functional aspects of the transfer appliance.

Tip

You can only use the Data Transfer Utility to set up your host machine.

Step 1: Initializing Authentication

Initialize authentication to allow the host machine to communicate with the data transfer appliance. Use the values returned from the Configure Networking command. See Configuring Your Transfer Appliance Networking for details.

To initialize authentication

The host machine can now communicate with the transfer appliance.

To show the status of and storage details about the connected appliance

Step 2: Configuring Appliance Encryption

You must configure the appliance to use encryption. Oracle Cloud Infrastructure creates a strong passphrase for each appliance. The Data Transfer Utility securely collects the strong passphrase from Oracle Cloud Infrastructure and sends that passphrase to the Data Transfer service.

If your environment requires Internet-aware applications to use network proxies, ensure that you set up the required Linux environment variables. See the HTTP proxy information in the Data Transfer Utility topic.

Important

If you are working with multiple appliances at the same time, be sure the job ID and appliance label you specify in this step matches the physical appliance you are currently working with. You can get the serial number associated with the job ID and appliance label using the Data Transfer Utility or the Console. You can find the serial number of the physical appliance on the back of the device on the agency label.

To configure transfer appliance encryption

Step 3: Unlocking the Transfer Appliance

Before you can write data to the transfer appliance, you must unlock the appliance. Unlocking the transfer appliance requires the strong passphrase that is created by Oracle Cloud Infrastructure for each appliance. Unlocking can be accomplished in two different ways:

  • If you provide the --job-id and --appliance-label when running the unlock command, the Data Transfer Utility retrieves the passphrase from Oracle Cloud Infrastructure and sends it to the transfer appliance during the unlock operation.
  • You can query Oracle Cloud Infrastructure for the passphrase and provide that passphrase when prompted during the unlock operation.

Important

It can take up to 10 minutes to unlock a transfer appliance the first time. Subsequent unlocks are not as time consuming.

To have the Data Transfer Utility retrieve the passphrase to unlock the transfer appliance
To query Oracle Cloud Infrastructure for the passphrase to provide to unlock the transfer appliance

Writing Data to the Transfer Appliance

Appliance data transfer supports NFS-v3 to write data to the appliance. In preparation for writing data, create and configure a dataset to write to. A dataset is a collection of files that are treated similarly. You can write up to 100 million files onto the appliance for transfer. We currently support one dataset per transfer appliance.

Important

Files written to the appliance must be world readable and the directories must be world readable and world executable. If files and directories do not match this criteria, you will not be able to seal the appliance.

You can only copy regular files to transfer appliances. Special files (for example, links, sockets, and pipes) cannot be copied directly. To transfer special files, create a tar archive of these files and copy the tar archive to the transfer appliance.

Tip

You can only use the Data Transfer Utility to create, configure, and allow access to a dataset over NFS.

Step 1: Creating an NFS dataset

To create an NFS dataset

Step 2: Configuring export settings on the dataset

To configure export settings on an NFS dataset

Step 3: Activating the dataset

Activation creates the NFS export, making the dataset accessible to NFS clients.

To activate the NFS dataset

Step 4: Confirming that the NFS share is visible

To confirm that the NFS share is visible

Step 5: Mounting the NFS share

To mount the NFS share

After the NFS share is mounted, you can write data to the share.

Step 6: Writing data to the share

Copy your data to the transfer appliance using normal file system tools.

Important

Files written to the appliance must be world readable and the directories must be world readable and world executable. If files and directories do not match this criteria, you will not be able to seal the appliance.

Step 7: Deactivating the dataset

After you are done writing data, deactivate the dataset. Deactivation removes the NFS export on the dataset, disallowing any further writes.

To deactivate the NFS dataset

Step 8: Sealing the dataset

Sealing a dataset stops all writes to the dataset. Sealing a dataset is a long running process that can take some time to complete. The completion time depends upon the number of files and total amount of data that was copied to the transfer appliance.

If you issue the seal command without the --wait option, the seal operation is triggered and runs in the background. You are returned to the command prompt and can use the seal-status command to monitor the sealing status. If you issue the seal command with the --wait option, the seal operation is triggered and continues to provide status updates until sealing completion.

Important

Files written to the appliance must be world readable and the directories must be world readable and world executable. If files and directories do not match this criteria, you will not be able to seal the appliance.

The sealing operation generates a manifest across all files in the dataset. The manifest contains an index of all of the copied files and generated data integrity hashes.

If the seal process fails with reasons like Failed to walk filesystem - java.nio.file.AccessDeniedException: /data/la-rc-ds/dir1/dir2 or Number of files with no read perms : 1, reactivate the dataset and correct permissions for the files and directories.

To seal the NFS dataset
To monitor the dataset sealing process

Finalizing the Transfer Appliance

Tip

You can only use the Data Transfer Utility to finalize the transfer appliance.

Finalizing an appliance tests and copies the following to the transfer appliance:

  • Upload user configuration credentials
  • Private PEM key details
  • Name of the upload bucket

The credentials, certificate, and bucket are required for Oracle to be able to upload your data to Oracle Cloud Infrastructure Object Storage. When you finalize a transfer appliance, you can no longer access the appliance for dataset operations unless you unlock the appliance. See Reopening a Dataset if you need to unlock an appliance that was finalized.

Important

If you are working with multiple appliances at the same time, be sure the job ID and appliance label you specify in this step matches the physical appliance you are currently working with. You can get the serial number associated with the job ID and appliance label using the Data Transfer Utility or the Console. You can find the serial number of the physical appliance on the back of the device on the agency label.

To finalize the appliance

Reopening a Dataset

Tip

You can only use the Data Transfer Utility to reopen a dataset.

If changes are necessary after sealing a dataset or finalizing an appliance, you must reopen the dataset to modify the contents. Make the required changes and again seal the dataset. Resealing the dataset generates a new manifest.

Note

If an appliance is rebooted or power cycled, follow the instructions in this topic to reopen the dataset.

Step 1: Unlocking the Transfer Appliance

Before you can write data to the transfer appliance, you must unlock the appliance. Unlocking the transfer appliance requires the strong passphrase that is created by Oracle Cloud Infrastructure for each appliance. Unlocking can be accomplished in two different ways:

  • If you provide the --job-id and --appliance-label when running the unlock command, the Data Transfer Utility retrieves the passphrase from Oracle Cloud Infrastructure and sends it to the transfer appliance during the unlock operation.
  • You can query Oracle Cloud Infrastructure for the passphrase and provide that passphrase when prompted during the unlock operation.
To have the Data Transfer Utility retrieve the passphrase to unlock the transfer appliance
To query Oracle Cloud Infrastructure for the passphrase to provide to unlock the transfer appliance

Step 2: Reopening the Transfer Appliance

Reopen the dataset to write data to the transfer appliance again.

To reopen an NFS dataset

Step 3: Repeat Steps to Write Data to the Appliance

Repeat the same tasks you performed when you originally wrote data to the appliance beginning with activating the dataset in the Writing Data to the Transfer Appliance section.

Shutting Down the Transfer Appliance

Shut down the appliance before packing up and shipping the appliance back to Oracle.

To shut down the appliance

Packing and Shipping Transfer Appliance to Oracle

Return the appliance to Oracle within 30 days. If you need the transfer appliance beyond the standard 30-day window, you can file a Service Request (SR) to ask for an extension of up to 60 days.

Important

Review and follow the instructions that were provided in the transit case with the appliance.

To pack and ship the appliance

Canceling a Transfer Appliance

If you change your mind about uploading your data to Oracle Cloud Infrastructure, you must cancel the transfer appliance. You can only cancel a transfer appliance after you ship the appliance back to Oracle. You can cancel one transfer appliance, while allowing the file upload from other appliances.

Oracle does not process canceled transfer appliances. Oracle retains the canceled transfer appliance and wipes all the data from the device.

To cancel a transfer device using the Data Transfer Utility

Monitoring the Status of Your Transfer Appliance Return Shipment

The shipping vendor notifies Oracle when your transfer appliance is picked up and shipped back for upload to Oracle Cloud Infrastructure Object Storage.

To monitor the status of your transfer appliance return shipment using the Console

Reviewing the Transfer Appliance Log Files

Oracle creates transfer log files for each transfer appliance uploaded. The logs are placed in the bucket where the transfer device data was uploaded. The log file compares the transfer appliance's manifest file to the contents of the target Oracle Cloud Infrastructure Object Storage bucket after file upload.

Note

If you chose to upload your data to an Archive Storage bucket, you must first restore the log file object before you can download that file for review.

The top of the log report summarizes the overall file processing status:

P - Present: The file is present in both the device and the target bucket
M - Missing: The file is present in the device but not the target bucket. It was likely uploaded and then deleted by another user before the summary was generated.
C - Name Collision: The file is present in the manifest but a file with the same name but different contents is present in the target bucket.
U - Unreadable: The file is not readable from the disk
N - Name Too Long: The file name on disk is too long and could not be uploaded

Complete file upload details follow the summary.

This screenshot shows an example of the data transfer log file.

If you upload more than 100,000 files, the upload details are broken into multiple pages. You can only download the first page from the Console. Download the rest of the pages directly from the Object Storage bucket. The subsequent pages have the same object name as the first page, but have an enumerated suffix.

Closing a Transfer Job

Tip

You can use the Console or the Data Transfer Utility to close a transfer job.

Typically, you would close a transfer job when no further transfer job activity is required or possible. Closing a transfer job requires that the status of all associated transfer appliances be returned, canceled, or deleted.

To close a transfer job using the Console
To close a transfer job using the Data Transfer Utility