Oracle Cloud Infrastructure Documentation

Copying Data to the Appliance

Copy phase indicator for appliance transfer

This topic describes the tasks associated with copying data from the Data Host to the appliance using the Control Host. The Data Administrator role typically performs these tasks. See Roles and Responsibilities.

Note

You can only run Oracle Cloud Infrastructure CLI commands from a Linux host. This differs from running CLI commands for other Oracle Cloud Infrastructure Services on a variety of host operating systems. Appliance-based commands require validation that is only available on Linux hosts.

Information Prerequisites

Before performing any disk copying tasks, you must obtain the following information:

  • Appliance IP address - typically is provided by the Infrastructure Engineer.
  • IAM login information, Data Transfer Utilityconfiguration files, transfer job ID, and job label - typically is provided by the Project Sponsor.

Generate and Upload RSA Key Pairs

The Data Administrator is responsible for generating and uploading the RSA key pairs. Do not share these generated RSA keys between users. See Creating a Key Pair.

Setting Up an HTTP Proxy Environment

You might need to set up an HTTP proxy environment on the Control Host to allow access to the public internet. This proxy environment allows the Oracle Cloud Infrastructure CLI to communicate with the Data Transfer Appliance Management Service and the appliance over a local network connection. If your environment requires internet-aware applications to use network proxies, configure the Control Host to use your environment's network proxies by setting the standard Linux environment variables on your Control Host.

Assume that your organization has a corporate internet proxy at http://www-proxy.myorg.com and that the proxy is an HTTP address at port 80. You would set the following environment variable:

export HTTPS_PROXY=http://www-proxy.myorg.com:80

 

If you configured a proxy on the Control Host and the transfer appliance is directly connected to that host, the Control Host tries unsuccessfully to communicate with the transfer appliance using a proxy. Set a no_proxy environment variable for the appliance. For example, if the appliance is on a local network at 10.0.0.1, you would set the following environment variable:

export NO_PROXY=10.0.0.1

Install and Using the Oracle Cloud Infrastructure Command Line Interface

The Oracle Cloud Infrastructure Command Line Interface (CLI) provides a set of command line-based tools for configuring and running Appliance-Based Data Transfer. Use the Oracle Cloud Infrastructure CLI as an alternative to running commands from the Console. Sometimes you must use the CLI to complete certain tasks as there is no Console equivalent.

Installation and configuration of the CLIs is described in detail in Command Line Interface (CLI).

Using the CLI

The CLI must be run as the root user.

You can specify CLI options using the following commands:

  • --option <value>or
  • --option=<value>

The basic CLI syntax is:

oci dts <resource> <action> <options>

This syntax is applied to the following:

  • oci dts is the shortened CLI command name
  • job is an example of a <resource>
  • create is an example of an <action>
  • Other strings are <options>

The following commands to create a transfer job shows a typical CLI command construct.

oci dts job create --compartment-id ocid1.compartment.oc1..<unique_ID> --display-name "mycompany transfer1" ‑‑bucket mybucket --device-type appliance 

or

oci dts job create --compartment-id=ocid1.compartment.oc1..<unique_ID> --display-name="mycompany transfer1" ‑‑bucket=mybucket --device-type=appliance 
Note

In the previous examples, provide a friendly name for the transfer job using the ‑‑display‑name option. Avoid entering

Firewall Access

If you have a restrictive firewall in the environment where you are using the Oracle Cloud Infrastructure CLI , you may need to open your firewall configuration to the following IP address ranges: 140.91.0.0/16.

Initializing Authentication to the Appliance

Tip

You can only use the Oracle Cloud Infrastructure CLI to initialize authentication.

Initialize authentication to allow the host machine to communicate with the appliance. Use the values returned from the Configure Networking command. See Configuring the Transfer Appliance Networking for details.

To initialize authentication using the CLI

The Control Host can now communicate with the appliance.

To show the status of and storage details about the connected appliance using the CLI

Configuring Appliance Encryption

Configure the appliance to use encryption. Oracle Cloud Infrastructure creates a strong passphrase for each appliance. The command securely collects the strong passphrase from Oracle Cloud Infrastructure and sends that passphrase to the Data Transfer service.

If your environment requires Internet-aware applications to use network proxies, ensure that you set up the required Linux environment variables. See for more information.

Important

If you are working with multiple appliances at the same time, be sure the job ID and appliance label you specify in this step matches the physical appliance you are currently working with. You can get the serial number associated with the job ID and appliance label using the Console or the Oracle Cloud Infrastructure CLI. You can find the serial number of the physical appliance on the back of the device on the agency label.

Tip

You can only use the Oracle Cloud Infrastructure CLI to configure encryption.

To configure appliance encryption using the CLI

Unlocking the Appliance

Before you can write data to the transfer appliance, you must unlock the appliance. Unlocking the transfer appliance requires the strong passphrase that is created by Oracle Cloud Infrastructure for each appliance. Unlocking can be accomplished in two different ways:

  • If you provide the --job-id and --appliance-label when running the unlock command, the data transfer system retrieves the passphrase from Oracle Cloud Infrastructure and sends it to the transfer appliance during the unlock operation.
  • You can query Oracle Cloud Infrastructure for the passphrase and provide that passphrase when prompted during the unlock operation.

Important

It can take up to 10 minutes to unlock an appliance the first time. Subsequent unlocks are not as time consuming.

To retrieve the passphrase to unlock the appliance using the CLI
To query Oracle Cloud Infrastructure for the passphrase to provide to unlock the appliance using the CLI

Creating NFS Datasets

A dataset is a collection of files that are treated similarly. You can write up to 100 million files onto the appliance for migration to Oracle Cloud Infrastructure. We currently support one dataset per appliance. Appliance-Based Data Transfer supports NFS versions 3, 4, and 4.1 to write data to the appliance. In preparation for writing data, create and configure a dataset to write to. See Datasets for complete details on all tasks related to datasets.

To create a dataset using the CLI

Activating the Dataset

Activation creates the NFS export, making the dataset accessible to NFS clients.

To activate the dataset

Configuring Export Settings on the Dataset

To configure export settings on a dataset

Setting Your Data Host as an NFS Client

Set up your Data Host as an NFS client:

  • For Debian or Ubuntu, install the nfs-common package. For example:
    sudo apt-get install nfs-common
  • For Oracle Linux or Red Hat Linux, install the nfs-utils package. For example:
    sudo yum install nfs-utils

Mounting the NFS Share

To mount the NFS share

After the NFS share is mounted, you can write data to the share.

Copying Files to the NFS Share

Copy your file to the appliance using normal file system tools.

Important

You can only copy regular files to transfer appliances. Special files (for example, symbolic links, device special, sockets, and pipes) cannot be copied directly. To transfer special files, create a tar archive of these files and copy the tar archive to the transfer appliance.

Deactivating the Dataset

Note

Deactivating the dataset is only required if you are running appliance commands using the Data Transfer Utility. If you are using the Oracle Cloud Infrastructure CLI to run your Appliance-Based Data Transfer, you can skip this step and proceed to Sealing the Dataset.

After you are done writing data, deactivate the dataset. Deactivation removes the NFS export on the dataset, disallowing any further writes.

To deactivate the dataset

Sealing the Dataset

Sealing a dataset stops all writes to the dataset. Sealing a dataset is a long running process that can take some time to complete. The completion time depends upon the number of files and total amount of data that was copied to the appliance.

If you issue the seal command without the --wait option, the seal operation is triggered and runs in the background. You are returned to the command prompt and can use the seal-status command to monitor the sealing status. If you issue the seal command with the --wait option, the seal operation is triggered and continues to provide status updates until sealing completion.

Important

You can only copy regular files to transfer appliances. Special files (for example, symbolic links, device special, sockets, and pipes) cannot be copied directly. To transfer special files, create a tar archive of these files and copy the tar archive to the transfer appliance.

The sealing operation generates a manifest across all files in the dataset. The manifest contains an index of the copied files and generated data integrity hashes.

To seal the dataset using the CLI
To monitor the dataset sealing process using the CLI

Note

If changes are necessary after sealing a dataset or finalizing an appliance, you must reopen the dataset to modify the contents. See Reopening a Dataset.

Downloading the Dataset Seal Manifest

After sealing the dataset, you can optionally download the dataset's seal manifest to a user-specified location. The manifest file contains the checksum details of all the files. The transfer site uploader consults the manifest file to determine the list of files to upload to object storage. For every uploaded file, it validates that the checksum reported by object storage matches the checksum in manifest. This validation ensures that no files got corrupted in transit.

To download the dataset seal manifest file using the CLI

Finalizing the Appliance

Tip

You can only use the CLI commands to finalize the appliance.

Finalizing an appliance tests and copies the following to the appliance:

The credentials, API key, and bucket are required for Oracle to be able to upload your data to Oracle Cloud Infrastructure Object Storage. When you finalize an appliance, you can no longer access the appliance for dataset operations unless you unlock the appliance. See Reopening a Dataset if you need to unlock an appliance that was finalized.

Important

If you are working with multiple appliances at the same time, be sure the job ID and appliance label you specify in this step matches the physical appliance you are currently working with. You can get the serial number associated with the job ID and appliance label using the Console or the Oracle Cloud Infrastructure CLI. You can find the serial number of the physical appliance on the back of the device on the agency label.

To finalize the appliance

Note

If changes are necessary after sealing a dataset or finalizing an appliance, you must reopen the dataset to modify the contents. See Reopening a Dataset.

What's Next

You are now ready to ship your appliance with the copied data to Oracle. See Shipping the Disk.