Copying Data to the Import Appliance

Copy phase indicator for appliance transfer

This topic describes the tasks associated with copying data from the Data Host to the import appliance using the Control Host. The Data Administrator role typically performs these tasks. See Roles and Responsibilities.

Note

You can only run Oracle Cloud Infrastructure CLI commands from a Linux host. This differs from running CLI commands for other Oracle Cloud Infrastructure Services on a variety of host operating systems. Appliance-based commands require validation that is only available on Linux hosts.

Information Prerequisites

Before performing any import appliance copying tasks, you must obtain the following information:

  • Appliance IP address - typically is provided by the Infrastructure Engineer.
  • IAM login information, Data Transfer Utilityconfiguration files, transfer job ID, and appliance label - typically is provided by the Project Sponsor.

Setting Up an HTTP Proxy Environment

You might need to set up an HTTP proxy environment on the Control Host to allow access to the public internet. This proxy environment allows the Oracle Cloud Infrastructure CLI to communicate with the Data Transfer Appliance Management Service and the import appliance over a local network connection. If your environment requires internet-aware applications to use network proxies, configure the Control Host to use your environment's network proxies by setting the standard Linux environment variables on your Control Host.

Assume that your organization has a corporate internet proxy at http://www-proxy.myorg.com and that the proxy is an HTTP address at port 80. You would set the following environment variable:

export HTTPS_PROXY=http://www-proxy.myorg.com:80

If you configured a proxy on the Control Host and the appliance is directly connected to that host, the Control Host tries unsuccessfully to communicate with the appliance using a proxy. Set a no_proxy environment variable for the appliance. For example, if the appliance is on a local network at 10.0.0.1, you would set the following environment variable:

export NO_PROXY=10.0.0.1

Setting Firewall Access

If you have a restrictive firewall in the environment where you are using the Oracle Cloud Infrastructure CLI, you may need to open your firewall configuration to the following IP address ranges: 140.91.0.0/16.

Initializing Authentication to the Appliance

Note

You can only use the Oracle Cloud Infrastructure CLI to initialize authentication.

Initialize authentication to allow the host machine to communicate with the import appliance. Use the values returned from the Configure Networking command. See Configuring the Transfer Appliance Networking for details.

To initialize authentication using the CLI
To show the status of and storage details about the connected import appliance using the CLI

Configuring Import Appliance Encryption

Configure the import appliance to use encryption. Oracle Cloud Infrastructure creates a strong passphrase for each appliance. The command securely collects the strong passphrase from Oracle Cloud Infrastructure and sends that passphrase to the Data Transfer service.

If your environment requires Internet-aware applications to use network proxies, ensure that you set up the required Linux environment variables. See for more information.

Important

If you are working with multiple appliances at the same time, be sure the job ID and appliance label you specify in this step matches the physical appliance you are currently working with. You can get the serial number associated with the job ID and appliance label using the Console or the Oracle Cloud Infrastructure CLI. You can find the serial number of the physical appliance on the back of the device on the agency label.

Note

You can only use the Oracle Cloud Infrastructure CLI to configure encryption.

To configure import appliance encryption using the CLI

Unlocking the Import Appliance

You must unlock the appliance before you can write data to it. Unlocking the appliance requires the strong passphrase that is created by Oracle Cloud Infrastructure for each appliance. Unlocking can be accomplished in two different ways:

  • If you provide the --job-id and --appliance-label when running the unlock command, the data transfer system retrieves the passphrase from Oracle Cloud Infrastructure and sends it to the appliance during the unlock operation.
  • You can query Oracle Cloud Infrastructure for the passphrase and provide that passphrase when prompted during the unlock operation.

Important

It can take up to 10 minutes to unlock an appliance the first time. Subsequent unlocks are not as time consuming.

To retrieve the passphrase to unlock the import appliance using the CLI
To query Oracle Cloud Infrastructure for the passphrase to provide to unlock the import appliance using the CLI

Creating NFS Datasets

A dataset is a collection of files that are treated similarly. You can write up to 100 million files onto the import appliance for migration to Oracle Cloud Infrastructure. We currently support one dataset per appliance. Appliance-Based Data Import supports NFS versions 3, 4, and 4.1 to write data to the appliance. In preparation for writing data, create and configure a dataset to write to. See Datasets for complete details on all tasks related to datasets.

To create a dataset using the CLI

Configuring Export Settings on the Dataset

To configure export settings on a dataset

Activating the Dataset

Activation creates the NFS export, making the dataset accessible to NFS clients.

To activate the dataset

Setting Your Data Host as an NFS Client

Note

Only Linux machines can be used as Data Hosts.

Set up your Data Host as an NFS client:

  • For Debian or Ubuntu, install the nfs-common package. For example:
    sudo apt-get install nfs-common
  • For Oracle Linux or Red Hat Linux, install the nfs-utils package. For example:
    sudo yum install nfs-utils

Mounting the NFS Share

To mount the NFS share

Copying Files to the NFS Share

Copy your file to the import appliance using normal file system tools.

Important

You can only copy regular files to transfer appliances. Special files (for example, symbolic links, device special, sockets, and pipes) cannot be copied directly. To transfer special files, create a tar archive of these files and copy the tar archive to the transfer appliance.

Deactivating the Dataset

Note

Deactivating the dataset is only required if you are running appliance commands using the Data Transfer Utility. If you are using the Oracle Cloud Infrastructure CLI to run your Appliance-Based Data Import, you can skip this step and proceed to Sealing the Dataset.

After you are done writing data, deactivate the dataset. Deactivation removes the NFS export on the dataset, disallowing any further writes.

To deactivate the dataset

Sealing the Dataset

Sealing a dataset stops all writes to the dataset. This process can take some time to complete, depending upon the number of files and total amount of data copied to the import appliance.

If you issue the seal command without the --wait option, the seal operation is triggered and runs in the background. You are returned to the command prompt and can use the seal-status command to monitor the sealing status. Running the seal command with the --wait option results in the seal operation being triggered and continues to provide status updates until sealing completion.

Important

You can only copy regular files to transfer appliances. Special files (for example, symbolic links, device special, sockets, and pipes) cannot be copied directly. To transfer special files, create a tar archive of these files and copy the tar archive to the transfer appliance.

The sealing operation generates a manifest across all files in the dataset. The manifest contains an index of the copied files and generated data integrity hashes.

To seal the dataset using the CLI

At the command prompt on the host, run oci dts nfs-dataset seal to seal the NFS dataset.

oci dts nfs-dataset seal --name <dataset_name> [--wait]

For example:

oci dts nfs-dataset seal --name nfs-ds-1
Seal initiated. Please use seal-status command to get progress.

To monitor the dataset sealing process using the CLI

At the command prompt on the host, run oci dts nfs-dataset seal-status to monitor the dataset sealing process.

oci dts nfs-dataset seal-status --name <dataset_name>

For example:

oci dts nfs-dataset seal-status --name nfs-ds-1
{
  "data": { 
    "bytesProcessed": 2803515612507, 
    "bytesToProcess": 2803515612507, 
    "completed": true, 
    "endTimeInMs": 1591990408804, 
    "failureReason": null, 
    "numFilesProcessed": 182, 
    "numFilesToProcess": 182, 
    "startTimeInMs": 1591987136180, 
    "success": true 
  }
}

Note

If changes are necessary after sealing a dataset or finalizing an appliance, you must reopen the dataset to modify the contents. See Reopening a Dataset.

Downloading the Dataset Seal Manifest

After sealing the dataset, you can optionally download the dataset's seal manifest to a user-specified location. The manifest file contains the checksum details of all the files. The transfer site uploader consults the manifest file to determine the list of files to upload to object storage. For every uploaded file, it validates that the checksum reported by object storage matches the checksum in manifest. This validation ensures that no files got corrupted in transit.

To download the dataset seal manifest file using the CLI

Finalizing the Import Appliance

Note

You can only use the CLI commands to finalize the import appliance.

Finalizing an appliance tests and copies the following to the appliance:

  • Upload user configuration credentials
  • Private PEM key details
  • Name of the upload bucket

The credentials, API key, and bucket are required for Oracle to be able to upload your data to Oracle Cloud Infrastructure Object Storage. When you finalize an appliance, you can no longer access the appliance for dataset operations unless you unlock the appliance. See Reopening a Dataset if you need to unlock an appliance that was finalized.

Important

If you are working with multiple appliances at the same time, be sure the job ID and appliance label you specify in this step matches the physical appliance you are currently working with. You can get the serial number associated with the job ID and appliance label using the Console or the Oracle Cloud Infrastructure CLI. You can find the serial number of the physical appliance on the back of the device on the agency label.

To finalize the import appliance

What's Next

You are now ready to ship your import appliance with the copied data to Oracle. See Shipping the Import Appliance.