Copying Data to the Import Appliance

Copy phase indicator for appliance transfer

This topic describes the tasks associated with copying data from the Data Host to the import appliance using the Control Host. The Data Administrator role typically performs these tasks. See Roles and Responsibilities.

Note

You can only run Oracle Cloud Infrastructure CLI commands from a Linux host. This differs from running CLI commands for other Oracle Cloud Infrastructure Services on a variety of host operating systems. Appliance-based commands require validation that is only available on Linux hosts.

Information Prerequisites

Before performing any import appliance copying tasks, you must obtain the following information:

  • Appliance IP address - typically is provided by the Infrastructure Engineer.
  • IAM login information, Data Transfer Utilityconfiguration files, transfer job ID, and appliance label - typically is provided by the Project Sponsor.

Setting Up an HTTP Proxy Environment

You might need to set up an HTTP proxy environment on the Control Host to allow access to the public internet. This proxy environment allows the Oracle Cloud Infrastructure CLI to communicate with the Data Transfer Appliance Management Service and the import appliance over a local network connection. If your environment requires internet-aware applications to use network proxies, configure the Control Host to use your environment's network proxies by setting the standard Linux environment variables on your Control Host.

Assume that your organization has a corporate internet proxy at http://www-proxy.myorg.com and that the proxy is an HTTP address at port 80. You would set the following environment variable:

export HTTPS_PROXY=http://www-proxy.myorg.com:80

If you configured a proxy on the Control Host and the appliance is directly connected to that host, the Control Host tries unsuccessfully to communicate with the appliance using a proxy. Set a no_proxy environment variable for the appliance. For example, if the appliance is on a local network at 10.0.0.1, you would set the following environment variable:

export NO_PROXY=10.0.0.1

Setting Firewall Access

If you have a restrictive firewall in the environment where you are using the Oracle Cloud Infrastructure CLI, you may need to open your firewall configuration to the following IP address ranges: 140.91.0.0/16.

Initializing Authentication to the Appliance

Note

You can only use the Oracle Cloud Infrastructure CLI to initialize authentication.

Initialize authentication to allow the host machine to communicate with the import appliance. Use the values returned from the Configure Networking command. See Configuring the Transfer Appliance Networking for details.

To initialize authentication using the CLI

Perform this task using the following CLI. There is no Console equivalent.

oci dts physical-appliance initialize-authentication --job-id <job-id> --appliance-cert-fingerprint <fingerprint> --appliance-ip <ip_address> --appliance-label <appliance-label>

For example:

oci dts physical-appliance initialize-authentication --job-id ocid1.datatransferjob.oci1..exampleuniqueID --appliance-cert-fingerprint F7:1B:D0:45:DA:04:0C:07:1E:B2:23:82:E1:CA:1A:E9 --appliance-ip 10.0.0.1 --appliance-label XA8XM27EVH

When prompted, supply the access token and system. For example:

oci dts physical-appliance initialize-authentication --appliance-certfingerprint 86:CA:90:9E:AE:3F:0E:76:E8:B4:E8:41:2F:A4:2C:38 --applianceip 10.0.0.5 --jobid ocid1.datatransferjob.oc1..exampleuniqueID --appliance-label XAKKJAO9KT

Retrieving the Appliance serial id from Oracle Cloud Infrastructure.
Access token ('q' to quit):
Found an existing appliance. Is it OK to overwrite it? [y/n]y
Registering and initializing the authentication between the dts CLI and the appliance
Appliance Info :
  encryptionConfigured : false
  lockStatus           : NA
  finalizeStatus       : NA
  totalSpace           : Unknown
  availableSpace       : Unknown

The Control Host can now communicate with the import appliance.

To show the status of and storage details about the connected import appliance using the CLI

At the command prompt on the host, run oci dts physical-appliance show to show the status of the connected import appliance.

oci dts physical-appliance show

For example:

Appliance Info :
 encryptionConfigured : false
 lockStatus           : NA
 finalizeStatus       : NA
 totalSpace           : Unknown
 availableSpace       : Unknown

Configuring Import Appliance Encryption

Configure the import appliance to use encryption. Oracle Cloud Infrastructure creates a strong passphrase for each appliance. The command securely collects the strong passphrase from Oracle Cloud Infrastructure and sends that passphrase to the Data Transfer service.

If your environment requires Internet-aware applications to use network proxies, ensure that you set up the required Linux environment variables. See for more information.

Important

If you are working with multiple appliances at the same time, be sure the job ID and appliance label you specify in this step matches the physical appliance you are currently working with. You can get the serial number associated with the job ID and appliance label using the Console or the Oracle Cloud Infrastructure CLI. You can find the serial number of the physical appliance on the back of the device on the agency label.
Note

You can only use the Oracle Cloud Infrastructure CLI to configure encryption.
To configure import appliance encryption using the CLI

At the command prompt on the host, run oci dts physical-appliance configure-encryption to configure import appliance encryption.

oci dts physical-appliance configure-encryption --job-id <job_id> --appliance-label <appliance_label>

For example:

oci dts physical-appliance configure-encryption --job-id ocid1.datatransferjob.region1.phx..exampleuniqueID --appliance-label XA8XM27EVH
				
Moving the state of the appliance to preparing...
Passphrase being retrieved...
Configuring encryption...
Encryption configured. Getting physical transfer appliance info...
{
  "data": {
    "availableSpaceInBytes": "Unknown", 
    "encryptionConfigured": true, 
    "finalizeStatus": "NA", 
    "lockStatus": "LOCKED", 
    "totalSpaceInBytes": "Unknown"
  }
}

Unlocking the Import Appliance

You must unlock the appliance before you can write data to it. Unlocking the appliance requires the strong passphrase that is created by Oracle Cloud Infrastructure for each appliance. Unlocking can be accomplished in two different ways:

  • If you provide the --job-id and --appliance-label when running the unlock command, the data transfer system retrieves the passphrase from Oracle Cloud Infrastructure and sends it to the appliance during the unlock operation.
  • You can query Oracle Cloud Infrastructure for the passphrase and provide that passphrase when prompted during the unlock operation.
Important

It can take up to 10 minutes to unlock an appliance the first time. Subsequent unlocks are not as time consuming.
Note

You can only use the Oracle Cloud Infrastructure CLI to unlick the import appliance.
To unlock the appliance and send the passphrase to the appliance using the CLI
oci dts physical-appliance unlock --job-id <job_id> --appliance-label <label>

For example:

oci dts physical-appliance unlock --job-id ocid1.datatransferjob.oc1..exampleuniqueID --appliance-label XAKWEGKZ5T				
Retrieving the passphrase from Oracle Cloud Infrastructure
{
  "data": {
    "availableSpaceInBytes": "64.00GB", 
    "encryptionConfigured": true, 
    "finalizeStatus": "NOT_FINALIZED", 
    "lockStatus": "NOT_LOCKED", 
    "totalSpaceInBytes": "64.00GB"
  }
}
To query Oracle Cloud Infrastructure for the passphrase to provide to unlock the import appliance using the CLI
oci dts appliance get-passphrase --job-id <job_id> --appliance-label <label>

For example:

oci dts appliance get-passphrase --job-id ocid1.datatransferjob.oc1..exampleuniqueID --appliance-label XAKWEGKZ5T
				
{
  "data": {
    "encryption-passphrase": "passphrase"
  }
}

Run dts physical-appliance unlock without --job-id and --appliance-label and supply the passphrase when prompted to complete the task:

oci dts physical-appliance unlock

Creating NFS Datasets

A dataset is a collection of files that are treated similarly. You can write up to 100 million files onto the import appliance for migration to Oracle Cloud Infrastructure. We currently support one dataset per appliance. Appliance-Based Data Import supports NFS versions 3, 4, and 4.1 to write data to the appliance. In preparation for writing data, create and configure a dataset to write to. See Datasets for complete details on all tasks related to datasets.

To create a dataset using the CLI
oci dts nfs-dataset create --name <dataset_name>

For example:

oci dts nfs-dataset create --name nfs-ds-1 

Creating dataset with NFS export details nfs-ds-1				
{
  "data": {
    "datasetType": "NFS", 
    "name": "nfs-ds-1", 
    "nfsExportDetails": {
      "exportConfigs": null
    }, 
  "state": "INITIALIZED"
  }
}

Configuring Export Settings on the Dataset

To configure export settings on a dataset
oci dts nfs-dataset set-export --name <dataset_name> --rw true --world true

For example:

oci dts nfs-dataset set-export --name nfs-ds-1 --rw true --world true

Settings NFS exports to dataset nfs-ds-1				
{
  "data": {
    "datasetType": "NFS", 
    "name": "nfs-ds-1", 
    "nfsExportDetails": {
      "exportConfigs": [
        {
          "hostname": null, 
          "ipAddress": null, 
          "readWrite": true, 
          "subnetMaskLength": null, 
          "world": true
        }
      ]
    }, 
    "state": "INITIALIZED"
  }
}

Here is another example of creating the export to give read/write access to a subnet:

oci dts nfs-dataset set-export --name nfs-ds-1 --ip 10.0.0.0 --subnet-mask-length 24 --rw true --world false
				
Settings NFS exports to dataset nfs-ds-1			
{
  "data": {
    "datasetType": "NFS", 
    "name": "nfs-ds-1", 
    "nfsExportDetails": {
      "exportConfigs": [
        {
          "hostname": null, 
          "ipAddress": "10.0.0.0", 
          "readWrite": true, 
          "subnetMaskLength": "24", 
          "world": false
        }
      ]
    }, 
    "state": "INITIALIZED"
  }
}

Activating the Dataset

Activation creates the NFS export, making the dataset accessible to NFS clients.

To activate the dataset
oci dts nfs-dataset activate --name <dataset_name>

For example:

oci dts nfs-dataset activate --name nfs-ds-1
					
Fetching all the datasets
Activating dataset small-files
Dataset nfs-ds-1 activated

Setting Your Data Host as an NFS Client

Note

Only Linux machines can be used as Data Hosts.

Set up your Data Host as an NFS client:

  • For Debian or Ubuntu, install the nfs-common package. For example:
    sudo apt-get install nfs-common
  • For Oracle Linux or Red Hat Linux, install the nfs-utils package. For example:
    sudo yum install nfs-utils

Mounting the NFS Share

To mount the NFS share

At the command prompt on the Data Host, create the mountpoint directory:

mkdir -p /mnt/<mountpoint>

For example:

mkdir -p /mnt/nfs-ds-1

Next, use the mount command to mount the NFS share.

mount -t nfs <appliance_ip>:/data/<dataset_name> <mountpoint>

For example:

mount -t nfs 10.0.0.1:/data/nfs-ds-1 /mnt/nfs-ds-1
Note

The appliance IP address in this example (10.0.0.1) may be different that the one you use for your appliance.

After the NFS share is mounted, you can write data to the share.

Copying Files to the NFS Share

You can only copy regular files to transfer appliances. You cannot copy special files, such as symbolic links, device special, sockets, and pipes, directly to the Data Transfer Appliance.

To transfer special files, create a tar archive of these files and copy the tar archive to the Data Transfer Appliance. We recommend copying many small files using a tar archive. Copying a single compressed archive file should also take less time than running copy commands such as cp -r or rsync.

Here are some examples of creating a tar archive and getting it onto the Data Transfer Appliance:

  • Running a simple tar command:

    tar -cvzf /mnt/nfs-dts-1/filesystem.tgz filesystem/
  • Running a command to create a file with md5sum hashes for each file in addition to the tar archive:

    tar cvzf /mnt/nfs-dts-1/filesystem.tgz filesystem/ |xargs -I '{}' sh -c "test -f '{}' && md5sum '{}'"|tee tarzip_md5

    The tar archive file filesystem.tgz has a base64 md5sum once it is uploaded to OCI Object Storage. Store the tarzip_md5 file where you can retrieve it. After the compressed tar archive file is downloaded from Object Storage and unpacked, you can compare the individual files against the hashes in the file.

Deactivating the Dataset

Note

Deactivating the dataset is only required if you are running appliance commands using the Data Transfer Utility. If you are using the Oracle Cloud Infrastructure CLI to run your Appliance-Based Data Import, you can skip this step and proceed to Sealing the Dataset.

After you are done writing data, deactivate the dataset. Deactivation removes the NFS export on the dataset, disallowing any further writes.

To deactivate the dataset

At the command prompt on the host, run dts nfs-dataset deactivate to deactivate the NFS dataset.

oci dts nfs-dataset deactivate --name <dataset_name>

For example:

oci dts nfs-dataset deactivate --name nfs-ds-1

Sealing the Dataset

Sealing a dataset stops all writes to the dataset. This process can take some time to complete, depending upon the number of files and total amount of data copied to the import appliance.

If you issue the seal command without the --wait option, the seal operation is triggered and runs in the background. You are returned to the command prompt and can use the seal-status command to monitor the sealing status. Running the seal command with the --wait option results in the seal operation being triggered and continues to provide status updates until sealing completion.

Important

You can only copy regular files to transfer appliances. Special files (for example, symbolic links, device special, sockets, and pipes) cannot be copied directly. To transfer special files, create a tar archive of these files and copy the tar archive to the transfer appliance.

The sealing operation generates a manifest across all files in the dataset. The manifest contains an index of the copied files and generated data integrity hashes.

To seal the dataset using the CLI

oci dts nfs-dataset seal --name <dataset_name> [--wait]

For example:

oci dts nfs-dataset seal --name nfs-ds-1

Seal initiated. Please use seal-status command to get progress.

To monitor the dataset sealing process using the CLI

oci dts nfs-dataset seal-status --name <dataset_name>

For example:

oci dts nfs-dataset seal-status --name nfs-ds-1
					
{
  "data": { 
    "bytesProcessed": 2803515612507, 
    "bytesToProcess": 2803515612507, 
    "completed": true, 
    "endTimeInMs": 1591990408804, 
    "failureReason": null, 
    "numFilesProcessed": 182, 
    "numFilesToProcess": 182, 
    "startTimeInMs": 1591987136180, 
    "success": true 
  }
}
Note

If changes are necessary after sealing a dataset or finalizing an appliance, you must reopen the dataset to modify the contents. See Reopening a Dataset.

Downloading the Dataset Seal Manifest

After sealing the dataset, you can optionally download the dataset's seal manifest to a user-specified location. The manifest file contains the checksum details of all the files. The transfer site uploader consults the manifest file to determine the list of files to upload to object storage. For every uploaded file, it validates that the checksum reported by object storage matches the checksum in manifest. This validation ensures that no files got corrupted in transit.

To download the dataset seal manifest file using the CLI
oci dts nfs-dataset get-seal-manifest --name <dataset_name> --output-file <file_path>

For example:

oci dts nfs-dataset get-seal-manifest --name nfs-ds-1 --output-file ~/Downloads/seal-manifest

Finalizing the Import Appliance

Note

You can only use the CLI commands to finalize the import appliance.

Finalizing an appliance tests and copies the following to the appliance:

  • Upload user configuration credentials
  • Private PEM key details
  • Name of the upload bucket

The credentials, API key, and bucket are required for Oracle to be able to upload your data to Oracle Cloud Infrastructure Object Storage. When you finalize an appliance, you can no longer access the appliance for dataset operations unless you unlock the appliance. See Reopening a Dataset if you need to unlock an appliance that was finalized.

Important

If you are working with multiple appliances at the same time, be sure the job ID and appliance label you specify in this step matches the physical appliance you are currently working with. You can get the serial number associated with the job ID and appliance label using the Console or the Oracle Cloud Infrastructure CLI. You can find the serial number of the physical appliance on the back of the device on the agency label.
To finalize the import appliance
  1. Seal the dataset before finalizing the import appliance. See Sealing the Dataset.
  2. Open a command prompt on the host and run oci dts physical-appliance finalize to finalize an appliance.

    oci dts physical-appliance finalize --job-id <job_id> --appliance-label <appliance_label>

    For example:

    oci dts physical-appliance finalize --job-id ocid1.datatransferjob.region1.phx..exampleuniqueID --appliance-label XAKWEGKZ5T						
    Retrieving the upload summary object name from Oracle Cloud Infrastructure
    Retrieving the upload bucket name from Oracle Cloud Infrastructure
    Validating the upload user credentials
    Create object BulkDataTransferTestObject in bucket MyBucket using upload user
    Overwrite object BulkDataTransferTestObject in bucket MyBucket using upload user
    Inspect object BulkDataTransferTestObject in bucket MyBucket using upload user
    Read bucket metadata MyBucket using upload user
    Storing the upload user configuration and credentials on the transfer appliance
    Finalizing the transfer appliance...
    The transfer appliance is locked after finalize. Hence the finalize status will be shown as NA. Please unlock the transfer appliance again to see the correct finalize status
    Changing the state of the transfer appliance to FINALIZED
    {
      "data": {
        "availableSpaceInBytes": "Unknown",
        "encryptionConfigured": true,
        "finalizeStatus": "NA",
        "lockStatus": "LOCKED",
        "totalSpaceInBytes": "Unknown"
      }
    }
Note

If changes are necessary after sealing a dataset or finalizing an appliance, you must reopen the dataset to modify the contents. See Reopening a Dataset.