Oracle Cloud Infrastructure Documentation

Data Transfer Utility

This topic describes how to install and configure the Data Transfer Utility. In addition, this topic describes the syntax for the Data Transfer Utility commands.

Prerequisites

To install and use the Data Transfer Utility, you need the following:

  • An Oracle Cloud Infrastructure account.
  • Required Oracle Cloud Infrastructure users and groups with the required IAM policies.

    See Preparing for Data Transfer for details.

  • A host machine with the following installed:

    • Oracle Linux 6 or greater, Ubuntu 14.04 or greater, or SUSE 11 or greater
    • Java 1.8 or greater
    • hdparm 9.0 or later
  • For HDD data transfers, the host machine needs the following:

    • Cryptsetup 1.2.0 or greater
  • For appliance data transfers, the host machine needs the following:

    • Serial console terminal emulator software. We recommend using the following terminal emulator software:
      • PuTTY for Windows
      • ZOC for OS X
      • PuTTY or Minicom for Linux
    • Terminal emulator software settings as follows:

      • Baud Rate: 115200
      • Emulation: VT102
      • Handshaking: Disabled/off
      • RTS/DTS: Disabled/off
    • The host set up as an NFS client:

      • For Debian or Ubuntu, install the nfs-common package.
      • For Oracle Linux or Red Hat Linux, install the nfs-utils package.

You might need to set up an HTTP proxy environment to access to the public Internet for the Data Transfer Utility to communicate with the Data Transfer Appliance Management Service and to the transfer appliance over a local network connection. If your environment requires Internet-aware applications to use network proxies, configure the Data Transfer Utility to use your environment's network proxies by setting the standard Linux environment variables on your host machine.

Assume that your organization has a corporate Internet proxy at http://www-proxy.myorg.com and that the proxy is an HTTP address at port 80. You would set the following environment variable:

export HTTPS_PROXY=http://www-proxy.myorg.com:80

If you configured a proxy on the host machine and the transfer appliance is directly connected to that host, the host machine tries unsuccessfully to communicate with the transfer appliance using a proxy. You must set a no_proxy environment variable for the transfer appliance. For example, if the appliance is on a local network at 10.0.0.1, you would set the following environment variable:

export NO_PROXY=10.0.0.1

Installing the Data Transfer Utility

Download and install the Data Transfer Utility installer that corresponds to your host machine's operating system.

To install the Data Transfer Utility on Debian or Ubuntu
To install the Data Transfer Utility on Oracle Linux or Red Hat Linux

Configuring the Data Transfer Utility

Before using the Data Transfer Utility, you must create a base Oracle Cloud Infrastructure directory and two configuration files with the required credentials. One configuration file is for the data transfer administrator, the IAM user with the authorization and permissions to create and manage transfer jobs. The other configuration file is for the data transfer upload user, the temporary IAM user that Oracle uses to upload your data on your behalf.

Base Data Transfer Directory

Create a base Oracle Cloud Infrastructure directory:

mkdir /root/.oci/

Configuration File for the Data Transfer Administrator

Create a data transfer administrator configuration file /root/.oci/config with the following structure:

[DEFAULT]
user=<The OCID for the data transfer administrator>
fingerprint=<The fingerprint of the above user's public key>
key_file=<The _absolute_ path to the above user's private key file on the host machine>
tenancy=<The OCID for the tenancy that owns the data transfer job and bucket>
region=<The region where the transfer job and bucket should exist. Valid values are: us-ashburn-1, us-phoenix-1, eu-frankfurt-1, and uk-london-1.>

For example:

[DEFAULT]
user=ocid1.user.oc1..<unique_ID>
fingerprint=4c:1a:6f:a1:5b:9e:58:45:f7:53:43:1f:51:0f:d8:45
key_file=/home/user/ocid1.user.oc1..<unique_ID>.pem
tenancy=ocid1.tenancy.oc1..<unique_ID>
region=us-phoenix-1

For the data transfer administrator, you can create a single configuration file that contains different profile sections with the credentials for multiple users. Then use the ‑‑profile option to specify which profile to use in the command. Here is an example of a data transfer administrator configuration file with different profile sections:

[DEFAULT]
user=ocid1.user.oc1..<unique_ID>
fingerprint=4c:1a:6f:a1:5b:9e:58:45:f7:53:43:1f:51:0f:d8:45
key_file=/home/user/ocid1.user.oc1..<unique_ID>.pem
tenancy=ocid1.tenancy.oc1..<unique_ID>
region=us-phoenix-1
[PROFILE1]
user=ocid1.user.oc1..<unique_ID>
fingerprint=4c:1a:6f:a1:5b:9e:58:45:f7:53:43:1f:51:0f:d8:45
key_file=/home/user/ocid1.user.oc1..<unique_ID>.pem
tenancy=ocid1.tenancy.oc1..<unique_ID>
region=us-ashburn-1

Important

Creating an upload user configuration file with multiple profiles is not supported.

By default, the DEFAULT profile is used for all Data Transfer Utility commands. For example:

dts job create --compartment-id <compartment_id> --bucket <bucket_name> --display-name <display_name> --device-type <disk_or_appliance>

Instead, you can issue any Data Transfer Utility command with the --profile option to specify a different data transfer administrator profile. For example:

dts job create --compartment-id <compartment_id> --bucket <bucket_name> --display-name <display_name> --device-type <disk_or_appliance> --profile <profile_name>

Using the example configuration file above, the <profile_name> would be profile1.

Configuration File for the Data Transfer Upload User

Create a data transfer upload user /root/.oci/config_upload_user configuration file with the following structure:

[DEFAULT]
user=<The OCID for the data transfer upload user>
fingerprint=<The fingerprint of the above user's public key>
key_file=<The _absolute_ path to the above user's private key file on the host machine>
tenancy=<The OCID for the tenancy that owns the data transfer job and bucket>
region=<The region where the transfer job and bucket should exist. Valid values are: us-ashburn-1, us-phoenix-1, eu-frankfurt-1, and uk-london-1.>

For example:

[DEFAULT]
user=ocid1.user.oc1..<unique_ID>
fingerprint=4c:1a:6f:a1:5b:9e:58:45:f7:53:43:1f:51:0f:d8:45
key_file=/home/user/ocid1.user.oc1..<unique_ID>.pem
tenancy=ocid1.tenancy.oc1..<unique_ID>
region=us-phoenix-1

Configuration File Entries

The following table lists the basic entries that are required for each configuration file and where to get the information for each entry.

Note

Data Transfer Service Does Not Support Passphrases on Private Keys

While we recommend encrypting a private key with a passphrase when generating API signing keys, Data Transfer service does not support passphrases on the key file required for the config_upload_user. If you use a passphrase, Oracle personnel cannot upload your data.

Entry Description and Where to Get the Value Required?
user

OCID of the data transfer administrator or the data transfer upload user, depending on which profile you are creating. To get the value, see Required Keys and OCIDs.

Yes
fingerprint

Fingerprint for the key pair being used. To get the value, see Required Keys and OCIDs.

Yes
key_file

Full path and filename of the private key.

Important: The key pair must be in PEM format. For instructions on generating a key pair in PEM format, see Required Keys and OCIDs.

Yes
tenancy

OCID of your tenancy. To get the value, see Required Keys and OCIDs.

Yes
region

An Oracle Cloud Infrastructure region. See Regions and Availability Domains.

Data transfer is currently supported in us-ashburn-1, us-phoenix-1, eu-frankfurt-1, and uk-london-1.

Yes

You can verify the data transfer upload user credentials using the following command:

dts job verify-upload-user-credentials --bucket <bucket_name>

Configuration File Location

The location of the configuration files is /root/.oci/config.

Using the Data Transfer Utility

This section provides an overview of the syntax for the Data Transfer Utility.

Important

The Data Transfer Utility must be run as the root user.

There are two ways you can specify Data Transfer Utility command options:
--option <value>
Or
--option=<value>

Syntax

The basic Data Transfer Utility syntax is:

dts <resource> <action> <options>

This syntax is applied, as follows:

  • dts is the shortened utility command name
  • job is an example of a <resource>
  • create is an example of an <action>, and
  • other utility strings are <options>.

The following commands to create a transfer job shows a typical Data Transfer Utility construct.

dts job create --compartment-id ocid1.compartment.oc1..<unique_ID> --display-name "mycompany transfer1" ‑‑bucket mybucket --device-type appliance 

Or:

dts job create --compartment-id=ocid1.compartment.oc1..<unique_ID> --display-name="mycompany transfer1" ‑‑bucket=mybucket --device-type=appliance 
Note

In the previous examples, provide a friendly name for the transfer job using the ‑‑display‑name option. Avoid entering confidential information when providing resource names or descriptions.

Getting Help with Commands

You can get help using --help, -h, or -?. For example:

dts --help

Finding Out the Installed Version of the Data Transfer Utility

You can get the installed version of the Data Transfer Utility using --version or -v. For example:

dts --version

What's Next

You are now ready to perform data transfer-related tasks.