Oracle Cloud Infrastructure Documentation

Installing and Configuring the Data Transfer Utility

This topic describes how to install and configure the Data Transfer Utility. In addition, this topic describes the syntax for the Data Transfer Utility commands.

Prerequisites

To install and use the Data Transfer Utility for either disk-based or appliance-based transfers, obtain the following:

  • An Oracle Cloud Infrastructure account.

  • Required Oracle Cloud Infrastructure users and groups with the required IAM policies.

    See Preparing for Data Transfer for details.

  • A host machine with the following installed:

    • Oracle Linux 6 or greater, Ubuntu 14.04 or greater, or SUSE 11 or greater

    • Java 1.8 or Java 1.11

    • hdparm 9.0 or later

  • Firewall access: If you have a restrictive firewall in the environment where you are using the Data Transfer Utility, you may need to open your firewall configuration to the following IP address ranges: 140.91.0.0/16.

    You also need to open access to the object storage IP address ranges: 134.70.0.0/17.

 

Depending on the type of transfer you are performing, you need the following additional prerequisites.

Disk-Based Transfers

For HDD data transfers, the host machine requires Cryptsetup 1.2.0 or greater.

Appliance-Based Transfers

  • For appliance data transfers, the host machine requires the following:
    • Serial console terminal emulator software. We recommend using the following terminal emulator software:

      • PuTTY for Windows
      • ZOC for OS X
      • PuTTY or Minicom for Linux
    • Terminal emulator software settings as follows:

      • Baud Rate: 115200
      • Emulation: VT102
      • Handshaking: Disabled/off
      • RTS/DTS: Disabled/off
    • The host set up as an NFS client:

      • For Debian or Ubuntu, install the nfs-common package. For example:

        sudo apt-get install nfs-common
      • For Oracle Linux or Red Hat Linux, install the nfs-utils package. For example:

        sudo yum install nfs-utils

    You might need to set up an HTTP proxy environment to allow access to the public internet so the Data Transfer Utility can communicate with the Data Transfer Appliance Management Service and the transfer appliance over a local network connection. If your environment requires internet-aware applications to use network proxies, configure the Data Transfer Utility to use your environment's network proxies by setting the standard Linux environment variables on your host machine.

    Assume that your organization has a corporate internet proxy at http://www-proxy.myorg.com and that the proxy is an HTTP address at port 80. You would set the following environment variable:

    export HTTPS_PROXY=http://www-proxy.myorg.com:80

    If you configured a proxy on the host machine and the transfer appliance is directly connected to that host, the host machine tries unsuccessfully to communicate with the transfer appliance using a proxy. Set a no_proxy environment variable for the transfer appliance. For example, if the appliance is on a local network at 10.0.0.1, you would set the following environment variable:

    export NO_PROXY=10.0.0.1

    Installing the Data Transfer Utility

    Download and install the Data Transfer Utility installer that corresponds to your host machine's operating system.

    To install the Data Transfer Utility on Debian or Ubuntu
    To install the Data Transfer Utility on Oracle Linux or Red Hat Linux

    Configuring the Data Transfer Utility

    Before using the Data Transfer Utility, you must create a base Oracle Cloud Infrastructure directory and two configuration files with the required credentials. One configuration file is for the data transfer administrator, the IAM user with the authorization and permissions to create and manage transfer jobs. The other configuration file is for the data transfer upload user, the temporary IAM user that Oracle uses to upload your data on your behalf.

    Base Data Transfer Directory

    Create a base Oracle Cloud Infrastructure directory:

    mkdir /root/.oci/

    Configuration File for the Data Transfer Administrator

    Create a data transfer administrator configuration file /root/.oci/config with the following structure:

    [DEFAULT]
    user=<The OCID for the data transfer administrator>
    fingerprint=<The fingerprint of the above user's public key>
    key_file=<The _absolute_ path to the above user's private key file on the host machine>
    tenancy=<The OCID for the tenancy that owns the data transfer job and bucket>
    region=<The region where the transfer job and bucket should exist. Valid values are: us-ashburn-1, us-phoenix-1, eu-frankfurt-1, and uk-london-1.>

    For example:

    [DEFAULT]
    user=ocid1.user.oc1..<unique_ID>
    fingerprint=4c:1a:6f:a1:5b:9e:58:45:f7:53:43:1f:51:0f:d8:45
    key_file=/home/user/ocid1.user.oc1..<unique_ID>.pem
    tenancy=ocid1.tenancy.oc1..<unique_ID>
    region=us-phoenix-1

    For the data transfer administrator, you can create a single configuration file that contains different profile sections with the credentials for multiple users. Then use the ‑‑profile option to specify which profile to use in the command. Here is an example of a data transfer administrator configuration file with different profile sections:

    [DEFAULT]
    user=ocid1.user.oc1..<unique_ID>
    fingerprint=4c:1a:6f:a1:5b:9e:58:45:f7:53:43:1f:51:0f:d8:45
    key_file=/home/user/ocid1.user.oc1..<unique_ID>.pem
    tenancy=ocid1.tenancy.oc1..<unique_ID>
    region=us-phoenix-1
    [PROFILE1]
    user=ocid1.user.oc1..<unique_ID>
    fingerprint=4c:1a:6f:a1:5b:9e:58:45:f7:53:43:1f:51:0f:d8:45
    key_file=/home/user/ocid1.user.oc1..<unique_ID>.pem
    tenancy=ocid1.tenancy.oc1..<unique_ID>
    region=us-ashburn-1

    Important

    Creating an upload user configuration file with multiple profiles is not supported.

    By default, the DEFAULT profile is used for all Data Transfer Utility commands. For example:

    dts job create --compartment-id <compartment_id> --bucket <bucket_name> --display-name <display_name> --device-type <disk_or_appliance>

    Instead, you can issue any Data Transfer Utility command with the --profile option to specify a different data transfer administrator profile. For example:

    dts job create --compartment-id <compartment_id> --bucket <bucket_name> --display-name <display_name> --device-type <disk_or_appliance> --profile <profile_name>

    Using the example configuration file above, the <profile_name> would be profile1.

    Configuration File for the Data Transfer Upload User

    Create a data transfer upload user /root/.oci/config_upload_user configuration file with the following structure:

    [DEFAULT]
    user=<The OCID for the data transfer upload user>
    fingerprint=<The fingerprint of the above user's public key>
    key_file=<The _absolute_ path to the above user's private key file on the host machine>
    tenancy=<The OCID for the tenancy that owns the data transfer job and bucket>
    region=<The region where the transfer job and bucket should exist. Valid values are: us-ashburn-1, us-phoenix-1, eu-frankfurt-1, and uk-london-1.>

    For example:

    [DEFAULT]
    user=ocid1.user.oc1..<unique_ID>
    fingerprint=4c:1a:6f:a1:5b:9e:58:45:f7:53:43:1f:51:0f:d8:45
    key_file=/home/user/ocid1.user.oc1..<unique_ID>.pem
    tenancy=ocid1.tenancy.oc1..<unique_ID>
    region=us-phoenix-1

    Configuration File Entries

    The following table lists the basic entries that are required for each configuration file and where to get the information for each entry.

    Note

    Data Transfer Service Does Not Support Passphrases on Private Keys

    While we recommend encrypting a private key with a passphrase when generating API signing keys, Data Transfer service does not support passphrases on the key file required for the config_upload_user. If you use a passphrase, Oracle personnel cannot upload your data.

    Entry Description and Where to Get the Value Required?
    user

    OCID of the data transfer administrator or the data transfer upload user, depending on which profile you are creating. To get the value, see Required Keys and OCIDs.

    Yes
    fingerprint

    Fingerprint for the key pair being used. To get the value, see Required Keys and OCIDs.

    Yes
    key_file

    Full path and filename of the private key.

    Important: The key pair must be in PEM format. For instructions on generating a key pair in PEM format, see Required Keys and OCIDs.

    Yes
    tenancy

    OCID of your tenancy. To get the value, see Required Keys and OCIDs.

    Yes
    region

    An Oracle Cloud Infrastructure region. See Regions and Availability Domains.

    Data transfer is supported in us-ashburn-1, us-phoenix-1, eu-frankfurt-1, and uk-london-1.

    Yes

    You can verify the data transfer upload user credentials using the following command:

    dts job verify-upload-user-credentials --bucket <bucket_name>

    Configuration File Location

    The location of the configuration files is /root/.oci/config.

    Using the Data Transfer Utility

    This section provides an overview of the syntax for the Data Transfer Utility.

    Important

    The Data Transfer Utility must be run as the root user.

    You can specify Data Transfer Utility command options using the following commands:
    --option <value>
    Or
    --option=<value>

    Syntax

    The basic Data Transfer Utility syntax is:

    dts <resource> <action> <options>

    This syntax is applied to the following:

    • dts is the shortened utility command name
    • job is an example of a <resource>
    • create is an example of an <action>
    • Other utility strings are <options>

    The following commands to create a transfer job shows a typical Data Transfer Utility construct.

    dts job create --compartment-id ocid1.compartment.oc1..<unique_ID> --display-name "mycompany transfer1" ‑‑bucket mybucket --device-type appliance 

    Or:

    dts job create --compartment-id=ocid1.compartment.oc1..<unique_ID> --display-name="mycompany transfer1" ‑‑bucket=mybucket --device-type=appliance 
    Note

    In the previous examples, provide a friendly name for the transfer job using the ‑‑display‑name option. Avoid entering confidential information when providing resource names or descriptions.

    Getting Help with Commands

    You can get help with the different commands associated with Data Transfer Utility using dts by itself. For example:

    dts

    Finding Out the Installed Version of the Data Transfer Utility

    You can get the installed version of the Data Transfer Utility using --version or -v. For example:

    dts --version

    What's Next

    You are now ready to perform data transfer-related tasks.