Oracle Cloud Infrastructure Documentation

Installing the Data Transfer Utility for Disk-Based Data Transfers

This topic describes how to install and configure the Data Transfer Utility for use in disk-based data transfers. In addition, this topic describes the syntax for the Data Transfer Utility commands.

Important

With this release, the Data Transfer Utility only supports HDD-based data transfers. Use of the Data Transfer Utility for appliance-based transfers has been replaced with the Oracle Cloud Infrastructure command line interface (CLI). See Managing Appliance Data Transfers for more information.

The Data Transfer Utility is licensed under the Universal Permissive License 1.0 and the Apache License 2.0. Third-party content is separately licensed as described in the code.

Prerequisites

To install and use the Data Transfer Utility, obtain the following:

  • An Oracle Cloud Infrastructure account.

  • Required Oracle Cloud Infrastructure users and groups with the required IAM policies.

    See Preparing for Data Transfer for details.

  • A host machine with the following installed:

    • Oracle Linux 6 or greater, Ubuntu 14.04 or greater, or SUSE 11 or greater

    • Java 1.8 or Java 1.11

    • hdparm 9.0 or later

    • Cryptsetup 1.2.0 or greater

  • Firewall access: If you have a restrictive firewall in the environment where you are using the Data Transfer Utility, you may need to open your firewall configuration to the following IP address ranges: 140.91.0.0/16.

    You also need to open access to the object storage IP address ranges: 134.70.0.0/17.

Installing the Data Transfer Utility

Download and install the Data Transfer Utility installer that corresponds to your host machine's operating system.

To install the Data Transfer Utility on Debian or Ubuntu
To install the Data Transfer Utility on Oracle Linux or Red Hat Linux

Configuring the Data Transfer Utility

Before using the Data Transfer Utility, you must create a base Oracle Cloud Infrastructure directory and two configuration files with the required credentials. One configuration file is for the data transfer administrator, the IAM user with the authorization and permissions to create and manage transfer jobs. The other configuration file is for the data transfer upload user, the temporary IAM user that Oracle uses to upload your data on your behalf.

Base Data Transfer Directory

Create a base Oracle Cloud Infrastructure directory:

mkdir /root/.oci/

Configuration File for the Data Transfer Administrator

Create a data transfer administrator configuration file /root/.oci/config with the following structure:

[DEFAULT]
user=<The OCID for the data transfer administrator>
fingerprint=<The fingerprint of the above user's public key>
key_file=<The _absolute_ path to the above user's private key file on the host machine>
tenancy=<The OCID for the tenancy that owns the data transfer job and bucket>
region=<The region where the transfer job and bucket should exist. Valid values are: us-ashburn-1, us-phoenix-1, eu-frankfurt-1, and uk-london-1.>

For example:

[DEFAULT]
user=ocid1.user.oc1..<unique_ID>
fingerprint=4c:1a:6f:a1:5b:9e:58:45:f7:53:43:1f:51:0f:d8:45
key_file=/home/user/ocid1.user.oc1..<unique_ID>.pem
tenancy=ocid1.tenancy.oc1..<unique_ID>
region=us-phoenix-1

For the data transfer administrator, you can create a single configuration file that contains different profile sections with the credentials for multiple users. Then use the ‑‑profile option to specify which profile to use in the command. Here is an example of a data transfer administrator configuration file with different profile sections:

[DEFAULT]
user=ocid1.user.oc1..<unique_ID>
fingerprint=4c:1a:6f:a1:5b:9e:58:45:f7:53:43:1f:51:0f:d8:45
key_file=/home/user/ocid1.user.oc1..<unique_ID>.pem
tenancy=ocid1.tenancy.oc1..<unique_ID>
region=us-phoenix-1
[PROFILE1]
user=ocid1.user.oc1..<unique_ID>
fingerprint=4c:1a:6f:a1:5b:9e:58:45:f7:53:43:1f:51:0f:d8:45
key_file=/home/user/ocid1.user.oc1..<unique_ID>.pem
tenancy=ocid1.tenancy.oc1..<unique_ID>
region=us-ashburn-1

Important

Creating an upload user configuration file with multiple profiles is not supported.

By default, the DEFAULT profile is used for all Data Transfer Utility commands. For example:

dts job create --compartment-id <compartment_id> --bucket <bucket_name> --display-name <display_name> --device-type <disk>

Instead, you can issue any Data Transfer Utility command with the --profile option to specify a different data transfer administrator profile. For example:

dts job create --compartment-id <compartment_id> --bucket <bucket_name> --display-name <display_name> --device-type <disk> --profile <profile_name>

Using the example configuration file above, the <profile_name> would be profile1.

Configuration File for the Data Transfer Upload User

Create a data transfer upload user /root/.oci/config_upload_user configuration file with the following structure:

[DEFAULT]
user=<The OCID for the data transfer upload user>
fingerprint=<The fingerprint of the above user's public key>
key_file=<The _absolute_ path to the above user's private key file on the host machine>
tenancy=<The OCID for the tenancy that owns the data transfer job and bucket>
region=<The region where the transfer job and bucket should exist. Valid values are: us-ashburn-1, us-phoenix-1, eu-frankfurt-1, and uk-london-1.>

For example:

[DEFAULT]
user=ocid1.user.oc1..<unique_ID>
fingerprint=4c:1a:6f:a1:5b:9e:58:45:f7:53:43:1f:51:0f:d8:45
key_file=/home/user/ocid1.user.oc1..<unique_ID>.pem
tenancy=ocid1.tenancy.oc1..<unique_ID>
region=us-phoenix-1

Configuration File Entries

The following table lists the basic entries that are required for each configuration file and where to get the information for each entry.

Note

Data Transfer Service does not support passphrases on the key files for both data transfer administrator and data transfer upload user.

Entry Description and Where to Get the Value Required?
user

OCID of the data transfer administrator or the data transfer upload user, depending on which profile you are creating. To get the value, see Required Keys and OCIDs.

Yes
fingerprint

Fingerprint for the key pair being used. To get the value, see Required Keys and OCIDs.

Yes
key_file

Full path and filename of the private key.

Important: The key pair must be in PEM format. For instructions on generating a key pair in PEM format, see Required Keys and OCIDs.

Yes
tenancy

OCID of your tenancy. To get the value, see Required Keys and OCIDs.

Yes
region

An Oracle Cloud Infrastructure region. See Regions and Availability Domains.

Data transfer is supported in us-ashburn-1, us-phoenix-1, eu-frankfurt-1, and uk-london-1.

Yes

You can verify the data transfer upload user credentials using the following command:

dts job verify-upload-user-credentials --bucket <bucket_name>

Configuration File Location

The location of the configuration files is /root/.oci/config.

Using the Data Transfer Utility

This section provides an overview of the syntax for the Data Transfer Utility.

Important

The Data Transfer Utility must be run as the root user.

You can specify Data Transfer Utility command options using the following commands:

  • --option <value> or
  • --option=<value>

Syntax

The basic Data Transfer Utility syntax is:

dts <resource> <action> <options>

This syntax is applied to the following:

  • dts is the shortened utility command name
  • job is an example of a <resource>
  • create is an example of an <action>
  • Other utility strings are <options>

The following commands to create a transfer job shows a typical Data Transfer Utility construct.

dts job create --compartment-id ocid1.compartment.oc1..<unique_ID> --display-name "mycompany transfer1" ‑‑bucket mybucket --device-type disk

Or:

dts job create --compartment-id=ocid1.compartment.oc1..<unique_ID> --display-name="mycompany transfer1" ‑‑bucket=mybucket --device-type=disk
Note

In the previous examples, provide a friendly name for the transfer job using the ‑‑display‑name option. Avoid entering confidential information when providing resource names or descriptions.

Getting Help with Commands

You can get help with the different commands associated with Data Transfer Utility using dts by itself. For example:

dts

Finding Out the Installed Version of the Data Transfer Utility

You can get the installed version of the Data Transfer Utility using --version or -v. For example:

dts --version

What's Next

You are now ready to perform disk-based data transfers. See Managing Disk Data Transfers.