Preparing for Appliance Data Transfers

Learn about the tasks associated with preparation for the appliance-based data transfer.

Prepare phase indicator for appliance transfer

This topic describes the tasks associated with preparing for the Appliance-Based Data Import job. The Project Sponsor role typically performs these tasks. See Roles and Responsibilities.

Note

You can only run Oracle Cloud Infrastructure CLI commands from a Linux host. This differs from running CLI commands for other Oracle Cloud Infrastructure Services on a variety of host operating systems. Appliance-based commands require validation that is only available on Linux hosts.

Installing and Using the Oracle Cloud Infrastructure Command Line Interface

The Oracle Cloud Infrastructure Command Line Interface (CLI) provides a set of command line-based tools for configuring and running Appliance-Based Data Import jobs. Use the Oracle Cloud Infrastructure CLI as an alternative to running commands from the Console. Sometimes you must use the CLI to complete certain tasks as there is no Console equivalent.

Minimum Required CLI Version

The minimum CLI version required for Appliance-Based Data Import is 2.12.1.

Determining CLI Versions

Access the following URL to see the currently available version of the CLI:

https://github.com/oracle/oci-cli/blob/master/CHANGELOG.rst

Enter the following command at the prompt to see the version of the CLI currently installed on your machine:

oci --version

If you have a version on your machine older than the version currently available, install the latest version.

Note

Always update to the latest version of the CLI. The CLI is not updated automatically, and you can only access new or updated CLI features by installing the current version.

Linux Operating System Requirements

See Requirements for a list of the Linux operating systems that support the CLI.

Installing the CLI

Installation and configuration of the CLIs is described in detail in Command Line Interface (CLI).

Using the CLI

You can specify CLI options using the following commands:

  • --option value or

  • --option=value

The basic CLI syntax is:

oci dts resource action options

This syntax is applied to the following:

  • oci dts is the shortened CLI command name.

  • job is an example of a resource.

  • create is an example of an action.

  • Other strings are options.

The following command to create a transfer job shows a typical CLI command construct.

oci dts job create --compartment-id ocid1.compartment.oc1..exampleuniqueID --bucket MyBucket --display-name MyApplianceImportJob --device-type appliance
Note

In the previous examples, provide a friendly name for the transfer job using the ‑‑display‑name option.

Accessing Command Line Interface Help

All CLI help commands have an associated help component you can access from the command line. To view the help, enter any command followed by the --help or -h option. For example:


oci dts job --help
			
NAME
  dts_job -

DESCRIPTION
  Transfer disk or appliance job operations

AVAILABLE COMMANDS
  o change-compartment
  o close
  o create
  o delete
  o detach-devices-details
  ...

When you run the help option (--help or -h) for a specified command, all the subordinate commands and options for that level of CLI are displayed. If you want to access the CLI help for a specific subordinate command, include it in the CLI string, for example:


oci dts job create --help
			
NAME
  dts_job_create -

DESCRIPTION
  Creates a new transfer disk or appliance job.

USAGE
  oci dts job create [OPTIONS]

REQUIRED PARAMETERS
  --bucket [text]

Upload bucket name

--compartment-id, -c [text]

Compartment OCID

--device-type [text]

Creating the Required IAM Users, Groups, and Policies

Each service in Oracle Cloud Infrastructure integrates with IAM for authentication and authorization.

To use Oracle Cloud Infrastructure, you must be given the required type of access in a policy  written by an administrator, whether you're using the Console or the REST API with an SDK, CLI, or other tool. If you try to perform an action and get a message that you don't have permission or are unauthorized, confirm with your administrator the type of access you've been granted and which compartment  you should work in.

Access to resources is provided to groups using policies and then inherited by the users that are assigned to those groups. Data transfer requires the creation of two distinct groups:

  • Data transfer administrators who can create and manage transfer jobs.

  • Data transfer upload users who can upload data to Object Storage. For your data security, the permissions for upload users allow Oracle personnel to upload standard and multi-part objects on your behalf and inspect bucket and object metadata. The permissions do not allow Oracle personnel to inspect the actual data.

The Data Administrator is responsible for generating the required RSA keys needed for the temporary upload users. These keys should never be shared between users.

For details on creating groups, see Managing Groups.

An administrator creates these groups with the following policies:

  • The data transfer administrator group requires an authorization policy that includes the following:

    Allow group group_name to manage data-transfer-jobs in compartment compartment_name
    Allow group group_name to manage objects in compartment compartment_name
    Allow group group_name to manage buckets in compartment compartment_name

    Alternatively, you can consolidate the manage buckets and manage objects policies into the following:

    Allow group group_name to manage object-family in compartment compartment_name
  • The data transfer upload user group requires an authorization policy that includes the following:

    Allow group group_name to manage buckets in compartment compartment_name where all { request.permission='BUCKET_READ', target.bucket.name='<bucket_name>' }
    Allow group group_name to manage objects in compartment compartment_name where all { target.bucket.name='<bucket_name>', any { request.permission='OBJECT_CREATE', request.permission='OBJECT_OVERWRITE', request.permission='OBJECT_INSPECT' }}

To enable notifications, add the following policies:

Allow group group name to manage ons-topics in tenancy
Allow group group name to manage ons-subscriptions in tenancy
Allow group group name to manage cloudevents-rules in tenancy
Allow group group name to inspect compartments in tenancy

See Notifications and Overview of Events for more information.

The Oracle Cloud Infrastructure administrator then adds a user to each of the data transfer groups created. For details on creating users, see Managing Users.

Important

For security reasons, we recommend that you create a unique IAM data transfer upload user for each transfer job and then delete that user once your data is uploaded to Oracle Cloud Infrastructure.

Requesting Appliance Entitlement

If your tenancy is not entitled to use the Data Transfer Appliance, you must request the Data Transfer Appliance Entitlement before creating an appliance-based transfer job.

Important

The main buyer or administrator, who is at a VP level or higher, receives an email notification and is required to e-sign a Terms and Conditions document. After Oracle has confirmed signature of the document, you can create an appliance-based transfer job. The email for the DocuSign does not go to the requester unless they are the main buyer or administrator who is at least at a VP level.

It can take up to 24 hours before the Terms and Conditions are sent.

Establishing the Appliance Entitlement Policy

Use the following policy to enable users in a specific group to request a Data Transfer Appliance Entitlement in your tenancy.

Allow group <group_name> to {DTA_ENTITLEMENT_CREATE} in tenancy

Appliance Entitlement Eligibility

Your request for a Data Transfer Appliance Entitlement in your tenancy may be denied if you are a free trial customer. If your request is denied, upgrade to a full account. You can also contact your Oracle Customer Support Manager or Oracle Support to determine your options for obtaining the entitlement.

Using the Console

Open the Transfer Job page and click Request at the top. Otherwise, you are prompted to request the entitlement when attempting to create your first appliance-based transfer job.

Once requested, the status of your request is visible at the top of the Transfer Job page. For example:

Data Transfer Appliance Entitlement: Granted

It can take a while to get the Data Transfer Appliance Entitlement approved. After Oracle receives your request, a Terms and Conditions Agreement is sent to the account owner via DocuSign to use the appliance. The entitlement request is approved once the signature is received. The Data Transfer Appliance Entitlement is a tenancy-wide entitlement that you need to request once for each tenancy.

Using the CLI

Use the dts appliance request-entitlement command and required parameters to request an appliance entitlement.

oci dts appliance request-entitlement --compartment-id compartment_id 
--name name --email email [OPTIONS]

name is the name of the requester.

email is the email address of the requester.

For a complete list of flags and variable options for CLI commands, see the Command Line Reference.

For example:

oci dts appliance request-entitlement --compartment-id ocid.compartment.oc1..exampleuniqueID 
--name "Robert Smith --email rsmith@example.com
					
{
  "data": {
    "compartment-id": "ocid.compartment.oc1..exampleuniqueID",
    "creation-time": "2019-12-18T18:29:15+00:00",
	"defined-tags": {},
	"display-name": null,
	"freeform-tags": {},
	"id": "ocid1.datatransferapplianceentitlement.oc1..exampleuniqueID",
	"lifecycle-state": "CREATING",
	"lifecycle-state-details": "REQUESTED",
	"requestor-email": "rsmith@example.com",
	"requestor-name": "Robert Smith",
    "update-time": "2019-12-20T19:04:09+00:00"
  }
}

Showing the Status of an Appliance Entitlement Request

Run the oci dts appliance show-entitlement command and its required parameters to show the status of an appliance entitlement.

oci dts appliance show-entitlement --compartment-id compartment_ocid [OPTIONS]

For a complete list of flags and variable options for CLI commands, see the Command Line Reference.

For example:

oci dts appliance show-entitlement --compartment-id ocid.compartment.oc1..exampleuniqueID 
{
  "data": {
    "compartment-id": ""ocid.compartment.oc1..exampleuniqueID",
    "defined-tags": null,
    "display-name": null,
    "freeform-tags": null,
    "id": null,
    "lifecycle-state": "ACTIVE",
    "lifecycle-state-details": "APPROVED",
    "requestor-email": "rsmith@example.com",
    "requestor-name": "Robert Smith"
  }
}

Creating Object Storage Buckets

The Object Storage service is used to upload your data to Oracle Cloud Infrastructure. Object Storage stores objects in a container called a bucket within a compartment  in your tenancy. For details on creating the bucket to store uploaded data, see Object Storage Buckets.

Configuring Firewall Settings

The firewall port number is 443 for all data transfer methods.

Ensure that your local environment's firewall can communicate with the Data Transfer Service running on the IP address ranges for your Oracle Cloud Infrastructure region based on the following table. Also ensure that open access exists to the Object Storage IP address range. You only need to configure this IP access for the region where your data transfer job is associated.

Region

Data Transfer

Object Storage

US East (Ashburn)

140.91.0.0/16

134.70.24.0/21

US West (Phoenix)

129.146.0.0/16

134.70.8.0/21

US Gov East (Ashburn)

splat-api.us-langley-1.oraclegovcloud.com

objectstorage.us-gov-ashburn-1.oraclegovcloud.com

US Gov West (Phoenix)

splat-api.us-luke-1.oraclegovcloud.com

objectstorage.us-luke-1.oraclegovcloud.com

US DoD East (Ashburn)

splat-api.us-gov-ashburn-1.oraclegovcloud.com

objectstorage.us-gov-ashburn-1.oraclegovcloud.com

US DoD West (Phoenix)

splat-api.us-gov-phoenix-1.oraclegovcloud.com

objectstorage.us-gov-phoenix-1.oraclegovcloud.com

Brazil East (Sao Paulo)

140.204.0.0/16

134.70.84.0/22

Canada Southeast (Toronto)

140.204.0.0/16

134.70.116.0/22

Germany Central (Frankfurt)

130.61.0.0/16

134.70.40.0/21

India West (Mumbai)

140.204.0.0/16

134.70.76.0/22

Japan Central (Osaka)

140.204.0.0/16

134.70.112.0/22

Japan East (Tokyo)

140.204.0.0/16

134.70.80.0/22

South Korea Central (Seoul)

140.204.0.0/16

134.70.96.0/22

UK South (London)

132.145.0.0/16

134.70.56.0/21

Creating Transfer Jobs

This section describes how to create a transfer job as part of the preparation for the data transfer. See Appliance Import Transfer Jobs for complete details on all tasks related to transfer jobs.

A transfer job represents the collection of files that you want to transfer and signals the intention to upload those files to Oracle Cloud Infrastructure. Identify which compartment and Object Storage bucket to which Oracle is to upload your data. Create the transfer job in the same compartment as the upload bucket and supply a human-readable name for the transfer job.

Note

It is recommended that you create a compartment for each transfer job to minimize the required access your tenancy.

Creating a transfer job returns a transfer job ID that you specify in other transfer tasks. For example:

ocid1.datatransferjob.region1.phx..unique_ID

Using the Console

  1. Open the navigation menu and click Migration & Disaster Recovery. Under Data Transfer, click Imports. The Transfer Jobs page appears.

  2. Choose a Compartment you have permission to work in under List scope. All transfer jobs in that compartment are listed in tabular form.

  3. Click Create Transfer Job. The Create Transfer Job dialog box appears.

  4. Complete the following:

    • Job Name: Enter a name for the transfer job.

    • Bucket: Select the bucket that contains the transfer data from the list. All available buckets for the selected compartment are listed. If you want to select a bucket in a different compartment, click Change Compartment and select the compartment that contains the bucket you want.

    • Transfer Type Device: Select the Appliance option.

    • (Optional) Complete the tagging settings:

      • Tag namespace: Select a namespace from the list.

      • Tag key: Enter a tagging key.

      • Tag value: Enter a value for the tagging key.

      See Overview of Tagging for more information.

  5. Click Create Transfer Job.

Using the CLI

Use the oci dts job create command and required parameters to create an appliance import transfer job

oci dts job create --bucket bucket --compartment-id compartment_id 
--display-name display_name --device-type appliance [OPTIONS]

For a complete list of flags and variable options for CLI commands, see the Command Line Reference.

For example:

oci dts job create --bucket MyBucket1 --compartment-id ocid.compartment.oc1..exampleuniqueID 
--display-name MyApplianceImportJob --device-type appliance
				
{
  "data": {
    "attached-transfer-appliance-labels": [],
    "attached-transfer-device-labels": [],
    "attached-transfer-package-labels": [],
    "compartment-id": "ocid.compartment.oc1..exampleuniqueID",
    "creation-time": "2019-12-18T19:43:58+00:00",
    "defined-tags": {},
    "device-type": "APPLIANCE",
    "display-name": "MyApplianceImportJob",
    "freeform-tags": {},
    "id": "ocid1.datatransferjob.oc1..exampleuniqueID",
    "label": "JAKQVAGJF",
    "lifecycle-state": "INITIATED",
    "upload-bucket-name": "MyBucket1"
  },
  "etag": "2--gzip"
}

Optionally, you can specify one or more defined or freeform tags when you create a transfer job. For more information about tagging, see Resource Tags.

Defined Tags

To specify defined tags when creating a job:

oci dts job create --bucket bucket --compartment-id compartment_id --display-name display_name --device-type appliance --defined-tags '{ "tag_namespace": { "tag_key":"value" }}'

For example:

oci dts job create --bucket MyBucket1 --compartment-id ocid.compartment.oc1..exampleuniqueID --display-name MyApplianceImportJob --device-type appliance --defined-tags '{"Operations": {"CostCenter": "01"}}'
				
{
"data": {
  "attached-transfer-appliance-labels": [],
  "attached-transfer-device-labels": [],
  "attached-transfer-package-labels": [],
  "compartment-id": "ocid.compartment.oc1..exampleuniqueID",
  "creation-time": "2019-12-18T19:43:58+00:00",
  "defined-tags": {
    "operations": {
      "costcenter": "01"
    }
  },
  "device-type": "APPLIANCE",
  "display-name": "MyApplianceImportJob",
  "freeform-tags": {},
  "id": "ocid1.datatransferjob.oc1..exampleuniqueID",
  "label": "JAKQVAGJF",
  "lifecycle-state": "INITIATED",
  "upload-bucket-name": "MyBucket1"
  },
  "etag": "2--gzip"
}
Freeform Tags

To specify freeform tags when creating a job:

oci dts job create --bucket bucket --compartment-id compartment_id --display-name display_name --device-type appliance --freeform-tags '{ "tag_key":"value" }'				

For example:

oci dts job create --bucket MyBucket1 --compartment-id ocid.compartment.oc1..exampleuniqueID --display-name MyApplianceImportJob --device-type appliance --freeform-tags '{"Pittsburg_Team":"brochures"}'
				
{
"data": {
  "attached-transfer-appliance-labels": [],
  "attached-transfer-device-labels": [],
  "attached-transfer-package-labels": [],
  "compartment-id": "ocid.compartment.oc1..exampleuniqueID",
  "creation-time": "2019-12-18T19:43:58+00:00",
  "defined-tags": {},
  "device-type": "APPLIANCE",
  "display-name": "MyApplianceImportJob",
  "freeform-tags": {
	"Pittsburg_Team": "brochures"
  },
  "id": "ocid1.datatransferjob.oc1..exampleuniqueID",
  "label": "JAKQVAGJF",
  "lifecycle-state": "INITIATED",
  "upload-bucket-name": "MyBucket1"
  },
  "etag": "2--gzip"
}
Note

Users create tag namespaces and tag keys with the required permissions. These items must exist before you can specify them when creating a job. See Tags and Tag Namespace Concepts for details.

Multiple Tags

To specify multiple tags, comma separate the JSON-formatted key/value pairs:

oci dts job create --bucket bucket --compartment-id compartment_id --display-name display_name --device-type appliance --freeform-tags '{ "tag_key":"value" }', '{ "tag_key":"value" }'

Notifications

To include notifications, include the --setup-notifications option. See Setting Up Transfer Job Notifications from the CLI for more information on this feature.

Getting Transfer Job OCIDs

Each transfer job you create has a unique OCID within Oracle Cloud Infrastructure. For example:

ocid1.datatransferjob.oc1.phx.exampleuniqueID

You will need to forward this transfer job OCID to the Data Administrator.

Using the Console

  1. Open the navigation menu and click Migration & Disaster Recovery. Under Data Transfer, click Imports. The Transfer Jobs page appears.

  2. Choose a Compartment you have permission to work in under List scope. All transfer jobs in that compartment are listed in tabular form.

  3. Click the link under Transfer Jobs for the transfer job whose details you want to view. The transfer job's Details page appears.

  4. Find the OCID field in the Details page and click Show to display it or Copy to copy it to your computer.

Using the CLI

Use the oci dts job list command and required parameters to list the appliance import transfer jobs in your tenancy.

oci dts job list --compartment-id compartment_id [OPTIONS]

For a complete list of flags and variable options for CLI commands, see the Command Line Reference.

For example:

oci dts job list --compartment-id ocid.compartment.oc1..exampleuniqueID
					
{
  "data": [
    {
      "creation-time": "2019-12-18T19:43:58+00:00",
      "defined-tags": {},
      "device-type": "APPLIANCE",
      "display-name": "MyApplianceImportJob",
      "freeform-tags": {},
      "id": "ocid1.datatransferjob.oc1..exampleuniqueID",
      "label": "JAKQVAGJF",
      "lifecycle-state": "INITIATED",
      "upload-bucket-name": "MyBucket1"
    },
    {
      "creation-time": "2019-10-03T16:52:26+00:00",
      "defined-tags": {},
      "device-type": "DISK",
      "display-name": "MyDiskImportJob",
      "freeform-tags": {},
      "id": "ocid1.datatransferjob.oc1..exampleuniqueID",
      "label": "J2AWEOL5T",
      "lifecycle-state": "INITIATED",
      "upload-bucket-name": "MyBucket2"
    }
  ]
}

The ID for each transfer job is included in the return:

"id": "ocid.compartment.oc1..exampleuniqueID"
Tip

When you create a transfer job using the oci dts job create CLI, the transfer job ID is displayed in the CLI's return. You can also run the oci dts job show CLI for that specific job to get the ID.

Setting Up Transfer Job Notifications from the CLI

You can generate notifications that send messages regarding changes to a new or existing appliance-based transfer job through the CLI. Using this feature creates a topic, subscription for a list of email addresses, and a rule that notifies you on all events related to the export job's activities and changes in state. This method provides a more convenient way to generate notifications tailored to appliance-based transfer jobs.

The CLI command to set up transfer job notifications is different depending on whether you are creating a new transfer job or updating an existing transfer job. In both cases, running the CLI command prompts you to enter the email addresses of each notification subscriber as a comma separated list. Each recipient is sent an email with a link to confirm they want to receive the notifications.

You are prompted to enter those email addresses you want included in the notifications, separated by commas (","). When your list is complete, add a colon (":") followed by your own email address: user1@mycompany.com,user2@mycompany.com : myemail@mycompany.com.

For both of the notification commands, the following is returned:

If the commands fail to run, you can use the OCI CLI to do the setup manually:
export ROOT_COMPARTMENT_OCID=ocidv1:tenancy:oc1:exampleuniqueID
oci ons topic create --compartment-id $ROOT_COMPARTMENT_OCID --name DTSExportTopic --description "Topic for data transfer service export jobs"
oci ons subscription create --protocol EMAIL --compartment-id $ROOT_COMPARTMENT_OCID --topic-id $TOPIC_OCID --endpoint $EMAIL_ID
oci events rule create --display-name DTSExportRule --is-enabled true --compartment-id $ROOT_COMPARTMENT_OCID --actions '{"actions":[{"actionType":"ONS","topicId":"$TOPIC_OCID","isEnabled":true}]}' --condition '{"eventType":["com.oraclecloud.datatransferservice.addapplianceexportjob","com.oraclecloud.datatransferservice.deleteapplianceexportjob","com.oraclecloud.datatransferservice.updateapplianceexportjob","com.oraclecloud.datatransferservice.moveapplianceexportjob"]}' --description "Rule for data transfer service to send notifications for export jobs"
Creating topic for export
		

When Creating a Transfer Job

To set up notifications when creating a transfer job, include the --setup-notifications parameter as part of the oci dts job create command:

oci dts job create ... --setup-notifications

Setting Up Notifications for an Existing Export Job

To set up notifications for an existing transfer job:

oci dts job setup-notifications --job-id job_id

For example:

oci dts job setup-notifications --job-id ocid1.datatransferjob.oc1..exampleuniqueID

If the commands fail to run, you can use the OCI CLI to do the setup manually:
oci ons topic create --compartment-id ocid1.tenancy.oc1..exampleuniqueID --name MyImportJob --description "Topic for data transfer service import job with OCID ocid1.datatransferjob.oc1..exampleuniqueID"
oci ons subscription create --protocol EMAIL --compartment-id $ROOT_COMPARTMENT_OCID --topic-id $TOPIC_OCID --subscription_endpoint $EMAIL_ID
oci events rule create --display-name MyImportJob --is-enabled true--compartment-id ocid1.tenancy.oc1..exampleuniqueID --actions '{"actions":[{"actionType":"ONS","topicId":"$TOPIC_OCID","isEnabled":true}]}' --condition '{"eventType":"com.oraclecloud.datatransferservice.*transferjob","data":{"resourceId":"ocid1.datatransferjob.oc1..exampleuniqueID"}}' --description "Rule for data transfer service to send notifications for an import job with OCID ocid1.datatransferjob.oc1..exampleuniqueID"
Creating topic DTSImportJobTopic_2pwaqq

{
  "data": {
    "api-endpoint": "https://cell1.notification.us-phoenix-1.oraclecloud.com",
    "compartment-id": "ocid1.tenancy.oc1..exampleuniqueID",
    "defined-tags": {},
    "description": "Topic for data transfer service import job with OCID ocid1.datatransferjob.oc1..exampleuniqueID",
    "etag": null,
    "freeform-tags": {},
    "lifecycle-state": "ACTIVE",
    "name": "DTSImportJobTopic_2pwaqq",
    "time-created": "2020-07-15T18:26:07.179000+00:00",
    "topic-id": "ocid1.onstopic.oc1..exampleuniqueID"
  },
  "etag": "2e5a567d"
}

Enter email addresses to subscribe to as a comma separated list. Example: jdoe@mycompany.com,rroe@mycompany.com : jsmith@mycompany.com
Creating subscription for jsmith@mycompany.com
{
  "data": {
    "compartment-id": "ocid1.tenancy.oc1..exampleuniqueID",
    "created-time": 1594837577401,
    "defined-tags": {},
    "deliver-policy": "{\"maxReceiveRatePerSecond\":0,\"backoffRetryPolicy\":{\"initialDelayInFailureRetry\":60000,\"maxRetryDuration\":7200000,\"policyType\":\"EXPONENTIAL\"}}",
    "endpoint": "jsmith@mycompany.com",
    "etag": "cac2f405",
    "freeform-tags": {},
    "id": "ocid1.onssubscription.oc1..exampleuniqueID",
    "lifecycle-state": "PENDING",
    "protocol": "EMAIL",
    "topic-id": "ocid1.onstopic.oc1..exampleuniqueID"
  },
  "etag": "cac2f405"
}

Creating rule DTSImportJobRule_2pwaqq
{
  "data": {
    "actions": {
    "actions": [
      {
        "action-type": "ONS",
        "description": null,
        "id": "ocid1.eventaction.oc1..exampleuniqueID",
        "is-enabled": true,
        "lifecycle-message": null,
        "lifecycle-state": "ACTIVE",
        "topic-id": "ocid1.onstopic.oc1..exampleuniqueID"
      }
    ]
  },
    "compartment-id": "ocid1.tenancy.oc1..exampleuniqueID",
    "condition": "{\"eventType\":\"com.oraclecloud.datatransferservice.*transferjob\",\"data\":{\"resourceId\":\"ocid1.datatransferjob.oc1..exampleuniqueID\"}}",
    "defined-tags": {},
    "description": "Rule for data transfer service to send notifications for an import job with OCID ocid1.datatransferjob.oc1..exampleuniqueID",
    "display-name": "DTSImportJobRule_2pwaqq",
    "freeform-tags": {},
    "id": "ocid1.eventrule.oc1..exampleuniqueID",
    "is-enabled": true,
    "lifecycle-message": null,
    "lifecycle-state": "ACTIVE",
    "time-created": "2020-07-15T18:26:18.307000+00:00"
  },
  "etag": "aff873bfb4015b49902b97c7a6cc40588bf89b9e3deeb27b77ecce6d7a99768a"
}

Preparing Upload Configuration Files

The Project Sponsor is responsible for creating or obtaining configuration files that allow the uploading of user data to the import appliance. Send these configuration files to the Data Administrator where they can be placed in the Control Host (if there are separate Control and Data Hosts).The config file is for the data transfer administrator, the IAM user with the authorization and permissions to create and manage transfer jobs. The config_upload_user file is for the data transfer upload user, the temporary IAM user that Oracle uses to upload your data on your behalf.

Create a base Oracle Cloud Infrastructure directory and two configuration files with the required credentials.

Creating the Data Transfer Directory

Create a Oracle Cloud Infrastructure directory (.oci) on the same Control Host machine where the Oracle Cloud Infrastructure CLI is installed. For example:

mkdir /root/.oci/

The two configuration files (config and config_upload_user) are placed in what ever location you choose.

Note

You can store the configuration files anywhere on your Control Host. The root directory is only given as an example.

Creating the Data Transfer Administrator Configuration File

The data transfer administrator configuration file contains the required credentials for working with Oracle Cloud Infrastructure. You can create this file using a setup CLI or manually using a text editor.

Using the Setup CLI

Run the oci setup config command line utility to walk through the first-time setup process. The command prompts you for the information required for the configuration file and the API public/private keys. The setup dialog generates an API key pair and creates the configuration file.

For more information about how to find the required information, see:

Manual Setup

If you want to set up the API public/private keys yourself and write your own configuration file, see SDK and CLI Configuration File.

Tip

Use the oci setup keys command to generate a key pair to include in the config file.

Create the data transfer administrator configuration file /root/.oci/config with the following structure:

[DEFAULT]
user=<The OCID for the data transfer administrator>
fingerprint=<The fingerprint of the above user's public key>
key_file=<The _absolute_ path to the above user's private key file on the host machine>
tenancy=<The OCID for the tenancy that owns the data transfer job and bucket>
region=<The region where the transfer job and bucket should exist. Valid values are: 
supported regions>.

where supported regions are the regions listed in Data Transfer Supported Regions.

For example:

[DEFAULT]
user=ocid1.user.oc1..unique_ID
fingerprint=4c:1a:6f:a1:5b:9e:58:45:f7:53:43:1f:51:0f:d8:45
key_file=/home/user/ocid1.user.oc1..exampleuniqueID.pem
tenancy=ocid1.tenancy.oc1..unique_ID
region=us-phoenix-1

For the data transfer administrator, you can create a single configuration file that contains different profile sections with the credentials for multiple users. Then use the --profile option to specify which profile to use in the command. Here is an example of a data transfer administrator configuration file with different profile sections:

[DEFAULT]
user=ocid1.user.oc1..exampleuniqueID
fingerprint=4c:1a:6f:a1:5b:9e:58:45:f7:53:43:1f:51:0f:d8:45
key_file=/home/user/ocid1.user.oc1..exampleuniqueID.pem
tenancy=ocid1.tenancy.oc1..exampleuniqueID
region=us-phoenix-1
[PROFILE1]
user=ocid1.user.oc1..exampleuniqueID
fingerprint=4c:1a:6f:a1:5b:9e:58:45:f7:53:43:1f:51:0f:d8:45
key_file=/home/user/ocid1.user.oc1..exampleuniqueID.pem
tenancy=ocid1.tenancy.oc1..exampleuniqueID
region=us-ashburn-1

By default, the DEFAULT profile is used for all CLI commands. For example:

oci dts job create --compartment-id ocid.compartment.oc1..exampleuniqueID --bucket MyBucket --display-name MyDisplay --device-type appliance

Instead, you can issue any CLI command with the --profile option to specify a different data transfer administrator profile. For example:

oci dts job create --compartment-id ocid.compartment.oc1..exampleuniqueID --bucket MyBucket --display-name MyDisplay --device-type appliance --profile MyProfile

Using the example configuration file above, the profile_name would be profile1.

If you created two separate configuration files, use the following command to specify the configuration file to use:

oci dts job create --compartment-id compartment_id --bucket bucket_name --display-name display_name --config 
                
                

Creating the Data Transfer Upload User Configuration File

The config_upload_user configuration file is for the data transfer upload user, the temporary IAM user that Oracle uses to upload your data on your behalf. Create this configuration file with the following structure:

[DEFAULT]
user=<The OCID for the data transfer upload user>
fingerprint=<The fingerprint of the above user's public key>
key_file=<The _absolute_ path to the above user's private key file on the host machine>
tenancy=<The OCID for the tenancy that owns the data transfer job and bucket>
region=<The region where the transfer job and bucket should exist. Valid values are: 
supported regions>.

where supported regions are the regions listed in Data Transfer Supported Regions.

Configuration File Entries

The following table lists the basic entries that are required for each configuration file and where to get the information for each entry.

Note

Data Transfer Service does not support passphrases on the key files for both data transfer administrator and data transfer upload user.

Entry

Description and Where to Get the Value

Required?

user

OCID of the data transfer administrator or the data transfer upload user, depending on which profile you are creating. To get the value, see Required Keys and OCIDs.

Yes

fingerprint

Fingerprint for the key pair being used. To get the value, see Required Keys and OCIDs.

Yes

key_file

Full path and filename of the private key.

Important: The key pair must be in PEM format. For instructions on generating a key pair in PEM format, see Required Keys and OCIDs.

Yes

tenancy

OCID of your tenancy. To get the value, see Required Keys and OCIDs.

Yes

region

An Oracle Cloud Infrastructure region. See Regions and Availability Domains.

See Data Transfer Supported Regions for those regions that support Data Transfer.

Yes

You can verify the data transfer upload user credentials using the following command:

oci dts job verify-upload-user-credentials --bucket bucket_name

Requesting the Import Appliance

This section describes how to request an import appliance from Oracle for copying your data to Oracle Cloud Infrastructure See Import Appliances for complete details on all tasks related to transfer jobs.

Oracle Cloud Infrastructure customers can use import appliances to migrate data for free. You are only charged for Object Storage usage once the data is successfully transferred to your designated bucket. All appliance requests still require approval from Oracle.

Tip

To save time, identify the data you intend to upload and make data copy preparations before requesting the import appliance.

Creating an appliance request returns an Oracle-assigned appliance label. For example:

XA8XM27EVH

Using the Console

  1. Open the navigation menu and click Migration & Disaster Recovery. Under Data Transfer, click Imports. The Transfer Jobs page appears.

  2. Choose a Compartment you have permission to work in under List scope. All transfer jobs in that compartment are listed in tabular form.

  3. Click the transfer job for which you want to request an import appliance. The transfer job's Details page appears.

  4. Click Request Transfer Appliance under Transfer Appliances. The Request Transfer Appliance dialog box appears.

  5. Provide the shipping address details where you want the import appliance sent.

    • Company Name: Specify the name of the company that owns the data being migrated to Oracle Cloud Infrastructure.

    • Recipient Name: Specify the name of the recipient of the import appliance.

    • Recipient Phone Number: Specify the recipient's phone number.

    • Recipient Email Address: Specify the recipient's email address.

    • Care Of: Optional intermediary party responsible for transferring the import appliance shipment from the delivery vendor to the intended recipient.

    • Address Line 1: Specify the street address to where the import appliance is sent.

    • Address Line 2: Optional identifying address details like building, suite, unit, or floor information.

    • City/Locality: Specify the city or locality.

    • State/Province/Region: Specify the state, province, or region.

    • Zip/Postal Code: Specify the zip code or postal code.

    • Country: Select the country.

    • Minimum Storage Capacity: Select the minimum device storage capacity (when there is more than one option) that meets your needs. The amounts displayed can vary by region. See Data Transfer Appliance Specifications for more information. After you submit your request, an Oracle representative will work with you to obtain the device best suited to your needs.

  6. Click Request Transfer Appliance.

Using the CLI

Use the oci dts appliance request command and required parameters to request an import appliance.

oci dts appliance request --job-id job_id --addressee addressee 
--care-of care_of --address1 address_line1 --city-or-locality city_or_locality 
--state-province-region state_province_region --country country --zip-postal-code zip_postal_code 
--phone-number phone_number --email email [OPTIONS]

For a complete list of flags and variable options for CLI commands, see the Command Line Reference.

For example:

oci dts appliance request --job-id ocid1.datatransferjob.oc1..exampleuniqueID --addressee "Example, Inc." 
--care-of "Robert Smith" --address1 "2300 Oracle Way" --city-or-locality Austin --state-province-region TX 
--country USA --zip-postal-code 78741 --phone-number 6035550100 --email rsmith@example.com
				
{
  "data": {
    "appliance-delivery-tracking-number": null,
    "appliance-delivery-vendor": null,
    "appliance-return-delivery-tracking-number": null,
    "creation-time": "2020-05-20T22:08:13+00:00",
    "customer-received-time": null,
    "customer-returned-time": null,
    "customer-shipping-address": {
      "address1": "2300 Oracle Way",
      "address2": null,
      "address3": null,
      "address4": null,
      "addressee": "Example, Inc.",
      "care-of": "Robert Smith",
      "city-or-locality": "Austin",
      "country": "USA",
      "email": "rsmith@example.com",
      "phone-number": "6035550100",
      "state-or-region": "TX",
      "zipcode": "78741"
    },
    "delivery-security-tie-id": null,
    "label": "XAKWEGKZ5T",
    "lifecycle-state": "REQUESTED",
    "next-billing-time": null,
    "return-security-tie-id": null,
    "serial-number": null,
    "transfer-job-id": "ocid1.datatransferjob.oc1..exampleuniqueID",
    "upload-status-log-uri": "JAKQVAGJF/XAKWEGKZ5T/upload_summary.txt"
  }
}

When you submit an appliance request, Oracle generates a unique label (label": "XAKWEGKZ5T) to identify the import appliance and your request is sent to Oracle for approval and processing.

Setting Up Import Appliance Request Notifications

You can generate notifications that send messages regarding changes to your import appliance request by using the setup-notifications command through the CLI. Running this command creates a topic, subscription for list of email addresses, and also a rule that notifies you on all events related to the import appliance request's activities and changes in state. This method provides a more convenient way to generate notifications tailored to import appliance requests.

Running the CLI command prompts you to enter the email addresses of each notification subscriber as a comma separated list. Each recipient is sent an email with a link to confirm they want to receive the notifications.

Note

Setting up notifications from the CLI affects all import appliances in your tenancy. You cannot specify notifications for individual appliances.

Setting Up Notifications for a New Import Appliance Request

To include job notifications when requesting an import appliance, include the --setup-notifications option as part of the oci dts appliance request command:

oci dts appliance request --job-id ... --setup-notifications

Setting up Notifications for an Existing Import Appliance Request

To set up notifications for an existing import appliance request, run the oci dts appliance setup-notifications command on the appliance:

oci dts appliance setup-notifications --appliance-label appliance_label

Notifying the Data Administrator

When you have completed all the tasks in this topic, provide the Data Administrator of the following:

  • IAM login credentials

  • Oracle Cloud Infrastructure CLI configuration files

  • Transfer job ID

  • Appliance label

Validating the Preparation Phase

Perform the following command line interface (CLI) validation tasks at the end of this phase before continuing to the next phase. Performing the validation procedures described here assesses your environment and confirms that you have completed all necessary setup requirements successfully. Running these procedures also serves as a troubleshooting resource for you to ensure a smooth and successful data transfer.

Use the oci dts verify prepared command and required parameters to validate the Preparing phase tasks and configurations you made:

oci dts verify prepared --compartment-id compartment_ocid --job-id job_ocid --bucket bucket [OPTIONS]

Running this CLI command validates the following:

  • Configuration files (admin config, upload config)

  • Connectivity to Data Transfer

  • Connectivity to Object Storage

  • Required IAM Users, Groups, and Policies for admin user (create and manage buckets and transfer jobs)

  • Upload bucket belongs to compartment

  • Data transfer upload user can create, overwrite, inspect and delete objects from upload bucket in Object Storage

  • Approved Data Transfer appliance entitlement

  • Transfer job in the compartment has the expected bucket and a corresponding appliance request

  • Appliance request associated with transfer job

For example:

oci dts verify prepared --compartment-id ocid.compartment.oc1..exampleuniqueID --job-id ocid1.datatransferjob.oc1..exampleuniqueID --bucket MyBucket
Verifying requirements after 'Preparing for Appliance Data Transfers' task...
Checking Data Transfer Service connectivity... OK
Checking Object Storage connectivity... OK
Checking Required IAM Users, Groups, and Policies... OK
Checking Upload Bucket exists in Compartment... OK
Checking Upload User credentials...
Create object BulkDataTransferTestObject in bucket MyBucket using upload user
Overwrite object BulkDataTransferTestObject in bucket MyBucket using upload user
Inspect object BulkDataTransferTestObject in bucket MyBucket using upload user
Read bucket metadata MyBucket using upload user
Delete object BulkDataTransferTestObject in bucket MyBucket using admin user
OK
Checking Data Transfer Appliance Entitlement... OK
Checking Transfer Job and Appliance Request... OK
Verification successful.

Failure Scenarios

This section describes possible failure scenarios detected during validation:

  • Connectivity to Data Transfer service

    • Timed out request to Data Transfer endpoint

      Checking Data Transfer Service connectivity... datatransfer.r1.oracleiaas.com is not reachable.
      Please ensure there there are no firewall rules blocking connectivity.
      RequestException:
      {
          "client_version": "Oracle-PythonCLI/3.22.0",
          "logging_tips": "Please run the OCI CLI command using --debug flag to find more debug information.",
          "message": "The connection to endpoint timed out.",
          "request_endpoint": null,
          "target_service": "CLI",
          "timestamp": "2023-02-01T11:46:02.374857",
          "troubleshooting_tips": " See [https://docs.oracle.com/en-us/iaas/Content/API/Concepts/sdk_troubleshooting.htm] for more information about resolving this error. If you are unable to resolve this issue, run this CLI command with --debug option and contact Oracle support and provide them the full error message."
      }
  • Connectivity to Object Storage

    • Timed out request to the Object Storage endpoint

      Checking Object Storage connectivity... objectstorage.r1.oracleiaas.com is not reachable.
      Please ensure there there are no firewall rules blocking connectivity.
      RequestException:
      {
          "client_version": "Oracle-PythonCLI/3.22.0",
          "logging_tips": "Please run the OCI CLI command using --debug flag to find more debug information.",
          "message": "The connection to endpoint timed out while trying to reach https://objectstorage.r1.oracleiaas.com",
          "request_endpoint": " GET https://objectstorage.r1.oracleiaas.com",
          "target_service": "CLI",
          "timestamp": "2023-02-01T12:52:33.979490",
          "troubleshooting_tips": "Try running curl https://objectstorage.r1.oracleiaas.com. If the curl doesn't work, check your network setting or contact your network administrator. See [https://docs.oracle.com/en-us/iaas/Content/API/Concepts/sdk_troubleshooting.htm] for more information about resolving this error. If you are unable to resolve this issue, run this CLI command with --debug option and contact Oracle support and provide them the full error message."
      }
  • Required IAM Users, Groups, and Policies for admin user

    • No Object Storage authorization policy for admin user
      Checking Required IAM Users, Groups, and Policies... Fail
      Unauthorized access to Object Storage.
      Please ensure you have the required IAM Users, Groups, and Policies to access Object Storage.
    • No Data Transfer authorization policy for admin user

      Checking Required IAM Users, Groups, and Policies... Fail
      Unauthorized access to Data Transfer Service.
      Please ensure you have the required IAM Users, Groups, and Policies to access Data Transfer Service.
  • Upload bucket belongs to compartment

    • Upload bucket does not exist

      Checking Upload Bucket exists in Compartment... Fail
      Bucket not found.
      The bucket 'e2e_test_hdx' does not exist in namespace 'bdtstest1' or you are unauthorized to access it.
    • Upload bucket not in compartment

      
      Checking Upload Bucket exists in Compartment... Fail
      The bucket 'e2e_test_hdd' does not belong to compartment 'ocid1.compartment.region1..aaaaaaaaph2r5w24faket2vsp22hiboxjzk4k3s47d2ut37jghjagfb3abca'.
  • Data Transfer upload users can create, overwrite, inspect and delete objects from bucket in Object Storage

    • No authorization policy for the upload user

      Checking Upload User credentials...
      Failed to Create object BulkDataTransferTestObject in bucket e2e_test_hdd using upload user in tenancy bdtstest1 as upload user
      Fail
      ServiceError:
      {
          "client_version": "Oracle-PythonSDK/2.90.0+2664, Oracle-PythonCLI/3.22.0",
          "code": "BucketNotFound",
          "logging_tips": "Please run the OCI CLI command using --debug flag to find more debug information.",
          "message": "Either the bucket named 'e2e_test_hdd' does not exist in the namespace 'bdtstest1' or you are not authorized to access it",
          "opc-request-id": "sea-1:tpz7tEO-SLezjoTd506L8R5wm3HewQNHFfpg-EKsAINe6P8fFj4b_znPyfujFeqq",
          "operation_name": "put_object",
          "request_endpoint": "PUT https://objectstorage.r1.oracleiaas.com/n/bdtstest1/b/e2e_test_hdd/o/BulkDataTransferTestObject",
          "status": 404,
          "target_service": "object_storage",
          "timestamp": "2023-02-07T06:39:24.816918+00:00",
          "troubleshooting_tips": "See [https://docs.oracle.com/iaas/Content/API/References/apierrors.htm] for more information about resolving this error. If you are unable to resolve this issue, run this CLI command with --debug option and contact Oracle support and provide them the full error message."
      }
  • Approved Data Transfer appliance entitlement

    • No Data Transfer service entitlement

      Checking Data Transfer Appliance Entitlement... Fail
      Unable to find Data Transfer Appliance Entitlement in Compartment 'ocid1.compartment.region1..aaaaaaaaa5mxm3fd6tanvthad5oplmdcmxjqdsoehidua7iz3sxauih3deja'.
    • Unapproved Data Transfer entitlement

      Checking Data Transfer Appliance Entitlement... Fail
      Data Transfer Appliance Entitlement needs to be in an APPROVED lifecycle state.
  • Data Transfer import appliance-based transfer job exists

    • No Data Transfer appliance-based transfer job

      Checking Transfer Job and Appliance Request... Fail
      There is no Appliance Transfer Job 'ocid1.datatransferjob.oc1..exampleuniqueID' for Data Import into Bucket 'e2e_test_hdd'.
  • Appliance request associated with a valid transfer job

    • No appliance request

      Checking Transfer Job and Appliance Request... Fail
      There is no Transfer Appliance Request for Appliance Transfer Job 'ocid1.datatransferjob.oc1..exampleuniqueID'.