Monitoring the Import Appliance and Data Transfer

Monitor phase indicator for appliance transfer

This topic describes the monitoring tasks to do after sending the import appliance with the copied data back to Oracle for data transfer to Oracle Cloud Infrastructure. The Project Sponsor role typically performs these tasks. See Roles and Responsibilities.

Note

You can only run Oracle Cloud Infrastructure CLI commands from a Linux host. This differs from running CLI commands for other Oracle Cloud Infrastructure Services on a variety of host operating systems. Appliance-based commands require validation that is only available on Linux hosts.

Monitoring the Status of Your Import Appliance Return Shipment

The shipping vendor notifies Oracle when your import appliance is picked up and shipped back for upload to Oracle Cloud Infrastructure Object Storage.

To monitor the status of your import appliance return shipment using the Console
  1. Open the navigation menu. Under Core Infrastructure, go to Object Storage and click Data Transfer - Imports.
  2. Find the transfer job and associated import appliance that you shipped back to Oracle for data upload.
  3. Under Transfer Appliances, look at the Status field.
To monitor the status of your import appliance return shipment using the CLI
oci dts appliance show --job-id <job_id> --appliance-label <appliance_label>

For example:

oci dts appliance show --job-id ocid1.datatransferjob.oc1..exampleuniqueID --appliance-label XAKWEGKZ5T
					
{
  "data": {
    "appliance-delivery-tracking-number": null,
    "appliance-delivery-vendor": null,
    "appliance-return-delivery-tracking-number": null,
    "creation-time": "2020-05-20T22:08:13+00:00",
    "customer-received-time": null,
    "customer-returned-time": null,
    "customer-shipping-address": {
      "address1": "123 Main Street",
      "address2": null,
      "address3": null,
      "address4": null,
      "addressee": "MyCompany",
      "care-of": "John Doe",
      "city-or-locality": "Anytown",
      "country": "USA",
      "email": "john.doe@mycompany.com",
      "phone-number": "3115551212",
      "state-or-region": "NY",
      "zipcode": "12345"
    },
    "delivery-security-tie-id": "exampleuniqueID",
    "label": "XAKWEGKZ5T",
    "lifecycle-state": "PROCESSING",
    "next-billing-time": null,
    "return-security-tie-id": "exampleuniqueID",
    "serial-number": "exampleuniqueserialnumber",
    "transfer-job-id": "ocid1.datatransferjob.oc1..exampleuniqueID",
    "upload-status-log-uri": "JAKQVAGJF/XAKWEGKZ5T/upload_summary.txt"
  }
}

The import appliance status is indicated by the lifecycle-state attribute.

Import Appliance Status Values

Here are the import appliance status values, listed in alphabetic order:

CANCELLED
You can change your mind about uploading your data to Oracle Cloud Infrastructure Object Storage and cancel your import appliance. Ship the appliance back to Oracle and then cancel the appliance. Oracle always uses secure wipe tools on the boot and data areas whenever an appliance is returned.
COMPLETE
Oracle completed your import appliance data upload. Your data is available in your designated bucket in Oracle Cloud Infrastructure Object Storage.
CUSTOMER LOST
You have not returned the import appliance within the required 90 days.
DELIVERED
Oracle received a delivery confirmation from the shipping vendor that your import appliance was delivered. When the appliance is delivered, Oracle provides the date and time it was received in the appliance details. Appliance usage tracking begins.
ERROR
Oracle encountered an unrecoverable error trying to process your import appliance. Oracle cannot upload your data from the appliance. To protect your data, Oracle uses secure wipe tools on the boot and data areas any transfer appliance that cannot be processed. Complete another request for an appliance.
ORACLE PREPARING
Oracle approved your import appliance request. The status displays "Preparing" until the appliance is shipped to you.
ORACLE RECEIVED
Oracle received your import appliance shipment. The status displays "Oracle Received" until Oracle begins processing and uploading your data from the appliance.
ORACLE RECEIVED CANCELED
You canceled your import appliance after you shipped the appliance back to Oracle. Oracle received your canceled appliance does not upload any data from it.
PREPARING
You activated your import appliance. You can now copy your data onto the appliance. The status displays "Preparing" until you ship the appliance back to Oracle.
PROCESSING
Oracle is processing and uploading the data from your import appliance. The status displays "Processing"until Oracle completes uploading your data from the appliance.
REJECTED
Oracle denied your import appliance request.
Important

If your appliance request is denied and you have questions, contact your Sales Representative or file a Service Request (SR).
REQUESTED
You successfully completed your request for an import appliance. The status displays requested until Oracle approves your appliance request.
RETURN SHIPPED
Oracle received confirmation from the shipping vendor that you shipped your import appliance back to Oracle. The status displays "Return Shipped" until Oracle receives your appliance.
RETURN SHIPPED CANCELLED
You canceled your import appliance after the appliance was delivered to you or after you shipped the appliance back to Oracle. Oracle received confirmation from the shipping vendor that your canceled transfer appliance is on the way back to Oracle. The status displays "Return shipped cancelled" until Oracle receives your appliance.
SHIPPING
Oracle completed the necessary preparations and shipped your import appliance. When the appliance is shipped, Oracle provides the serial number of the appliance, the shipping vendor, and the tracking number in the appliance details. The status displays shipping until the appliance is delivered to you.

Reviewing the Upload Summary

Oracle creates upload summary log files for each uploaded appliance. These log files are placed in the bucket where data was uploaded to Oracle Cloud Infrastructure. The upload summary file compares the appliance's manifest file to the contents of the target Oracle Cloud Infrastructure Object Storage bucket after file upload.

Note

If you chose to upload your data to an Archive Storage bucket, you must first restore the log file object before you can download that file for review.

The top of the log report summarizes the overall file processing status:

P - Present: The file is present in both the device and the target bucket
M - Missing: The file is present in the device but not the target bucket. It was likely uploaded and then deleted by another user before the summary was generated.
C - Name Collision: The file is present in the manifest but a file with the same name but different contents is present in the target bucket.
U - Unreadable: The file is not readable from the disk
N - Name Too Long: The file name on disk is too long and could not be uploaded

Complete file upload details follow the summary.

This screenshot shows an example of the data transfer log file.

If you upload more than 100,000 files, the upload details are broken into multiple pages. You can only download the first page from the Console. Download the rest of the pages directly from the Object Storage bucket. The subsequent pages have the same object name as the first page, but have an enumerated suffix.

Verifying Uploaded File Integrity

To verify object data integrity of files uploaded to Object Storage from the Data Transfer Appliance, a cryptographic hash using MD5 is provided for all objects uploaded to Object Storage from the Data Transfer Appliance. Oracle Cloud Infrastructure provides the object hash value in base64 encoding.

To download files imported into Object Storage and verify their integrity, run the following CLI command:

oci os object get --namespace <object_storage_namespace> --bucket-name <bucket_name> --name <object_name> --file <file_location>

For example:

oci os object get --namespace MyNamespace --bucket-name MyBucket1 --name JLA12B3C/XAABC12EFG/upload_summary.txt –file upload_summary.txt
		
Downloading object  [####################################]  100%

Open the file and match the file names of the uploaded files with the MD5 column:

DTA download summary

In this example, file_1.txt has the MD5 sum of: EoN8s6dgT/9pGYA7Yx1klQ==

To download file_1.txt, run the following CLI command:

oci os object get --namespace <object_storage_namespace> --bucket-name <bucket_name> --name <object_name> --file <file_location>

For example:

oci os object get --namespace example_namespace --bucket-name bucket-1 --name file_1.txt –file file_1.txt
		
Downloading object  [####################################]  100%

Covert the base64 encoded hash value to hexadecimal, use the following command:

python -c 'print "BASE64-ENCODED-MD5-VALUE".decode("base64").encode("hex")'

For example:

python -c 'print "EoN8s6dgT/9pGYA7Yx1klQ==".decode("base64").encode("hex")'
		
12837cb3a7604fff6919803b631d6495

Now generate the md5sum on Linux and verify both values match:

md5sum <file_name>

For example:

md5sum file_1.txt
		
12837cb3a7604fff6919803b631d6495  file_1.txt

Verifying Multipart Uploaded Files

Large files are split into 1 GB parts when they are uploaded from the Data Transfer Appliance to Object Storage. You can verify the md5sum after downloading a file that was transferred in multiple parts using one of several available scripts. See the following for more information and links to these scripts:

https://github.com/oracle/oci-cli/issues/134

Viewing Data Transfer Metrics

After the import appliance with your copied data is received by Oracle and the data transfer begins, you can view the metrics associated with the transfer job in the Transfer Appliance Details page in chart or table format.

Tip

Set up your notifications to alert you when the data transfer from the appliance to Oracle Cloud Infrastructure is occurring. When the state changes from ORACLE_RECEIVED to PROCESSING, you can start viewing data transfer metrics. If you included the --setup-notification option when you made your appliance request from the CLI, this alert occurs automatically. See Notifications Overview for more information.

Select Metrics under Resources to display each of these measures:

  • Import Files Uploaded - Total number of files uploaded for import.
  • Import Bytes Uploaded - Total number of bytes uploaded for import.
  • Import Files Remaining - Total number of files remaining for import upload.
  • Import Bytes Remaining - Total number of bytes remaining for import upload.
  • Import Files in Error - Total number of files in error for import.
  • Import Upload Verification Progress - Progress of verification of files that have already been uploaded for import.

Select the Start Time and End Time for these measures, either by manually entering the days and times in their respective fields, or by selecting the Calendar feature and picking the times that way. As an alternative to selecting a start and end time, you can also select from a list of standard times (last hour, last 6 hours, and so forth) from the Quick Selects list for the period measured. The time period you specify applies to all the measures.

Specify the Interval (for example, 5 minutes, 1 hour) that each measure is recorded from the list.

Specify the Statistic being recorded (for example, Sum, Mean) for each measure from the list.

Tip

Mean is the most useful statistic for data transfer as it reflects an absolute value of the metric.

Choose additional actions from the Options list, including viewing the query in the Metrics Explorer, capturing the URL for the measure, and switching between chart and table view.

Click Reset Charts to delete any existing information in the charts and begin recording new metrics.

See Monitoring Overview for general information on monitoring your Oracle Cloud Infrastructure services.

Closing the Transfer Job

Close the transfer job when no further transfer job activity is required or possible. Closing a transfer job requires that the status of all associated import appliances be returned, canceled, or deleted.

To close a transfer job using the Console
  1. Open the navigation menu. Under Core Infrastructure, go to Object Storage and click Data Transfer - Imports.
  2. Find the data transfer package for which you want to display the details.
  3. Click the Actions icon (Actions icon), and then click View Details.

    Alternatively, click the hyperlinked name of the transfer job.

  4. Click Close Transfer Job.
To close a transfer job using the CLI
oci dts job close --job-id <job_id>

For example:

oci dts job close --job-id ocid1.datatransferjob.oc1..exampleuniqueID
{					
  "data": {
    "attached-transfer-appliance-labels": [],
    "attached-transfer-device-labels": [],
    "attached-transfer-package-labels": [],
    "compartment-id": "ocid.compartment.oc1..exampleuniqueID",
    "creation-time": "2020-05-20T22:00:43+00:00",
    "defined-tags": {},
    "device-type": "APPLIANCE",
    "display-name": "MyApplianceImportJob",
    "freeform-tags": {},
    "id": "ocid1.datatransferjob.oc1..exampleuniqueID",
    "label": "JGX4N1XLI",
    "lifecycle-state": "CLOSED",
    "upload-bucket-name": "MyBucket"
  },
  "etag": "1"
}

The lifecycle-state attribute value is "CLOSED."

What's Next

You have completed the process of setting up, running, and monitoring the Appliance-Based Data Import. If you determine that another appliance-based data transfers is required, repeat the procedure from the beginning.