Oracle Cloud Infrastructure Documentation

Troubleshooting

This topic describes various troubleshooting issues related to the Data Transfer Service.

Troubleshooting the Appliance

You can generate performance information for troubleshooting issues with the appliance through the terminal emulator on the host machine. Select Collect Appliance Diagnostic Information from the serial console configuration menu. The diagnostic tool generates system, network, storage, and performance data while the transfer job is running. It then forwards the data to the appliance serial console. Here you can scroll through the terminal to view it.

You can also use the log capture feature of the serial port emulator to capture the output. Serial port emulators often support the ability to copy the session to a file. Refer to the documentation of your serial port emulation package for instructions. Copying to a log file is useful if you need assistance from Oracle or if your emulation session does not allow you to scroll back and see all the output.

For each operation, the display shows exactly what command was executed and all the options.

Here is an example of the diagnostic output:

--------------------------------------------------------------------------------
- systemctl -l --type service --state=active                                   -
--------------------------------------------------------------------------------
UNIT LOAD ACTIVE SUB DESCRIPTION
auditd.service loaded active running Security Auditing Service
blk-availability.service loaded active exited Availability of block devices
chronyd.service loaded active running NTP client/server
console-diags@39-3147-1001.service loaded active running Diagnostic Collection Server for the XA (PID 3147/UID 1001)
crond.service loaded active running Command Scheduler
data-transfer-appliance.service loaded active running Data Transfer Appliance
data-transfer-console.service loaded active running Data Transfer Serial Console

Any problem with the diagnostic data collection results in the console output being written to the log file of the service. Failure of the commands is indicative of a serious problem, perhaps requiring the return of the appliance. Here is an example of the log:

Mar 6 17:55:33 localhost console-diags: {"Module": "main", "Type": "Info", "Message": "Received message {\"cmd\": \"collect\"}"}
Mar 6 17:55:33 localhost console-diags: {"Module": "main", "Type": "Info", "Message": "Setting up output file. First to remove all /tmp/xa-diags-results"}
Mar 6 17:55:33 localhost console-diags: {"Module": "main", "Type": "Info", "Message": "Removing /tmp/xa-diags-results.2019-03-06T17:54:56.000471"}

Initializing Appliance Fails Because of IP Address Issues

Initializing the Appliance can fail because of using the incorrect IP address. The IP address for initialize-auth can differ from the IP address obtained when running ping or ssl connect. If you experience an initialization failure, ensure that you are using the correct IP address for your Appliance and try initializing again.

Dataset Sealing Process Fails

The dataset sealing process can fail sometimes because there are special files in the dataset:

dts nfs-dataset seal-status --name nfs-ds-1

Seal Status :
success           : false
failureReason :
Number of special files : 5
startTime : 2019/03/26 11:52:37 PDT
endTime : 2019/03/26 11:52:39 PDT
numFilesToProcess : 0
numFilesProcessed : 0
bytesToProcess : 0.00 KB
bytesProcessed : 0.00 KB
bytesToProcess : 0.00 KB

At the command prompt on the host, reactivate the NFS dataset.

oci dts nfs-dataset activate --name <dataset_name>

Then run find to get the full list of all special files and the specific type of each one.

find <mountpoint> \! -type f \! -type d | xargs file

For example:

$ find /mnt/nfs-ds-1 \! -type f \! -type d | xargs file
/mnt/nfs-ds-1/myfile1: symbolic link to `/home/user1/myfile1'
/mnt/nfs-ds-1/myfile2: symbolic link to `/home/user1/myfile2'

Next, review the list and remove all special files from the NFS mount point.

find <mountpoint> \! -type f \! -type d | xargs rm

Deactivate the NFS dataset.

oci dts nfs-dataset deactivate --name <dataset_name>

Finally, reseal the dataset.

oci dts nfs-dataset seal --name <dataset_name>[--wait]

Monitor the seal progress. Wait for it to complete successfully and continue with the subsequent steps.

Initialize Authentication Fails with "connection refused" or "connection timed out"

If you try to configure networking using the appliance serial console but fail with a "connection refused" or "connection timed out" message, follow these troubleshooting steps.

Run the following command at the command prompt on the host:

ping <appliance_ip>

If a failure occurs, run the following command to verify appliance IP and the path to appliance.

ping -I <local_interface> <appliance_ip>

To determine expected interface, run ip route or an equivalent command. Verify that routing table is sane. Try running traceroute if you're not sure to see the network path to the appliance IP.

Run the following command:

curl -k https://<appliance_ip>

You should receive the response "Not found." This failure can indicate the IP address may be wrong. For example, nothing is listening on port 443.If you receive a failure message, run the following command:

openssl s_client -showcerts -connect <appliance_ip>:443

You should see a certificate issued for "Oracle Cloud Infrastructure" / "Data Transfer Appliance."

This command is similar to curl but does not use HTTPS and so proxies do not affect it. If this command works, and curl fails, then verify there are no proxy environmental variables.

Data Transfer Utility Fails with "invalid configuration file"

If you attempt to run Data Transfer commands and receive the error message "invalid configuration file," verify that the following files are present on your host and are correctly set up:

  • ~/.oci/config and
  • ~/.oci/config_upload_user

Both files must have "[DEFAULT]" as the first line. Use of the "~" character in a path is not valid in the file's contents.

Data Transfer Utility Fails with "Processing exception..." while communicating to Oracle Cloud Infrastructure

Check if your environment has proxies to the internet. If so, update them to the latest version and set "https_proxy." If you are using the appliance, set "no_proxy" environmental variables. See Prerequisites for more information on proxies.

Data Transfer Utility Fails Because of Lack of Exclusive Access to Disk

The Data Transfer Utility requires exclusive access to the disk. If you have any drivers that already claim exclusive access to the disk, then the Data Transfer Utility fails. For example, if you employ a devicemapper multipath driver over all your disk devices, you must first remove the disk used for the data transfer from the list of devices managed by the multipath driver.

Be sure that access to the disk is not done through any devicemapper or volume manager. During the data transfer, the expectation is that the file system is created on a "raw" device. Any layering or mapping through intermediate drivers or abstraction layers makes it impossible for the disk to be uploaded at the transfer site. The source of these failures can include drivers like multipath, md, striping, logical volume managers, and potentially others as well.

You can confirm that the Data Transfer Utility has exclusive access by attempting to manually format the disk being used for your data transfer. The Data Transfer Utility uses the cryptsetup utility to create an encrypted device. You can run cryptsetup from the command line (root privileges required):

cryptsetup luksFormat -c aes-xts-plain64 -s 512 -h sha512 --iter-time 2000 --use random /dev/<sdXX>

<sdXX> is the name of the disk being used for the data transfer.

When prompted, respond that you do want to encrypt the device. You are required to provide a passphrase. Any passphrase is acceptable as the cryptsetup utility can run on a disk repeatedly without any problems.

If the command succeeds, then you know that the Data Transfer Utility can gain exclusive ownership of the disk to do the necessary for the data transfer.