Oracle Cloud Infrastructure Documentation

Understanding Storage Gateway Performance

This topic covers the performance characteristics of Storage Gateway and the ways in which you can maximize performance.

Performance Characteristics

It is important to understand the basic performance characteristics of Storage Gateway:

  • Because there is transactional overhead for each file, Storage Gateway generally exhibits better performance with large files than with small files. Storage Gateway can only upload data as fast as your connection and the storage host allows. Storage Gateway buffers the data in local disk storage while waiting to upload to Oracle Cloud Infrastructure. Storage Gateway throttles application I/O so that the local disk space is not consumed with upload activities. Once a file is uploaded, the file's local copy can be removed from the cache to free up space. If file system cache space falls below 10 GB, application I/O and file/directory creation can fail.
  • Modifying a file involves uploading a whole new copy of that file. This behavior is not performant with frequently modified large files.
  • Storage Gateway does not support frequently modified files such as log, database, or virtual disks. Storage Gateway depends on the closing of a file to trigger an upload of that file. If a file never closes or is modified frequently, the upload event cannot successfully occur.
  • Upload throughput to Object Storage (WAN) is typically lower than NFS client throughput (LAN). As a result, Storage Gateway might accumulate a large amount of data that is pending upload. Storage Gateway might throttle ingestion rate in an attempt to limit cache build-up.

Factors That Impact Performance

Follow the documented best practices to get the maximum performance benefits from Storage Gateway: Best Practices for Using Storage Gateway.

In addition to having sufficient memory and file system cache space, we recommend using SSDs to help improve NFS ingestion rate.

Storage Gateway is tuned for maximum upload and download performance by default. No additional tuning is needed.

Performance Testing

While measuring performance is complex and open to variability, we have observed the following in performance benchmark tests with 10 Gb/s link speed:

Upload Speeds

Single File Upload – 100 - 120 MB/s (using a 1.3 TB file)

Multiple File Upload - 450 - 500 MB/s (using 4 to 8 files ranging from 10 GB to 50 GB in size)

Download Speeds

Single File Download – 300 - 350 MB/s (using a 1.3 TB file)

Multiple File Download - 700 - 750 MB/s (using 3 files ranging from 50 GB to 200 GB in size)

Note

Using FastConnect in conjunction with Storage Gateway makes optimal use of full link speed. We have customers using FastConnect with 10Gb/s link speed and have seen each gateway achieve 400 - 450 MB/s uploads to Oracle Cloud Infrastructure.

See FastConnect Overview for more information.

Testing Network Bandwidth

Storage Gateway provides a diagnostic command that you can use to test the bandwidth in your environment and ensure that you are getting the expected upload and download speeds. The amount of data transferred depends on these factors:

Bandwidth * Delay Product (bits) = total_available_bandwidth (bits/sec) x round_trip_time (sec)

Note

Different buckets will have different upload and download speeds.

The round_trip_time can vary by region.

To run the diagnostic command:

You need root permissions to run the diag command.

  1. Using ssh, log in to the host on which you installed Storage Gateway.
  2. Run the diag command, specifying the Storage Gateway file system name:

    [root@ocisg-ashburn opc]# sudo docker exec ocisg cat /mnt/gateway/<file_system_name>/:::diag:oci-network-speed-test

    The diag command responds with the average upload speed, for example:

    Average Upload Speed = 217 MB/s