Oracle Cloud Infrastructure Documentation

Understanding Storage Gateway Performance

This topic covers the performance characteristics of Storage Gateway and the ways in which you can maximize performance.

Performance Characteristics

It is important to understand the basic performance characteristics of Storage Gateway:

  • Because there is transactional overhead for each file, Storage Gateway generally exhibits better performance with large files than with small files. Storage Gateway can only upload data as fast as your connection and the storage host allows. Storage Gateway buffers the data in local disk storage while waiting to upload to Oracle Cloud Infrastructure. Once a file is uploaded, the file's local copy can be removed from the cache to free up space. If file system cache space falls below 10 GB, application I/O and file/directory creation can fail.
  • Modifying a file involves uploading a whole new copy of that file. This behavior is not performant with frequently modified large files.
  • Storage Gateway does not support frequently modified files such as log, database, or virtual disks. Storage Gateway depends on the closing of a file to trigger an upload of that file. If a file never closes or is modified frequently, the upload event cannot successfully occur.
  • Upload throughput to Object Storage (WAN) is typically slower than NFS client throughput (LAN). As a result, Storage Gateway can accumulate a large amount of data that is pending upload.
  • You can attach Storage Gateway to an existing Object Storage bucket that contains data. The service is optimized for this type of initialization. Storage Gateway can initialize about 700 thousand files per hour with a hard drive based cache or about 7 million files per hour with an NVMe SSD based cache.

Factors That Affect Performance

Follow the documented best practices to get the maximum performance benefits from Storage Gateway: Best Practices for Using Storage Gateway.

In addition to having sufficient memory and file system cache space, Oracle recommends that you use SSDs to help improve NFS ingestion rate.

Storage Gateway is tuned for maximum upload and download performance by default. No additional tuning is needed.

Performance Testing

While measuring performance is complex and open to variability, we have observed the following in performance benchmark tests with 10-Gb/s link speed:

Upload Speeds

  • Single-file upload: 100–120 MB/s (using a 1.3-TB file)
  • Multiple-file upload: 450–500 MB/s (using 4–8 files ranging in size from 10 GB to 50 GB)

Download Speeds

  • Single-file download: 300–350 MB/s (using a 1.3-TB file)
  • Multiple-files download: 700–750 MB/s (using 3 files ranging in size from 50 GB to 200 GB)
Note

Using FastConnect with Storage Gateway makes optimal use of full link speed. We have customers using FastConnect with 10-Gb/s link speed and have seen each gateway achieve 400–450 MB/s uploads to Oracle Cloud Infrastructure.

See FastConnect Overview for more information.

Testing Network Bandwidth

Storage Gateway provides a diagnostic command that you can use to test the bandwidth in your environment and ensure that you are getting the expected upload and download speeds. The amount of data transferred depends on these factors:

Bandwidth * Delay Product (bits) = total_available_bandwidth (bits/sec) x round_trip_time (sec)

Note

Different buckets can have different upload and download speeds.

The round_trip_time can vary by region.

To run the diagnostic command:

You need root permissions to run the diag command.

  1. Using SSH, log in to the host on which you installed Storage Gateway.
  2. Run the diag command, specifying the Storage Gateway file system name:

    [root@ocisg-ashburn opc]# sudo docker exec ocisg cat /mnt/gateway/<file_system_name>/:::diag:oci-network-speed-test

    The diag command responds with the average upload speed, for example:

    Average Upload Speed = 217 MB/s