Storage Gateway is a cloud storage gateway that lets you connect your on-premises applications with Oracle Cloud Infrastructure. Applications that can write data to an NFS target can also write data to the Oracle Cloud Infrastructure Object Storage, without requiring application modification to uptake the REST APIs.
Storage Gateway is the evolution of the Storage Software Appliance that was launched with Oracle Cloud Infrastructure Classic. Now that you’re migrating to Oracle Cloud Infrastructure Object Storage, you’ll use Storage Gateway, with its enhanced file-to-object transparency and improved scale and performance.
Storage Gateway and Oracle Cloud Infrastructure Concepts
The following summarizes key Storage Gateway and Oracle Cloud Infrastructure-related concepts.
- file system
- A Storage Gateway file system on a local host maps files and directories to objects with the same name in a corresponding Object Storage bucket in Oracle Cloud Infrastructure.
- file system cache
- Storage Gateway's configurable file system cache enables asynchronous and optimized movement of data to the cloud. The file system cache serves two roles for data storage and retrieval: a write buffer and a read cache. The write buffer contains data that has been copied to the disk cache and is queued to be uploaded to Oracle Cloud Infrastructure. The read cache contains frequently retrieved data that’s accessible locally for read operations.
- Proper file system cache configuration is critical to Storage Gateway performance. See Configuring the Cache for File Systems for details.
- The metadata associated with a Storage Gateway file is stored as custom metadata for the corresponding object in Oracle Cloud Infrastructure Object Storage. Examples of file metadata include: object id, creation date, modification date, size, and permissions. Storage Gateway caches all metadata for the file system locally.
- NFS is an established and widely adopted distributed file system protocol for uptaking network storage. NFS lets client computers mount file systems on remote servers and access those remote file systems over the network as though they were local file systems. Storage Gateway performs the NFS to REST API translation to interact with Oracle Cloud Infrastructure Object Storage.
- Oracle Cloud Infrastructure
- Oracle Cloud Infrastructure is a set of complementary cloud services that lets you build and run a wide range of applications and services in a highly available hosted environment. Oracle Cloud Infrastructure offers high-performance compute capabilities (as physical hardware instances) and storage capacity in a flexible overlay virtual network that is securely accessible from your on‑premise network.
- A tenancy is a secure and isolated partition within Oracle Cloud Infrastructure where you can create, organize, and administer your cloud resources.
- Oracle Cloud Infrastructure Object Storage and Archive Storage
- Oracle Cloud Infrastructure offers two distinct storage tiers for you to store your unstructured data. Use the Object Storage Standard tier for data to which you need fast, immediate, and frequent access. Use the Archive Storage Archive tier for data to which you seldom or rarely access, but that must be retained and preserved for long periods of time. Both storage tiers use the same manageable resources (for example, objects and buckets). The difference is that when you upload a file to Archive Storage, the object is immediately archived. Before you can access an archived object, you must first restore the object to the Standard tier.
- Either storage tier is simple to use, performs well, and scales to an unlimited capacity.
- An Object Storage bucket is a logical container for storing objects. A file system created in Storage Gateway maps to a corresponding bucket by the same name in Object Storage. A bucket is associated with a single Oracle Cloud Infrastructure A collection of related resources that can be accessed only by certain groups that have been given permission by an administrator in your organization. that has An IAM document that specifies who has what type of access to your resources. It is used in different ways: to mean an individual statement written in the policy language; to mean a collection of statements in a single, named "policy" document (which has an Oracle Cloud ID (OCID) assigned to it); and to mean the overall body of policies your organization uses to control access to resources. that determine what actions a user can perform on a bucket and on all the objects in the bucket.
- An individual file or directory written to an Storage Gateway file system on an NFS share, creates an identically named object in the target Object Storage bucket. An object is composed of the object itself and metadata about the object.
- A logical entity that serves as a top-level container for all Oracle Cloud Infrastructure Object Storage buckets and objects, allowing you to control bucket naming within your tenancy. Each tenancy is provided one unique and uneditable Object Storage namespace that is global, spanning all compartments and regions. Bucket names must be unique within your tenancy.
- A collection of Oracle Cloud Infrastructure-related resources that can be accesses only by users and groups who are explicitly granted access permission by an administrator. Compartments help you organize resources and make it easier to control the access to those resources. Object Storage automatically creates a root compartment when a compartment is provisioned. An administrator can then create more compartments within the root compartment and add access rules for those compartments. A bucket can only exist in one compartment.
To simplify this Storage Gateway documentation, we generically refer to Object Storage to mean that you can direct your applications to store data in a bucket in either the Standard or Archive storage tier.
How Storage Gateway Works
Storage Gateway is installed in an Oracle Cloud Infrastructure compute instance or as a Linux Docker instance on one or more hosts in your on-premises data center. Applications store and retrieve objects from Oracle Cloud Infrastructure Object Storage through file systems that you create in Storage Gateway.
Storage Gateway exposes an NFS mount point that can be mounted to any host that supports an NFSv4 client. The Storage Gateway mount point maps to an Object Storage bucket.
There is file to object transparency between Storage Gateway and Object Storage:
- A Storage Gateway file system directory on a local host maps to a bucket with an identical name in Oracle Cloud Infrastructure Object Storage.
- Any file written to a Storage Gateway file system is written as an object with the identical name in the associated Object Storage bucket. Associated file attributes are stored as object metadata.
- You can access Object Storage objects directly using the native APIs, SDKs, third-party tools, the HDFS connector, and the Oracle Cloud Infrastructure CLI and Console. You then use the Refresh operation in Storage Gateway to ingest any data that was added or modified directly in Object Storage.
Enterprise applications typically work with files in nested directories. Within Object Storage, buckets and the objects in those buckets exist in a flat hierarchy. Storage Gateway flattens the directory hierarchy into nested object prefixes in Object Storage. See Interacting With Object Storage for details.
The following summarizes some of the ways that you can use Storage Gateway.
- data transfer
- Use Storage Gateway to move data from your on-premises data to Oracle Cloud Infrastructure. The Storage Gateway is not a replacement for general-purpose network attached storage (NAS), though it behaves similarly to NAS. Use the Storage Gateway integrated Cloud Sync feature to transfer and synchronize data to Oracle Cloud Infrastructure.
- cloud tiering
- Use Storage Gateway to expand the capacity of on-premises storage solutions without capital expenditures. Configuring and connecting a Storage Gateway file system with a large cache to Oracle Cloud Infrastructure Object Storage provides unlimited scale to create a workflow in which files get automatically moved to the cloud and only retrieved on demand. Even though on-demand retrieval is slower than access to local storage, capital expenditures or changes to existing tools and software is not required.
- Use Storage Gateway to move files to Oracle Cloud Infrastructure Archive Storage as a cost-effective backup solution. You can move individual files and compressed or uncompressed ZIP or TAR archives. Storing secondary copies of data is an ideal use case for Storage Gateway.
- Storage Gateway is ideal for archive use cases.
- disaster recovery
Storage Gateway lets traditional applications move data to a highly durable object storage. When there is a need to recover data, a new instance of Storage Gateway is created and data can be easily recovered.
Storage Gateway does not support the following uses and workloads.
- general-purpose network storage
- Storage Gateway isn't a general-purpose storage filer and must not be used as a replacement for traditional network storage appliances.
- file sync and share
- Though Storage Gateway is an effective data mover, it’s not a replacement for file sync and share services. Evaluate Oracle services like Oracle Document Cloud service if you need file sync and share functionality.
- content collaboration
- Storage Gateway does not support multiple Storage Gateway instances simultaneously reading from and writing to a single Object Storage bucket. Do not use Storage Gateway as a tool for distributed teams to collaborate on creating and managing content.
- frequently modified files
- Do not use Storage Gateway if you expect your data to be modified frequently. Each time a file is modified and closed, Storage Gateway creates a new version that is uploaded to Object Storage as a new object. Frequently modified data would result in substantial inefficiency, both in terms of consuming upload/download bandwidth and capacity utilization.
- renaming large directory trees
- Renaming directories in the Storage Gateway works well for a small directory tree. However, renaming a parent directory with many children can be very slow. The service updates the object ID of every corresponding child object in the object store to reflect the new path. If you do start a rename, ensure that the action finishes by monitoring the Pending Uploads field in the Storage Gateway user interface.
- admin password
- Because Storage Gateway administrators can create, modify, and delete file systems, follow these password guidelines:
- Set a strong password.
- Ensure that the password is secure.
- Share passwords with others only on a need-to-know basis.
- Storage Gateway runs inside a Docker container for security and isolation. Follow these Docker-related guidelines and recommendations:
- Avoid or minimize Docker instance operations.
- Avoid logging in to the Docker container. If there is a genuine requirement to log in to the Docker container, use extreme caution to avoid service disruption. Do not change the docker configuration or the docker instance unless instructed to do so by Oracle support personal.
- Although the NFS protocol controls access to the file system from clients, Storage Gateway file systems are also locally mounted inside docker container. To prevent unauthorized access to file system data, ensure that a docker container is accessible only by an administrator or an authorized user.
- Configure the Docker host to limit user access to the Storage Gateway docker container.
- Files and directories in a Docker container are also visible in the Docker host—typically file systems and directories that are provisioned in the Docker host and are mapped to the container. Set the appropriate ownership and modes to ensure that only an administrator or an authorized user can access these folders. We recommend the following:
- A dedicated Storage Gateway host
- Limit who can access the Storage Gateway host
- Set firewall rules to limit access to the Docker host and Docker container
- Implement backup and retention policies for the files associated with Storage Gateway.
- access control
- Default file system export options are too permissive. Set more restrictive export options so that only trusted NFS clients can access the file system data and metadata. Modify the advanced file system settings for NFS Allowed Hosts and NFS Export Options to restrict to access to a file system. In addition to NFS protocol security, you can also set up and configure a firewall on the host to further control access to the file system. UID/GID/modes control access to files and directories. Set the appropriate ownership mode to protect sensitive data.
- Object Storage
- Files in a file system are uploaded to Oracle Cloud Infrastructure and stored as objects in an Object Storage bucket. Associated file attributes are stored as object metadata. Access control for Object Storage is different from access control for a traditional file system. Anyone with permission to read or modify any object in the bucket can read/modify all objects in the bucket. To protect sensitive data, set up Oracle Cloud Infrastructure IAM policies to limit who can access objects in the bucket.
- Storage Gateway transfers data to Oracle Cloud Infrastructure using HTTPS, which encrypts data packets in flight between Storage Gateway and the cloud. Data written to Object Storage is always automatically encrypted in the cloud.
See Service Limits for a list of applicable limits and instructions for requesting a limit increase.
Other limits include:
- Ensure that the number of file systems per Storage Gateway doesn't exceed 10. For best performance, host each file system on a dedicated Storage Gateway.
- Ensure that the number of objects stored in an Storage Gateway file system doesn’t exceed 100 million. For datasets that consist of more than 100 million objects, distribute the objects across multiple Storage Gateways.
- Ensure that you configure adequate local storage for file system cache. Storage Gateway warns you if you have configured less than the recommended 500 GB.
The minimum amount of memory required for any Storage Gateway file system is 16 GB.
- For file systems with the number of files up to 50 million, the required amount of memory is 32 GB.
- For large file systems with the number of files up to 100 million, the required amount of memory is 64 GB.
- The number of files in cache is limited to 20,000, regardless of the specified cache size in bytes.
- To improve the efficiency of file ingest and cloud upload operations, and to reduce the number of objects in the namespace, bin-pack or zip small files before writing them to Storage Gateway.
Storage Gateway Release Notes
Release Notes provide version-specific release information and any important Storage Gateway issues that you need to be aware of: