Oracle Cloud Infrastructure Documentation

Network Setup for Exadata DB Systems

Before you set up an Exadata DB system, you must set up a virtual cloud network (VCN) and other Networking service components. This topic describes the recommended configuration for the VCN and several related requirements for the Exadata DB system.

VCN and Subnets

To launch an Exadata DB system, you must have:

  • A VCN in the region where you want the DB system
  • At least two subnets in the VCN. The two subnets are:

    • Client subnet
    • Backup subnet

In general, Oracle recommends using A subnet that spans all availability domains (ADs) in the region. Oracle recommends using regional subnets because they are more flexible and make it easier to implement failover across ADs. Compare with AD-specific subnets., which span all One or more isolated, fault-tolerant Oracle data centers that host cloud resources such as instances, volumes, and subnets. A region contains one or more availability domains. in the region. If you instead use A subnet that is specific to a particular availability domain (AD). Historically all subnets were AD-specific. Compare with regional subnets, which Oracle recommends over AD-specific subnets., both the client and backup subnets must be in the same availability domain. The important thing to know for your DB system is that the resources you create in the two subnets must be in the same availability domain. For more information, see About Regional Subnets.

You will create custom Virtual route table for your VCN that provides mapping for the traffic from subnets via gateways to external destinations.and A list of virtual firewall rules for your VCN. Security lists consist of security rules that apply to traffic coming in and out of a VNIC. Security lists are configured at the subnet level, which means all VNICs in a given subnet are subject to the same security rules. for each subnet. More information follows about that.

Option 1: Public Client Subnet with Internet Gateway

This option can be useful when doing a proof-of-concept or development work. You can use this setup in production if you want to use an An optional virtual router that you can add to your VCN. It provides a path for network traffic between your VCN and the internet. with the VCN, or if you have services that run only on a public network and need access to the database. See the following diagram and description.

This image shows the network setup with a public client subnet.

You set up:

Important

See this known issue for information about configuring route rules with service gateway as the target on route tables associated with public subnets.

Option 2: Private Subnets

This option is recommended for a production system. Both subnets are private and cannot be reached from the internet. See the following diagram and description.

This image shows the network setup with a private client subnet.

You set up:

  • Subnets:

    • Private client subnet.
    • Private backup subnet.
  • Gateways for the VCN:

  • Route tables:

    • Custom route table for the private client subnet, with two rules:

      • A rule for the on-premises network's CIDR, and target = DRG.
      • A rule for the service CIDR label called All <region> Services in Oracle Services Network, and target = the service gateway. The Oracle Services Network is a conceptual network in Oracle Cloud Infrastructure that is reserved for Oracle services. The rule enables the client subnet to reach the regional Oracle YUM repo for OS updates. Also see Option 2: Service Gateway Access to Both Object Storage and YUM Repos.
      • A rule for 0.0.0.0/0, and target = NAT gateway.
    • Separate custom route table for the private backup subnet, with one rule:
      • The same rule as for the client subnet: for the service CIDR label called All <region> Services in Oracle Services Network, and target = the service gateway. This rule enables the backup subnet to reach the regional Object Storage for backups.
  • Security lists:

  • Static route on the DB system's compute nodes (to enable access to Object Storage by way of the backup subnet).

Requirements for IP Address Space

If you're setting up Exadata DB systems (and thus VCNs) in more than one region, make sure the IP address space of the VCNs does not overlap. This is important if you want to set up disaster recovery with Oracle Data Guard.

The two subnets you create for the Exadata DB system must not overlap with 192.168.128.0/20.

The following table lists the minimum required subnet sizes, depending on the Exadata rack size. For the client subnet, each node requires two IP addresses, and in addition, three addresses are reserved for Single Client Access Names (SCANs). For the backup subnet, each node requires one address.

Tip

The Networking service reserves three IP addresses in each subnet. Allocating a larger space for the subnet than the minimum required (for example, at least /25 instead of /28) can reduce the relative impact of those reserved addresses on the subnet's available space.

Rack Size Client Subnet: # Required IP Addresses Client Subnet: Minimum Size Backup Subnet: # Required IP Addresses Backup Subnet: Minimum Size
Quarter

(2 addresses * 2 nodes) + 3 for SCANs + 3 reserved in subnet = 10

 

/28 (16 IP addresses) (1 address * 2 nodes) + 3 reserved in subnet = 5 /29 (8 IP addresses)
Half (2 * 4 nodes) + 3 + 3 = 14 /28 (16 IP addresses) (1 * 4 nodes) + 3 = 7 /29 (8 IP addresses)
Full (2* 8 nodes) + 3 + 3 = 22 /27 (32 IP addresses) (1 * 8 nodes) + 3 = 11 /28 (16 IP addresses)

VCN Creation Wizard: Not for Production

The Networking section of the Console includes a handy wizard that creates a VCN along with related resources. It can be useful if you just want to try launching an instance. However, the wizard automatically chooses the address ranges and creates public subnets and an internet gateway. You may not want this for your production network, so Oracle recommends you create the VCN and other resources individually yourself instead of using the wizard.

DNS: Short Names for the VCN, Subnets, and DB System

For the nodes to communicate, the VCN must use the Internet and VCN Resolver. It enables hostname assignment to the nodes, and DNS resolution of those hostnames by resources in the VCN. It enables round robin resolution of the database's SCANs. It also enables resolution of important service endpoints required for backing up databases, patching, and updating the cloud tooling on an Exadata DB system. The Internet and VCN Resolver is the VCN's default choice for DNS in the VCN. For more information, see DNS in Your Virtual Cloud Network and also DHCP Options.

When you create the VCN, subnets, and Exadata, you must carefully set the following identifiers, which are related to DNS in the VCN:

  • VCN domain label
  • Subnet domain label
  • Hostname prefix for the Exadata DB system

These values make up the node's fully qualified domain name (FQDN):

<hostname_prefix>-######.<subnet_domain_label>.<vcn_domain_label>.oraclevcn.com

For example:

exacs-abcde1.clientpvtad1.acmevcniad.oraclevcn.com

In this example, you assign exacs as the hostname prefix when you create the Exadata DB system. The Database service automatically appends a hyphen and a five-letter string with the node number at the end. For example:

  • Node 1: exacs-abcde1.clientpvtad1.acmevcniad.oraclevcn.com
  • Node 2: exacs-abcde2.clientpvtad1.acmevcniad.oraclevcn.com
  • Node 3: exacs-abcde3.clientpvtad1.acmevcniad.oraclevcn.com
  • And so on

Requirements for the hostname prefix:

  • Maximum 12 characters
  • Cannot be the string localhost

Requirements for the VCN and subnet domain labels:

  • Recommended maximum: 14 characters each. The actual underlying requirement is a total of 28 characters across both domain labels (excluding the period between the labels). For example, both of these are acceptable: subnetad1.verylongvcnphx or verylongsubnetad1.vcnphx. For simplicity, the recommendation is 14 characters each.
  • No hyphens.
  • Recommended: include the region name in the VCN's domain label, and include the availability domain name in the subnet's domain label.

<12_chars_max>-######.<14_chars_max>.<14_chars_max>.oraclevcn.com

In general, the FQDN has a maximum total limit of 63 characters.

The preceding maximums are not enforced when you create the VCN and subnets. However, if the labels exceed the maximum, the Exadata deployment fails.

DNS: Between On-Premises Network and VCN

To enable the use of hostnames when on-premises hosts and VCN resources communicate with each other, you have two options:

  • Set up an instance in the VCN to be a custom DNS server. For an example of an implementation of this scenario with the Oracle Terraform provider, see Hybrid DNS Configuration.
  • Manage hostname resolution yourself manually.

Node Access to Object Storage: Static Route

Access to Oracle Cloud Infrastructure Object Storage is required for backing up databases, patching, and updating the cloud tooling on an Exadata DB system. Regardless of how you set up the VCN with that access (for example, with a service gateway), you must configure a static route to Object Storage on each of the compute nodes in the cluster. This is required because, by default, all traffic in an Exadata DB system is routed through the data network. You need the traffic destined for Object Storage to be routed instead through the backup interface (BONDETH1).

Important

You must configure a static route for Object Storage access on each compute node in an Exadata DB system. Otherwise, attempts to back up databases, patch, or update tooling on the system might fail.

Object Storage IP allocations
To configure a static route for Object Storage access

Service Gateway for the VCN

Your VCN needs access to both Object Storage for backups and Oracle YUM repos for OS updates.

Depending on whether you use option 1 or option 2 described previously, you use the service gateway in different ways. See the next two sections.

Option 1: Service Gateway Access Only to Object Storage
Option 2: Service Gateway Access to Both Object Storage and YUM Repos

Security Lists for the Subnets

Oracle recommends that you use at least two security lists with each of the subnets.

For the client subnet:

  • VCN's default security list (with all its default rules). This security list contains several basic rules that are necessary for the DB system (such as ingress SSH access to the DB system, and general egress from the DB system).
  • Custom security list specifically for the client subnet (see Rules for Client Subnet's Custom Security List).

For the backup subnet:

You must create the two custom security lists yourself. The VCN's default security list is automatically created when you create the VCN. When you create a subnet, you can choose which security lists the subnet will use, or you can choose the security lists after the subnet is created.

Warning

Do not remove the default egress rule from the default security list. If you do, make sure to instead include a replacement egress rule in the client subnet's security list, like so:

  • Stateless: No (all rules must be stateful)
  • Destination Type: CIDR
  • Destination CIDR: 0.0.0.0/0
  • IP Protocol: All

Rules for Client Subnet's Custom Security List

Create the following security list rules in the client subnet's custom security list and associate the security list with the client subnet.

Ingress rule 1: Allows ONS and FAN traffic from within the client subnet
Ingress rule 2: Allows SQL*NET traffic from within the client subnet

Important

The two preceding rules only cover connections initiated from within the client subnet. If you have a client that resides outside the VCN, Oracle recommends setting up two additional similar rules that instead have the Source CIDR set to the public IP address of the client.

The next four rules (two ingress, two egress) allow TCP and ICMP traffic inside the client subnet and enable the nodes to communicate with each other. If TCP connectivity fails across the nodes, the Exadata DB system fails to provision.

Ingress rule 3: Allows all TCP traffic inside the client subnet
Ingress rule 4: Allows all ICMP traffic inside the client subnet
Egress rule 1: Allows all TCP traffic inside the client subnet
Egress rule 2: Allows all ICMP traffic inside the client subnet

The next egress rule is redundant with the default egress rule in the default security list. It is optional but recommended in case the default security list's egress rule is inadvertently changed. The rule is important because it allows connections to the Oracle YUM repos.

Egress rule 3: Allows all egress traffic

Rule for Backup Subnet's Custom Security List

The following rule enables the DB system to communicate with Object Storage through the service gateway. It is redundant with the default egress rule in the default security list. It is optional but recommended in case the default security list's rules are inadvertently changed.

Put this rule in the backup subnet's custom security list and associate the security list with the backup subnet.

Egress rule: Allows access to Object Storage