Network Setup for Exadata DB Systems

Before you set up an Exadata DB system, you must set up a virtual cloud network (VCN) and other Networking service components. This topic describes the recommended configuration for the VCN and several related requirements for the Exadata DB system.

VCN and Subnets

To launch an Exadata DB system, you must have:

  • A VCN in the region where you want the DB system
  • At least two subnets in the VCN. The two subnets are:

    • Client subnet
    • Backup subnet

In general, Oracle recommends using regional subnets , which span all availability domains  in the region. If you instead use AD-specific subnets , both the client and backup subnets must be in the same availability domain. The important thing to know for your DB system is that the resources you create in the two subnets must be in the same availability domain. For more information, see Overview of VCNs and Subnets.

You will create custom route tables  for each subnet. You will also create security rules  to control traffic to and from the client network and backup network of the Exadata compute notes. More information follows about those items.

Option 1: Public Client Subnet with Internet Gateway

This option can be useful when doing a proof-of-concept or development work. You can use this setup in production if you want to use an internet gateway  with the VCN, or if you have services that run only on a public network and need access to the database. See the following diagram and description.

This image shows the network setup with a public client subnet.

You set up:

Important

See this known issue for information about configuring route rules with service gateway as the target on route tables associated with public subnets.

Option 2: Private Subnets

Oracle recommends this option for a production system. Both subnets are private and cannot be reached from the internet. See the following diagram and description.

This image shows the network setup with a private client subnet.

You set up:

  • Subnets:

    • Private client subnet.
    • Private backup subnet.
  • Gateways for the VCN:

  • Route tables:

    • Custom route table for the private client subnet, with two rules:

      • A rule for the on-premises network's CIDR, and target = DRG.
      • A rule for the service CIDR label called All <region> Services in Oracle Services Network, and target = the service gateway. The Oracle Services Network is a conceptual network in Oracle Cloud Infrastructure that is reserved for Oracle services. The rule enables the client subnet to reach the regional Oracle YUM repo for OS updates. Also see Option 2: Service Gateway Access to Both Object Storage and YUM Repos.
      • A rule for 0.0.0.0/0, and target = NAT gateway.
    • Separate custom route table for the private backup subnet, with one rule:
      • The same rule as for the client subnet: for the service CIDR label called All <region> Services in Oracle Services Network, and target = the service gateway. This rule enables the backup subnet to reach the regional Object Storage for backups.
  • Security rules to enable the desired traffic to and from the Exadata nodes. See Security Rules for the Exadata System.
  • Static route on the DB system's compute nodes (to enable access to Object Storage by way of the backup subnet).

Requirements for IP Address Space

If you're setting up Exadata DB systems (and thus VCNs) in more than one region, make sure the IP address space of the VCNs does not overlap. This is important if you want to set up disaster recovery with Oracle Data Guard.

The two subnets you create for the Exadata DB system must not overlap with 192.168.128.0/20.

The following table lists the minimum required subnet sizes, depending on the Exadata rack size. For the client subnet, each node requires two IP addresses, and in addition, three addresses are reserved for Single Client Access Names (SCANs). For the backup subnet, each node requires one address.

Tip

The Networking service reserves three IP addresses in each subnet. Allocating a larger space for the subnet than the minimum required (for example, at least /25 instead of /28) can reduce the relative impact of those reserved addresses on the subnet's available space.
Rack Size Client Subnet: # Required IP Addresses Client Subnet: Minimum Size Backup Subnet: # Required IP Addresses Backup Subnet: Minimum Size
Base System or Quarter Rack

(2 addresses * 2 nodes) + 3 for SCANs + 3 reserved in subnet = 10

 
/28 (16 IP addresses) (1 address * 2 nodes) + 3 reserved in subnet = 5 /29 (8 IP addresses)
Half Rack (2 * 4 nodes) + 3 + 3 = 14 /28 (16 IP addresses) (1 * 4 nodes) + 3 = 7 /29 (8 IP addresses)
Full Rack (2* 8 nodes) + 3 + 3 = 22 /27 (32 IP addresses) (1 * 8 nodes) + 3 = 11 /28 (16 IP addresses)

VCN Creation Wizard: Not for Production

The Networking section of the Console includes a handy wizard that creates a VCN along with related resources. It can be useful if you just want to try launching an instance. However, the wizard automatically creates a public subnet and an internet gateway. You may not want this for your production network, so Oracle recommends you create the VCN and other resources individually yourself instead of using the wizard.

DNS: Short Names for the VCN, Subnets, and DB System

For the nodes to communicate, the VCN must use the Internet and VCN Resolver. It enables hostname assignment to the nodes, and DNS resolution of those hostnames by resources in the VCN. It enables round robin resolution of the database's SCANs. It also enables resolution of important service endpoints required for backing up databases, patching, and updating the cloud tooling on an Exadata DB system. The Internet and VCN Resolver is the VCN's default choice for DNS in the VCN. For more information, see DNS in Your Virtual Cloud Network and also DHCP Options.

When you create the VCN, subnets, and Exadata, you must carefully set the following identifiers, which are related to DNS in the VCN:

  • VCN domain label
  • Subnet domain label
  • Hostname prefix for the Exadata DB system

These values make up the node's fully qualified domain name (FQDN):

<hostname_prefix>-######.<subnet_domain_label>.<vcn_domain_label>.oraclevcn.com

For example:

exacs-abcde1.clientpvtad1.acmevcniad.oraclevcn.com

In this example, you assign exacs as the hostname prefix when you create the Exadata DB system. The Database service automatically appends a hyphen and a five-letter string with the node number at the end. For example:

  • Node 1: exacs-abcde1.clientpvtad1.acmevcniad.oraclevcn.com
  • Node 2: exacs-abcde2.clientpvtad1.acmevcniad.oraclevcn.com
  • Node 3: exacs-abcde3.clientpvtad1.acmevcniad.oraclevcn.com
  • And so on

Requirements for the hostname prefix:

  • Maximum 12 characters
  • Cannot be the string localhost

Requirements for the VCN and subnet domain labels:

  • Recommended maximum: 14 characters each. The actual underlying requirement is a total of 28 characters across both domain labels (excluding the period between the labels). For example, both of these are acceptable: subnetad1.verylongvcnphx or verylongsubnetad1.vcnphx. For simplicity, the recommendation is 14 characters each.
  • No hyphens or underscores.
  • Recommended: include the region name in the VCN's domain label, and include the availability domain name in the subnet's domain label.

<12_chars_max>-######.<14_chars_max>.<14_chars_max>.oraclevcn.com

In general, the FQDN has a maximum total limit of 63 characters.

The preceding maximums are not enforced when you create the VCN and subnets. However, if the labels exceed the maximum, the Exadata deployment fails.

DNS: Between On-Premises Network and VCN

To enable the use of hostnames when on-premises hosts and VCN resources communicate with each other, you have two options:
  • Set up an instance in the VCN to be a custom DNS server. For an example of an implementation of this scenario with the Oracle Terraform provider, see Hybrid DNS Configuration.
  • Manage hostname resolution yourself manually.

Node Access to Object Storage: Static Route

To be able to back up databases, and patch and update cloud tools on an Exadata DB system, you must configure access to Oracle Cloud Infrastructure Object Storage. Regardless of how you configure the VCN with that access (for example, with a service gateway), you may also need to configure a static route to Object Storage on each of the compute nodes in the cluster. This is only required if you are not using automatic backups. If you are using customized backups using the backup APIs, then you must route traffic destined for Object Storage through the backup interface (BONDETH1). This is not necessary if you are using the automatic backups created with the Console, APIs, or CLIs.

Important

You must configure a static route for Object Storage access on each compute node in an Exadata DB system if you are not creating automatic backups with the Console, APIs, or CLIs. Otherwise, attempts to back up databases, and patch or update tools on the system, can fail.
Object Storage IP allocations

Oracle Cloud Infrastructure Object Storage uses the CIDR block IP range 134.70.0.0/17 for all regions. This range was introduced in April and May of 2018.

As of June 1, 2018, Object Storage no longer supports the following discontinued IP ranges. Oracle recommends that you remove these older IP addresses from your access-control lists, firewall rules, and other rules after you have adopted the new IP ranges.

The discontinued IP ranges are:

  • Germany Central (Frankfurt): 130.61.0.0/16
  • UK South (London): 132.145.0.0/16
  • US East (Ashburn): 129.213.0.0/16
  • US West (Phoenix): 129.146.0.0/16
To configure a static route for Object Storage access
  1. SSH to a compute node in the Exadata DB system.

    ssh -i <private_key_path> opc@<node_ip_address>
  2. Log in as opc and then sudo to the root user. Use sudo su - with a hyphen to invoke the root user's profile.

    
    login as: opc
    			
    [opc@dbsys ~]$ sudo su - 
    
  3. Identify the gateway configured for the BONDETH1 interface.

    [root@dbsys ~]# grep GATEWAY /etc/sysconfig/network-scripts/ifcfg-bondeth1 |awk -F"=" '{print $2}'
    
    
    10.0.4.1
  4. Add the following static rule for BONDETH1 to the /etc/sysconfig/network-scripts/route-bondeth1 file:

    10.0.X.0/XX dev bondeth1 table 211
    default via <gateway> dev bondeth1 table 211
    134.70.0.0/17 via <gateway_from_previous_step> dev bondeth1
  5. Restart the interface.

    
    [root@dbsys ~]# ifdown bondeth1; ifup bondeth1;

    The file changes from the previous step take effect immediately after the ifdown and ifup commands run.

  6. Repeat the preceding steps on each compute node in the Exadata DB system.

Service Gateway for the VCN

Your VCN needs access to both Object Storage for backups and Oracle YUM repos for OS updates.

Depending on whether you use option 1 or option 2 described previously, you use the service gateway in different ways. See the next two sections.

Option 1: Service Gateway Access Only to Object Storage

You configure the backup subnet to use the service gateway for access only to Object Storage. As a reminder, here's the diagram for option 1:

This image shows the network setup with a public client subnet.

In general, you must:

  • Perform the tasks for setting up a service gateway on a VCN, and specifically enable the service CIDR label called OCI <region> Object Storage.
  • In the task for updating routing, add a route rule to the backup subnet's custom route table. For the destination service, use OCI <region> Object Storage and target = the service gateway.
  • In the task for updating security rules in the subnet, perform the task on the backup network's network security group (NSG) or custom security list. Set up a security rule with the destination service set to OCI <region> Object Storage. See Rule Required Specifically for the Backup Network.
Option 2: Service Gateway Access to Both Object Storage and YUM Repos

You configure both the client subnet and backup subnet to use the service gateway for access to the Oracle Services Network, which includes both Object Storage and the Oracle YUM repos.

Important

See this known issue for information about accessing Oracle YUM services through the service gateway.

As a reminder, here's the diagram for option 2:

This image shows the network setup with a private client subnet.

In general, you must:

  • Perform the tasks for setting up a service gateway on a VCN, and specifically enable the service CIDR label called All <region> Services in Oracle Services Network.
  • In the task for updating routing in each subnet, add a rule to each subnet's custom route table. For the destination service, use All <region> Services in Oracle Services Network and target = the service gateway.
  • In the task for updating security rules for the subnet, perform the task on the backup network's network security group (NSG) or custom security list. Set up a security rule with the destination service set to OCI <region> Object Storage. See Rule Required Specifically for the Backup Network. Note that the client subnet already has a broad egress rule that covers access to the YUM repos.

Here are a few additional details about using the service gateway for option 2:

  • Both the client subnet and backup subnet use the service gateway, but to access different services. You cannot enable both the OCI <region> Object Storage service CIDR label and the All <region> Services in Oracle Services Network for the service gateway. To cover the needs of both subnets, you must enable All <region> Services in Oracle Services Network for the service gateway. The VCN can have only a single service gateway.
  • Any route rule that targets a given service gateway must use an enabled service CIDR label and not a CIDR block as the destination for the rule. That means for option 2, the route tables for both subnets must use All <region> Services in Oracle Services Network for their service gateway rules.
  • Unlike route rules, security rules can use either any service CIDR label (whether the VCN has a service gateway or not) or a CIDR block as the source or destination CIDR for the rule. Therefore, although the backup subnet has a route rule that uses All <region> Services in Oracle Services Network, the subnet can have a security rule that uses OCI <region> Object Storage. See Rule Required Specifically for the Backup Network.

Security Rules for the Exadata System

This section lists the security rules to use with your Exadata system. Security rules control the types of traffic allowed for the client network and backup network of the Exadata's compute nodes. The rules are divided into three sections.

There are different ways to implement these rules. For more information, see Ways to Implement the Security Rules.

Rules Required for Both the Client Network and Backup Network

This section has several general rules that enable essential connectivity for hosts in the VCN.

If you use security lists to implement your security rules, be aware that the rules that follow are included by default in the default security list. Update or replace the list to meet your particular security needs. The two ICMP rules (general ingress rules 2 and 3) are required for proper functioning of network traffic within the Oracle Cloud Infrastructure environment. Adjust the general ingress rule 1 (the SSH rule) and the general egress rule 1 to allow traffic only to and from hosts that require communication with resources in your VCN.

General ingress rule 1: Allows SSH traffic from anywhere
  • Stateless: No (all rules must be stateful)
  • Source Type: CIDR
  • Source CIDR: 0.0.0.0/0
  • IP Protocol: TCP
  • Source Port Range: All
  • Destination Port Range: 22
General ingress rule 2: Allows Path MTU Discovery fragmentation messages

This rule enables hosts in the VCN to receive Path MTU Discovery fragmentation messages. Without access to these messages, hosts in the VCN can have problems communicating with hosts outside the VCN.

  • Stateless: No (all rules must be stateful)
  • Source Type: CIDR
  • Source CIDR: 0.0.0.0/0
  • IP Protocol: ICMP
  • Type: 3
  • Code: 4
General ingress rule 3: Allows connectivity error messages within the VCN

This rule enables the hosts in the VCN to receive connectivity error messages from each other.

  • Stateless: No (all rules must be stateful)
  • Source Type: CIDR
  • Source CIDR: Your VCN's CIDR
  • IP Protocol: ICMP
  • Type: 3
  • Code: All
General egress rule 1: Allows all egress traffic
  • Stateless: No (all rules must be stateful)
  • Destination Type: CIDR
  • Destination CIDR: 0.0.0.0/0
  • IP Protocol: All

Rules Required Specifically for the Client Network

The following security rules are important for the client network.

Important

  • Client ingress rules 1 and 2 only cover connections initiated from within the client subnet. If you have a client that resides outside the VCN, Oracle recommends setting up two additional similar rules that instead have the Source CIDR set to the public IP address of the client.
  • Client ingress rules 3 and 4 and client egress rules 1 and 2 allow TCP and ICMP traffic inside the client network and enable the nodes to communicate with each other. If TCP connectivity fails across the nodes, the Exadata DB system fails to provision.
Client ingress rule 1: Allows ONS and FAN traffic from within the client subnet

The first rule is recommended and enables the Oracle Notification Services (ONS) to communicate about Fast Application Notification (FAN) events.

  • Stateless: No (all rules must be stateful)
  • Source Type: CIDR
  • Source CIDR: Client subnet's CIDR
  • IP Protocol: TCP
  • Source Port Range: All
  • Destination Port Range: 6200
  • Description: An optional description of the rule.
Client ingress rule 2: Allows SQL*NET traffic from within the client subnet

This rule is for SQL*NET traffic and is required in these cases:

  • If you need to enable client connections to the database
  • If you plan to use Oracle Data Guard
  • Stateless: No (all rules must be stateful)
  • Source Type: CIDR
  • Source CIDR: Client subnet's CIDR
  • IP Protocol: TCP
  • Source Port Range: All
  • Destination Port Range: 1521
  • Description: An optional description of the rule.
Client ingress rule 3: Allows all TCP traffic inside the client subnet
  • Stateless: No (all rules must be stateful)
  • Source Type: CIDR
  • Source CIDR: Client subnet's CIDR
  • IP Protocol: TCP
  • Source Port Range: All
  • Destination Port Range: All
  • Description: An optional description of the rule.
Client ingress rule 4: Allows all ICMP traffic inside the client subnet
  • Stateless: No (all rules must be stateful)
  • Source Type: CIDR
  • Source CIDR: Client subnet's CIDR
  • IP Protocol: ICMP
  • Type and Code: All
  • Description: An optional description of the rule.
Client egress rule 1: Allows all TCP traffic inside the client subnet
  • Stateless: No (all rules must be stateful)
  • Destination Type: CIDR
  • Destination CIDR: Client subnet's CIDR
  • IP Protocol: TCP
  • Source Port Range: All
  • Destination Port Range: All
  • Description: An optional description of the rule.
Client egress rule 2: Allows all ICMP traffic inside the client subnet
  • Stateless: No (all rules must be stateful)
  • Destination Type: CIDR
  • Destination CIDR: Client subnet's CIDR
  • IP Protocol: ICMP
  • Type and Code: All
  • Description: An optional description of the rule.
Client egress rule 3: Allows all egress traffic (allows connections to the Oracle YUM repos)

Client egress rule 3 is important because it allows connections to the Oracle YUM repos. It is redundant with the general egress rule in Security Rules for the Exadata System (and in the default security list). It is optional but recommended in case the general egress rule (or default security list) is inadvertently changed.

  • Stateless: No (all rules must be stateful)
  • Destination Type: CIDR
  • Destination CIDR: 0.0.0.0/0
  • IP Protocol: All
  • Description: An optional description of the rule.

Rule Required Specifically for the Backup Network

The following security rule is important for the backup network because it enables the DB system to communicate with Object Storage through the service gateway (and optionally with the Oracle YUM repos if the client network doesn't have access to them). It is redundant with the general egress rule in Security Rules for the Exadata System (and in the default security list). It is optional but recommended in case the general egress rule (or default security list) is inadvertently changed.

Backup egress rule: Allows access to Object Storage
  • Stateless: No (all rules must be stateful)
  • Destination Type: Service
  • Destination Service:

    • The service CIDR label called OCI <region> Object Storage
    • If the client network does not have access to the Oracle YUM repos, use the service CIDR label called All <region> Services in Oracle Services Network
  • IP Protocol: TCP
  • Source Port Range: All
  • Destination Port Range: 443 (HTTPS)
  • Description: An optional description of the rule.

Ways to Implement the Security Rules

The Networking service offers two ways to implement security rules within your VCN:

For a comparison of the two methods, see Comparison of Security Lists and Network Security Groups.

If you use network security groups

If you choose to use network security groups (NSGs), here is the recommended process:

  1. Create an NSG for the client network. Add the following security rules to that NSG:

  2. Create a separate NSG for the backup network. Add the following security rules to that NSG:

  3. When the database administrator creates the Exadata DB system, they must choose several networking components (for example, which VCN and subnets to use):
    • When they choose the client subnet, they can also choose which NSG or NSGs to use. Make sure they choose the client network's NSG.
    • When they choose the backup subnet, they can also choose which NSG or NSGs to use. Make sure they choose the backup network's NSG.

You could instead create a separate NSG for the general rules. Then when the database administrator chooses which NSGs to use for the client network, make sure they choose both the general NSG and the client network NSG. Similarly for the backup network, they choose both the general NSG and the backup network NSG.

If you use security lists

If you choose to use security lists, here is the recommended process:

  1. Configure the client subnet to use the required security rules:

    1. Create a custom security list for the client subnet and add the rules listed in Rules Required Specifically for the Client Network.
    2. Associate the following two security lists with the client subnet:

  2. Configure the backup subnet to use the required security rules:

    1. Create a custom security list for the backup subnet and add the rules listed in Rule Required Specifically for the Backup Network.
    2. Associate the following two security lists with the backup subnet:

Later when the database administrator creates the Exadata DB system, they must choose several networking components. When they select the client subnet and backup subnet that you've already created and configured, the security rules are automatically enforced for the nodes created in those subnets.

Warning

Do not remove the default egress rule from the default security list. If you do, make sure to instead include the following replacement egress rule in the client subnet's security list:

  • Stateless: No (all rules must be stateful)
  • Destination Type: CIDR
  • Destination CIDR: 0.0.0.0/0
  • IP Protocol: All