Dynamic Routing Gateways

This topic describes how to manage a dynamic routing gateway (DRG) . This topic uses the terms dynamic routing gateway and DRG interchangeably. The Console uses the term Dynamic Routing Gateway, whereas for brevity the API uses DRG.

A DRG is a virtual router to which you can attach the following resources:

A DRG can have multiple network attachments of each of the following types:

  • VCN attachments: you can attach multiple VCNs to a single DRG. Each VCN can be in the same tenancy as the DRG, or in a different tenancy (provided appropriate policies are set).
  • RPC attachments: you can peer a DRG to other DRGs using remote peering connections. The other DRG can be in other regions or tenancies, or in the same region.
  • IPSEC_TUNNEL attachments: you can use Site-to-Site VPN to attach two or more IPSec tunnels to your DRG to connect to on-premises networks. This is also allowed across tenancies.
  • VIRTUAL_CIRCUIT attachments: you can attach one or more FastConnect virtual circuits to your DRG to connect to on-premises networks.
  • LOOPBACK attachments: you can use Site-to-Site VPN to encrypt FastConnect virtual circuits. See Loopback Attachments for details.

Creating DRG route tables and DRG route distributions allows you to define routing policies that route traffic between attachments. Routes can be dynamically imported and exported through these attachments. A route table must be associated with an attachment for that table's configuration to be applied, but an unassociated routing table can exist. DRG route distributions are of an explicit type (either import or export) and do not inherit an action that depends on where they are associated.

Overview of Dynamic Routing Gateways

A DRG acts as a virtual router, providing a path for traffic between your on-premises networks and VCNs, and can also be used to route traffic between VCNs. Using different types of attachments, custom network topologies can be constructed using components in different regions and tenancies. Each DRG attachment has an associated route table which is used to route packets entering the DRG to their next hop. In addition to static routes, routes from the attached networks are dynamically imported into DRG route tables using optional import route distributions.

Working with DRGs and DRG Attachments

Note

A DRG created before April 2021 isn't able to perform transit routing between on-premises networks and multiple VCNs, or provide peering between VCNs. If you require that functionality and you see an Upgrade DRG button in the console, click it. Upgrading the DRG resets all existing BGP sessions and temporarily interrupts traffic from the on-premises network. Once it starts, you can't roll back the upgrade. See Upgrading a DRG.

When creating a DRG, you must specify the compartment where you want the DRG to reside. Placing the DRG in a compartment helps to limit access control. If you're not sure which compartment to use, put the DRG in the same compartment as a VCN you use regularly. For more information, see Access Control.

You might optionally assign a friendly name to the DRG. It doesn't have to be unique, and you can change it later. Oracle automatically assigns the DRG a unique identifier called an Oracle Cloud ID (OCID). For more information, see Resource Identifiers.

To use a DRG, it must be attached to other network resources. In the API, the process of attaching creates a DrgAttachment object with its own OCID. The DrgAttachment has a type field which denotes the type of object being attached to the DRG. The type field can be set to one of the following values:

  • VCN
  • VIRTUAL_CIRCUIT
  • IPSEC_TUNNEL
  • REMOTE_PEERING_CONNECTION
  • LOOPBACK (See Loopback Attachments for more details.)

To attach a VCN to a DRG, use the CreateDrgAttachment operation or the console to explicitly create the DRG attachment object. Attachments for virtual circuits, IPSec tunnels, and remote peering connections are created (and deleted) automatically on your behalf when you create (or delete) the network object.

Working with DRG Route Tables and Route Distributions

A packet enters a DRG through an attachment and is routed using rules in the DRG route table assigned to that attachment. You can assign the same route table to multiple DRG attachments or create a dedicated route table for each attachment depending on the routing policies you want.

When you create a DRG, two default route tables are created for you: one for VCN attachments and one for all other attachments. When a route table is set as the default route table for an attachment type, the table is assigned to newly created attachments of that type unless an alternate table is explicitly specified. Route tables specified as the default for any type cannot be deleted. Ensure that a route table is not currently set as a default route table for an attachment type before trying to delete it.

A VCN attachment has two route tables: One DRG routing table for traffic entering the DRG, and one VCN routing table for traffic entering the VCN. The DRG route table exists in the DRG and is used to route packets entering the DRG through the attachment. The VCN route table is used to route packets entering the VCN through the attachment. If a VCN routing table is not defined, a hidden implicit table always provides connectivity to all subnets in the VCN.

Dynamic Route Import Distributions

A distribution is a list of declarative statements that contain a match criteria (such as an OCID or an attachment type) and an action. You can use route distributions to specify how routes get imported from or exported to a DRG attachment.

DRG route tables contain both static and dynamic routes. Static routes are inserted into tables using the API, while dynamic routes are imported from attachments and inserted using an import route distribution. When a statement's criteria matches on an attachment, the routes associated with the network object being attached to the DRG are dynamically imported into the DRG route table assigned to the containing distribution. If the statement is removed from the distribution, the routes are withdrawn from the DRG route tables. Statements in a route distribution are evaluated in priority order: the lowest number has the highest priority. The order in which statements are evaluated doesn't affect the preference set for the routes they import.

When building route distribution statements in the console, you can create a statement whose match type is "Match All". In the API, encode a "match all" statement by setting the match criteria to the empty list.

How do dynamic routes arrive at an attachment?

Routes to your on-premises networks are advertised from the CPE to IPSec tunnel and virtual circuit attachments using BGP. Routes are dynamically propagated from an RPC attachment on one DRG to an RPC attachment on another DRG through connected RPCs. Dynamic routes in a VCN include either all of the subnet CIDRs or all of the VCN CIDRs and all static route CIDRs configured on the VCN route table associated with the DRG attachment. The vcnRouteType property of the VCN attachment determines whether the subnet CIDRs or VCN CIDRs are propagated into the VCN attachment. The default behavior is to import the subnet CIDRs, but this behavior can be modified when creating or updating the VCN attachment.

Dynamic Route Export Distributions

When an attachment is assigned to a DRG route table, the contents of that table can be dynamically exported to the attachment. If the default export route distribution is assigned to an attachment, the entire contents of the attachment's assigned DRG route table are dynamically exported to the attachment. If you want to disable dynamic route exports to an attachment, use the API operation removeExportDrgRouteDistribution to set the attachment's exportDrgRouteDistributionId field to NULL. Dynamic route export to VCN attachments is not supported.

Route Propagation Restrictions

Routes imported from an IPSec tunnel or virtual circuit are never exported to other IPSec tunnel or virtual circuit attachments, regardless of how the export route distribution is configured. In a similar vein, packets which enter a DRG through an IPSec tunnel or virtual circuit attachment can never leave through an IPSec tunnel or virtual circuit attachment. Packets are dropped if routing is configured such that packets originating from IPSec tunnel or virtual circuit attachments are sent to IPSec tunnel or virtual circuit attachments.

ECMP

Equal-cost multi-path routing (ECMP) is a feature which allows flow-based load balancing of network traffic over multiple FastConnect virtual circuits or multiple IPSec tunnels (but not a mix of circuit types) using BGP. ECMP allows active-active load balancing and failover of network traffic between a maximum of eight circuits.

Oracle utilizes the protocol, destination IP, source IP, destination port, and source port to distinguish flows for load balancing purposes using a consistent and deterministic algorithm. Therefore, multiple flows are necessary to utilize all available bandwidth.

ECMP is off by default and can be enabled on a per-route table basis. Oracle only considers routes with identical route preference as eligible for ECMP forwarding. See Route Conflicts for more.

Route Source

DRG routes originate as either static routes or as dynamic routes from VCN, IPSec tunnel, FastConnect virtual circuit, or RPC attachments. This origin defines their source, which is an immutable characteristic of the route. In the API, the source is referred to as the routeProvenance of a DrgRouteRule.

Routes are propagated between DRGs using RPC attachments.

Routes with a source of IPSEC_TUNNEL or VIRTUAL_CIRCUIT are not exported to IPSec tunnel or virtual circuit attachments, regardless of the attachment's export distribution.

Routing a Subnet's Traffic to a DRG

The basic routing scenario sends traffic from a subnet in the VCN to the DRG. For example, if you're sending traffic from the subnet to your on-premises network, you set up a rule in the subnet's route table. The rule's destination CIDR is the CIDR for the on-premises network (or a subnet within), and the rule's target is the DRG. For more information, see VCN Route Tables.

Required IAM Policy

Peering VCNs using a DRG requires specific IAM permissions. See IAM Policies for Routing Between VCNs for details on the permissions needed.

DRG Versions

DRGs created before May 17, 2021 use the legacy software, and can be upgraded to the most recent version. DRGs created after that have the upgraded features by default.

The following summarizes the difference between an upgraded and legacy DRG:

A legacy DRG:

  • Has no programmable route tables. It has a default routing behavior where all traffic is forwarded from on-premises to an associated VCN and from the VCN to on-premises.
  • Can attach to a single VCN. The DRG can only be used for remote VCN peering using an RPC.
  • Can attach FastConnect or Site-to-Site VPN, or both. You can only reach resources in the local region using these connections.
  • Can support an RPC connection with a remote DRG-VCN pair in the same tenancy. See Remote VCN Peering using a Legacy DRG to learn about remote peering using a legacy DRG.

An upgraded DRG:

  • Has two route tables by default, and more can be added later.
  • Can have many VCNs attached to it within the same region. Local VCN to VCN traffic can pass through a mutually connected DRG instead of an LPG. See Local VCN Peering Through an Upgraded DRG for details.
  • Can attach to on-premises using FastConnect orSite-to-Site VPN, or both. You can reach resources in both local and remote regions using these connections.
  • Supports an RPC connection with a DRG/VCN pair in the same tenancy or another tenancy. See Remote VCN Peering through an Upgraded DRG to learn about remote peering using an upgraded DRG.

The rest of this article has recently been updated to reflect the capabilities of an upgraded DRG, as have the common networking scenarios. Remote VCN Peering using a Legacy DRG is the only routing or peering scenario specific to a legacy DRG.

Before you Upgrade a DRG

To take advantage of enhanced DRG features, upgrade your DRG. The DRG upgrade process is automated, but you must have the required permissions to initiate an upgrade.

Upgrading a DRG is a one-way process with no option to roll back to a legacy DRG after the upgrade process has been initiated.

Expect there to be a traffic outage for all the DRG's attachments during the upgrade process. Each attachment is updated in turn, forcing each upgrading attachment into a provisioning state where it will no longer forward traffic. If you have redundant connections, traffic will failover to other circuits while one is undergoing upgrade. If you only have one circuit, it could go down for about 20 minutes. Any existing BGP sessions for your on-premises connections (FastConnect or Site-to-Site VPN) are also reset while the attachment and any Site-to-Site VPN IPSec tunnels are also brought offline.

For example, if your DRG has two FastConnect virtual circuit attachments, one virtual circuit is upgraded first, causing it to drop connectivity while traffic can pass over the other virtual circuit. After that update has finished, the upgrade process upgrades the second virtual circuit attachment and the completed virtual circuit is brought back online. Expect the upgrade process to last up to 30 minutes per on-premises attachment.

Remote peering connections do not need to be reset like virtual circuit or IPSec tunnel connections do, and do not take the same length of time to upgrade.

Note

Expect a traffic outage during the upgrade process for any components attached to the DRG. Oracle recommends upgrading your DRG during a maintenance window.

After the DRG upgrade process has completed, any Site-to-Site VPN IPSec tunnels are brought back online and all BGP sessions for FastConnect and Site-to-Site VPN are re-established. By default, the upgraded DRG has two autogenerated DRG route tables and import route distributions enabled for your attachments. These resources are designed for backward compatibility with your legacy DRG and allow for all previous communication to resume in the same manner as before the upgrade without any additional user intervention.

For step-by-step instructions on how to upgrade your DRG refer to Upgrading a DRG.

Note

If the DRG upgrade process gets stuck for any reason, create a service request ticket, and mark the ticket as high severity.

Scenarios

We have provided some detailed networking scenarios to help you understand the role of a DRG in the Networking service and how the components work together in general.

DRG Routing

Route Conflicts

If two routes with identical CIDRs are observed the same DRG route table, the DRG resolves the conflict using the following criteria:

  1. Static routes always have higher preference than dynamic routes.

    Note

    You cannot manually specify two static routes with the same CIDR in the same DRG route table, but it's possible for more than one route with the same CIDR to be imported dynamically.
  2. Conflicts between dynamic routes are resolved by first analyzing the route's AS Path:
  3. Otherwise, the attachment type that imported the route is evaluated according to the following priority based on the attachment type:
    1. VCN: the DRG makes an arbitrary but stable selection.
    2. VIRTUAL_CIRCUIT: If ECMP is disabled, the DRG makes an arbitrary but stable selection. If ECMP is enabled, all routes will be added to the route table and the DRG makes routing choices using ECMP. The maximum supported ECMP width inside a DRG is 8.
    3. IPSEC_TUNNEL: If ECMP is disabled, the DRG makes an arbitrary but stable selection.If ECMP is enabled, all routes will be added to the route table and the DRG makes routing choices using ECMP. The maximum supported ECMP width inside a DRG is 8.
    4. REMOTE_PEERING_CONNECTION: The DRG will choose the route with the lowest network distance.

      If two routes have identical network distances, the DRG selects the route with the highest priority route source (STATIC > VCN > VIRTUAL_CIRCUIT> IPSEC_TUNNEL > RPC).

      If two routes have the same route source, the DRG makes an arbitrary but stable selection.

  4. If conflicting routes are imported from attachments of the same type, the conflict is resolved differently depending on the attachment type:
    1. VCN attachments: If identical CIDRs are imported from two VCN attachments, only one is selected using an arbitrary but stable decision procedure.
    2. VIRTUAL_CIRCUIT and IPSEC_TUNNEL attachments: If multiple routes with the same CIDR and different AS_PATH lengths are imported into a DRG route table, the route with the lowest AS_PATH length is selected. Otherwise, one route is chosen using an arbitrary but stable decision procedure.
    3. RPC attachments: If identical CIDRs are imported from two RPC attachments, one of them is chosen using an arbitrary stable decision procedure.

You can observe the results of conflict resolution by listing the contents of the route table. Deprecated routes are marked with the "conflict" status.

Limitations

Some functions might appear to be possible, but the following routing functions are not currently allowed:

  • Explicit creation or deletion of RPC, IPSec tunnel, or virtual circuit attachments
  • Static routes in DRG route tables with next-hop of IPSec tunnel or virtual circuit attachments
  • Use of non-default export route distributions
  • Dynamic route export to VCN attachments
  • Routes propagating via RPC through more than 4 DRGs

Using BGP to prefer routes from Oracle to your on-premises network

This section describes in greater detail how the BGP AS_PATH attribute can be used to influence route selection in the context of a single DRG route table.

If the routes for the different paths are the same, Oracle uses the shortest AS path when sending traffic to your on-premises network, regardless of which path was used to initiate the connection to Oracle. Therefore asymmetric routing is allowed. Asymmetric routing here means that Oracle's response to a request can follow a different path than the request. For example, depending on how your edge device (also called your customer-premises equipment, or CPE) is configured, you could send a request over Site-to-Site VPN, but the Oracle response could come back over FastConnect. If you want to force routing to be symmetric, Oracle recommends using BGP and AS path prepending with your routes to influence which path Oracle uses when responding to and initiating connections.

Oracle implements AS path prepending to establish preference on which path to use if your edge device advertises the same route and routing attributes over multiple different connection types between your on-premises network and VCN. The details are summarized in the following table. Unless you're influencing routing in some other way, when the same route is advertised over multiple paths to the DRG at the Oracle end of the connections, Oracle prefers the paths in the following order:

Oracle preference Path Details of how Oracle prefers the path Resulting AS path for the route
1 FastConnect Oracle prepends no ASNs to the routes that your edge device advertises, for a total AS path length of 1. Your ASN
2 Site-to-Site VPN with BGP routing Oracle prepends a single private ASN on all the routes that your edge device advertises over Site-to-Site VPN with BGP, for a total AS path length of 2. Private ASN, Your ASN
3 Site-to-Site VPN with static routing Oracle prepends 3 private ASNs on the static routes that you've provided (Oracle advertises those routes to the dynamic routing gateway (DRG) at the Oracle end of Site-to-Site VPN). This results in a total AS path length of 3. Private ASN, Private ASN, Private ASN

The preceding table assumes you are sending a single autonomous system number in your AS path. Oracle honors the complete AS path you send. If you use static routing, and also send an AS path that has "Your ASN" plus two or more other ASNs, it can cause unexpected behavior because Oracle's routing preference might change.

While policy-based VPN static routing behavior is documented earlier, Oracle also recommends that if you use FastConnect connections with VPN backup, you employ BGP on your IPSec route-based VPN. This strategy allows you to have full control of failover behavior.

Other relevant links