Maintaining Virtual Circuits
Learn about planned maintenance of FastConnect virtual circuits.
Oracle performs regular maintenance on the routers dedicated for use with FastConnect virtual circuits. This maintenance allows Oracle to enhance overall device operational stability by replacing faulty hardware, applying patches, and more. These maintenance activities are crucial for service improvement. Each maintenance task is planned carefully and scheduled in advance to minimize any impact on your services. This article outlines what happens during a FastConnect virtual circuit maintenance and what steps you should take to minimize service outages due to planned or unplanned maintenance.
Oracle recommends that you always configure a primary and secondary connection to Oracle Cloud Infrastructure. The secondary connection can either be a redundant FastConnect virtual circuit or an IPsec connection. In case the IPSec connection is being used as the secondary path, then ensure that the Site-to-Site VPN IPSec tunnels are configured to use BGP Routing. While using such connections, Oracle Cloud always prioritizes FastConnect over IPSec tunnels using the AS Path prepend mechanism.
The primary and secondary connections should be established on different physical devices to offer reliable connectivity from on-premises to OCI resources. While creating a secondary FastConnect connection withFastConnect Direct, use the "specify router proximity" option from the OCI console to make the secondary connection land on a different physical device. For Partner connections, work with your FastConnect partner to provision a secondary virtual circuit on a redundant partner cross-connect. This would help you to have an uninterrupted connection if there are any planned or unplanned events. For information on redundancy practices, please follow the Connectivity Redundancy Guide (PDF).
High availability in Virtual Circuits
High availability in virtual circuits is achieved through redundant connections between OCI and the on-premises network. Implementing high availability keeps the network intact during any outage or planned activities. In case you are using the Oracle partner connectivity model, Oracle handles the redundancy of the physical connections between the partner and Oracle, and the redundancy of routers in the FastConnect locations. You are expected to handle the redundancy of the physical connection between the on-premises network and the Oracle partner. For other FastConnect connectivity models like Third-party and colocation, you are responsible for ensuring redundancy between FastConnect routers and your own edge devices by configuring redundant virtual circuits using different physical FastConnect routers provided by Oracle in every region and FastConnect POP location.
The below network topologies show redundant virtual circuits used in the Oracle partner scenario, Third-party provider or colocate with Oracle scenario, and IPSec VPN as a backup for FastConnect.
For more information, review FastConnect Redundancy Best Practices.
Maintenance Events and Notifications
Planned maintenance for FastConnect services is carefully scheduled to focus on one FastConnect router at a time to ensure uninterrupted connectivity over virtual circuits during maintenance. This approach ensures that there is always at least one available path to access the OCI network when redundant circuits are in place, minimizing any potential disruption.
During maintenance, Oracle sends the well-known RFC8326 "BGP GRACEFUL SHUTDOWN 65535:0" message to your CPE edge devices along with AS path prepending. If the CPE device acknowledges this message then the local preference on the CPE device will be set to zero to ensure that the path going under maintenance is de-preferred. The AS path change is done by prepending AS 31898 (three times) to the BGP routes going from OCI toward the CPE. Sending this message allows the traffic to gracefully shift to the redundant path.
You need to ensure that any on-premises devices in the path are set up to acknowledge the AS path prepend or BGP Graceful Shutdown Community message. Also, ensure that redundancy is configured properly to shift the traffic to an alternate path, in case the primary path is de-preferred. It's important to check with the service provider to confirm if they are set up to allow AS path prepending and BGP Community messages on the connection if they are managing the your network.
If your network does not allow the above actions, you are likely to experience asymmetric routing and packet drops during maintenance activities.
Setting the local preference to zero on the CPE device after receiving the graceful shutdown community might be vendor specific. Validate with your equipment vendors that your CPE device has this feature built in or not. If not, configure an inbound routing policy to set the local preference on the CPE to zero, based on receiving the graceful shutdown community message from OCI.
OCI routers enable AS path prepending when it's also done on CPE devices. There is a possibility of asymmetric routing if traffic shifting on CPE and OCI internal routers does not happen at the same time, which can happen if there is a delay in shifting traffic. To eliminate such issues, we recommend that you enable support for asymmetric routing in CPE devices.
When planned maintenance is scheduled, you will be notified at least 14 days prior to the maintenance windows through Console Announcements and also email notifications if you are subscribed for email notifications. Email notification contacts are added and managed by the Service Administrator. You will be notified of service outages and security incidents using the same mechanisms.
Verifying Virtual Circuit Failover
When you first provision your redundant connections, validate they are working correctly before you place them into production. Repeat the validation the same on a regular basis (every 6 months, every year, etc.) or before scheduled maintenance windows to ensure the failover is still working correctly as changes can be made in your environment after the initial failover test that can break the failover. If you only test it when you first provision the redundant connectivity, you run the risk of finding out it's not working when an actual outage occurs which may be too late. Also, don't forget to validate that failing back to the primary works.
The failover validation process has two stages, each of which will be seen during OCI router maintenance:
- De-prefer the primary path using local preference and AS path prepend then checking if traffic shifts to the secondary path or not. The Connectivity Redundancy Guide (PDF) explains how the AS path prepend and local preference settings can be used to prioritize a particular path. This should be the main fail-over test that you perform, as the path de-prefer process is run by OCI during the maintenance window before the BGP session shutdown.
- Shut down the primary BGP session between the on-premises and OCI networks. To shut down the BGP session, deactivate the virtual circuit from the OCI console. This will force traffic to flow through the secondary connection.
You can bring up the primary path by reverting the above changes and then checking if the traffic is forwarded back to the primary path. Oracle recommends testing failover using both methods suggested above to ensure the failover mechanism is working smoothly.
For Oracle partner connectivity models, if you have multiple virtual circuits you have the option to validate failover using the above-mentioned methods. If you only have one virtual circuit you will not have an option to test failover as the redundancy only exists between Oracle FastConnect router and the provider.
If your on-premises network uses stateful firewalls, you will be prone to issues during failover, so it is even more important to ensure traffic will failover properly.
Traffic statistics can be monitored on the OCI console. The bits received and bits sent metrics should only increment on the path that is currently active. You can monitor the health, capacity, and performance of your FastConnect connection by using metrics, alarms, and notifications. For more information, see https://docs.oracle.com/iaas/Content/Monitoring/home.htm and https://docs.oracle.com/iaas/Content/Notification/home.htm.