Configuring Oracle-Managed Infrastructure Maintenance

Oracle performs the updates to all of the Oracle-managed infrastructure components on Exadata Cloud@Customer.

You may manage contacts who are notified regarding infrastructure maintenance, set a maintenance window to determine the time your quarterly infrastructure maintenance will begin, and also view scheduled maintenance runs and the maintenance history of your Exadata Cloud@Customer in the Oracle Cloud Infrastructure Console. For details regarding the infrastructure maintenance process and configuring the maintenance controls refer to the following:

About Oracle Managed Exadata Cloud@Customer Infrastructure Maintenance Updates

Oracle performs patches and updates to all of the Oracle-managed system components on Exadata Cloud@Customer.

Oracle patches and updates include the physical database server hosts, Exadata Storage Servers, Network Fabric Switches, management switch, power distribution units (PDUs), integrated lights-out management (ILOM) interfaces, and Control Plane Servers. This is referred to as Exadata Cloud@Customer infrastructure maintenance.

In all but rare exceptional circumstances, you receive advance communication about these updates to help you plan for them. If there are corresponding recommended updates for your VM cluster virtual machines (VMs), then Oracle provides notifications about them.

Wherever possible, scheduled updates are performed in a manner that preserves service availability throughout the update process. However, there can be some noticeable impact on performance and throughput while individual system components are unavailable during the update process.

For example, database server patching typically requires a reboot. In such cases, wherever possible, the database servers are restarted in a rolling manner, one at a time, to ensure that the service remains available throughout the process. However, each database server is unavailable for a short time while it restarts, and the overall service capacity diminishes accordingly. If your applications cannot tolerate the restarts, then take mitigating action as needed. For example, shut down an application while database server patching occurs.

Overview of the Quarterly Infrastructure Maintenance Process

By default, infrastructure maintenance updates the Exadata database server hosts in a rolling fashion, followed by updating the storage servers.

You can also choose non-rolling maintenance to update database and storage servers. The non-rolling maintenance method first updates your storage servers at the same time, then your database servers at the same time. Although non-rolling maintenance minimizes maintenance time, it incurs full system downtime while the storage servers and database servers are being updated.

Rolling infrastructure maintenance begins with the Exadata database server hosts. For the rolling maintenance method, database servers are updated one at a time. Each of the database server host's VMs is shut down, the host is updated, restarted, and then the VMs are started, while other database servers remain operational. This rolling maintenance impact older applications not written to handle a rolling instance outage. This process continues until all servers are updated.

After database server maintenance is complete, storage server maintenance begins. For the rolling maintenance method, storage servers are updated one at a time and do not impact VM cluster VM's availability. However, the rolling storage server maintenance can result in reduced IO performance as storage servers are taken offline (reducing available IO capacity) and resynced when brought back online (small overhead on database servers). Properly sizing the database and storage infrastructure to accommodate increased work distributed to database and storage servers not under maintenance will minimize (or eliminate) any performance impact.

Note that while databases are expected to be available during the rolling maintenance process, the automated maintenance verifies Oracle Clusterware is running but does not verify that all database services and pluggable databases (PDBs) are available after a server is brought back online. The availability of database services and PDBs after maintenance can depend on the application service definition. For example, a database service, configured with certain preferred and available nodes, may be relocated during the maintenance and wouldn't automatically be relocated back to its original node after the maintenance completes. Oracle recommends reviewing the documentation on Achieving Continuous Availability for Your Applications on Exadata Cloud Systems to reduce the potential for impact to your applications. By following the documentation's guidelines, the impact of infrastructure maintenance will be only minor service degradation as database servers are sequentially updated.

Oracle recommends that you follow the Maximum Availability Architecture (MAA) best practices and use Data Guard to ensure the highest availability for your critical applications. For databases with Data Guard enabled, Oracle recommends that you separate the maintenance windows for the infrastructure instances running the primary and standby databases. You may also perform a switchover prior to the maintenance operations for the infrastructure instance hosting the primary database. This allows you to avoid any impact on your primary database during infrastructure maintenance.

Prechecks are performed on the Exadata Cloud@Customer infrastructure components prior to the start of the maintenance window. The goal of the prechecks is to identify issues that may prevent the infrastructure maintenance from succeeding. The Exadata infrastructure and all components remain online during the prechecks. An initial precheck is run approximately 10 days prior to the maintenance start and another precheck is run approximately 24 hours prior to maintenance start. If the prechecks identify an issue that requires rescheduling the maintenance notification is sent to the maintenance contacts.

The time taken to update infrastructure components varies depending on the number of database servers and storage servers in the Exadata infrastructure, the maintenance method, and whether custom action has been enabled. The approximate times provided are estimates. Time for custom action, if configured, is not included in the estimates below. Database server maintenance time may vary depending on the time required to shutdown each VM before the update and then start each VM and associated resources after the update of each node before proceeding to the next node. The storage server maintenance time will vary depending on the time required for the ASM rebalance, which is not included in the estimates below. If issues are encountered during maintenance this may also delay completion beyond the approximate time listed. In such a situation, if Oracle cloud operations determine resolution would extend beyond the expected window, they will send a notification and may reschedule the maintenance.

Note

The timeframes mentioned below can change if Oracle cloud operations determine that additional maintenance work is needed. If additional time is necessary, Oracle will send a customer notification in advance to inform customers that additional time will be required for the next quarterly maintenance window.
  • Rolling:
    • Each database server takes 90 minutes on average.
    • Each storage server takes 60 minutes on average.
    • Each InfiniBand or RoCE fabric switch takes 30 minutes on average.
    • The approximate total time for infrastructure maintenance is as follows:

      • Base and Quarter Rack (2 Database Servers/3 Storage Servers): Approximately 7 hours

        2 Database Servers X 90 = 180 minutes

        3 Storage Servers X 60 = 180 minutes

        2 InfiniBand or RoCE Fabric Switch X 30 = 60 minutes

      • Half Rack (4 Database Servers/6 Storage Servers): Approximately 13 hours

        4 Database Servers X 90 = 360 minutes

        6 Storage Servers X 60 = 360 minutes

        2 InfiniBand or RoCE Fabric Switch X 30 = 60 minutes

      • Full Rack (8 Database Servers/12 Storage Servers): Approximately 26 hours

        8 Database Servers X 90 = 720 minutes

        12 Storage Servers X 60 = 720 minutes

        2 InfiniBand or RoCE Fabric Switch X 30 = 60 minutes

  • Non-Rolling:
    • All database servers take 180 minutes on average.
    • All storage servers take 60 minutes on average.
    • Storage Servers and Database servers are brought back online prior to starting fabric switch maintenance.
    • Network fabric switches are still updated in a rolling method and take 30 minutes each on average.
    • The approximate total time for infrastructure maintenance is 5 hours regardless of shape:
      • All Database Servers = 180 minutes
      • All Storage Servers = 60 minutes
      • 2 InfiniBand or RoCE Fabric Switch = 60 minutes

Infrastructure Maintenance Contacts

Maintenance contacts are required for service request based communications for hardware replacement and other maintenance events.

Add a primary maintenance contact and optionally add a maximum of nine secondary contacts. Both the primary and secondary contacts receive all notifications about hardware replacement, network issues, and software maintenance runs.

You can promote any secondary contacts as the primary anytime you want. When you promote a secondary contact to primary, the current primary contact will be demoted automatically to secondary.

For more information, see: Using the Console to Create Infrastructure and Managing Infrastructure Maintenance Contacts.

Using the Console to Configure Oracle-Managed Infrastructure Updates

Full Exadata infrastructure software updates are scheduled on a quarterly basis. In addition, important security updates are scheduled monthly. While you cannot opt-out of these infrastructure updates, Oracle alerts you in advance through the Cloud Notification Portal and allows scheduling flexibility to help you plan for them.

For quarterly infrastructure maintenance, you can set a maintenance window to determine when the maintenance will begin. You can also edit the maintenance method, enable custom action, and view the scheduled maintenance runs and the maintenance history of your Exadata Cloud@Customer in the Oracle Cloud Infrastructure Console. For security maintenance, you may edit the scheduled start time within the 21-day window.

For more information, see the following:
  • View or Edit Quarterly Infrastructure Maintenance Preferences for Exadata Cloud@Customer Infrastructure
  • View or Edit a Scheduled Quarterly Maintenance for Exadata Cloud@Customer Infrastructure
  • View or Edit a Scheduled Security Maintenance for Exadata Cloud@Customer Infrastructure
  • View the Maintenance History of Exadata Cloud@Customer Infrastructure
  • View and Edit Quarterly Maintenance While Maintenance is In Progress or Waiting for Custom Action
  • View or Edit a Scheduled Security Maintenance for Exadata Cloud@Customer Infrastructure

View or Edit Quarterly Infrastructure Maintenance Preferences for Exadata Cloud@Customer Infrastructure

To edit your Oracle Exadata Database Service on Cloud@Customer infrastructure quarterly maintenance preferences, be prepared to provide values for the infrastructure configuration. The changes you make will only apply to future maintenance runs, not those already scheduled.

  1. Open the navigation menu. Under Oracle Database, click Exadata Database Service on Cloud@Customer.
  2. Select Region and Compartment, and provide the region and the compartment where the Oracle Exadata infrastructure you want to edit is located.
  3. Click Exadata Infrastructure.
  4. Click the name of the Exadata infrastructure that you want to edit.

    The Infrastructure Details page displays information about the selected Oracle Exadata infrastructure.

  5. Click Edit Maintenance Preferences.

    Edit Maintenance Preferences page is displayed.

    Note

    Changes made to maintenance preferences apply only to future maintenance, not the maintenance that has already been scheduled. To modify scheduled maintenance, see View or Edit a Scheduled Maintenance for Exadata Cloud@Customer Infrastructure.

  6. On the Edit Maintenance Preferences page, configure the following:
    • Choose a maintenance method:
      • Rolling: By default, Exadata Infrastructure is updated in a rolling fashion, one server at a time with no downtime.
      • Non-rolling: Update database and storage servers at the same time. The non-rolling maintenance method minimizes maintenance time but incurs full system downtime.
    • Enable custom action before performing maintenance on DB servers: Enable custom action only if you want to perform additional actions outside of Oracle’s purview. For maintenance configured with a rolling software update, enabling this option will force the maintenance run to wait for a custom action with a configured timeout before starting maintenance on each DB server. For maintenance configured with non-rolling software updates, the maintenance run will wait for a custom action with a configured timeout before starting maintenance across all DB servers. The maintenance run, while waiting for the custom action, may also be resumed prior to the timeout.
      • Custom action timeout (in minutes): Timeout available to perform custom action before starting maintenance on the DB Servers.

        Default: 30 minutes

        Maximum: 120 minutes

    • Maintenance schedule:
      • No preference: The system assigns a date and start time for infrastructure maintenance.
      • Specify a schedule: Choose your preferred month, week, weekday, start time, and lead time for infrastructure maintenance.
        • Under Maintenance months, specify at least one month for each quarter during which Exadata infrastructure maintenance will take place. You can select more than one month per quarter. If you specify a long lead time for advanced notification (for example, 4 weeks), you may wish to specify 2 or 3 months per quarter during which maintenance runs can occur. This will ensure that your maintenance updates are applied in a timely manner after accounting for your required lead time. Lead time is discussed in the following steps.
        • Optional. Under Week of the month, specify which week of the month, maintenance will take place. Weeks start on the 1st, 8th, 15th, and 22nd days of the month, and have a duration of 7 days. Weeks start and end based on calendar dates, not days of the week. Maintenance cannot be scheduled for the fifth week of months that contain more than 28 days. If you do not specify a week of the month, Oracle will run the maintenance update in a week to minimize disruption.
        • Optional. Under Day of the week, specify the day of the week on which the maintenance will occur. If you do not specify a day of the week, Oracle will run the maintenance update on a weekend day to minimize disruption.
        • Optional. Under Start hour, specify the hour during which the maintenance run will begin. If you do not specify a start hour, Oracle will pick the least disruptive time to run the maintenance update.
        • Under Lead Time, specify the minimum number of weeks ahead of the maintenance event you would like to receive a notification message. Your lead time ensures that a newly released maintenance update is scheduled to account for your required minimum period of advanced notification.
  7. Click Save Changes.

    If you switch from rolling to non-rolling maintenance method, then Confirm Non-rolling Maintenance Method dialog is displayed.

    1. Enter the name of the infrastructure in the field provided to confirm the changes.
    2. Click Save Changes.

View or Edit a Scheduled Quarterly Maintenance for Exadata Cloud@Customer Infrastructure

Learn how to view and edit the time of the next scheduled maintenance.

  1. Open the navigation menu. Under Oracle Database, click Exadata Database Service on Cloud@Customer.
  2. Choose your Compartment.
  3. Click Exadata Infrastructure.
  4. In the list of Exadata Infrastructures, find the infrastructure you want to set the next scheduled maintenance window for and click its highlighted name.

    The Infrastructure Details page displays information about the selected Oracle Exadata infrastructure.

    Note

    An information block is displayed 6 hours before the start of a maintenance run, regardless of whether you've chosen rolling or non-rolling maintenance method. When the maintenance begins, it is automatically removed.

  5. On the Infrastructure Details page, under Maintenance, click the view link in the Next Quarterly Maintenance field.

    The Exadata Infrastructure Maintenance page is displayed.

  6. On the Exadata Infrastructure Maintenance page, scheduled maintenance details are listed.

    Target DB Server Version and Target Storage Server Version: These fields display the Exadata software version to be applied by the scheduled maintenance. The version applied will be the most recent certified update for Exadata infrastructures in the cloud. If the next quarterly update is not yet certified when the maintenance is scheduled, then the versions may show "LATEST" until the new quarterly update becomes available. Once the update becomes available the new version will be displayed.

    To find information on the Database Server Exadata software version or the Storage Server Exadata software version, see My Oracle Support note Exadata Database Machine and Exadata Storage Server Supported Versions (Doc ID 888828.1).

  7. To change the next scheduled maintenance settings, click Edit Maintenance Run.
    On the Edit Maintenance page, do the following:
    • Select a maintenance method, Rolling or Non-rolling.
      Note

      If you select the Non-rolling option, an information block appears stating that components will be updated simultaneously, resulting in full system downtime.

    • Enable custom action before performing maintenance on DB servers: Enable custom action only if you want to perform additional actions outside of Oracle’s purview. For maintenance configured with a rolling software update, enabling this option will force the maintenance run to wait for a custom action with a configured timeout before starting maintenance on each DB server. For maintenance configured with non-rolling software updates, the maintenance run will wait for a custom action with a configured timeout before starting maintenance across all DB servers. The maintenance run, while waiting for the custom action, may also be resumed prior to the timeout.
      • Custom action timeout (in minutes): Maximum timeout available to perform custom action before starting maintenance on the DB Servers.

        Default: 30 minutes

        Maximum: 120 minutes

    • To reschedule the next quarterly maintenance run, enter a date and time in the Scheduled Start time field.
      The following restrictions apply:
      • Oracle expects to be able to perform infrastructure maintenance at least once per quarter. You should not defer maintenance beyond the end of a maintenance quarter unless unexpected issues prevent your accommodating it before the next maintenance quarter.
      • In the event unexpected issues prevent your accommodating the scheduled infrastructure maintenance run, you can reschedule the infrastructure maintenance to another date no more than 180 days from the prior infrastructure maintenance. Since normal maintenance should be performed quarterly, this provides approximately 90 additional days for you to reschedule the infrastructure maintenance. Oracle strongly recommends you not schedule maintenance at or close to the 180 day limit, as you will have no flexibility to reschedule further if additional unexpected issues arise.
      • If a new maintenance release is announced prior to your rescheduled maintenance run, the newer release will be applied on your specified date.
      • You can reschedule your maintenance to take place earlier than it is currently scheduled. You cannot reschedule the maintenance if the current time is within 2 hours of the scheduled maintenance start time.
      • Oracle reserves certain dates each quarter for internal maintenance operations, and you cannot schedule your maintenance on these dates.
    • Click Save Changes.
  8. To view estimated maintenance time details for various components, click the View link is displayed in the Total Estimated Maintenance Time field.

    The View link is displayed in the Total Estimated Maintenance Time field only if the Maintenance Method is Rolling.

    The Estimated Maintenance Time Details page is displayed with details that include:
    • Total Estimated Maintenance Time
    • Database Servers Estimated Maintenance Time
    • Storage Servers Estimated Maintenance Time
    • Network Switches Estimated Maintenance Time
    • Order in which components are updated. In rolling maintenance, components are updated in the sequence displayed
    1. To view the number of VMs that will be restarted as part of Database Server maintenance, click the Show details link.

      The VM Location dialog is displayed.

    2. In the VM Cluster Name field, you can find out what VM cluster a particular VM belongs to.
    3. Click Close.
  9. Click Close to close the Estimated Maintenance Time Details page.

View the Maintenance History of Exadata Cloud@Customer Infrastructure

Learn how to view the maintenance history for an Exadata Cloud@Customer Infrastructure.

  1. Open the navigation menu. Under Oracle Database, click Exadata Database Service on Cloud@Customer.
  2. Choose your Compartment.
  3. Click Exadata Infrastructure.
  4. In the list of Exadata Infrastructures, find the infrastructure you want to view the maintenance history and click its highlighted name.

    The Infrastructure Details page displays information about the selected Oracle Exadata infrastructure.

  5. On the Infrastructure Details page, under Maintenance, click the view link in the Next Quarterly Maintenance field.

    The Exadata Infrastructure Maintenance page is displayed.

  6. On the Exadata Infrastructure Maintenance page, click Maintenance History to see a list of past maintenance events including details on their completion state and the target database and storage server versions.

    To find information on the Database Server Exadata software version or the Storage Server Exadata software version, see My Oracle Support note Exadata Database Machine and Exadata Storage Server Supported Versions (Doc ID 888828.1).

Monitor Infrastructure Maintenance Using Lifecycle State Information

The lifecycle state of your Exadata Infrastructure resource enables you to monitor when the maintenance of your infrastructure resource begins and ends.

In the Oracle Cloud Infrastructure Console, you can see lifecycle state details messages on the Exadata Infrastructure Details page when a tooltip is displayed beside the Status field. You can also access these messages using the ListExadataInfrastructures API, and using tools based on the API, including SDKs and the OCI CLI.

During infrastructure maintenance operations, you can expect the following:
  • If you specify a maintenance window, then patching begins at your specified start time. The infrastructure resource's lifecycle state changes from Available to Maintenance in Progress.
    Note

    The prechecks are now done prior to the start of the maintenance.
  • When Exadata database server maintenance starts, the infrastructure resource's lifecycle state is Maintenance in Progress, and the associated lifecycle state message is, The underlying infrastructure of this system (dbnodes) is being updated.
  • When storage server maintenance starts, the infrastructure resource's lifecycle state is Maintenance in Progress, and the associated lifecycle state message is, The underlying infrastructure of this system (cell storage) is being updated and this will not impact Database availability.
  • After storage server maintenance is complete, the networking switches are updated one at a time, in a rolling fashion.
  • When maintenance is complete, the infrastructure resource's lifecycle state is Available, and the Console and API-based tools do not provide a lifecycle state message.

Receive Notifications about Your Infrastructure Maintenance Updates

There are two ways to receive notifications. One is through email to infrastructure maintenance contacts and the other one is to subscribe to the maintenance events and get notified.

Oracle schedules maintenance run of your infrastructure based on your scheduling preferences and sends email notifications to all your infrastructure maintenance contacts. You can login to the console and view details of the schedule maintenance run. Appropriate maintenance related events will be generated as Oracle prepares for your scheduled maintenance run, for example, precheck, patching started, patching end, and so on. For more information about all maintenance related events, see Oracle Exadata Cloud@Customer Events. In case, if there are any failures, then Oracle reschedules your maintenance run, generates related notification, and notifies your infrastructure maintenance contacts.

For more information about Oracle Cloud Infrastructure Events, see Overview of Events. To receive additional notifications other than the ones sent to infrastructure maintenance contacts, you can subscribe to infrastructure maintenance events and get notified using the Oracle Notification service, see Notifications Overview.