Use the Oracle Cloud Infrastructure Monitoring service to actively and passively monitor cloud resources using the Metrics and Alarms features. Learn how Monitoring works.
The Monitoring service uses metrics to monitor resources and alarms to
notify you when these metrics meet alarm-specified triggers.
Metrics are emitted to the Monitoring service as raw data points , or timestamp-value pairs, along with dimensions and metadata. Metrics come from various sources:
Metric data posted to the Monitoring service is only
presented to you or consumed by the Oracle Cloud Infrastructure features
that you enable to use metric data.
When you query a metric, the Monitoring service returns aggregated data according to the specified parameters. You can specify a range (such as the last 24 hours), statistic , and interval . The Console displays one monitoring chart per metric for selected resources. The aggregated data in each chart reflects the selected statistic and interval. API requests can optionally filter by dimension and specify a resolution . API responses include the metric name along with its source compartment and metric namespace . You can feed the aggregated data into a visualization or graphing library.
Metric and alarm data is accessible from the Console, CLI, and API. For retention periods, see Storage Limits.
The Alarms feature of the Monitoring service publishes alarm messages to configured destinations, such as topics in Notifications and streams in Streaming.
Metrics Feature Overview 🔗
The Metrics feature relays metric data about the health, capacity, and performance of cloud resources.
A metric is a measurement related to health, capacity, or performance of a resource . Resources, services, and applications emit metrics to the Monitoring service. Common metrics reflect data related to:
Availability and latency
Application uptime and downtime
Completed transactions
Failed and successful operations
Key performance indicators (KPIs), such as sales and engagement quantifiers
By querying Monitoring for this data, you can understand how well the systems and processes are working to achieve the service levels you commit to your customers. For example, you can monitor the CPU utilization and disk reads of compute instances . You can then use this data to decide when to provision more instances to handle increased load, troubleshoot issues with the instance, or better understand system behavior.
Example Metric: Failure Rate 🔗
For application health, one of the common KPIs is failure rate, for which a common definition is the number of failed transactions divided by total transactions. This KPI is typically delivered through application monitoring and management software.
As a developer, you can capture this KPI from applications using custom metrics. Record observations every time an application transaction takes place and then post that data to the Monitoring service. In this case, set up metrics to capture failed transactions, successful transactions, and transaction latency (time spent per completed transaction).
Alarms Feature Overview 🔗
Use alarms to monitor the health, capacity, and performance of cloud resources.
The Alarms feature of the Monitoring service works with the configured destination service to notify you when metrics meet alarm-specified triggers. The previous illustration depicts the flow, starting with resources emitting metric data points to Monitoring. When triggered, an alarm sends an alarm message to the configured destination. For Notifications, messages are sent to subscriptions in the configured topic. For Streaming, messages are sent to the configured stream. (This illustration doesn't cover raw and aggregated metric data. For these details, see the "Monitoring Overview" illustration at the top of this page.)
When configured, repeat notifications remind you of a continued firing state at the configured repeat interval. You're also notified when an alarm transitions back to the OK state, or when an alarm is reset.
Alarm Evaluations 🔗
Monitoring evaluates alarms once per minute to find alarm status.
When the alarm splits notifications, Monitoring evaluates each tracked metric stream. If the evaluation of that metric stream indicates a new FIRING status or other qualifying event, then Monitoring sends an alarm message.
Data point values for this statistic could be anything from null (absent) to 100.
Data point evaluations:
For any data point value greater than 85, the evaluation is true (1). A true evaluation means that the trigger rule condition has been met.
For any data point value that isn't greater than 85, the evaluation is false (0).
The alarm doesn't fire until the trigger rule condition is met for three successive minutes. This configuration is the alarm's trigger delay (pendingDuration), set as PT3M.
The alarm updates its state to OK when the breaching condition has been clear for the most recent minute.
The following image shows an aggregated metric stream for the example alarm. Each data point is indicated by a square.
The following table shows consecutive alarm evaluations for the example alarm. The alarm is evaluated on a moving window of three one-minute intervals.
Evaluation period timestamp
Minutes in period
Data point evaluations*
Status
3
[1, 2, 3]
[0, 0, 0]
OK
4
[2, 3, 4]
[0, 0, 1]
OK
5
[3, 4, 5]
[0, 1, 1]
OK
6
[4, 5, 6]
[1, 1, 1]
FIRING
7
[5, 6, 7]
[1, 1, 1]
FIRING
8
[6, 7, 8]
[1, 1, 0]
OK
9
[7, 8, 9]
[1, 0, 0]
OK
10
[8, 9, 10]
[0, 0, 0]
OK
*A value of one (1) means that the trigger rule condition is met.
About the Internal Reset Period 🔗
The internal reset period determines when an alarm stops checking for an absent metric that triggered the Firing state in the previous evaluation. When the metric is absent for the entire period, later alarm evaluations ignore the indicated metric stream. If no other metric streams are causing the Firing state for the alarm, then the alarm transitions to OK and sends a RESET message. By default, the RESET message arrives after 13 minutes (internal reset period plus the default slack period of 3 minutes). You can customize the slack period.
The length of the internal reset period is globally configured at 10 minutes, which causes the alarm history to show a 10-minute difference.
The beginning of an internal reset period depends on the alarm type. For threshold alarms, the internal reset period starts when the first absence is detected. For absence alarms, the internal reset period starts after completion of the absence detection period (default of 2 hours, can be customized).
Data Points Gathered During an Internal Reset Period 🔗
Each evaluation during the ten-minute internal reset period accounts for all data points in that period.
For example, consider a metric stream (A) that exceeds the threshold (dashed red line in following diagrams). The alarm fires (F). When a lack of emitted data points is detected, an internal reset period begins.
The following diagram shows a single internal reset period for metric stream A, from the times t5 to t15. At time t16, metric stream A is no longer evaluated.
The following diagram shows two internal reset periods for metric stream A, from the times t3 to t5, and from t6 to t16. A emits a data point at t6, starting another internal reset period. At time t17, metric stream A is no longer evaluated.
Threshold Alarm Example 🔗
A threshold alarm reports on metric streams that occur outside the threshold. When a previously problematic metric stream is absent, the alarm starts the internal reset period for the metric stream.
In this example, four metric streams are evaluated by a threshold alarm. The Console shows the initial Firing (1:30) and Ok (1:51) transition states. The internal reset period occurs while the alarm is in Firing state.
The internal reset period and other significant events in this example are described in the following table.
No emission is detected for resource1. The alarm starts the internal reset period for resource1.
--
1:38
Firing
--
No emission is detected for resource2. The alarm starts the internal reset period for resource2.
--
1:45
Firing
--
The internal reset period ends for resource1, so the alarm no longer checks for emissions from resource1. However, the alarm is still Firing because resource2 is still in its own internal reset period.
--
1:48
Ok
Ok
The internal reset period ends for resource2, so the alarm no longer checks for emissions from resource2. Emissions from the remaining resources (resource3 and resource4) are within threshold.
RESET (sent after the three-minute slack period, at about 1:51)
Absence Alarm Example 🔗
An absence alarm reports on absent metric streams. When a metric stream is absent, the alarm starts the absence detection period for the metric stream (default of two hours, can be customized). After completion of the absence detection period, the alarm starts the internal reset period for the metric stream.
In this example, a metric stream is evaluated by an absence alarm that uses the default two-hour absence detection period and default three-minute slack period. The Console shows the initial Firing (2:00) and Ok (4:10) transition states. The internal reset period occurs while the alarm is in Firing state.
The internal reset period and other significant events in this example are described in the following table.
No emission is detected for resource-z. The alarm starts the absence detection period for resource-z.
OK_TO_FIRING
4:00
Firing
--
The absence detection period for resource-z ends. The alarm starts the internal reset period for resource-z.
--
4:10
Ok
Ok
The internal reset period ends for resource-z, so the alarm no longer checks for emissions from resource-z. No metric streams are monitored by the alarm any more, so the alarm transitions to Ok state.
RESET (sent after the three-minute slack period, at about 4:13)
Time Needed to Reflect Alarm Updates 🔗
Updates to alarms take up to five minutes to be reflected everywhere.
The message type indicates the reason that the message was sent.
Note
The specified message type is sent at the indicated time plus the alarm's configured trigger delay, if any.
Repeat messages are also sent if configured in the alarm.
The following table lists the alarm state and transition for each message type.
Message type
State
Transition
Comments
OK_TO_FIRING
FIRING
from OK to FIRING
FIRING_TO_OK
OK
from FIRING to OK
REPEAT
FIRING
--
This message type is sent when the alarm maintains the FIRING state, and the alarm is configured for repeat notifications.
RESET
OK
from FIRING to OK
Important: When a RESET status change occurs, look at the health of the resource.
This message type is sent when the alarm transitions to the OK state after one or more internal resets. An internal reset occurs when a metric stream that caused the alarm to transition to the FIRING state is continuously absent for the full internal reset period. A metric stream that's internally reset is no longer tracked by the alarm.
Possible causes for an absent metric stream: The resource that was emitting the metric might have been moved or terminated, or the metric might be emitted only on failure. For more information about the internal reset period, see About the Internal Reset Period.
The following concepts are essential to working with Monitoring.
aggregated data
The result of applying a statistic and interval to a selection of raw data points for a metric. For example, you can apply the statisticmax and interval1h (one hour) to the last 24 hours of raw data points for the metricCpuUtilization. Aggregated data is displayed in default metric charts in the Console. You can also build metric queries for specific sets of aggregated data. For instructions, see Viewing Default Metric Charts and Building Metric Queries.
alarm
The alarm query to evaluate and the notification destination to use when the alarm is in the firing state, along with other alarm properties.
The Monitoring Query Language (MQL) expression to evaluate for the alarm. An alarm query must specify a metric, statistic, interval, and a trigger rule (threshold or absence). The Alarms feature of the Monitoring service interprets results for each returned time series as a Boolean value, where zero represents false and a nonzero value represents true. A true value means that the trigger rule condition has been met.
A timestamp-value pair for the specified metric. Example: 2022-05-10T22:19:00Z, 10.4
A data point is either raw or aggregated. Raw data points are posted by the metric namespace to the Monitoring service using the PostMetricData operation. The frequency of the data points posted varies by metric namespace. For example, a custom namespace might send data points for a metric at a 20-second frequency.
Aggregated data points are the result of applying a statistic and interval to raw data points. The interval of the aggregated data points is specified in the SummarizeMetricsData request. For example, a request specifying the statisticsum and interval1h (one hour) returns a sum value for each hour of available raw data points for the metric.
dimension
A qualifier provided in a metric definition. Example: Resource identifier (resourceId), provided in the definitions of oci_computeagent metrics. Use dimensions to filter or group metric data. Example dimension name-value pair for filtering by availability domain: availabilityDomain = "VeBZ:PHX-AD-1"
The time period between each posted raw data point for a metric. (Raw data points are posted by the metric namespace to the Monitoring service.) While frequency varies by metric, default service metrics typically have a frequency of 60 seconds (one data point posted per minute). See also resolution.
interval
The time window used to convert the set of raw data points.
The timestamp of the aggregated data point corresponds to the end of the time window during which raw data points are assessed. For example, for a five-minute interval, the timestamp "2:05" corresponds to the five-minute time window from 2:00:n to 2:05:00.
Supported values for interval depend on the specified time range in the metric query (not applicable to alarm queries). More interval values are supported for smaller time ranges. For example, if you select one hour for the time range, then all interval values are supported. If you select 90 days for the time range, then only interval values between 1 hour and 1 day are supported.
The content that the Alarms feature of the Monitoring service publishes to topics in the alarm's configured notification destinations. A message is sent when the alarm transitions to another state, such as from OK to FIRING.
A reference provided in a metric definition. Example: unit (bytes), provided in the definition of the oci_computeagent metricDiskBytesRead. Use metadata to determine additional information about a metric. For metric definitions, see Supported Services.
metric
A measurement related to health, capacity, or performance of a resource. Example: The oci_computeagentmetricCpuUtilization, which measures usage of a compute instance. For metric definitions, see Supported Services.
Note
Metric resources don't have OCIDs .
metric definition
A set of references, qualifiers, and other information provided by a metric namespace for a metric. For example, the oci_computeagent metricDiskBytesRead is defined by dimensions (such as resource identifier) and metadata (specifying bytes for unit) as well as identification of its metric namespace (oci_computeagent). Each posted set of data points carries this information. Use the ListMetricData API operation to get metric definitions. For metric definitions, see Supported Services.
Indicator of the resource , service, or application that emits the metric. Provided in the metric definition. For example, the CpuUtilizationmetric definition emitted by the Oracle Cloud Agent software on compute instances lists the metric namespaceoci_computeagent as the source of the CpuUtilizationmetric. For metric definitions, see Supported Services.
The following image depicts metric streams in a chart. Each line in the chart corresponds to a metric stream.
For example, consider a compartment containing three compute instances in the AD-1 availability domain (including two in the ipexample instance pool) and a fourth instance in the AD-2 availability domain. In this example, the CPU Utilization metric chart shows four lines (one per instance). When filtered by the AD-1 availability domain, the chart shows three lines. When further filtered by the ipexample instance pool, the chart shows two lines.
Details for sending messages when the alarm transitions to another state, such as from OK to FIRING. The details and setup might vary by destination service. Available destination services include Notifications and Streaming.
For the Notifications service, specify a topic. (If you're creating the topic for the alarm, also specify one or more subscription protocols (such as PagerDuty).
Software used by a compute instance to post raw data points to the Monitoring service. Automatically installed with the latest versions of supported images. See Enabling Monitoring for Compute Instances.
query
The Monitoring Query Language (MQL) expression and associated information (such as metric namespace) to evaluate for returning aggregated data. The query must specify a metric, statistic, and interval.
The period between time windows, or the regularity at which time windows shift. For example, use a resolution of 1m to retrieve aggregations every minute.
Note
For metric queries, the interval you select drives the default resolution of the request, which determines the maximum time range of data returned.
For alarm queries, the specified interval has no effect on the resolution of the request. The only valid value of the resolution for an alarm query request is 1m.For more information about the resolution parameter as used in alarm queries, see Alarm.
As shown in the following illustration, resolution controls the start time of each aggregation window relative to the previous window while interval controls the length of the windows. Both requests apply the statistic max to the data within each five-minute window (from the interval), resulting in a single aggregated data point representing the highest CPUutilization counter for that window. Only the resolution value differs. This resolution changes the regularity at which the aggregation windows shift, or the start times of successive aggregation windows. Request A doesn't specify a resolution and thus uses the default value equal to the interval (5 minutes). This request's five-minute aggregation windows are thus taken from the sets of data points emitted from 0:n to 5:00, 5:n to 10:00, and so forth. Request B specifies a 1-minute resolution, so its five-minute aggregation windows are taken from the set of data points emitted every minute from 0:n to 5:00, 1:n to 6:00, and so forth.
A custom string provided with a custom metric that can be used as a filter or to aggregate results. The resource group must exist in the definition of the posted metric. Only one resource group can be applied per metric.
The Monitoring service is available in all Oracle Cloud Infrastructure commercial regions. See About Regions and Availability Domains for the list of available regions, along with associated locations, region identifiers, region keys, and availability domains.
Supported Services 🔗
The following services have resources or components that can emit metrics to Monitoring:
Most types of Oracle Cloud Infrastructure resources have a unique, Oracle-assigned identifier called an Oracle Cloud ID (OCID). For information about the OCID format and other ways to identify your resources, see Resource Identifiers., see Resource Identifiers.
Note
Metric resources don't have OCIDs .
Ways to Access Monitoring 🔗
You can access Oracle Cloud Infrastructure (OCI) by using the Console (a browser-based interface), REST API, or OCI CLI. Instructions for using the Console, API, and CLI are included in topics throughout this documentation.For a list of available SDKs, see Software Development Kits and Command Line Interface.
Console: To access Monitoring using the Console, you must use a supported browser. To go to the Console sign-in page, open the navigation menu at the top of this page and select Infrastructure Console. You are prompted to enter your cloud tenant, your user name, and your password.
Open the navigation menu and select Observability & Management. Under Monitoring, select Service Metrics.
API: To access Monitoring through APIs, use Monitoring API for metrics and alarms and Notifications API for notifications (used with alarms).
Each service in Oracle Cloud Infrastructure integrates with IAM for authentication and authorization, for all interfaces (the Console, SDK or CLI, and REST API).
An administrator in an organization needs to set up groups , compartments , and policies that control which users can access which services, which resources, and the type of access. For example, the policies control who can create new users, create and manage the cloud network, create instances, create buckets, download objects, and so on. For more information, see Managing Identity Domains. For specific details about writing policies for each of the different services, see Policy Reference.
If you're a regular user (not an administrator) who needs to use the Oracle Cloud Infrastructure resources that the company owns, contact an administrator to set up a user ID for you. The administrator can confirm which compartment or compartments you can use.
For more information about user authorizations for monitoring, see IAM Policies.
Administrators: For common policies that give groups access to metrics, see Metric Access for Groups. For common alarm policies, see Alarm Access for Groups. To authorize resources, such as instances, to make API calls, add the resources to a dynamic group. Use the dynamic group's matching rules to add the resources, and then create a policy that allows that dynamic group access to metrics. See Metric Access for Resources.
Limits on Monitoring 🔗
See Monitoring Limits for a list of applicable limits and instructions for requesting a limit increase.
The maximum number of messages per alarm evaluation depends on the alarm destination. Limits are associated with the Oracle Cloud Infrastructure service used for the destination.
Monitoring tracks 200,000 metric streams per alarm for qualifying events. For more information about alarm evaluations, see Alarm Evaluations on this page.
For example, consider the following evaluations of an alarm that splits notifications among 200 metric streams, using a topic as its destination.
Alarm evaluation (time)
Metric stream transition
Generated messages
Sent messages
Dropped messages
00:01:00
110 metric streams transition from OK to FIRING.
110
60
50
00:02:00
90 metric streams transition from OK to FIRING.
90
60
30
When a topic or stream is overused, it can result in delayed alarm notifications. Overuse can occur when multiple resources are using that topic or stream.
Best Practices to Work Within Limits 🔗
When you expect a high volume of alarm notifications, follow these best practices to help prevent exceeding alarm message limits and associated delays.
Reserve a single topic or stream for use with a high-volume alarm. Don't use one topic or stream for multiple high-volume alarms.
If you expect more than 60 messages per minute, specify Streaming as the alarm destination.
If alarm messages exceed the stream space, then update the alarm to use a different stream that has more partitions. For example, if the original stream contains five partitions, create a stream with ten partitions and then update the alarm to use the new stream.
Note
To avoid missing messages, continue consuming the original stream until no more messages are received.