Managing Logs and Log Groups

This topic describes how to manage logs and log groups.

Overview of Logs and Log Groups

Logs contain critical diagnostic information that tells you how your resources are performing and being accessed. You can enable logging on supported resources. To see a list of supported resources grouped by service, see Supported Services.

Log groups are logical containers for organizing logs. Logs must always be inside log groups. You must create a log group to enable a log.

Use log groups to limit access to sensitive logs with IAM policy. With log groups, you don't have to rely on complex compartment hierarchies to secure your logs. For example, say the default log group in a single compartment is where you store logs for the entire tenancy. You grant access to the compartment for log administrators with IAM policy as you normally would. However, let's say some projects contain personally identifiable information (PII) and those logs should only be viewed by a select group of log administrators. Log groups allow you to put logs that contain PII into a separate log group, and then use IAM policy to restrict access to all but a few log administrators.

Log groups IAM policy

Required IAM Policy

To use Oracle Cloud Infrastructure, you must be granted security access in a policy  by an administrator. This access is required whether you're using the Console or the REST API with an SDK, CLI, or other tool. If you get a message that you don’t have permission or are unauthorized, verify with your administrator what type of access you have and which compartment  you should work in.

Administrators: for policy samples specific to logs and log groups, see Required Permissions for Working with Logs and Log Groups.

If you're new to policies, see Getting Started with Policies and Common Policies. If you want to know more about writing policies for Logging, see Details for Logging.

Required Permissions for Working with Logs and Log Groups

To enable service logs in a resource, a user must be granted manage access on the log group and access to the resource. In general, inspect access on the resource is enough, but check for specific resources. This provides permission to update the resource and permission for the log group that contains the log.

Logs and log groups use the log-group resource-type, but to search the contents of logs you must use a different resource-type.

Managing log groups and log objects

To manage groups or objects, use verbs for log-groups:

Allow group A to use log-groups in compartment C
Allow group B to mange log-groups in compartment C
Allow group D to read log-groups in compartment C

This allows users in group A to create, update, or delete log groups and log objects in compartment C.

To provision agent configurations

Three different types of access are needed:

  1. Access to operate on configurations.

    2. Access to operate on log groups.

    3. Inspect capabilities on dynamic groups or groups.

To have access to configurations, the policy should be:
Allow group B to use unified-configuration in compartment X
To create, update, or delete custom logs used as a destination in a configuration

This policy allows users in compartment B to create, update, or delete configurations in compartment X.

In order to provide a destination for the logs incoming from the configuration, you need log-groups access:
Allow group B to use log-groups in compartment X
To assign a configuration to a set of instances
To assign a configuration to a set of instances, you need inspect access to the dynamic group or group that identifies the instances:
Allow group B {IDENTITY_DYNAMIC_GROUP_INSPECT} in tenancy  / Allow group B {IDENTITY_GROUP_INSPECT} in tenancy
Enable instances to push logs into the Logging service
In order to allow instances to push logs, they need to have access to get a configuration and push the logs. This permission is controlled by log-content (where X is the compartment where the configurations are located):
Allow dynamic group production-fleet to use log-content in compartment X
To view logs
To view logs in the Console (Search), the following is required:
Allow group Searchers to read log-content in compartment X
Example log search policy

To allow a group to read the contents of indexed logs:

allow group GroupA to read log-groups in tenancy
allow group GroupA to read log-content in tenancy
Example policies for logs and log groups

In these examples, policy statements use GroupA as the name of the group.

To allow a group to simply view the log groups in the tenancy (or in a compartment) requires inspect access:

allow group GroupA to inspect log-groups in tenancy

To allow a group to read metadata for logs or log groups requires read access:

allow group GroupA to read log-groups in tenancy

To allow a group to update log groups or the logs in them, requires use access:

allow group GroupA to use log-groups in tenancy

To enable a log on a resource (or to create and delete log groups and the logs in them) requires manage access:

allow group GroupA to manage log-groups in tenancy

To allow usage of a specific log group or groups , use a where clause with the target.loggroup.id variable. For example:

Allow group GroupA to manage loggroups in tenancy where 
target.loggroup.id='ocid1.loggroup.oc1.phx.<uniqueID>'

To specify multiple log groups:

Allow group GroupA to manage log-groups in tenancy where any {target.loggroup.id='ocid1.loggroup'}
Custom logs

For custom logs the following is required. This policy is needed to allow the user to search logs via the Console Search page:

allow group userGroup1 to read log-content in compartment c
Note

While it is implicit that this is only for custom logs, and it is also true for all logs. LOG_CONTENT_READ allows reading logs from both custom and OCI service logs. It is identical in behavior to this policy:
allow group GroupA to read log-content in tenancy

The following is needed for the agent that uses the instance principal on the virtual machine to send logs:

allow dynamicgroup1 to use log-content in compartment c
Note

If user group is being used instead of a dynamic group for pushing custom logs, replace the dynamic group name with the user group name in these policies.

For custom logs, if you use allow group dynamicGroup1 to use log-content in compartment c, the instances in that dynamic group get permission to download the configuration, send logs, and can search logs.

IAM Policy Requirements for Resources

In addition to the permissions to work with the log group, to add service logs to a resource you must have the update permission for the resource. For many resources, the update permission is granted with the use verb. For example, users who can use buckets in CompartmentA, can also enable logging on a bucket in CompartmentA.

However, some resources don't include permission to update a resource with the use verb. For example, to update a rule for the Events service, you must have the full manage permission. To enable a log on an Events rule (or any other resource that doesn't include the update permission with the use verb), you must have the manage permission.

To allow a group to enable logging for these resources without granting the full permissions of manage, you can add a policy statement to grant only the <RESOURCE>_UPDATE permission (or, in the case of the Events service, <RESOURCE>_MODIFY) from the manage verb. For example, to allow a group EventUsers to enable logs on Events rules in CompartmentA, you could write a policy like:

Allow group EventUsers to read cloudevents-rules in compartment CompartmentA
Allow group EventUsers to manage cloudevents-rules in compartment CompartmentA 
 where request.permission='EVENTRULE_MODIFY'

For information about resource permissions, see Policy Reference.

VCN Flow Logs IAM Policy

In addition to Required Permissions for Working with Logs and Log Groups, subnet read and update permissions are required for managing VCN Flow Logs.

These subnet permissions can be provided by one of the following policies, listed in order from broader to narrowed privileges:

Allow group FlowLogsEnablers to manage virtual-network-family in tenancy 

or

Allow group FlowLogsEnablers to manage subnets in tenancy

or

Allow group FlowLogsEnablers to {SUBNET_READ, SUBNET_UPDATE} in tenancy

This group is similar to what is described for EventUsers in IAM Policy Requirements for Resources.

Example Scenario

Your company has an Operations department. Within the Operations department are several cost centers. You want to be able to tag resources that belong to the Operations department with the appropriate cost center.

  1. Create a log group called "confidential".
  2. Add logs with sensitive data to the "confidential" log group.

An employee named Alice already belongs to the group BucketManagers. Alice can manage buckets in CompartmentA. You want Alice and other members of BucketManagers group to be able to enable logs on buckets in CompartmentA.

To grant the BucketManagers group access to the sensitive data log group (and only the sensitive data log group), add the following statements to the BucketManagers policy:

Allow group BucketManagers to manage log-groups in compartment CompartmentA where 
target.loggroups.id='ocid1.lumloggroup.oc1.phx.<uniqueID>'

Alice can now enable logs to bucket resources in CompartmentA.

Move Log Groups to a Different Compartment

You can move log groups from one compartment to another. When you move a log group to a new compartment, all the logs in the log group move with the log group to the new compartment. After you move the log group to the new compartment, the policies in the new compartment apply immediately, and affect access to the log group and any logs the log group contains.

For more information, see Moving Resources to a Different Compartment.

Log and Log Group Names

For log group names, the first character must start with a letter. Otherwise, the following guidelines apply to both log and log group names:

  • Use from 1 to 256 characters.
  • Valid characters are letters (upper or lower case), numbers, hyphens, underscores, and periods.
  • Log and log group names are case-sensitive. Logging handles write-log and WRITE-log as separate logs.
Caution

Avoid entering confidential information when assigning descriptions, tags, or friendly names to your cloud resources through the Oracle Cloud Infrastructure Console, API, or CLI.

Legacy Archival

Legacy automatic log archival was previously available before general release. This is a toggle configurable for every log. If enabled, it automatically creates a bucket in your compartment, and places a copy of your log there. New and improved functionality is now available in Service Connector Hub.

Using the Console

The Logs table lists both custom logs and service logs (indicated by the Log type field). The table is organized in terms of the following fields:

  • Log name
  • Log type
  • Status
  • Details
  • Created

From this page you can click the Log name entry to go to the Log details page, or click the linked resource in Details to go directly to the resource. For example, if the log is for the Load Balancer service, clicking the link opens the Load Balancer Details page. From the action menu, you can edit the log, disable logging, change the log group, view tags, or delete the log.

To view the contents of logs
  1. Open the navigation menu. Under Solutions and Platform, go to Logging, and then click Logs.
  2. Under List Scope, Compartment, choose a compartment you have permission to work in.
  3. Under Log name, click the name of the log you want to view. The log detail page opens. This page displays the following on the Log Information tab:
    • OCID
    • Compartment
    • Log Group (click to view the log group contents)
    • Created date and time in UTC format
    • Retention Period
      Note

      Retention period can be set in 30-day increments, up to a maximum of 180 days.
    • Legacy Archival mode
    • Status (Creating, Active, Updating, Inactive, Deleting, Deleted)
    • The Tags tab shows associated tags for this log.
    • Under Log Details, following is displayed: the Log Type (whether Service or Custom), Service (for service logs), Log Category, and Resource (click to view the linked resource).

      In the Explore Log resource, log data is displayed in a very similar manner as the Log Data on the Search page. You can apply some simple filters, such as sorting by newest or oldest from the Sort field, or filtering by time from the corresponding Filter by Time field.

  4. Click Explore with Log Search, which allows you to view this log on the Search page directly. After clicking this link, the Search page opens with the Select Logs to Search field populated with the log in the filter settings. At this point you can perform more analysis and investigation related to this log directly on the Search page. For more information, see Searching Logs.

    In addition to these functions in the Explore Log resource, you can also click the Metrics resource to view interactive charts for either a chosen time period (Start Time and End Time), or pre-selected ranges from Quick Selects. The charts display the Bytes Ingested (total bytes of log entries ingested) and the Search Success (number of successful search queries issued by the user).

    Clicking anywhere in a chart displays a larger version of the chart. You can perform several chart actions from the Options menu (both in the Metrics resource and in the zoomed-in view):
    • View Query in Metrics Explorer: Opens the chart in the Monitoring Metrics Explorer. See Metrics Explorer page for more information.
    • Copy Chart URL
    • Copy Query (MQL)
    • Create an Alarm on this Query: Opens the Monitoring Create Alarm page. See To create an alarm for more information.
    • Table View: Displays a tabular summary of the chart data. Select Chart View to switch back to the chart.
To delete a log
  1. Open the navigation menu. Under Solutions and Platform, go to Logging, and then click Logs.
  2. Under List Scope, Compartment, choose a compartment you have permission to work in.
  3. Under Log name, click the name of the log you want to view. The log detail page opens.
  4. Click Delete. A confirmation dialog is displayed regarding the delete operation.
  5. Confirm by clicking Delete.

    From the main Logs page, for the log you want to delete, you can also click the the Actions icon (three dots), and then click Delete.

To move a log to a new log group
  1. Open the navigation menu. Under Solutions and Platform, go to Logging, and then click Logs.
  2. Under List Scope, Compartment, choose a compartment you have permission to work in.
  3. Under Log name, click the name of the log you want to view. The log detail page opens.
  4. Click Change Log Group. The Move to a Different Log Group dialog is displayed.
  5. From the Choose New Group drop-down, select the new group from the list, and click Chang Log Group. The new log group membership is reflected on the Log Information tab's Log Group field.

    From the main Logs page, for the log you want to move to a new group, you can also click the the Actions icon (three dots), and then click Change Log Group.

To enable or disable an existing log
  1. Open the navigation menu. Under Solutions and Platform, go to Logging, and then click Logs.
  2. Under List Scope, Compartment, choose a compartment you have permission to work in.
  3. Under Log name, click the name of the log you want to view. The log detail page opens.
  4. Click Disable Log/Enable Log. A confirmation dialog is displayed regarding the disabling or enabling of the log.
  5. Confirm by clicking Disable Log/Enable Log. The log detail page changes its status and displays Inactive (for a disabled log) or Active (for an enabled log) in the status field, both on the log detail page and the Logs page.

    From the main Logs page, for the log you want to enable or disable, you can also click the the Actions icon (three dots), and then click Disable Logging/Enable Logging.

To create a log group
  1. Open the navigation menu. Under Solutions and Platform, go to Logging, and then click Log Groups.
  2. Choose a compartment you have permission to work in and click Create Log Group. The Create Log Group panel is displayed.
  3. Enter the following:
    • Compartment: The compartment in which you want to create the log group. This field is pre-filled based on your compartment choice.
    • Name: A name for this log group. The first character of a log group name must be a letter. For more, see Log and Log Group Names.

    • Description: A friendly description.
    • Optionally, enter tagging information.
  4. Click Create. The log group detail page is then displayed. From this page you can:
    • Edit the group
    • Move resources
    • Add tags
    • Delete the log group
    • View log group information and tags
    • View log group resources (explore the log group, view the logs included in the log group, create custom or service logs, and view metrics)

      The Metrics resource in a log group detail page functions the same as in a log detail page. See To view the contents of logs for more information.

To edit a log group
  1. Open the navigation menu. Under Solutions and Platform, go to Logging, and then click Log Groups.
  2. Choose a compartment you have permission to work in and click the linked log group name under Log Group in the table. The log group detail page is displayed.
  3. Click Edit. The Edit Log Group panel is displayed. From here you can change the log group name and its description in the associated fields. See Log and Log Group Names for more information on naming.

    From the main Log Groups page, for the log group you want to edit, you can also click the the Actions icon (three dots), and then click Edit.

  4. Make your changes and click Update.
Note

You cannot move, edit, or delete the default log group.
To delete a log group
  1. Open the navigation menu. Under Solutions and Platform, go to Logging, and then click Log Groups.
  2. Choose a compartment you have permission to work in and click the linked log group name under Log Group in the table. The log group detail page is displayed.
    Tip

    You cannot delete a log group that contains logs.
  3. Click Delete. A confirmation dialog is displayed regarding the delete operation.
  4. Confirm by clicking Delete. The log group is removed from the Log Groups page.

    From the main Log Groups page, for the log group you want to delete, you can also click the the Actions icon (three dots), and then click Delete.

Note

You cannot move, edit, or delete the default log group.
To move a log group to a different compartment
  1. Open the navigation menu. Under Solutions and Platform, go to Logging, and then click Log Groups.
  2. Choose a compartment you have permission to work in and click the linked log group name under Log Group in the table. The log group detail page is displayed.
  3. Click Move Resource. The Move Resource to a Different Compartment dialog is displayed.
  4. Chose the new compartment and then click Move Resource.

    From the main Log Groups page, for the log group you want to move to a new compartment, you can also click the the Actions icon (three dots), and then click Move Resource.

Note

You cannot move, edit, or delete the default log group.
To list the logs in a log group
  1. Open the navigation menu. Under Solutions and Platform, go to Logging, and then click Log Groups.
  2. Choose a compartment you have permission to work in.
  3. For the log group you want to inspect for logs, click the name of the log group under Log Group in the table. The log group detail page is displayed.
  4. In Resources, click Logs to display a list of all the logs contained in the log group. This resource table of logs functions in the same manner as the main Logs page.

Using the Command Line Interface (CLI)

For more information on installing the CLI, see Quickstart, and logging-ingestion, logging, and logging-search for command documentation.

Agent Configuration Commands

The following are agent configuration-related commands:

To create a unified agent configuration registration
Open a command prompt and run:
oci logging agent-configuration create --compartment-id | -c <compartment_OCID>, 
 --is-enabled <boolean>, --service-configuration <service_configure>
To create a unified agent log configuration registration
Open a command prompt and run:
oci logging agent-configuration create-log-configuration --compartment-id | -c <compartment_OCID>,
 --is-enabled <boolean>
To get a unified agent configuration for an ID
Open a command prompt and run:
oci logging agent-configuration get --config-id <agent_configuration_OCID>
To list all agent configurations in the specified compartment
Open a command prompt and run:
oci logging agent-configuration list --compartment-id | -c <compartment_OCID>
To update an existing unified agent configuration
Open a command prompt and run:
oci logging agent-configuration update --config-id <agent_configuration_OCI>, 
 --display-name <configuration_name>, --is-enabled <Boolean>, --service-configuration <service_configure>

This call fails if the log group does not exist.

To update an existing unified agent log configuration
Open a command prompt and run:
oci logging agent-configuration update-log-configuration --config-id <agent_configuration_OCI>, 
 --display-name <configuration_name>, --is-enabled <Boolean>

This call fails if the log group does not exist.

To move a unified agent configuration into a different compartment within the same tenancy
Open a command prompt and run:
oci logging agent-configuration change-compartment --config-id <agent_configuration_OCID>

When provided, the If-Match is checked against the ETag values of the resource. For information about moving resources between compartments, see Moving Resources to a Different Compartment.

To delete a unified agent configuration
Open a command prompt and run:
oci logging agent-configuration delete --config-id <agent_configuration_OCID>

Log Commands

The following are log commands:

Create a log within a specified log group
Open a command prompt and run:
oci logging log create --display-name <log_name>, --log-group-id <log_group_OCID>, 
 --log-type <SERVICE_or_CUSTOM>
This call fails if the log group has already been created with the same displayName or (service, resource, category) triplet.
Get the log object configuration for the log object OCID
Open a command prompt and run:
oci logging log get --log-group-id <log_group_OCID>, --log-id <log_OCID>
List the specified log group's log objects
Open a command prompt and run:
oci logging log list --log-group-id <log_group_OCID>
Move a log into a different log group within the same tenancy
Open a command prompt and run:
oci logging log change-log-group --log-group-id <log_group_OCID>, --log-id <log_OCID>

When provided, the If-Match is checked against the ETag values of the resource.

To delete a log object in a log group
If you have an issue with deleting a log object, open a command prompt and run the following command to delete it:
oci logging log delete --log-group-id <log_group_OCID>, --log-id <log_OCID>
To ingest logs associated with a logId
Open a command prompt and run:
oci logging-ingestion put-logs --log-entry-batches, --log-id, --specversion
List all services supporting logging
Open a command prompt and run:
oci logging service list
Update an existing log object with the associated configuration
Open a command prompt and run:
oci logging log update --log-group-id <log_group_OCID>, --log-id <log_OCID>

This call fails if the log object does not exist.

Log Group Commands

The following are log group-related commands:

Create a new log group with a unique display name
Open a command prompt and run:
oci logging log-group create --compartment-id | -c <compartment_OCID>, --display-name <log_group_name>

This call fails if the log group is already created with same displayName in the compartment.

Get a specified log group's information
Open a command prompt and run:
oci logging log-group get --log-group-id <log_group_OCID>
List all log groups for the specified compartment or tenancy
Open a command prompt and run:
oci logging log-group list --compartment-id | -c <compartment_OCID>
Updates an existing log group with the associated configuration
Open a command prompt and run:
oci logging log-group update --log-group-id <log_group_OCID>

This call fails if the log group does not exist.

Move a log group into a different compartment within the same tenancy
Open a command prompt and run:
oci logging log-group change-compartment --log-group-id <log_group_OCID>

When provided, the If-Match is checked against the ETag values of the resource. For information about moving resources between compartments, see Moving Resources to a Different Compartment.

To delete a specified log group
Open a command prompt and run:
oci logging log-group delete --log-group-id <log_group_OCID>

Work Request Commands

The following are work request-related commands:

Get the details of a work request with a given ID
Open a command prompt and run:
oci logging work-request get --work-request-id <work_request_OCID>
List the work requests in a compartment
Open a command prompt and run:
oci logging work-request list --compartment-id | -c <compartment_OCID>
List the errors for a given work request
Open a command prompt and run:
oci logging work-request-error list --work-request-id <work_request_OCID>
List the logs for a given work request
Open a command prompt and run:
oci logging work-request-log list --work-request-id <work_request_OCID>
Delete a work request that has not yet started
Open a command prompt and run:
oci logging work-request delete --work-request-id <work_request_OCID>

Object Storage Example

The following is an Object Storage example:
To create a log group and create a log in Object Storage
Open a command prompt and run:
oci logging log-group create --compartment-id <compartment_OCID> 
 --display-name CLITestLogGroup
oci logging log create --display-name object_log_write --log-group-id <log_group_OCID>
 --log-type SERVICE --is-enabled true --configuration file://~/.oci/objectstorage_configuration.json
objectstorage_configuration.json:
{
  "archiving": {
    "isEnabled": true
  },
  "compartmentId": "ocid1.compartment.oc1..<unique_ID>",                 
  "source": [
    {
      "category":"write",
      "parameters": null,
      "resource": "bucket-cli-sample",    
      "service": "objectstorage",       
      "sourceType": "OCISERVICE"
    }
  ]
}

VCN Flow Logs Example

The following are sample commands related to VCN Flow Logs:

To create a log group
Open a command prompt and run:
oci logging log-group create --compartment-id <compartment_OCID> --display-name <log_group_name> 
 --description <description>
To create a flowlogs log object (enable Flow Logs)
Open a command prompt and run:
oci logging log create --display-name <log_display_name> --log-group-id <log_group_OCID> 
 --description <description> --log-type SERVICE --is-enabled <Boolean> 
 --configuration file://input.json

Sample configuration file:

{
    "compartment-id":"...",               # CompartmentId of where the subnet resource is present.
    "source": {
        "resource": "ocid1.subnet.....",  # OCID of subnet for which flowlogs is enabled.
        "service": "flowlogs",            # "flowlogs" is the official service name and it should be all lowercase.
        "source-type": "OCISERVICE",      # OCISERVICE is the name of the Logging source-type.
        "category": "all"
    }
}
To disable a flowlogs log object (disable Flow Logs)
Open a command prompt and run:
oci logging log update --log-group-id <log_group_OCID> --log-id <log_OCID> --is-enabled false
To delete the log object
Open a command prompt and run:
oci logging log delete --log-id <log_OCID>

Functions Example

To enable Functions logging

To enable Functions logging, open a command prompt and run:

oci logging log create --display-name cli_test --log-group-id ocid1.loggroup.oc1.phx.<log_group_OCID> --log-type SERVICE --is-enabled true --configuration file://fnconfig.json

Sample fnconfig.json configuration file:

{
    "compartment-id":"ocid1.compartment.oc1..<compartment_OCID>"
    "source": {
        "resource": "ocid1.fnapp.oc1.phx.<unique_ID>",
        "service": "functions",
        "source-type": "OCISERVICE",
        "category": "invoke"
    }
}