Custom Logs

Custom logs are logs that contain diagnostic information from custom applications, other cloud providers, or an on-premise environment. Custom logs can be ingested in two ways:
  1. By using PutLogs to ingest custom logs directly. Also see the Logging Ingestion API and REST APIs for more information.
  2. By configuring the Unified Monitoring Agent. See Installing the Agent for instructions.

Custom logs can be viewed in the Oracle Cloud Infrastructure Compute instance page, and have an associated Logs resource. They can also be viewed on the Logging Search page, Logs page, or within an associated Log Groups detail page. Custom logs are also supported in bare metal instances.

The agent can be installed on many machines, and it pulls logs from local directories, where your apps or systems emit logs. The agent can also parse your logs for you. All of this is configured in the agent configuration. You can create an agent configuration separately and then associate a custom log with it, or create a custom log and then later create its agent configuration.

An agent configuration is the central mechanism for defining:
  • What hosts you want logs from.
  • What specific logs you want from the hosts.
  • Additional parsers.
  • The custom log destination.

Creating a custom log is a two-step process, in that you create the custom log object first, and then secondly, create its associated agent configuration. See Creating Custom Logs for more information on creating custom logs and agent configurations, and Agent Management for more information on setting up and managing the agent.

Creating Custom Logs

To create custom logs:

  1. Open the navigation menu. Under Solutions and Platform, go to Logging, and then click Logs.
  2. Under List Scope, Compartment, choose a compartment you have permission to work in.
  3. In Custom Log Name, enter a name for the custom log.
  4. From Compartment, choose a compartment you have permission to work in.
  5. From Log Group, select a log group to place the custom log into.
  6. Optionally, select a log retention value from Log Retention, and add any applicable tags in Add Tags.
  7. Click Create Log Object. The Create Agent Configuration panel is displayed, where you can create a new configuration next, to define the parameters for the associated log data (the default), or add it later.
  8. In Name and compartment, enter a Configuration Name in the corresponding field, and select a Compartment you have permissions to work in.
  9. In Choose Host Groups, which allows you to define which VMs apply to this configuration, select a Group Type from the drop-down list, whether Dynamic Group or User Group.

    For the Dynamic Group case, Dynamic Group refers to a group of instances, which you can create in the IAM feature of the Console. See About Dynamic Groups for more information. These Dynamic Groups can be selected from the Groups field when setting up Dynamic Group settings.

    For the User Group case, select the group from the Groups field. User Groups also refer to the IAM Groups feature of the Console. See Managing Groups for more information.

    Click Add Host Group to add additional groups. You can add a combination of Group Types for the agent configuration, that is, both Dynamic Groups and User Groups can be set up in the configuration.

    Note

    A maximum of five groups per configuration are allowed, and a host can be in a maximum of five different groups.
  10. Next, in the configuration, you need to define the format of the logs (that is, what logs do you want to watch for) in Configure Log Inputs. Select an Input Type form the drop-down list, whether Windows Event Log or Log Directory.
    • For Windows Event Log, enter an Input Name and select an Event Channels option from the drop-down list.
    • For Log Directory, enter an Input Name and a Path in the corresponding fields. For example, /<log_path>/<log_name>. Multiple paths can be entered.
    Click Advanced Parser Options, which opens the Advanced Parser Options panel. This allows you to specify how to parse the log, according to the following parsers. Some of the parsers require further input and have additional options, depending on the type chosen.
    • AUDITD
    • JSON
    • TSV
    • CSV
    • NONE (the default)
    • SYSLOG
    • APACHE2
    • APACHE_ERROR
    • MSGPACK
    • REGEXP
    • MULTILINE
    For example for JSON, you must select a Time Type value from its drop-down, while optionally, you can specify additional event time and null field settings. Meanwhile for REGEXP, you specify the regular expression for matching logs, along with the time format. See Log Inputs and Parsers for more information.
  11. After configuring the log inputs and the parser, you can optionally specify any tag settings. Click Submit to save your changes, and create the custom log and its associated agent configuration.

In summary, the agent configuration defines what instances the configuration applies to (Choose Host Groups), which log files are obtained and what parser (if any) is used (Configure Log Inputs), and to what log object in the Oracle Cloud Infrastructure system that the records are pushed to (Select log destination). The latter is already set up since this was set during the custom log creation step.

The custom log object is now created, as well as the agent configuration, which pulls data from instances, and pushes into the custom log object.