Troubleshooting Connectors

Use troubleshooting information to identify and address common issues that can occur while working with Connector Hub.

No Data is Being Moved

Troubleshoot issues with moving data through connectors.

The connector isn't moving data. For example connectors, see Connector Hub Scenarios.

Cause: Data doesn't exist

The source doesn't contain data matching the query in the connector's source configuration.

To find out if data exists at the source, do one of the following:

  • Get service logs for the connector. (If needed, enable logs first.) Following is an example log message that indicates a successful connector run, including the amount of data moved:
    Service connector run succeeded - <number> messages (<number> bytes) written to target
  • For source logs, search the logs using the query from the connector's source configuration.
  • For source metrics, query the metrics.
  • For source streams, query the stream's default producer metrics.

Remedy: Update the connector

Change the configuration for the source service to specify data that exists. See Updating a Connector.

Cause: Service-level error

An error at the source service, target service, or Connector Hub service is preventing movement of data.

To find out if a service-level error exists, do one of the following:

  • Get service logs for the connector. (If needed, enable logs first.) Following is an example log message that indicates a service-level error:
    Connector run failed due to <type> error, Error Code : XXX <exception>
  • View the following metric charts for the connector:
    • Errors at source
    • Errors at target
    • Connector Hub errors

Remedy: Update the connector

Change the configuration for the source service or target service. See Updating a Connector.

Remedy: Contact support

If you see 5xx or -1 in Connector Hub errors, then contact support. See Getting Help and Contacting Support.

Cause: Missing authorization

You don't have authorization to write to the target service.

To find out if authorization is missing, get service logs for the connector. (If needed, enable logs first.) Following is an example log message that indicates missing authorization:

Connector run failed due to <type> error, Error Code : 404 NotAuthorizedOrNotFound

Remedy: Get authorization

Ensure you have authorization, either through the default policy offered when creating or updating the connector or through a group-based policy. See Access to Source, Task, and Target Services.
Note

Your accepted default policies might take a few minutes to propagate to regions that aren't your home region. The connector doesn't move data until the policies are propagated.

Deactivation for Unknown Reasons

Troubleshoot a deactivated connector.

A connector's status is Deactivated and you didn't deactivate it.

Someone Else Deactivated the Connector

The connector was deactivated by someone else:

  • Another user at your organization
  • Oracle Cloud Infrastructure

    Any connector that continuously fails for seven days is deactivated by the service team at Oracle Cloud Infrastructure. Such a long-term continuous failure can indicate invalid configuration of the connector's source or target.

Reactivate the Connector

  1. Update the connector to ensure valid configuration of its source and target.
  2. Reactivate the connector.
  3. Confirm that the connector successfully moves data by checking for expected results at the target service.

Can't View Query in Basic Mode

Troubleshoot issues viewing the query in Basic mode in the Console.

The Console doesn't allow selection of Basic mode after you edit the query in Advanced mode.

Cause: The query is too complex

The query includes one or more of the following items not supported in Basic mode:

  • OR operator (except with type-based filters when you only use Audit logs)
  • Functions (for example: isNull())
  • select
  • summarize

Remedy: Update the query

Update the query so that it only includes elements supported in Basic mode:

  • Audit logs only: Type-based filters can use the OR operator. Other filters must use the AND operator.
    Example:
    ((type = value1 OR type = value2) AND field = value3 AND field1 = value4)
  • Any combination of logs (Service logs, custom logs, and Audit logs): Filters joined with the AND operator.
    Example:
    (field = value AND field1 != value1)

How Do I Know When Issues Occur?

Identify when issues occur with connectors.

Look for the following indicators of issues with connectors.

  • Data freshness for a single connector: Look for unexpected lapses of time between data movement.
    1. Open the navigation menu and click Analytics & AI. Under Messaging, click Connector Hub.
    2. Choose a Compartment.
    3. Click the name of the connector that you want.
    4. Under Resources, click Metrics.
    5. Review the Data freshness metric chart.
  • Data freshness across connectors: Look for unexpected lapses of time between data movement.
    1. Open the navigation menu and click Observability & Management. Under Monitoring, click Service Metrics.
    2. Choose the Compartment that contains the connectors you want to view data freshness for.
    3. For Metric namespace, select oci_service_connector_hub.

    4. Review the following metric charts: 
      • Data freshness
  • Logging source: If the connector retrieves data from a log, then it might be attempting more than the maximum amount of hourly retrieval of data per connector (1 GB). Log data at the target isn't delivered if this issue continues to occur past 24 hours (the maximum duration for catching missed data in previous transmissions by the connector). To determine if this issue is occurring, create alarms to monitor the following indicators.

    Note

    For steps to edit alarm queries in MQL, see Editing the MQL Expression When Updating an Alarm.
    Indicator (Metric) Alarm query in MQL, with comments
    Data older than 12 hours (Data freshness)
    DataFreshness[1h].mean() > 43200000

    Comments:

    • The value 43200000 is the number of milliseconds in 12 hours.
    • Ignore occasional failures. We recommend setting the alarm trigger delay to 30 minutes or more. With this configuration, the alarm only alerts you when multiple consecutive failures occur over the specified time period.
    Error at source (any error) (Errors at source)
    ErrorsAtSource[15m].groupby(errorCode,connectorId).min() > 0

    Comments:

    • Ignore occasional failures. We recommend setting the alarm trigger delay to 30 minutes or more. With this configuration, the alarm only alerts you when multiple consecutive failures occur over the specified time period.
    • Results are grouped by error code and connector.
    Internal errors at source that don't resolve after 15 minutes (5xx) (Errors at source)
    ErrorsAtSource[15m]{errorCode =~ "5*"}.groupby(connectorId).sum() > 0 && 
    ErrorsAtSource[15m].groupby(connectorId).min() > 0

    Comments:

    • Internal errors might indicate an issue at the source, which could delay delivery of data.
    • To trigger the alarm at shorter intervals, change the interval ([15m]).
    • Ignore occasional failures. We recommend setting the alarm trigger delay to 30 minutes or more. With this configuration, the alarm only alerts you when multiple consecutive failures occur over the specified time period.
    Throttling errors at source (429) (Errors at source)
    ErrorsAtSource[15m]{errorCode = "429"}.groupby(connectorId).sum() >0 && 
    ErrorsAtSource[15m].groupby(connectorId).min() > 0

    Comments:

    • For more information on throttling errors, see documented limits for the relevant service.
    • For example, for throttling errors related to the Streaming source, see Limits on Streaming Resources. Throttling at the Streaming source occurs when a connector attempts to read a stream from a partition, other calls to the same partition are also occurring, and the number of calls exceeds service limits.
    • Ignore occasional failures. We recommend setting the alarm trigger delay to 30 minutes or more. With this configuration, the alarm only alerts you when multiple consecutive failures occur over the specified time period.
    Service communication errors at source (-1) (Errors at source)
    ErrorsAtSource[15m]{errorCode = "-1"}.groupby(connectorId).sum() >0 && 
    ErrorsAtSource[15m].groupby(connectorId).min() > 0
    404 error at source (Errors at source)
    ErrorsAtSource[15m]{errorCode = "404"}.groupby(connectorId).sum() >0

    Comments:

    • Either the connector can't access the log or it doesn't exist.
    • Ensure that policies exist for accessing the specified log. See Access to Source, Task, and Target Services.
    • Confirm that the specified log exists by searching for it in Logging.
    • Ignore occasional failures. We recommend setting the alarm trigger delay to 30 minutes or more. With this configuration, the alarm only alerts you when multiple consecutive failures occur over the specified time period.
    Zero (0) bytes read (when data is expected) (Bytes read from source)
    BytesReadFromSource[15m].groupby(connectorId).sum() == 0

    Comments:

    • If errors aren't occurring at source, target, or task, then the log might not exist. Confirm that the specified log exists by searching for it in Logging.
    • Ignore occasional failures. We recommend setting the alarm trigger delay to 30 minutes or more. With this configuration, the alarm only alerts you when multiple consecutive failures occur over the specified time period.