Managing Kafka Connect Configurations

This topic describes how to work with Kafka Connect configurations. The Streaming API calls these configurations harnesses.

Required IAM Policy

To use Oracle Cloud Infrastructure, you must be granted security access in a policy  by an administrator. This access is required whether you're using the Console or the REST API with an SDK, CLI, or other tool. If you get a message that you don’t have permission or are unauthorized, verify with your administrator what type of access you have and which compartment  you should work in.

To allow a group to manage Kafka Connect configurations, you need to create the correct policy in your tenancy. For example:

allow group KafkaAdmins to manage connect-harnesses in tenancy

For administrators: The policy in Let streaming users manage streams lets the specified group do everything with streaming and related Streaming service resources.

If you're new to policies, see Getting Started with Policies and Common Policies. If you want to dig deeper into writing policies for the Streaming service, see Details for the Streaming Service in the IAM policy reference.

Using the Console

To create a Kafka Connect Configuration
  1. Open the navigation menu. Click Solutions and Platform, locate Analytics, and then click Streaming.
  2. Click on Kafka Connect Configurations on the left side of the screen.

    A list of existing Kafka Connect configurations is displayed.

  3. Click Create Kafka Connect Configuration to display the Create Kafka Connect Configuration page.
  4. Select a compartment in the Compartment drop-down list.
  5. Enter a name for the configuration in the Kafka Connect Configuration Name text box.
  6. TagsIf you have permissions to create a resource, then you also have permissions to apply free-form tags to that resource. To apply a defined tag, you must have permissions to use the tag namespace. For more information about tagging, see Resource Tags. If you are not sure if you should apply tags, then skip this option (you can apply tags later) or ask your administrator.
  7. Click Create Kafka Connect Configuration.
  8. The details page for the new Kafka Connect configuration has a read-only text box labeled Kafka Connect Storage Topics.
    1. Click Copy to copy the connector configuration so you can paste it into the connect-distributed.properties file for your Kafka connector. For more information, see the official Kafka Connect documentation.
To delete a Kafka Connect Configuration
  1. Open the navigation menu. Click Solutions and Platform, locate Analytics, and then click Streaming.
  2. Click on Kafka Connect Configurations on the left side of the screen.

    A list of existing Kafka Connect configurations is displayed.

  3. You can delete a Kafka Connect configuration in two ways:
    1. Click the the Actions icon (three dots) on the right side of the configuration you want to delete and select Delete Kafka Connect Configuration.
    2. Click the configuration you want to delete. The Kafka Connect Configuration Details page displays. Click Delete.
  4. Confirm when prompted.
To move a Kafka Connect Configuration to a different compartment
  1. Open the navigation menu. Click Solutions and Platform, locate Analytics, and then click Streaming.
  2. Click on Kafka Connect Configurations on the left side of the screen.

    A list of existing Kafka Connect configurations is displayed.

  3. You can move a Kafka Connect configuration in two ways:
    1. Click the the Actions icon (three dots) on the right side of the configuration you want to move and select Move Resource.
    2. Click the configuration you want to move. The Kafka Connect Configuration Details screen displays. Click Move Resource.
  4. In the Move Resource to a Different Compartment dialog box, choose the destination compartment from the drop-down list.
  5. Click Move Resource.

Using the Command Line Interface (CLI)

For information about using the CLI, see Command Line Interface (CLI). For a complete list of flags and options available for CLI commands, see the Command Line Reference.

Note

The examples in this section use the full syntax for all parameters, for example --compartment-id. For some parameters, there are shortened versions that you can use instead, like -c. See the CLI online help for instances of a shortened parameter associated with a command.
To get a list of Kafka Connect configurations

oci streaming admin connect-harness list --compartment-id <compartment_OCID>

For example:

oci streaming admin connect-harness list --compartment-id exampleuniqueID
{
  "data": [
    {
      "compartment-id": "ocid1.tenancy.oc1..exampleuniqueID",
      "defined-tags": {},
      "freeform-tags": {},
      "id": "ocid1.connectharness.oc1.phx.exampleuniqueID",
      "lifecycle-state": "ACTIVE",
      "name": "MyKafkaConnectConfig",
      "time-created": "2020-08-31T17:26:09.640000+00:00"
    }
  ]
}

By default, getting a list of Kafka Connect configurations returns up to the first 10 configurations in the compartment.

To create a Kafka Connect configuration

oci streaming admin connect-harness create --name <kafka_connect_configuration_name> --compartment-id <compartment_OCID>

For example:

oci streaming admin connect-harness create --name MyKafkaConnectConfig --compartment-id exampleuniqueID
{
  "data": {
    "compartment-id": "ocid1.tenancy.oc1..exampleuniqueID",
    "defined-tags": {},
    "freeform-tags": {},
    "id": "ocid1.connectharness.oc1.phx.exampleuniqueID",
    "lifecycle-state": "CREATING",
    "lifecycle-state-details": null,
    "name": "MyKafkaConnectConfig",
    "time-created": "2020-11-04T17:36:44.323000+00:00"
  },
  "etag": "\"48886b1f-333f-4142-a335-ed42e88b37f8-0e7d329d-c6df-47fc-b814-f1237b414281\""
}
To view Kafka Connect configuration details

oci streaming admin connect-harness get --connect-harness-id <kafka_connect_configuration_OCID>

For example:

oci streaming admin connect-harness get --connect-harness-id exampleuniqueID
{
  "data": {
    "compartment-id": "ocid1.tenancy.oc1..exampleuniqueID",
    "defined-tags": {},
    "freeform-tags": {},
    "id": "ocid1.connectharness.oc1.phx.exampleuniqueID",
    "lifecycle-state": "ACTIVE",
    "lifecycle-state-details": null,
    "name": "MyKafkaConnectConfig",
    "time-created": "2020-08-31T17:26:09.640000+00:00"
  },
  "etag": "\"5cca5a19-ad06-42f4-b11e-3945d1b0ece5-a3cec7b6-7b84-4056-b411-10fd9f4b1763\""
}
To update a Kafka Connect configuration

oci streaming admin connect-harness update --connect-harness-id <kafka_connect_configuration_OCID> --defined-tags <JSON_tags>

For example:

oci streaming admin connect-harness update --connect-harness-id exampleuniqueid --defined-tags "{\"tagNamespace\": {\"tagKey1\": \"tagValue1\",\"tagKey2\": \"tagValue2\"}}"
WARNING: Updates to freeform-tags and defined-tags will replace any existing values. Are you sure you want to continue? [y/N]: y 

Select y and press Enter. The Kafka Connect configuration is updated:

{
  "data": {
    "compartment-id": "ocid1.tenancy.oc1..exampleuniqueID",
    "defined-tags": {
      "tagNamespace": {
        "tagKey1": "tagValue1",
        "tagKey2": "tagValue2"
      }
    },
    "freeform-tags": {},
    "id": "ocid1.connectharness.oc1.phx.exampleuniqueID",
    "lifecycle-state": "UPDATING",
    "lifecycle-state-details": null,
    "name": "MyKafkaConnectConfig",
    "time-created": "2020-08-31T17:26:09.640000+00:00"
  },
  "etag": "\"f3b74455-2584-4f20-a8bd-24068b1d0cba-2f66ecee-2e34-4c92-90d6-5c643b2be3fb\""
}
To delete a Kafka Connect configuration
Caution

Kafka Connect configurations are deleted immediately. You cannot recover a deleted configuration.

oci streaming admin connect-harness delete --connect-harness-id <kafka_connect_configuration_OCID>

For example:

oci streaming admin connect-harness delete --connect-harness-id ocid1.connectharness.oc1.phx.exampleuniqueID
Are you sure you want to delete this resource? [y/N]:

Select y and press Enter. The Kafka Connect configuration is deleted with no further prompting.

To move a Kafka Connect configuration to a different compartment

oci streaming admin connect-harness change-compartment --connect-harness-id <kafka_connect_configuration_OCID> --compartment-id <target_compartment_OCID>

For example:

oci streaming admin connect-harness change-compartment --connect-harness-id ocid1.connectharness.oc1.phx.exampleuniqueID --compartment-id ocid1.compartment.oc1..exampleuniqueID
{
  "etag": "\"69aae3a1-e5f0-4289-bbbf-950c0fdddf2e-cbdd76e0-9a97-49be-a747-812ba28248d1\""
}