Managing Stream Pools

This topic describes how to work with stream pools.

Overview

Stream pools are logical groupings for streams. Every stream needs to be a member of a stream pool. If you don't create a stream pool, the Streaming service uses a default pool to contain your streams.

You can use stream pools to:

  • Organize streams into groups matching your organizational structure or a specific solution
  • Restrict access to a specified virtual cloud network (VCN) inside your tenancy so that streams in the pool are not accessible through the internet
  • Specify whether the data in the pool's streams should be encrypted using your own Vault encryption key or an Oracle-managed key

There is no limit to the number of stream pools you can create.

Note

Stream names must be unique within a stream pool.

Stream Pools and Apache Kafka

Stream pools serve as the root of a virtual Apache Kafka cluster when you use Kafka with Streaming. All streams within the pool share the same Kafka configuration, encryption, and access control settings. Every action on that virtual cluster is scoped to that stream pool.

You can configure the stream pool to automatically create streams, or Kafka topics, and call KafkaAdminClient::createTopic to create a stream or topic in that stream pool.

For more information, see Using Streaming with Apache Kafka.

Required IAM Policy

To use Oracle Cloud Infrastructure, you must be granted security access in a policy  by an administrator. This access is required whether you're using the Console or the REST API with an SDK, CLI, or other tool. If you get a message that you don’t have permission or are unauthorized, verify with your administrator what type of access you have and which compartment  you should work in.

For administrators: The policy in Let streaming users manage streams lets the specified group do everything with streaming and related Streaming service resources.

To set up a private endpoint, you must have access to a VCN with a private subnet where DNS resolution is enabled. For general information about policies and permissions to do this, see IAM Policies for Networking. Specifically, you need use permissions for a VNIC, a network security group, if you specify one, and a subnet. For example:

allow user group ServiceWriters to use vnics in compartment ABC
allow user group ServiceWriters to use network-security-groups in compartment ABC
allow user group ServiceWriters to use subnets in compartment XYZ

To use your own encryption key, you must let the Streaming service use a Vault key to encrypt data in streams in this stream pool. For example:

allow service streaming to use keys in compartment ABC where target.key.id = '<key_OCID>'

The preceding policy also requires a companion policy to let Streaming use a key on behalf of a user group to create a stream pool that uses the key for cryptographic purposes. For example:

allow user group StreamWriters to use key-delegate in compartment ABC where target.key.id = '<key_OCID>'

If you're new to policies, see Getting Started with Policies and Common Policies. If you want to dig deeper into writing policies for the Streaming service, see Details for the Streaming service in the IAM policy reference.

Using the Console

To create a stream pool
  1. Open the navigation menu. Click Solutions and Platform, locate Analytics, and then click Streaming.
  2. Click on Stream Pools on the left side of the screen.

    A list of existing stream pools is displayed.

  3. Click Create Stream Pool to display the Create Stream Pool page.
  4. Enter a name for the stream pool in the Stream Pool Name text box.
  5. Select a compartment from the Resource Compartment drop-down list.
  6. In the Configure Stream Pool panel:
    1. Select Endpoint Type: Click Public Endpoint or Private Endpoint, depending on whether you want to restrict traffic to streams in this stream pool to a private endpoint that does not require traffic to traverse the internet. To create a private endpoint, you need access to a virtual cloud network (VCN) with a private subnet. Select a VCN with a private subnet where DNS resolution is also enabled, and then select the subnet. Optionally, if you want to assign a specific private IP address, you must choose one that belongs to the subnet's CIDR. By default, the Networking service assigns a random private IP address on your behalf and applies no security rules to the stream pool. For more information about VCNs and subnets, see VCNs and Subnets.
    2. Configure Encryption Settings: By default, Encrypt using Oracle-managed keys is selected. If you want to encrypt the data in the streams in this stream pool using your own Vault encryption key, click Encrypt using customer-managed keys. To use the Vault service for your encryption needs, you need access to a vault and key and you need to allow the Streaming service to use the key.
      1. Vault: Choose the vault that contains the master encryption key you want to use from the drop-down list.
      2. Master Encryption Key: Choose the master encryption key you want to use from the drop-down list.

      For more information, see Stream Security. For more information about encryption with a Vault key that you manage, see Overview of Vault and Managing Keys.

  7. If you would like to add tags or intend to use Kafka with this stream pool, click Show Advanced Options.
  8. Add TagsIf you have permissions to create a resource, then you also have permissions to apply free-form tags to that resource. To apply a defined tag, you must have permissions to use the tag namespace. For more information about tagging, see Resource Tags. If you are not sure if you should apply tags, then skip this option (you can apply tags later) or ask your administrator.
  9. To use the stream pool with Kafka, select the Auto Create Topics checkbox and configure your stream settings:
    1. Add a number of hours for the stream's retention period in Default Retention Period (hours) text box.
    2. Specify the Default Number of Partitions for the stream.
    3. Select the View Kafka settings after the stream pool is created checkbox to display the Kafka Connection settings for the stream pool when it is created.
  10. Click Create.
To edit a stream pool
  1. Open the navigation menu. Click Solutions and Platform, locate Analytics, and then click Streaming.
  2. Click on Stream Pools on the left side of the screen.

    A list of existing stream pools is displayed.

  3. Click the stream pool you want to edit to bring up the stream pool details page.
  4. Click Edit Settings.
  5. Click Edit Settings or Cancel when finished.
To delete a stream pool
  1. Open the navigation menu. Click Solutions and Platform, locate Analytics, and then click Streaming.
  2. Click on Stream Pools on the left side of the screen.

    A list of existing stream pools is displayed.

  3. You can delete a stream pool in two ways:
    1. Click the the Actions icon (three dots) on the right side of the stream pool you want to delete and select Delete Stream Pool.
    2. Click the stream pool you want to delete. The stream pool details screen displays. Click Delete.
  4. Confirm when prompted.
To change or remove the master encryption key assigned to an existing stream pool
  1. Open the navigation menu. Click Solutions and Platform, locate Analytics, and then click Streaming.
  2. Click Stream Pools.
  3. Click a stream pool to display the stream details page.
  4. In Stream Pool Information, next to Encryption Key, do one of the following:
    • To stop using an Oracle-managed key in favor of a Vault master encryption key that you manage, click Assign, select a vault and encryption key you have access to, and then click Assign.
    • To select a different Vault master encryption key that you manage, click Update, select a vault and encryption key you have access to, and then click Update.
    • Click Unassign to remove the assigned Vault master encryption key and let Oracle manage the encryption key, and then click Unassign again to confirm the removal of the existing key assignment.
To move a stream pool to a different compartment
  1. Open the navigation menu. Click Solutions and Platform, locate Analytics, and then click Streaming.
  2. Click on Stream Pools on the left side of the screen.

    A list of existing stream pools is displayed.

  3. You can move a stream pool in two ways:
    1. Click the the Actions icon (three dots) on the right side of the stream pool you want to move and select Move Resource.
    2. Click the stream pool you want to move. The stream pool details screen displays. Click Move Resource.
  4. In the Move Resource to a Different Compartment dialog box, choose the destination compartment from the drop-down list.
  5. Click Move Resource.

Using the Command Line Interface (CLI)

For information about using the CLI, see Command Line Interface (CLI). For a complete list of flags and options available for CLI commands, see the Command Line Reference.

Note

The examples in this section use the full syntax for all parameters, for example --compartment-id. For some parameters, there are shortened versions that you can use instead, like -c. See the CLI online help for instances of a shortened parameter associated with a command.
To get a list of stream pools

oci streaming admin stream-pool list --compartment-id <compartment_OCID>

For example:

oci streaming admin stream-pool list --compartment-id ocid1.tenancy.oc1..exampleuniqueID
{
  "data": [
    {
      "compartment-id": "ocid1.tenancy.oc1..exampleuniqueID",
      "defined-tags": {},
      "freeform-tags": {},
      "id": "ocid1.streampool.oc1.phx.exampleuniqueID",
      "is-private": false,
      "lifecycle-state": "ACTIVE",
      "name": "MyStreamPool",
      "time-created": "2020-10-09T22:21:16.931000+00:00"
    },
    {
      "compartment-id": "ocid1.tenancy.oc1..exampleuniqueID",
      "defined-tags": {},
      "freeform-tags": {},
      "id": "ocid1.streampool.oc1.phx.exampleuniqueID",
      "is-private": true,
      "lifecycle-state": "ACTIVE",
      "name": "example-stream-pool-02",
      "time-created": "2020-04-03T07:00:56.196000+00:00"
    },
    {
      "compartment-id": "ocid1.tenancy.oc1..exampleCompartmentOCID",
      "defined-tags": {},
      "freeform-tags": {},
      "id": "ocid1.streampool.oc1.phx.exampleStreamPoolOCID",
      "is-private": false,
      "lifecycle-state": "ACTIVE",
      "name": "example-stream-pool-01,
      "time-created": "2020-04-03T05:28:26.025000+00:00"
    }
  ]
}

By default, getting a list of streams returns up to the first 10 streams in the compartment.

To create a stream pool

oci streaming admin stream-pool create --name <stream_pool_name> --compartment-id <compartment_OCID>

For example:

oci streaming admin stream-pool create --name MyStreamPool --compartment-id ocid1.tenancy.oc1..exampleuniqueID
{
  "data": {
    "compartment-id": "ocid1.tenancy.oc1..exampleuniqueID",
    "custom-encryption-key": {
      "key-state": "NONE",
      "kms-key-id": null
    },
    "defined-tags": {},
    "endpoint-fqdn": null,
    "freeform-tags": {},
    "id": "ocid1.streampool.oc1.phx.exampleuniqueID",
    "is-private": false,
    "kafka-settings": {
      "auto-create-topics-enable": false,
      "bootstrap-servers": null,
      "log-retention-hours": 24,
      "num-partitions": 1
    },
    "lifecycle-state": "CREATING",
    "lifecycle-state-details": null,
    "name": "MyStreamPool",
    "private-endpoint-settings": {
      "nsg-ids": null,
      "private-endpoint-ip": null,
      "subnet-id": null
    },
    "time-created": "2020-11-02T23:01:59.429000+00:00"
  },
  "etag": "\"b0066564-4bf4-4e27-9255-9055e69a7808-03668273-b0d5-4b8b-9370-74522c29eb56\""
}
Tip

Provide input for --custom-encryption-key-details, --private-endpoint-details, and --kafka-settings as valid formatted JSON. See Passing Complex Input and Using a JSON File for Complex Input for information about JSON formatting.
To view stream pool details

oci streaming admin stream-pool get --stream-pool-id <stream_pool_OCID>

For example:

oci streaming admin stream-pool get --stream-pool-id ocid1.streampool.oc1.phx.exampleuniqueID
{
  "data": {
    "compartment-id": "ocid1.tenancy.oc1..exampleuniqueID",
    "custom-encryption-key": {
      "key-state": "NONE",
      "kms-key-id": null
    },
    "defined-tags": {},
    "endpoint-fqdn": "cell-1.streaming.us-phoenix-1.oci.oraclecloud.com",
    "freeform-tags": {},
    "id": "ocid1.streampool.oc1.phx.exampleuniqueID",
    "is-private": false,
    "kafka-settings": {
      "auto-create-topics-enable": false,
      "bootstrap-servers": "cell-1.streaming.us-phoenix-1.oci.oraclecloud.com:9092",
      "log-retention-hours": 24,
      "num-partitions": 1
    },
    "lifecycle-state": "ACTIVE",
    "lifecycle-state-details": null,
    "name": "MyStreamPool",
    "private-endpoint-settings": {
      "nsg-ids": null,
      "private-endpoint-ip": null,
      "subnet-id": null
    },
    "time-created": "2020-11-02T23:01:59.429000+00:00"
  },
  "etag": "\"6934531c-efaa-40ba-b083-94eb2350d737-a8b10bda-09cc-45e1-800b-b66b4bc29353\""
}
To update a stream pool

oci streaming admin stream-pool update --stream-pool-id <stream_pool_OCID>

For example:

oci streaming admin stream-pool update --stream-pool-id ocid1.streampool.oc1.phx.exampleuniqueID --name MyUpdatedStreamPool
{
  "data": {
    "compartment-id": "ocid1.tenancy.oc1..exampleuniqueID",
    "custom-encryption-key": {
      "key-state": "NONE",
      "kms-key-id": null
    },
    "defined-tags": {},
    "endpoint-fqdn": "cell-1.streaming.us-phoenix-1.oci.oraclecloud.com",
    "freeform-tags": {},
    "id": "ocid1.streampool.oc1.phx.exampleuniqueID",
    "is-private": false,
    "kafka-settings": {
      "auto-create-topics-enable": false,
      "bootstrap-servers": "cell-1.streaming.us-phoenix-1.oci.oraclecloud.com:9092",
      "log-retention-hours": 24,
      "num-partitions": 1
    },
    "lifecycle-state": "UPDATING",
    "lifecycle-state-details": null,
    "name": "MyUpdatedStreamPool",
    "private-endpoint-settings": {
      "nsg-ids": null,
      "private-endpoint-ip": null,
      "subnet-id": null
    },
    "time-created": "2020-11-02T23:01:59.429000+00:00"
  },
  "etag": "\"6ad44a83-4804-4cb5-87ae-2100d3a7012c-9679fcb9-37b7-48c5-9114-d514f132d363\""
}
Tip

Provide input for --custom-encryption-key-details, --private-endpoint-details, and --kafka-settings as valid formatted JSON. See Passing Complex Input and Using a JSON File for Complex Input for information about JSON formatting.
To delete a stream pool
Caution

The stream pool and all streams within the pool are deleted immediately. You cannot recover a deleted stream pool.

oci streaming admin stream-pool delete --stream-pool-id <stream_pool_OCID>

For example:

oci streaming admin stream-pool delete --stream-pool-id ocid1.streampool.oc1.phx.exampleuniqueID
Are you sure you want to delete this resource? [y/N]:

Select y and press Enter. The stream pool is deleted with no further prompting.

To move a stream pool to a different compartment

oci streaming admin stream-pool change-compartment --stream-pool-id <stream_pool_OCID> --compartment-id <target_compartment_OCID>

For example:

oci streaming admin stream-pool change-compartment --stream-pool-id ocid1.streampool.oc1.phx.exampleuniqueID --compartment-id ocid1.compartment.oc1..exampleuniqueID
{
  "etag": "\"b9abe5ea-d473-4451-8cf4-9b058fa9b435-a670293b-d0d9-4fff-9c54-4a862418e353\""
}