Managing Streams

This topic describes how to configure, create, view, test, and delete streams.

Overview

Before publishing messages to a stream, or consuming messages from a stream, you must first create a stream. You can use the Oracle Cloud Infrastructure Console or the Streaming API to create your stream.

When creating a stream, consider the expected stream throughput, message retention period, partition key strategy, and how your stream will be consumed. Most configuration values cannot be changed after the stream has been created. For example, after a stream is created, you can't change the message retention time or number of partitions.

Partitioning a Stream

To take full advantage of the Streaming service's ability to operate at scale, configure the number of partitions in the stream based on the following considerations.

Throughput

When you create a stream, you must specify how many partitions the stream has. The expected throughput of your application can help you determine the number of partitions for your stream. Multiply the average message size by the maximum number of messages written per second to estimate your expected throughput. Since a single partition is limited to a 1 MB per second write rate, a higher throughput requires additional partitions to avoid throttling. Keep additional Streaming limits in mind when making your design decisions.

Tip

To help you manage application spikes, we recommend allocating partitions slightly higher than your maximum throughput.

Publishing

The content of the messages you intend to publish to a stream can also help you determine how many partitions your stream should have.

A message is published to a partition in the stream. If there is more than one partition, the partition where the message is published is calculated using the message's key. Messages with the same key go to the same partition. Messages with different keys might go to different partitions or to the same partition.

For more information, see Publishing Messages.

Consumer Groups

If your stream will be consumed by one or more consumer groups, you should factor that into your decision on how many partitions it should have. Partition reads are balanced among the instances in a consumer group.

Consumer groups can only utilize a single instance at a time if the stream has only one partition. If your stream has multiple partitions, you can scale the number of instances up to the number of the partitions and have one instance in the group reading from one partition in the stream.

For more information, see Using Consumer Groups.

Stream Pooling

When you create your stream, you need to specify whether it should become a member of an existing stream pool, or a member of a new, automatically created stream pool. See Managing Stream Pools for more information.

Stream Security

Streaming data is encrypted both at rest and in transit. Private endpoints within your virtual cloud network (VCN) can be used to restrict access to your streams so they cannot be accessed through the internet.

Both encryption and private access are configured at the stream pool level to make managing groups of streams easier.

Encryption

By default, all encryption-related matters are handled by Oracle, but you can manage your own encryption keys using Oracle Cloud Infrastructure Vault. Vault allows you to bring your own Advanced Encryption Standard (AES) symmetric keys and manage, rotate, disable, and delete them as needed.

Because encryption keys are managed at the stream pool level, you can use a different encryption key for each logical stream grouping or virtual Kafka cluster.

To use your own encryption key:

For more information, see Overview of Vault and Managing Keys.

Private Endpoints

Private endpoints associate a private IP address within a VCN to the stream pool, allowing Streaming traffic to avoid traversing the internet.

To create a private endpoint for Streaming, you need access to a VCN with a private subnet when you create the stream pool. See About Private Endpoints and VCNs and Subnets for more information.

To use private endpoints:

Required IAM Policy

To use Oracle Cloud Infrastructure, you must be granted security access in a policy  by an administrator. This access is required whether you're using the Console or the REST API with an SDK, CLI, or other tool. If you get a message that you don’t have permission or are unauthorized, verify with your administrator what type of access you have and which compartment  you should work in.

For administrators: The policy in Let streaming users manage streams lets the specified group do everything with streaming and related Streaming service resources.

If you're new to policies, see Getting Started with Policies and Common Policies. If you want to dig deeper into writing policies for the Streaming service, see Details for the Streaming Service in the IAM policy reference.

Using the Console

To create a stream
  1. Open the navigation menu. Click Solutions and Platform, locate Analytics, and then click Streaming.

    A list of existing streams is displayed.

  2. Click Create Stream at the top of the list.
  3. Stream Name: Required. Specify a friendly name for the stream. It does not have to be unique within the compartment, but it must be unique to the stream pool. The stream name cannot be changed. Avoid entering confidential information.
  4. Compartment: Choose the compartment in which the stream will be created. To change the compartment, select a different compartment from the drop-down list.
  5. Stream Pool: Choose the stream pool that will contain your stream.
    1. If your chosen compartment has an existing stream pool, you can select it from the drop-down list or click Create new stream pool and configure the stream pool manually.
    2. If no stream pool exists in the chosen compartment, select Auto-create a default stream pool or click Create a new stream pool and configure the stream pool manually.
  6. In the Define Stream Settings panel:
    1. Retention (in Hours): Enter the number of hours (from 24 to 168) to retain messages. The default value is 24.
    2. Number of Partitions: Enter the number of partitions for the stream. The maximum number is based on the limits for your tenancy.

      Tip

      The maximum Total Write Rate and Total Read Rate of your stream are displayed as you adjust the number of partitions.
  7. Click Show Advanced Options to optionally define TagsIf you have permissions to create a resource, then you also have permissions to apply free-form tags to that resource. To apply a defined tag, you must have permissions to use the tag namespace. For more information about tagging, see Resource Tags. If you are not sure if you should apply tags, then skip this option (you can apply tags later) or ask your administrator.
  8. Click Create.
To delete a stream
  1. Open the navigation menu. Click Solutions and Platform, locate Analytics, and then click Streaming.
  2. You can delete a stream in two ways:
    1. Click the the Actions icon (three dots) on the right side of the stream you want to delete and select Delete Stream.
    2. Click the stream you want to delete. The stream details screen displays. Click Delete.
  3. Confirm when prompted.
To produce a test message
  1. Open the navigation menu. Click Solutions and Platform, locate Analytics, and then click Streaming.
  2. Click a stream to display the stream details page.
  3. Click Produce Test Message.
  4. On the Test Stream dialog, enter the text-only message to produce in the Datatext box.
  5. Click Produce.
To show recent messages on a stream
  1. Open the navigation menu. Click Solutions and Platform, locate Analytics, and then click Streaming.
  2. Click a stream to display the stream details page.
  3. In the Recent Messages panel, click Load messages.
To move a stream to a different compartment
  1. Open the navigation menu. Click Solutions and Platform, locate Analytics, and then click Streaming.
  2. Click a stream to display the stream details page.
  3. Find the stream you want to move in the list, click the the Actions icon (three dots), and then click Move Resource.
  4. Choose the destination compartment from the list.
  5. Click Move Resource.

Using the Command Line Interface (CLI)

For information about using the CLI, see Command Line Interface (CLI). For a complete list of flags and options available for CLI commands, see the Command Line Reference.

Note

The examples in this section use the full syntax for all parameters, for example --compartment-id. For some parameters, there are shortened versions that you can use instead, like -c. See the CLI online help for instances of a shortened parameter associated with a command.
To get a list of streams in a compartment

oci streaming admin stream list --compartment-id <compartment_OCID>

For example:

oci streaming admin stream list --compartment-id ocid1.tenancy.oc1..exampleuniqueID
{
  "data": [
    {
      "compartment-id": "ocid1.tenancy.oc1..exampleuniqueID",
      "defined-tags": {},
      "freeform-tags": {},
      "id": "ocid1.stream.oc1.phx.exampleuniqueID",
      "lifecycle-state": "ACTIVE",
      "messages-endpoint": "https://cell-1.streaming.us-phoenix-1.oci.oraclecloud.com",
      "name": "example_stream_2",
      "partitions": 1,
      "stream-pool-id": "ocid1.streampool.oc1.phx.exampleuniqueID",
      "time-created": "2020-08-21T21:19:35.707000+00:00"
    },
    {
      "compartment-id": "ocid1.tenancy.oc1..exampleuniqueID",
      "defined-tags": {},
      "freeform-tags": {},
      "id": "ocid1.stream.oc1.phx.exampleuniqueID",
      "lifecycle-state": "DELETED",
      "messages-endpoint": "https://cell-1.streaming.us-phoenix-1.oci.oraclecloud.com",
      "name": "example_stream_1",
      "partitions": 5,
      "stream-pool-id": "ocid1.streampool.oc1.phx.exampleuniqueID",
      "time-created": "2020-07-16T20:59:32.904000+00:00"
    }
  ]
}

By default, getting a list of streams returns up to the first 10 streams in the compartment.

To get a list of streams in a stream pool

oci streaming admin stream list --stream-pool-id <stream_pool_OCID>

For example:

oci streaming admin stream list --stream-pool-id ocid1.streampool.oc1.phx.exampleuniqueID
{
  "data": [
    {
      "compartment-id": "ocid1.tenancy.oc1..exampleuniqueID",
      "defined-tags": {},
      "freeform-tags": {},
      "id": "ocid1.stream.oc1.phx.exampleuniqueID",
      "lifecycle-state": "ACTIVE",
      "messages-endpoint": "https://cell-1.streaming.us-phoenix-1.oci.oraclecloud.com",
      "name": "example_stream_2",
      "partitions": 1,
      "stream-pool-id": "ocid1.streampool.oc1.phx.exampleuniqueID",
      "time-created": "2020-08-21T21:19:35.707000+00:00"
    },
    {
      "compartment-id": "ocid1.tenancy.oc1..exampleuniqueID",
      "defined-tags": {},
      "freeform-tags": {},
      "id": "ocid1.stream.oc1.phx.exampleuniqueID",
      "lifecycle-state": "DELETED",
      "messages-endpoint": "https://cell-1.streaming.us-phoenix-1.oci.oraclecloud.com",
      "name": "example_stream_1",
      "partitions": 5,
      "stream-pool-id": "ocid1.streampool.oc1.phx.exampleuniqueID",
      "time-created": "2020-07-16T20:59:32.904000+00:00"
    }
  ]
}

By default, getting a list of streams returns up to the first 10 streams in the stream pool.

To create a stream

You can create a stream in a compartment or a stream pool. The --compartment-id and --stream-pool-id parameters cannot be specified at the same time.

Caution

Avoid entering confidential information when assigning descriptions, tags, or friendly names to your cloud resources through the Oracle Cloud Infrastructure Console, API, or CLI.
oci streaming admin stream create --name <stream_name> --partitions <number_of_partitions> --compartment-id <compartment_OCID>
oci streaming admin stream create --name <stream_name> --partitions <number_of_partitions> --stream-pool-id <stream_pool_OCID>

For example:

oci streaming admin stream create --name MyStream --partitions 5 --compartment-id ocid1.tenancy.oc1..exampleuniqueID
{
  "data": {
    "compartment-id": "ocid1.tenancy.oc1..exampleuniqueID",
    "defined-tags": {},
    "freeform-tags": {},
    "id": "ocid1.stream.oc1.phx.exampleuniqueID",
    "lifecycle-state": "CREATING",
    "lifecycle-state-details": null,
    "messages-endpoint": "https://cell-1.streaming.us-phoenix-1.oci.oraclecloud.com",
    "name": "MyStream",
    "partitions": 5,
    "retention-in-hours": 24,
    "stream-pool-id": "ocid1.streampool.oc1.phx.exampleuniqueID",
    "time-created": "2020-11-02T19:12:22.385000+00:00"
  },
  "etag": "\"d72d8103-f1ae-442a-822d-10f86cd097c5-25e61a9b-cc08-4fad-9908-40c9636d31d8\""
}
To view stream details

oci streaming admin stream get --stream-id <stream_OCID>

For example:

oci streaming admin stream get --stream-id ocid1.stream.oc1.phx.exampleuniqueID
{
  "data": {
    "compartment-id": "ocid1.tenancy.oc1..exampleuniqueID",
    "defined-tags": {},
    "freeform-tags": {},
    "id": "ocid1.stream.oc1.phx.exampleuniqueID",
    "lifecycle-state": "ACTIVE",
    "lifecycle-state-details": null,
    "messages-endpoint": "https://cell-1.streaming.us-phoenix-1.oci.oraclecloud.com",
    "name": "MyStream",
    "partitions": 5,
    "retention-in-hours": 24,
    "stream-pool-id": "ocid1.streampool.oc1.phx.exampleuniqueID",
    "time-created": "2020-11-02T19:12:22.385000+00:00"
  },
  "etag": "\"0613d634-86ab-4446-973f-268d175313d4-12e9725e-5574-4f6b-995b-7dcc80271666\""
}
To delete a stream
Caution

Stream contents are deleted immediately. You cannot recover a deleted stream.

oci streaming admin stream delete --stream-id <stream_OCID>

For example:

oci streaming admin stream --delete --stream-id ocid1.stream.oc1.phx.exampleuniqueID
Are you sure you want to delete this resource? [y/N]:

Select y and press Enter. The stream is deleted with no further prompting.

To move a stream to a different stream pool

oci streaming admin stream update --stream-id <stream_OCID> --stream-pool-id <stream_pool_OCID>

For example:

oci streaming admin stream update --stream-id ocid1.stream.oc1.phx.exampleuniqueID --stream-pool-id ocid1.streampool.oc1.phx.exampleuniqueID
{
  "data": {
    "compartment-id": "ocid1.tenancy.oc1..exampleuniqueID",
    "defined-tags": {},
    "freeform-tags": {},
    "id": "ocid1.stream.oc1.phx.exampleuniqueID",
    "lifecycle-state": "UPDATING",
    "lifecycle-state-details": null,
    "messages-endpoint": "https://cell-1.streaming.us-phoenix-1.oci.oraclecloud.com",
    "name": "MyStream",
    "partitions": 1,
    "retention-in-hours": 24,
    "stream-pool-id": "ocid1.streampool.oc1.phx.exampleuniqueID",
    "time-created": "2020-11-02T19:12:22.385000+00:00"
  },
  "etag": "\"25b49cb2-f0c0-4421-b39f-899846b9d7c9-0c3a8572-1719-44ba-840d-19843352c9ba\""
}
To move a stream to a different compartment

oci streaming admin stream change-compartment --stream-id <stream_OCID> --compartment-id <compartment_OCID>

For example:

oci streaming admin stream change-compartment --stream-id ocid1.stream.oc1.phx.exampleuniqueID --compartment-id ocid1.tenancy.oc1..exampleuniqueID
{
  "data": {
    "compartment-id": "ocid1.tenancy.oc1..exampleuniqueID",
    "defined-tags": {},
    "freeform-tags": {},
    "id": "ocid1.stream.oc1.phx.exampleuniqueID",
    "lifecycle-state": "UPDATING",
    "lifecycle-state-details": null,
    "messages-endpoint": "https://cell-1.streaming.us-phoenix-1.oci.oraclecloud.com",
    "name": "MyStream",
    "partitions": 5,
    "retention-in-hours": 24,
    "stream-pool-id": "ocid1.streampool.oc1.phx.exampleuniqueID",
    "time-created": "2020-11-02T19:12:22.385000+00:00"
  },
  "etag": "\"25b49cb2-f0c0-4421-b39f-899846b9d7c9-0c3a8572-1719-44ba-840d-19843352c9ba\""
}