Publishing Messages

This topic covers how to emit messages to a stream.

Messages

Messages are published to a single partition in a stream. If there is more than one partition in the stream, the decision of which partition to publish the message to depends on whether your producers are using the Streaming API and PutMessages, or taking advantage of Streaming's Kafka compatibility and using the Kafka API. If your producers are using the Streaming API, partitioning is handled server-side by the Streaming service. If your producers are using the Kafka API, partitioning is handled client-side by Kafka.

Server-side Partitioning

A message is composed of a key and a value. Both the key and the value are byte arrays. If you don't explicitly supply a key, one is generated automatically.

The partition where the message is published is calculated using the message's key. If the key is null, the partition is calculated using a random 16-byte value.

Passing a null key puts the message in a random partition. For messages with a null key, do not expect messages with same value to go on the same partition, since the partitioning scheme may change. If you want to ensure that messages with the same value go to the same partition, you should use the same key for those messages.

Handling Large Messages

If your messages are larger than the 1 MB limit, you can either use chunking or send the message by using Oracle Cloud Infrastructure Object Storage.

  • Chunking: You can split large payloads into multiple, smaller chunks that the Streaming service can accept. The chunks are stored in the service in the same way that ordinary (non-chunked) messages are stored. The only difference is that the consumer must keep the chunks and combine them into the message when all the chunks have been collected. The chunks in the partition can be interwoven with ordinary messages.
  • Object Storage: A large payload is placed in Object Storage and only the pointer to that data is transferred. The receiver recognizes this type of pointer payload, transparently reads the data from Object Storage, and provides it to the end user.

Batching and Throttling

We recommend message batching for the following reasons:

  • Reduces the number of Put requests sent to the service, which avoids throttling
  • Enables better throughput

The size of a batch of messages shouldn't exceed 1 MB. If this limit is exceeded, the throttling mechanism is triggered. The throttling mechanism for PutMessages is activated when writes exceed 1 MB per second per partition. See Limits on Streaming Resources for more information.

Required IAM Policy

To use Oracle Cloud Infrastructure, you must be granted security access in a policy  by an administrator. This access is required whether you're using the Console or the REST API with an SDK, CLI, or other tool. If you get a message that you don’t have permission or are unauthorized, verify with your administrator what type of access you have and which compartment  you should work in.

For administrators: The policy in Let streaming users manage streams lets the specified group do everything with streaming and related Streaming service resources.

If you're new to policies, see Getting Started with Policies and Common Policies. If you want to dig deeper into writing policies for the Streaming service, see Details for the Streaming Service in the IAM policy reference.

Using the Console

To publish a message
  1. Open the navigation menu. Click Solutions and Platform, locate Analytics, and then click Streaming.
  2. In the list of available streams, click on the stream that you want to publish to.
  3. Click Produce Test Message.
  4. Type the message in the Data text box.
  5. Click Produce.

Using the Command Line Interface (CLI)

For information about using the CLI, see Command Line Interface (CLI). For a complete list of flags and options available for CLI commands, see the Command Line Reference.

To publish a message

oci streaming stream message put --stream-id <stream_id> --messages <JSON_messages> --endpoint <messages_endpoint>
Tip

Provide input for --messages as valid formatted JSON. See Passing Complex Input and Using a JSON File for Complex Input for information about JSON formatting.

For example, file.txt contains the properly formatted JSON. Its values are Base64-encoded:

[
  {
    "key": "a2V5MQ==",
    "value": "dmFsdWUx"
  },
  {
    "key": "a2V5Mg==",
    "value": "dmFsdWUy"
  }
]

The --messages parameter takes the file as its value:

oci streaming stream message put --stream-id ocid1.stream.oc1.phx.exampleuniqueID --messages file://file.txt --endpoint https://cell-1.streaming.us-phoenix-1.oci.oraclecloud.com  
{
  "data": {
    "entries": [
      {
        "error": null,
        "error-message": null,
        "offset": 0,
        "partition": "0",
        "timestamp": "2020-11-03T21:35:03.837000+00:00"
      },
      {
        "error": null,
        "error-message": null,
        "offset": 1,
        "partition": "0",
        "timestamp": "2020-11-03T21:35:03.837000+00:00"
      }
    ],
    "failures": 0
  }
}