Messages are published to a single partition in a stream. If there is more than one
partition in the stream, the decision of which partition to publish the message to
depends on whether your producers are using the Streaming API and PutMessages, or
taking advantage of Streaming's Kafka compatibility and using the Kafka API.
If your producers are using the Streaming API, partitioning is handled server-side by the
Streaming service. If your producers are using the
Kafka API, partitioning is handled client-side by Kafka.
Server-side Partitioning 🔗
The partition where a message is published is calculated using the message's key. If
the key is null, the partition is calculated using a random 16-byte value. You cannot
specify which partition a key uses.
Passing a null key puts the message in a random partition. If a user publishes the same
message twice, it could go to different partitions, since a completely new key is
generated. Do not expect all messages with a null key to go to the same partition. If
you want to ensure that messages with the same value go to the same partition, you
should use the same key for those messages.
If your messages are larger than the 1 MB limit, you can either use chunking or send
the message by using Oracle Cloud Infrastructure Object Storage.
Chunking: You can split large payloads into multiple, smaller chunks that the
Streaming service can accept. The chunks are
stored in the service in the same way that ordinary (non-chunked) messages are
stored. The only difference is that the consumer must keep the chunks and combine
them into the message when all the chunks have been collected. The chunks in the
partition can be interwoven with ordinary messages.
Object Storage: A large payload is placed in Object Storage and only the pointer to that data
is transferred. The receiver recognizes this type of pointer payload,
transparently reads the data from Object Storage, and provides it to the end user.
Batching and Throttling 🔗
We recommend batching messages to avoid throttling and enable better throughput. The size
of a batch of messages shouldn't exceed 1 MB. If this limit is exceeded, the message
fails to validate.
The throttling mechanism for PutMessages is activated when data write rates exceed 1 MB per second per
partition. There is no limitation on the number of writes to a stream, as long as you
are under the 1 MB per second per partition throughput.
To use Oracle Cloud Infrastructure, an administrator must be a member of a group granted security access in a policy by a tenancy administrator. This access is required whether you're using the Console or the REST API with an SDK, CLI, or other tool. If you get a message that you don't have permission or are unauthorized, verify with the tenancy administrator what type of access you have and which compartment your access works in.