Table of Contents

Consuming Chunked Messages

When messages are split into smaller parts (chunks) by the producer, the consumer is responsible for reassembling them into the original message before delivering it to the subscribers. Silverback handles chunked messages automatically, so in most cases you don't need to write any special consumer-side logic (if the messages are produced using Silverback or another framework that adds compatible headers).

Chunking diagram
The messages are being split into small chunks and reassembled on the consumer side.

Consumer Configuration

No special configuration is required on the consumer to support chunked messages.

Note

All chunks belonging to the same message are acknowledged/committed at once, when the full message has been reassembled and processed by the subscriber. Similarly, when an exception is thrown, the error policy is applied to all chunks.

Important

The chunks belonging to the same message are expected to be contiguous and in strict order. When using Kafka, the chunks belonging to the same message are also supposed to be in the same partition.

Important

MQTT Notes

  • Chunking requires MQTT 5.
  • The chunks must be contiguous in all the messages consumed by the client, not only in their topic: any other message received between the chunks, from any topic of the client, discards the incomplete message, which is acknowledged and lost. Consume the chunked messages with a dedicated client, subscribed only to their topic, and not through a shared subscription, which spreads them across the consumers.
  • Chunking can't be combined with batch processing (see Batch Processing Consumed Messages). A client with batch processing rejects each chunk with a SequenceException, which the error policy of the endpoint handles separately for each chunk (e.g. skipping the chunks, with a callback to keep track of them). The messages fitting in a single chunk aren't affected.
  • With QoS 1 or 2, the broker stops delivering when too many messages are waiting for acknowledgment. The number of chunks of a message must therefore not exceed the ReceiveMaximum of the client (see LimitUnacknowledgedPublications in Other Connection Settings) and the limit enforced by the broker. Otherwise, the consumer stalls until the sequence timeout elapses and then discards the message.

Additional Resources