Table of Contents

Producing Chunked Messages

When dealing with large messages, it might be necessary to split them into smaller parts to avoid issues with the broker or the consumer. This process is called chunking and is especially useful with Kafka, which has a maximum message size limit, or with the MQTT brokers limiting the message size.

Chunking diagram: a large message is split into smaller chunks.
The messages are being split into small chunks.

The producer automatically adds the necessary headers to the messages to allow the consumer to reassemble the chunks. The consumer will then automatically reassemble the message before pushing it to subscribers and doesn't require any additional configuration.

Producer Configuration

services.AddSilverback()
    .WithConnectionToMessageBroker(options => options.AddKafka())
    .AddKafkaClients(clients => clients
        .WithBootstrapServers("PLAINTEXT://localhost:9092")
        .AddProducer("producer1", producer => producer
            .Produce<MyMessage>("endpoint1", endpoint => endpoint
                .ProduceTo("my-topic")
                .EnableChunking(500000))));
Important

The chunks belonging to the same message must be contiguous. It is therefore recommended to have a single producer per endpoint or partition. If using Kafka see also Kafka Keys and Partitioning.

Important

With MQTT, chunking requires MQTT 5 and can't be enabled for retained messages (see Retain), since the broker retains only the last message of each topic: a client subscribing later would receive the last chunk alone.

Additional Resources