Producer

In some cases when high throughput is important you might want to skip the IPublisher and take advantage of the several options offered by the IProducer interface.

Producing pre serialized messages

A pre-serialized message can be produced via the normal Produce/ProduceAsync or the RawProduce/RawProduceAsync methods. The difference is that the latter skip the entire Silverback behaviors pipeline (note that it means that no chunking or other features will kick in).

Non-blocking overloads

These are especially important for Kafka, since the underlying library is able to batch the outgoing messages for efficiency and that improves the throughput a lot. They will complete as soon as the message has been enqueued and invoke a callback when it is successfully produced (or when it fails / times out). These overloads exist for Produce, ProduceAsync, RawProduce and RawProduceAsync. (Note that )

public class ProducerService
{
    private readonly IProducer _producer;
    private readonly ILogger _logger;
    
    public ProducerService(
        IBroker broker,
        ILogger<ProducerService> logger)
    {
        _producer = broker.GetProducer("some-topic");
        _logger = logger;
    }
    
    public async Task Produce(byte[] rawMessage)
    {
        for (int i = 0; i < 100000; i++)
        {
            _producer.RawProduce(
                rawMessage,
                null,
                () => _logger.LogInformation($"Produced {i}"),
                ex => _logger.LogError(ex, $"Failed to produce {i}");
        }
    }
}

Note

The non-async overload with callback functions is generally the fastest option with Kafka.