Table of Contents

Kafka - Transactional Producer

This sample demonstrates how to produce messages atomically using Kafka transactions, committing some transactions and aborting others.

What This Sample Shows

  • Enabling transactions on the producer with EnableTransactions, specifying the transactional id.
  • Starting a transaction with InitKafkaTransaction, publishing a set of 5 messages and committing it with Commit.
  • Aborting a transaction by disposing it without committing: an exception is thrown every few iterations (whenever the first number is a multiple of 3000), so the transaction is never committed.
  • Running two hosted services that use transactions concurrently within the same application.
  • Consuming the topic with a plain consumer, which retries each message twice and then skips it in case of error.

How to Run

Start the infrastructure as described in Samples, then run the two applications from the repository root.

dotnet run --project ./samples/Kafka/TransactionalProducer.Producer/.
dotnet run --project ./samples/Kafka/TransactionalProducer.Consumer/.

The producer logs when each transaction is initialized, when its messages are produced and when it is committed. Each transaction stays open for 3 seconds before being committed or aborted, and the consumer receives the messages only once their transaction is committed.

Common

The message being exchanged is defined in a common project.

namespace Silverback.Samples.Kafka.TransactionalProducer.Common;

public class SampleMessage
{
    public int Number { get; set; }
}

Full source code: https://github.com/BEagle1984/silverback/tree/main/samples/Kafka/TransactionalProducer.Common

Producer

The producer uses two hosted services to publish the messages in the background, each one wrapping every set of messages in a Kafka transaction.

using Microsoft.Extensions.DependencyInjection;
using Silverback.Configuration;
using Silverback.Messaging.Configuration;

namespace Silverback.Samples.Kafka.TransactionalProducer.Producer;

public class Startup
{
    public void ConfigureServices(IServiceCollection services)
    {
        // Enable Silverback
        services.AddSilverback()

            // Use Apache Kafka as message broker
            .WithConnectionToMessageBroker(options => options.AddKafka())

            // Delegate the broker clients configuration to a separate class
            .AddBrokerClientsConfigurator<BrokerClientsConfigurator>();

        // Add the hosted service that produces the random sample messages
        services.AddHostedService<ProducerBackgroundService>();
        services.AddHostedService<ProducerBackgroundService2>();
    }

    public void Configure()
    {
    }
}

Full source code: https://github.com/BEagle1984/silverback/tree/main/samples/Kafka/TransactionalProducer.Producer

Consumer

The consumer processes the messages and outputs their value to the standard output.

using Microsoft.Extensions.DependencyInjection;
using Silverback.Configuration;
using Silverback.Messaging.Configuration;

namespace Silverback.Samples.Kafka.TransactionalProducer.Consumer;

public class Startup
{
    public void ConfigureServices(IServiceCollection services)
    {
        // Enable Silverback
        services.AddSilverback()

            // Use Apache Kafka as message broker
            .WithConnectionToMessageBroker(options => options.AddKafka())

            // Delegate the broker clients configuration to a separate class
            .AddBrokerClientsConfigurator<BrokerClientsConfigurator>()

            // Register the subscribers
            .AddSingletonSubscriber<SampleMessageSubscriber>();
    }

    public void Configure()
    {
    }
}

Full source code: https://github.com/BEagle1984/silverback/tree/main/samples/Kafka/TransactionalProducer.Consumer

Additional Resources