Table of Contents

Kafka - File Streaming

This sample demonstrates how to deal with raw binary contents and large messages, to transfer some files through Kafka.

What This Sample Shows

  • Producing a file as a BinaryMessage that wraps the FileStream, so that the file is never entirely loaded into memory.
  • Extending BinaryMessage with a custom property mapped to a header (x-filename) via the [Header] attribute.
  • Splitting the files into chunks of 512 kB with EnableChunking and producing them to a fixed partition, keeping the chunks contiguous.
  • Manually assigning all partitions on the consumer side, to avoid a rebalance in the middle of a sequence of chunks.
  • Streaming the consumed content straight to a file and retrying the whole chunks sequence up to 5 times in case of error.

How to Run

Start the infrastructure as described in Samples, then run the two applications from the repository root.

dotnet run --project ./samples/Kafka/BinaryFileStreaming.Producer/.
dotnet run --project ./samples/Kafka/BinaryFileStreaming.Consumer/.

The producer exposes a Swagger UI at the /swagger path of the address it logs at startup. Use it to call the binary-file or custom-binary-file endpoint, passing the path of a local file.

Note

The producer's launch profile binds to port 0, so a free port is picked at every start. Look for the Now listening on line in the console output to find the address.

The consumer saves the received files into a temp folder, two levels above its working directory (samples/temp when run from the project folder).

Producer

The producer exposes two REST API that receive the path of a local file to be streamed. The second API uses a custom BinaryMessage to forward further metadata (the file name in this example).

using Microsoft.AspNetCore.Builder;
using Microsoft.Extensions.DependencyInjection;
using Silverback.Configuration;
using Silverback.Messaging.Configuration;

namespace Silverback.Samples.Kafka.BinaryFileStreaming.Producer;

public class Startup
{
    public void ConfigureServices(IServiceCollection services)
    {
        // Enable Silverback
        services.AddSilverback()

            // Use Apache Kafka as message broker
            .WithConnectionToMessageBroker(options => options.AddKafka())

            // Delegate the broker clients configuration to a separate class
            .AddBrokerClientsConfigurator<BrokerClientsConfigurator>();

        // Add API controllers and SwaggerGen
        services.AddControllers();
        services.AddSwaggerGen();
    }

    public void Configure(IApplicationBuilder app)
    {
        // Enable middlewares to serve generated Swagger JSON and UI
        app.UseSwagger().UseSwaggerUI(
            uiOptions =>
            {
                uiOptions.SwaggerEndpoint(
                    "/swagger/v1/swagger.json",
                    $"{GetType().Assembly.FullName} API");
            });

        // Enable routing and endpoints for controllers
        app.UseRouting();
        app.UseEndpoints(endpoints => { endpoints.MapControllers(); });
    }
}

Full source code: https://github.com/BEagle1984/silverback/tree/main/samples/Kafka/BinaryFileStreaming.Producer

Consumer

The consumer simply streams the file to a temporary folder in the local file system.

using Microsoft.Extensions.DependencyInjection;
using Silverback.Configuration;
using Silverback.Messaging.Configuration;
using Silverback.Samples.Kafka.BinaryFileStreaming.Consumer.Subscribers;

namespace Silverback.Samples.Kafka.BinaryFileStreaming.Consumer;

public class Startup
{
    public void ConfigureServices(IServiceCollection services)
    {
        // Enable Silverback
        services.AddSilverback()

            // Use Apache Kafka as message broker
            .WithConnectionToMessageBroker(options => options.AddKafka())

            // Delegate the broker clients configuration to a separate class
            .AddBrokerClientsConfigurator<BrokerClientsConfigurator>()

            // Register the subscribers
            .AddSingletonSubscriber<BinaryFileSubscriber>();
    }

    public void Configure()
    {
    }
}

Full source code: https://github.com/BEagle1984/silverback/tree/main/samples/Kafka/BinaryFileStreaming.Consumer

Additional Resources