Silverback 3 Release Notes
3.8.0
Released on 2022-09-20
What's New
- Topic name resolvers can be used to filter the messages to be produce: returning
nullwill discard the message
Fixes
- Fix error policies not being triggered consistently when batch consuming
- Make
IntegrationSpyfully thread-safe - Prevent errors when the IKafkaPartitionsRevokedCallback is invoked during application shutdown
- Improve error handling during connection to MQTT
3.7.3
Released on 2022-07-21
Fixes
- Support topic names with symbols (e.g. hyphens) in mocked MQTT broker
3.7.2
Released on 2022-07-01
Fixes
- Correctly invoke the IKafkaOffsetCommittedCallback when auto commit is disabled [#167]
3.7.1
Released on 2022-06-27
Fixes
- Improve message streams abort process to avoid first chance exceptions (e.g. during dispose)
3.7.0
Released on 2022-05-18
What's New
- Implement basic support for Kafka transactions via KafkaTransactionalProducer (Note: this is just a first step and a more comprehensive implementation is planned for the upcoming release 4.0.0)
- Skip chunking when processing single chunk messages
Fixes
- Fix possible race condition in consumer pipeline
3.6.1
Released on 2022-03-24
Fixes
- Handle race condition in BatchSequence with timeout
- Limit consumer status history
3.6.0
Released on 2022-03-04
What's New
- Handle
IAsyncEnumerable<T>returned by the subscriber and republished the contained messages - Enrich Kafka messages moved by the MoveMessageErrorPolicy adding some extra headers containing some information about the source topic, partition, offset, etc. (see Default Headers)
- Allow filters such as the KafkaGroupIdFilterAttribute or MqttClientIdFilterAttribute to be added to the subscribers at runtime via the configuration API
- Add overload for
Publishmethod in the error policies that forwards the exception as well as the envelope - Throw
TimeoutExceptionfrom KafkaTestingHelper and MqttTestingHelper - Improve MQTT connection related logs (info for successful reconnect and add broker name to log messages)
- Support shared subscriptions in mocked MQTT broker
Fixes
- Ensure each consumed message gets a unique traceId (when the traceparent header is not present)
- Fix memory leak in consumer
- Fully validate messages, including nested objects
3.5.0
Released on 2021-10-31
What's New
- Log
MqttClientinternal events (see Logging and Tracing) - Upgrade to Confluent.Kafka 1.8.2
- Upgrade to MQTTnet 3.0.16
- Upgrade to RabbitMQ.Client 6.2.2
- Update several dependencies
Fixes
- Fix MqttConsumer reconnection issues
- Handle edge cases related to MQTT acknowledgment timeout in MqttConsumer
- Allow max retries specification and error policies chains with MQTT V3
3.4.0
Released on 2021-09-27
What's New
- Support encryption key rotation (see Encrypt Produced Messages)
3.3.1
Released on 2021-09-03
Fixes
- Fix
AddHeaders<TMessage>andWithKafkaKey<TMessage>not being correctly invoked by allIProducer.ProduceandIProducer.ProducerAsyncoverloads - Add endpoint friendly name to all logs
3.3.0
Released on 2021-08-29
What's New
- Optimize in-memory mocked Kafka (avoid spawning too many threads)
- Support multiple brokers (with overlapping topic names) in mocked Kafka and MQTT
- Add message validation for both producer and consumer (see Validating the Produced Messages and Validating the Consumed Messages)
- Add new
AddInboundoverloads specifying message type for a more compact configuration when using the typed deserializer (see Serializing the Produced Messages)
Fixes
- Invoke the Kafka partition EOF callback for all connected consumers
- Ignore null or empty Kafka key in producer
3.2.0
Released on 2021-07-13
What's New
- Add new Kafka partition EOF callback to be notified when the end of a partition is reached by the consumer (see Broker Callbacks and IKafkaPartitionEofCallback)
- Allow multiple calls to
IKafkaConsumerEndpointBuilder.ConfigureorIKafkaProducerEndpointBuilder.Configurefor the same endpoint - Observe a grace period in the ConsumersHealthCheck to prevent false positives during a normal Kafka rebalance
- Add optional friendly name to the endpoints (see
IEndpointBuilder<TBuilder>.WithNameandEndpoint.FriendlyName) - Allow filtering the endpoints targeted by the ConsumersHealthCheck (see AddConsumersCheck(IHealthChecksBuilder, ConsumerStatus, TimeSpan?, string, HealthStatus?, IEnumerable<string>?))
3.1.1
Released on 2021-06-28
Fixes
- Invoke broker callbacks during the application shutdown to allow custom code to be run when disconnecting
3.1.0
Released on 2021-06-28
What's New
- Add new ways to configure headers and kafka key (see Setting Headers, Reading Headers, and Kafka Keys and Partitioning)
- New callbacks for Kafka log events (see Broker Callbacks)
- Improve consumer status tracking introducing ConsumerStatus.Ready
- Revert the Kafka consumer status from
ReadytoConnectedwhenever partitions are revoked or a poll timeout occurs - Adapt consumer health check to monitor the new status and report unhealthy if not
Ready(see Consumer Health Check)
- Revert the Kafka consumer status from
- Try to automatically recover from Kafka maximum poll interval exceed errors
- Improve Kafka static partition assignment with resolver function and fetching the available partitions (see Kafka Keys and Partitioning)
- Upgrade to Confluent.Kafka 1.7.0
- Upgrade to MQTTnet 3.0.15
Fixes
- Prevent possible race condition causing messages to be skipped when a
RetryPolicykicks in for messages from multiple Kafka partitions simultaneously - Prevent
ObjectDisposedExceptionto be thrown when Kafka events (e.g. statistics) are fired during the application shutdown - Prevent
ObjectDisposedExceptionto be thrown whenConsumer.Disposeis called multiple times - Properly clear the trace context (
Activity) when reconnecting the consumer to prevent the newly started consume loop to be tracked under the current message traceId - Fix wrong prefix in MQTT log event names
3.0.1
Released on 2021-04-04
Fixes
- Fix
IOutboxWriterlifecycle [#128]
3.0.0
Released on 2021-03-14
What's New
- Add support for MQTT (see Connecting to a Message Broker, Producing Messages, Consuming Messages, ...)
- Simplify configuration and reduce boilerplate
- Simplify subscribers registration and get rid of the
ISubscriberinterface (see Using the Message Bus) - Scan subscribers automatically at startup to reduce cost of first message
- Connect brokers and handle graceful shutdown automatically (see Connecting to a Message Broker)
- Improve endpoints configuration API (see Connecting to a Message Broker)
- Add
IServiceCollection.ConfigureSilverbackextension method to conveniently split the configuration code (see Connecting to a Message Broker)
- Simplify subscribers registration and get rid of the
- Refactor Silverback.Integration to support streaming
- Create IMessageStreamEnumerable<TMessage>
- Improve chunking support in conjunction with streaming, requiring only one chunk at a time to be loaded into memory
- Redesign sequences handling to support chunking, batch consuming and future sequences as well
- Improve Kafka partitions handling (see Kafka Keys and Partitioning)
- Process partitions independently and concurrently
- Add setting to produce to a specific partition
- Add setting to manually assign the consumer partitions
- Add option to throw an exception if no subscriber is handling a message that was published to the internal bus or was consumed from a message broker (see
throwIfUnhandledargument in the IPublisher methods andThrowIfUnhandledproperty in theIConsumerEndpoint) - Handle null messages as Tombstone/Tombstone<TMessage> (see Consuming Tombstone Messages)
- Replace Newtonsoft.Json with System.Text.Json to improve serialization and deserialization performance (the old serializers have been moved into the Silverback.Integration.Newtonsoft package, see Serializing the Produced Messages)
- Improve outbound routing customization options with endpoint name resolvers (see Producing Messages)
- Add non-blocking
Produce/ProduceAsync/RawProduce/RawProduceAsyncoverloads to IProducer, better suitable for higher throughput scenarios - Refactor broker event handlers (see Broker Callbacks)
- Expose IConsumer.StopAsync and IConsumer.StartAsync methods to pause and resume consumers
- Add log levels configuration (see Logging and Tracing)
- Improve (distributed) tracing (see Logging and Tracing)
- Allow header names customization (see Default Headers)
- Add consumer status information and statistics (see Connecting to a Message Broker)
- Add basic consumer health check (see Consumer Health Check)
- Allow broker behaviors to be registered as transient, meaning that an instance will be created per each producer or consumer (see Producer Behaviors Pipeline and Consumer Behaviors Pipeline)
- Improve code quality
- Enhance CI pipeline to use Roslyn analyzers
- Integrate SonarCloud
- Improve integration tests
- Increase automated tests coverage
- Enable nullable reference types and adjust all API
- Document the entire public API
- Released some utilities to help writing automated tests involving Silverback.Integration (see Testing)
- Upgrade to Confluent.Kafka 1.6.2
- Upgrade to RabbitMQ.Client 6.2.1
Fixes
- Fix
OutboxWorkernot publishing custom headers [#102]
Breaking Changes
- Refactored IPublisher
- Removed the overloads to publish a batch of messages (see Using the Message Bus)
- Cannot subscribe to collection of messages anymore (see Using the Message Bus), unless they are consumed from a message broker (see Consuming Messages)
- The chunks belonging to the same message must be contiguous (interleaved messages are at the moment not supported anymore) and in the same partition in case of Kafka
- Removed
ISubscriberinterface - Removed
BusConfigurator(moved all the configuration into theISilverbackBuilderextension methods)- Replaced
BusConfigurator.ConnectwithISilverbackBuilder.AddEndpointsConfiguratorandISilverbackBuilder.AddEndpoints(orISilverbackBuilder.AddKafkaEndpointsetc.) to configure the endpoints, while the broker is connected automatically at startup (see Connecting to a Message Broker) - Replaced
BusConfigurator.Subscribemethods withISilverbackBuilder.AddDelegateSubscriber(see Using the Message Bus) - Replaced
BusConfigurator.HandleMessagesOfTypemethods withISilverbackBuilder.HandleMessageOfType(see Using the Message Bus) BusConfigurator.ScanSubscribersis not needed anymore since it gets called automatically at startup (from an IHostedService)
- Replaced
- Removed
IServiceCollection.Add*Subscriber,IServiceCollection.Add*Behavior,IServiceCollection.Add*BrokerBehavior,IServiceCollection.AddEndpointsConfigurator,IServiceCollection.Add*OutboundRouterextension methods, use the same methods on theISilverbackBuilder(usingIServiceCollection.ConfigureSilverbackto get an instance if theISilverbackBuilderif necessary, as shown in Setting Up Silverback) - Removed
IBrokerOptionsBuilder.Add*BrokerBehavior,IBrokerOptionsBuilder.RegisterConfigurator,IBrokerOptionsBuilder.Add*OutboundRouterextension methods, use the same methods on theISilverbackBuilder(usingIServiceCollection.ConfigureSilverbackto get an instance if theISilverbackBuilderif necessary, as shown in Setting Up Silverback) - Reorganized the
Silverback.Messaging.Configurationnamespace moving some broker specific types underSilverback.Messaging.Configuration.Kafka,Silverback.Messaging.Configuration.RabbitorSilverback.Messaging.Configuration.Mqtt - The visibility of some types has been changed to internal to favor a cleaner and clearer API where the public types are well documented and their backward compatibility is valued
- Removed Silverback prefix from exceptions name
- Removed the
IRequest<TResponse>interface (it was implemented by both IQuery<TResult> and ICommand<TResult>) - Changed Impl methods suffix with Core, this affects some virtual members in the
Brokerand other base classes IConsumer.Receivedevent replaced by a callback delegateIBroker.GetConsumerandIBrokerCollection.GetConsumermethods renamed toIBroker.AddConsumerandIBrokerCollection.AddConsumerIQueueProducerandIQueueConsumerrenamed toIOutboxWriterandIOutboxReader- The messages with a null body are by default mapped to a Tombstone/Tombstone<TMessage> (see Consuming Tombstone Messages)
- Database:
- Moved all entities (used with Entity Framework Core) to the
Silverback.Database.Modelnamespace - Replaced
InboundMessageentity withInboundLogEntry - Replaced
OutboundMessageentity withOutboxMessage - Removed
TemporaryMessageChunk - Modified schema of
StoredOffsetentity
- Moved all entities (used with Entity Framework Core) to the
- Moved and renamed some internally used types (e.g.
QueuedMessage,DbQueuedMessage, ...) - Complete redesign of the error policies
- Removed
IMessageIdProviderand all related logic: theIdorMessageIdproperty will not be automatically initialized anymore and its value will not be used as identifier for the outbound message anymore WithConnectionTo<>,WithConnectionToKafka,WithConnectionToRabbitMQandWithInMemoryBrokerhave been removed, please use the newWithConnectionToMessageBrokerandAddKafka/AddRabbitmethods (see Connecting to a Message Broker)- Replaced the internal messages for the Kafka events such as partitions revoked/assigned, offset commit, error, log and statistics with event handler interfaces (see Broker Callbacks)
- Deprecated Silverback.Integration.InMemory, use Silverback.Integration.Kafka.Testing, Silverback.Integration.RabbitMQ.Testing, etc. instead
- Renamed
PartitioningKeyMemberAttributeto KafkaKeyMemberAttribute - Silverback.Integration.Configuration has been discontinued
- Renamed
Settingsproperty toOptionsin the default JsonMessageSerializer (since the switch to System.Text.Json) - Removed
LogWithLevelmethod from SkipMessageErrorPolicy, use the newWithLogLevelsconfiguration instead - Removed
Paralleloption from SubscribeAttribute - Renamed
Offsetto a more genericBrokerMessageIdentifierin the Silverback.Integration abstractions (including the envelopes) - Some changes to the behaviors:
- Renamed
HandletoHandleAsyncin the IBehavior, IProducerBehavior and IConsumerBehavior - Changed signature of the
HandleAsyncmethod (see Using the Message Bus, Producer Behaviors Pipeline and Consumer Behaviors Pipeline) - Changed some sort indexes and introduced some new broker behaviors, you may need to adjust the sort index of your custom behaviors (see Producer Behaviors Pipeline and Consumer Behaviors Pipeline for the updated list of built-in behaviors)
- Renamed
- Replaced
IBroker.ConnectandIBroker.DisconnectwithIBroker.ConnectAsyncandIBroker.DisconnectAsync - Some major changes to batch consuming:
- Removed all batch events (
BatchStartedEvent,BatchCompleteEvent,BatchProcessedEvent,BatchAbortedEvent), refer to Consuming Messages to learn how to leverage the new IMessageStreamEnumerable<TMessage> - Setting the batch size to 1 doesn't disable batching anymore, set the
Batchtonullin the ConsumerEndpoint to disable it - When batching is enabled the messages can be subscribed only via the IMessageStreamEnumerable<TMessage> (see Consuming Messages), the subscribers to the single messages will not be invoked
- Removed all batch events (
- ChunkSettings moved from
Silverback.Messaging.LargeMessagesnamespace toSilverback.Messaging.Sequences.Chunking - Replaced
CoreEventIds,IntegrationEventIds,KafkaEventIdsandRabbitEventIdswith CoreLogEvents, IntegrationLogEvents, KafkaLogEvents andRabbitLogEvents(see also Logging and Tracing) - Deprecated support for Entity Framework 2, only the version 3.0.1 of Silverback.Core.EntityFrameworkCore will work with Silverback 3.0.0
- Modified message encryption for chunked messages and it will not be compatible with previous versions of Silverback (affects chunking+encryption only)