Logging and Tracing
Silverback writes its logs through the standard Microsoft.Extensions.Logging abstractions, so they flow into whatever logging provider your application uses (console, Serilog, OpenTelemetry, etc.). Each event has a stable EventId that can be used to filter or route the logs.
On top of that, Silverback creates Activity objects to support distributed tracing (see Tracing).
Log Events
Silverback logs several events that can be used to monitor the application and troubleshoot issues. The tables below list the events and their default level.
Core
| Id | Level | Message | Reference |
|---|---|---|---|
| 11 | Debug | Discarding result of type {Type} because doesn't match expected return type {ExpectedType} | SubscriberResultDiscarded |
| 41 | Information | Starting background service {BackgroundService} | BackgroundServiceStarting |
| 42 | Error | Background service {BackgroundService} execution failed | BackgroundServiceException |
| 51 | Information | Background service {BackgroundService} stopped | RecurringBackgroundServiceStopped |
| 52 | Debug | Background service {BackgroundService} sleeping for {Delay} ms | RecurringBackgroundServiceSleeping |
| 61 | Information | Lock {LockName} acquired | LockAcquired |
| 62 | Information | Lock {LockName} released | LockReleased |
| 63 | Error | Lock {LockName} lost | LockLost |
| 64 | Error | Failed to acquire lock {LockName} | AcquireLockFailed |
| 65 | Information | Failed to acquire lock {LockName} | AcquireLockConcurrencyException |
| 66 | Error | Failed to release lock {LockName} | ReleaseLockFailed |
Integration
| Id | Level | Message | Reference |
|---|---|---|---|
| 1001 | Information | Processing consumed message | EndpointName: {EndpointName}, BrokerMessageId: {BrokerMessageId} | ProcessingConsumedMessage |
| 1002 | Error | Error occurred processing consumed message | EndpointName: {EndpointName}, BrokerMessageId: {BrokerMessageId} | ProcessingConsumedMessageError |
| 1003 | Critical | Fatal error occurred processing consumed message; the client will be disconnected | EndpointName: {EndpointName}, BrokerMessageId: {BrokerMessageId} | ProcessingConsumedMessageFatalError |
| 1004 | Critical | Fatal error occurred processing consumed message; the client will be disconnected | ConsumerName: {ConsumerName} | ConsumerFatalError |
| 1005 | Information | Message produced | EndpointName: {EndpointName}, BrokerMessageId: {BrokerMessageId} | MessageProduced |
| 1006 | Warning | Error occurred producing message | EndpointName: {EndpointName} | ErrorProducingMessage |
| 1007 | Debug | Message filtered | EndpointName: {EndpointName} | OutboundMessageFiltered |
| 1008 | Information | Message filtered | EndpointName: {EndpointName}, BrokerMessageId: {BrokerMessageId} | InboundMessageFiltered |
| 1011 | Debug | Message {BrokerMessageId} added to {SequenceType} {SequenceId} | Length: {SequenceLength} | MessageAddedToSequence |
| 1012 | Debug | Started new {SequenceType} {SequenceId} | SequenceStarted |
| 1013 | Debug | {SequenceType} {SequenceId} completed | Length: {SequenceLength} | SequenceCompleted |
| 1014 | Debug | {SequenceType} {SequenceId} processing has been aborted | Length: {SequenceLength}, Reason: {Reason} | SequenceProcessingAborted |
| 1015 | Error | Error occurred processing {SequenceType} {SequenceId} | Length: {SequenceLength} | SequenceProcessingError |
| 1016 | Warning | Aborted incomplete {SequenceType} {SequenceId} | Length: {SequenceLength} | IncompleteSequenceAborted |
| 1017 | Warning | Skipped incomplete sequence {SequenceId} (missing first message) | IncompleteSequenceSkipped |
| 1018 | Warning | Error occurred executing timeout for {SequenceType} {SequenceId} | SequenceTimeoutError |
| 1019 | Error | Error occurred disposing {SequenceType} {SequenceId} | SequenceDisposeError |
| 1021 | Error | Error occurred initializing broker clients | BrokerClientsInitializationError |
| 1022 | Debug | {ClientType} initializing | ClientName: {ClientName} | BrokerClientInitializing |
| 1023 | Debug | {ClientType} initialized | ClientName: {ClientName} | BrokerClientInitialized |
| 1024 | Debug | {ClientType} disconnecting | ClientName: {ClientName} | BrokerClientDisconnecting |
| 1025 | Information | {ClientType} disconnected | ClientName: {ClientName} | BrokerClientDisconnected |
| 1026 | Error | Error occurred initializing {ClientType} | ClientName: {ClientName} | BrokerClientInitializeError |
| 1027 | Error | Error occurred disconnecting {ClientType} | ClientName: {ClientName} | BrokerClientDisconnectError |
| 1028 | Warning | Failed to reconnect {ClientType}; retry in {RetryDelay} ms | ClientName: {ClientName} | BrokerClientReconnectError |
| 1031 | Error | Error occurred (re)starting {ConsumerType} | ConsumerName: {ConsumerName} | ConsumerStartError |
| 1032 | Error | Error occurred stopping {ConsumerType} | ConsumerName: {ConsumerName} | ConsumerStopError |
| 1033 | Error | {ConsumerType} commit failed | ConsumerName: {ConsumerName}, BrokerMessageId: {BrokerMessageId} | ConsumerCommitError |
| 1034 | Error | {ConsumerType} rollback failed | ConsumerName: {ConsumerName}, BrokerMessageId: {BrokerMessageId} | ConsumerRollbackError |
| 1041 | Debug | Created {ClientType} | ClientName: {ClientName} | BrokerClientCreated |
| 1042 | Debug | Created {ConsumerType} | ConsumerName: {ConsumerName} | ConsumerCreated |
| 1043 | Debug | Created {ProducerType} | ProducerName: {ProducerName} | ProducerCreated |
| 1051 | Information | The message(s) will be processed again | EndpointName: {EndpointName}, BrokerMessageId: {BrokerMessageId} | RetryMessageProcessing |
| 1052 | Information | The message will be moved to {TargetEndpointName} | EndpointName: {EndpointName}, BrokerMessageId: {BrokerMessageId} | MessageMoved |
| 1053 | Information | The message(s) will be skipped | EndpointName: {EndpointName}, BrokerMessageId: {BrokerMessageId} | MessageSkipped |
| 1054 | Warning | The message belongs to a {SequenceType} and cannot be moved | EndpointName: {EndpointName}, BrokerMessageId: {BrokerMessageId} | CannotMoveSequence |
| 1061 | Warning | Error occurred rolling back, retry error policy cannot be applied; the consumer will be reconnected | EndpointName: {EndpointName}, BrokerMessageId: {BrokerMessageId} | RollbackToRetryFailed |
| 1062 | Warning | Error occurred rolling back transaction or committing offset, skip error policy cannot be applied; the consumer will be reconnected | EndpointName: {EndpointName}, BrokerMessageId: {BrokerMessageId} | RollbackToSkipFailed |
| 1071 | Debug | Storing message into outbox | EndpointName: {EndpointName} | StoringIntoOutbox |
| 1072 | Trace | Reading batch of {ReadBatchSize} messages from outbox | ReadingMessagesFromOutbox |
| 1073 | Trace | Outbox empty | OutboxEmpty |
| 1074 | Debug | Processing outbox message {CurrentMessageIndex} | ProcessingOutboxStoredMessage |
| 1075 | Error | Failed to produce message from outbox | EndpointName: {EndpointName} | ErrorProducingOutboxStoredMessage |
| 1076 | Error | Error occurred processing outbox | ErrorProcessingOutbox |
| 1081 | Warning | Invalid message produced: {ValidationErrors} | EndpointName: {EndpointName} | InvalidMessageProduced |
| 1082 | Warning | Invalid message consumed: {ValidationErrors} | EndpointName: {EndpointName}, BrokerMessageId: {BrokerMessageId} | InvalidMessageConsumed |
| 1101 | Critical | Invalid configuration for endpoint {EndpointName} | InvalidEndpointConfiguration |
| 1102 | Critical | Error occurred configuring endpoints | Configurator: {EndpointsConfiguratorName} | EndpointConfiguratorError |
| 1103 | Error | Error occurred invoking callback handlers | CallbackError |
| 1104 | Critical | Failed to configure endpoint {EndpointName} | EndpointBuilderError |
| 1999 | Trace | The actual message will vary | Tracing |
Kafka
| Id | Level | Message | Reference |
|---|---|---|---|
| 2011 | Debug | Consuming message {Topic}[{Partition}]@{Offset} | ConsumerName: {ConsumerName} | ConsumingMessage |
| 2012 | Information | Partition EOF reached: {Topic}[{Partition}]@{Offset} | ConsumerName: {ConsumerName} | EndOfPartition |
| 2013 | Warning | Error occurred trying to pull next message; the consumer will try to recover | ConsumerName: {ConsumerName} | KafkaExceptionAutoRecovery |
| 2014 | Error | Error occurred trying to pull next message; the consumer will be stopped (auto recovery disabled for consumer) | ConsumerName: {ConsumerName} | KafkaExceptionNoAutoRecovery |
| 2016 | Trace | Consuming canceled | ConsumerName: {ConsumerName} | ConsumingCanceled |
| 2017 | Information | No message consumed within the stall detection threshold ({StallDetectionThreshold}); the consumer will restart | ConsumerName: {ConsumerName} | StaleConsumer |
| 2022 | Warning | Message produced to {Topic}[{Partition}] but not acknowledged | ProducerName: {ProducerName} | ProduceNotAcknowledged |
| 2031 | Information | Assigned partition {Topic}[{Partition}]@{Offset} | ConsumerName: {ConsumerName} | PartitionStaticallyAssigned |
| 2032 | Information | Assigned partition {Topic}[{Partition}] | ConsumerName: {ConsumerName} | PartitionAssigned |
| 2033 | Debug | {Topic}[{Partition}] offset will be reset to {Offset} | ConsumerName: {ConsumerName} | PartitionOffsetReset |
| 2034 | Information | Revoked partition {Topic}[{Partition}]@{Offset} | ConsumerName: {ConsumerName} | PartitionRevoked |
| 2035 | Debug | Partition {Topic}[{Partition}] paused at offset {Offset} | ConsumerName: {ConsumerName} | PartitionPaused |
| 2036 | Debug | Partition {Topic}[{Partition}] resumed | ConsumerName: {ConsumerName} | PartitionResumed |
| 2037 | Debug | Successfully committed offset {Topic}[{Partition}]@{Offset} | ConsumerName: {ConsumerName} | OffsetCommitted |
| 2038 | Error | Error occurred committing offset {Topic}[{Partition}]@{Offset}: '{ErrorReason}' ({ErrorCode}) | ConsumerName: {ConsumerName} | OffsetCommitError |
| 2039 | Warning | Error in Kafka consumer: '{ErrorReason}' ({ErrorCode}) | ConsumerName: {ConsumerName} | ConfluentConsumerError |
| 2040 | Error | Fatal error in Kafka consumer: '{ErrorReason}' ({ErrorCode}) | ConsumerName: {ConsumerName} | ConfluentConsumerFatalError |
| 2041 | Debug | Kafka consumer statistics received: {Statistics} | ConsumerName: {ConsumerName} | ConsumerStatisticsReceived |
| 2042 | Debug | Kafka producer statistics received: {Statistics} | ProducerName: {ProducerName} | ProducerStatisticsReceived |
| 2043 | Error | Statistics JSON couldn't be deserialized | StatisticsDeserializationError |
| 2044 | Warning | Error occurred preparing the consumer to be closed | ConsumerName: {ConsumerName} | ConsumerLeaveGroupError |
| 2045 | Warning | The consumer wasn't ready to be closed within {Timeout}; it will be closed anyway | ConsumerName: {ConsumerName} | ConsumerLeaveGroupTimeout |
| 2060 | Warning | {SysLogLevel} from Confluent.Kafka consumer: '{LogMessage}'; the consumer will try to recover | ConsumerName: {ConsumerName} | PollTimeoutAutoRecovery |
| 2061 | Error | {SysLogLevel} from Confluent.Kafka consumer: '{LogMessage}'; auto recovery disabled for consumer | ConsumerName: {ConsumerName} | PollTimeoutNoAutoRecovery |
| 2070 | Trace | Transactions initialized | ProducerName: {ProducerName}, TransactionalId: {TransactionalId} | TransactionsInitialized |
| 2071 | Trace | Transaction started | ProducerName: {ProducerName}, TransactionalId: {TransactionalId} | TransactionStarted |
| 2072 | Information | Transaction committed | ProducerName: {ProducerName}, TransactionalId: {TransactionalId} | TransactionCommitted |
| 2073 | Information | Transaction aborted | ProducerName: {ProducerName}, TransactionalId: {TransactionalId} | TransactionAborted |
| 2074 | Debug | Offset {Topic}[{Partition}]@{Offset} sent to transaction | ProducerName: {ProducerName}, TransactionalId: {TransactionalId} | OffsetSentToTransaction |
| 2199 | Warning | Error in Kafka producer: '{ErrorReason}' ({ErrorCode}) | ProducerName: {ProducerName} | ConfluentProducerError |
| 2200 | Error | Fatal error in Kafka producer: '{ErrorReason}' ({ErrorCode}) | ProducerName: {ProducerName} | ConfluentProducerFatalError |
| 2201 | Critical | {SysLogLevel} from Confluent.Kafka producer: '{LogMessage}' | ProducerName: {ProducerName} | ConfluentProducerLogCritical |
| 2202 | Error | {SysLogLevel} from Confluent.Kafka producer: '{LogMessage}' | ProducerName: {ProducerName} | ConfluentProducerLogError |
| 2203 | Warning | {SysLogLevel} from Confluent.Kafka producer: '{LogMessage}' | ProducerName: {ProducerName} | ConfluentProducerLogWarning |
| 2204 | Information | {SysLogLevel} from Confluent.Kafka producer: '{LogMessage}' | ProducerName: {ProducerName} | ConfluentProducerLogInformation |
| 2205 | Debug | {SysLogLevel} from Confluent.Kafka producer: '{LogMessage}' | ProducerName: {ProducerName} | ConfluentProducerLogDebug |
| 2211 | Critical | {SysLogLevel} from Confluent.Kafka consumer: '{LogMessage}' | ConsumerName: {ConsumerName} | ConfluentConsumerLogCritical |
| 2212 | Error | {SysLogLevel} from Confluent.Kafka consumer: '{LogMessage}' | ConsumerName: {ConsumerName} | ConfluentConsumerLogError |
| 2213 | Warning | {SysLogLevel} from Confluent.Kafka consumer: '{LogMessage}' | ConsumerName: {ConsumerName} | ConfluentConsumerLogWarning |
| 2214 | Information | {SysLogLevel} from Confluent.Kafka consumer: '{LogMessage}' | ConsumerName: {ConsumerName} | ConfluentConsumerLogInformation |
| 2215 | Debug | {SysLogLevel} from Confluent.Kafka consumer: '{LogMessage}' | ConsumerName: {ConsumerName} | ConfluentConsumerLogDebug |
| 2301 | Warning | Error in Kafka admin client: '{ErrorReason}' ({ErrorCode}) | ConfluentAdminClientError |
| 2302 | Error | Fatal error in Kafka admin client: '{ErrorReason}' ({ErrorCode}) | ConfluentAdminClientFatalError |
| 2311 | Critical | {SysLogLevel} from Confluent.Kafka admin client: '{LogMessage}' | ConfluentAdminClientLogCritical |
| 2312 | Error | {SysLogLevel} from Confluent.Kafka admin client: '{LogMessage}' | ConfluentAdminClientLogError |
| 2313 | Warning | {SysLogLevel} from Confluent.Kafka admin client: '{LogMessage}' | ConfluentAdminClientLogWarning |
| 2314 | Information | {SysLogLevel} from Confluent.Kafka admin client: '{LogMessage}' | ConfluentAdminClientLogInformation |
| 2315 | Debug | {SysLogLevel} from Confluent.Kafka admin client: '{LogMessage}' | ConfluentAdminClientLogDebug |
MQTT
| Id | Level | Message | Reference |
|---|---|---|---|
| 4011 | Debug | Consuming message {BrokerMessageId} from topic {Topic} | ConsumerName: {ConsumerName} | ConsumingMessage |
| 4012 | Warning | Failed to acknowledge message {BrokerMessageId} from topic {Topic} | ConsumerName: {ConsumerName} | AcknowledgeFailed |
| 4013 | Warning | No matching subscribers found for the produced message | Topic: {Topic} | NoMatchingSubscribers |
| 4014 | Information | Batch processing enabled: the broker must allow at least {MaxPendingMessages} unacknowledged messages (batch size {BatchSize}, max degree of parallelism {MaxDegreeOfParallelism}) | ConsumerName: {ConsumerName} | BatchProcessingEnabled |
| 4021 | Warning | Error occurred connecting to MQTT | ClientName: {ClientName}, ClientId: {ClientId}, Broker: {Broker} | ConnectError |
| 4022 | Debug | Error occurred retrying to connect to MQTT | ClientName: {ClientName}, ClientId: {ClientId}, Broker: {Broker} | ConnectRetryError |
| 4023 | Warning | Connection to MQTT lost; the client will try to reconnect | ClientName: {ClientName}, ClientId: {ClientId}, Broker: {Broker} | ConnectionLost |
| 4024 | Information | Connection to MQTT reestablished | ClientName: {ClientName}, ClientId: {ClientId}, Broker: {Broker} | Reconnected |
| 4031 | Debug | Producer queue processing canceled | ClientName: {ClientName}, ClientId: {ClientId} | ProducerQueueProcessingCanceled |
| 4041 | Information | Consumer subscribed to {TopicPattern} | ClientName: {ClientName}, ClientId: {ClientId}, ConsumerName: {ConsumerName} | ConsumerSubscribed |
| 4042 | Error | Subscription to {TopicPattern} rejected by the broker with result code {ResultCode} | ClientName: {ClientName}, ClientId: {ClientId} | SubscriptionRejected |
| 4043 | Warning | Error occurred unsubscribing before disconnecting from MQTT | ClientName: {ClientName}, ClientId: {ClientId}, Broker: {Broker} | UnsubscribeError |
| 4044 | Warning | Unsubscription from {TopicPattern} rejected by the broker with result code {ResultCode} | ClientName: {ClientName}, ClientId: {ClientId} | UnsubscriptionRejected |
| 4101 | Error | Error from MqttClient ({Source}): '{LogMessage}' | MqttClientLogError |
| 4102 | Warning | Warning from MqttClient ({Source}): '{LogMessage}' | MqttClientLogWarning |
| 4103 | Information | Information from MqttClient ({Source}): '{LogMessage}' | MqttClientLogInformation |
| 4104 | Trace | Verbose from MqttClient ({Source}): '{LogMessage}' | MqttClientLogVerbose |
In-Memory Storage
| Id | Level | Message | Reference |
|---|---|---|---|
| 6001 | Warning | Ambient transaction not supported by the in-memory outbox implementation; the messages will be stored ignoring the transaction | OutboxTransactionUnsupported |
Customizing the Log Levels
The level of each Silverback log event can be overridden using WithLogLevels. The level can be set statically or determined dynamically, for example depending on the exception type.
services
.AddSilverback()
.WithLogLevels(levels => levels
.SetLogLevel(CoreLogEvents.LockAcquired.EventId, LogLevel.Debug)
.SetLogLevel(
IntegrationLogEvents.ProcessingConsumedMessageError.EventId,
(exception, defaultLevel) => exception is TimeoutException ? LogLevel.Warning : defaultLevel));
The event ids are exposed by the CoreLogEvents, IntegrationLogEvents, KafkaLogEvents, MqttLogEvents and StorageMemoryLogEvents static classes.
Tip
To reduce the noise, you can also use the standard logging filters to change the minimum level of the Silverback categories.
Tracing
An Activity is created:
- in the Consumer when a message is received (initialized with the
traceparentheader, if submitted) - in the Producer when a message is being sent (submitting the
Activity.Idin thetraceparentheader) - when a subscriber is being invoked (either internally or from a consumer)
This allows tracing the message handling and follow it across different services (distributed tracing). See Distributed Tracing and Producers and Distributed Tracing and Consumers for details about the trace propagation.
The following table summarizes the activities and the tags being set (the tags in square brackets are set only when applicable).
| Source | Activity | Description / Tags |
|---|---|---|
Silverback.Integration.Produce |
Silverback.Integration.Produce |
A message is being produced to a message broker. Tags:
|
Silverback.Integration.Consume |
Silverback.Integration.Consume |
A consumed message is being processed. Tags:
|
Silverback.Core.Subscribers |
Silverback.Core.Subscribers.InvokeSubscriber |
A subscriber is being invoked to process a message. Tags:
|
Note
The produce and consume activities are always started, even if no listener is registered, so that the trace context is propagated and available in the logs. The subscriber activity is started only when the Silverback.Core.Subscribers source is listened to.
Exporting the Traces
Register the Silverback sources to export the activities, for example with OpenTelemetry:
services
.AddOpenTelemetry()
.WithTracing(tracing => tracing
.AddSource(
"Silverback.Integration.Produce",
"Silverback.Integration.Consume",
"Silverback.Core.Subscribers")
.AddOtlpExporter());
Including the Trace in the Logs
The generic host enables the TraceId, SpanId and ParentId activity tracking options by default, adding them to the logging scope. Enable the scopes in the logging provider to write them along with the log entries (including the Silverback ones):
builder.Logging.AddSimpleConsole(options => options.IncludeScopes = true);
info: Silverback.Messaging.Consuming.Transaction.TransactionHandlerConsumerBehavior[1001]
=> SpanId:6b9a3e2c41f0d815, TraceId:4bf92f3577b34da6a3ce929d0e0e4736, ParentId:00f067aa0ba902b7
Processing consumed message | EndpointName: my-topic, BrokerMessageId: [2]@42
Outside the generic host, set the options explicitly:
services.AddLogging(logging => logging
.Configure(options => options.ActivityTrackingOptions =
ActivityTrackingOptions.TraceId |
ActivityTrackingOptions.SpanId |
ActivityTrackingOptions.ParentId)
.AddSimpleConsole(options => options.IncludeScopes = true));
Tip
Serilog (3.1 or later) captures the TraceId and SpanId automatically, so they can be added to the output template (e.g. {TraceId}), while the OpenTelemetry logging provider correlates the logs with the traces out of the box.