Table of Contents

Logging and Tracing

Silverback writes its logs through the standard Microsoft.Extensions.Logging abstractions, so they flow into whatever logging provider your application uses (console, Serilog, OpenTelemetry, etc.). Each event has a stable EventId that can be used to filter or route the logs.

On top of that, Silverback creates Activity objects to support distributed tracing (see Tracing).

Log Events

Silverback logs several events that can be used to monitor the application and troubleshoot issues. The tables below list the events and their default level.

Core

Id Level Message Reference
11 Debug Discarding result of type {Type} because doesn't match expected return type {ExpectedType} SubscriberResultDiscarded
41 Information Starting background service {BackgroundService} BackgroundServiceStarting
42 Error Background service {BackgroundService} execution failed BackgroundServiceException
51 Information Background service {BackgroundService} stopped RecurringBackgroundServiceStopped
52 Debug Background service {BackgroundService} sleeping for {Delay} ms RecurringBackgroundServiceSleeping
61 Information Lock {LockName} acquired LockAcquired
62 Information Lock {LockName} released LockReleased
63 Error Lock {LockName} lost LockLost
64 Error Failed to acquire lock {LockName} AcquireLockFailed
65 Information Failed to acquire lock {LockName} AcquireLockConcurrencyException
66 Error Failed to release lock {LockName} ReleaseLockFailed

Integration

Id Level Message Reference
1001 Information Processing consumed message | EndpointName: {EndpointName}, BrokerMessageId: {BrokerMessageId} ProcessingConsumedMessage
1002 Error Error occurred processing consumed message | EndpointName: {EndpointName}, BrokerMessageId: {BrokerMessageId} ProcessingConsumedMessageError
1003 Critical Fatal error occurred processing consumed message; the client will be disconnected | EndpointName: {EndpointName}, BrokerMessageId: {BrokerMessageId} ProcessingConsumedMessageFatalError
1004 Critical Fatal error occurred processing consumed message; the client will be disconnected | ConsumerName: {ConsumerName} ConsumerFatalError
1005 Information Message produced | EndpointName: {EndpointName}, BrokerMessageId: {BrokerMessageId} MessageProduced
1006 Warning Error occurred producing message | EndpointName: {EndpointName} ErrorProducingMessage
1007 Debug Message filtered | EndpointName: {EndpointName} OutboundMessageFiltered
1011 Debug Message {BrokerMessageId} added to {SequenceType} {SequenceId} | Length: {SequenceLength} MessageAddedToSequence
1012 Debug Started new {SequenceType} {SequenceId} SequenceStarted
1013 Debug {SequenceType} {SequenceId} completed | Length: {SequenceLength} SequenceCompleted
1014 Debug {SequenceType} {SequenceId} processing has been aborted | Length: {SequenceLength}, Reason: {Reason} SequenceProcessingAborted
1015 Error Error occurred processing {SequenceType} {SequenceId} | Length: {SequenceLength} SequenceProcessingError
1016 Warning Aborted incomplete {SequenceType} {SequenceId} | Length: {SequenceLength} IncompleteSequenceAborted
1017 Warning Skipped incomplete sequence {SequenceId} (missing first message) IncompleteSequenceSkipped
1018 Warning Error occurred executing timeout for {SequenceType} {SequenceId} SequenceTimeoutError
1021 Error Error occurred initializing broker clients BrokerClientsInitializationError
1022 Debug {ClientType} initializing | ClientName: {ClientName} BrokerClientInitializing
1023 Debug {ClientType} initialized | ClientName: {ClientName} BrokerClientInitialized
1024 Debug {ClientType} disconnecting | ClientName: {ClientName} BrokerClientDisconnecting
1025 Information {ClientType} disconnected | ClientName: {ClientName} BrokerClientDisconnected
1026 Error Error occurred initializing {ClientType} | ClientName: {ClientName} BrokerClientInitializeError
1027 Error Error occurred disconnecting {ClientType} | ClientName: {ClientName} BrokerClientDisconnectError
1028 Warning Failed to reconnect {ClientType}; retry in {RetryDelay} ms | ClientName: {ClientName} BrokerClientReconnectError
1031 Error Error occurred (re)starting {ConsumerType} | ConsumerName: {ConsumerName} ConsumerStartError
1032 Error Error occurred stopping {ConsumerType} | ConsumerName: {ConsumerName} ConsumerStopError
1033 Error {ConsumerType} commit failed | ConsumerName: {ConsumerName}, BrokerMessageId: {BrokerMessageId} ConsumerCommitError
1034 Error {ConsumerType} rollback failed | ConsumerName: {ConsumerName}, BrokerMessageId: {BrokerMessageId} ConsumerRollbackError
1041 Debug Created {ClientType} | ClientName: {ClientName} BrokerClientCreated
1042 Debug Created {ConsumerType} | ConsumerName: {ConsumerName} ConsumerCreated
1043 Debug Created {ProducerType} | ProducerName: {ProducerName} ProducerCreated
1051 Information The message(s) will be processed again | EndpointName: {EndpointName}, BrokerMessageId: {BrokerMessageId} RetryMessageProcessing
1052 Information The message will be moved to {TargetEndpointName} | EndpointName: {EndpointName}, BrokerMessageId: {BrokerMessageId} MessageMoved
1053 Information The message(s) will be skipped | EndpointName: {EndpointName}, BrokerMessageId: {BrokerMessageId} MessageSkipped
1054 Warning The message belongs to a {SequenceType} and cannot be moved | EndpointName: {EndpointName}, BrokerMessageId: {BrokerMessageId} CannotMoveSequence
1061 Warning Error occurred rolling back, retry error policy cannot be applied; the consumer will be reconnected | EndpointName: {EndpointName}, BrokerMessageId: {BrokerMessageId} RollbackToRetryFailed
1062 Warning Error occurred rolling back transaction or committing offset, skip error policy cannot be applied; the consumer will be reconnected | EndpointName: {EndpointName}, BrokerMessageId: {BrokerMessageId} RollbackToSkipFailed
1071 Debug Storing message into outbox | EndpointName: {EndpointName} StoringIntoOutbox
1072 Trace Reading batch of {ReadBatchSize} messages from outbox ReadingMessagesFromOutbox
1073 Trace Outbox empty OutboxEmpty
1074 Debug Processing outbox message {CurrentMessageIndex} ProcessingOutboxStoredMessage
1075 Error Failed to produce message from outbox | EndpointName: {EndpointName} ErrorProducingOutboxStoredMessage
1076 Error Error occurred processing outbox ErrorProcessingOutbox
1081 Warning Invalid message produced: {ValidationErrors} | EndpointName: {EndpointName} InvalidMessageProduced
1082 Warning Invalid message consumed: {ValidationErrors} | EndpointName: {EndpointName}, BrokerMessageId: {BrokerMessageId} InvalidMessageConsumed
1101 Critical Invalid configuration for endpoint {EndpointName} InvalidEndpointConfiguration
1102 Critical Error occurred configuring endpoints | Configurator: {EndpointsConfiguratorName} EndpointConfiguratorError
1103 Error Error occurred invoking callback handlers CallbackError
1104 Critical Failed to configure endpoint {EndpointName} EndpointBuilderError
1999 Trace The actual message will vary Tracing

Kafka

Id Level Message Reference
2011 Debug Consuming message {Topic}[{Partition}]@{Offset} | ConsumerName: {ConsumerName} ConsumingMessage
2012 Information Partition EOF reached: {Topic}[{Partition}]@{Offset} | ConsumerName: {ConsumerName} EndOfPartition
2013 Warning Error occurred trying to pull next message; the consumer will try to recover | ConsumerName: {ConsumerName} KafkaExceptionAutoRecovery
2014 Error Error occurred trying to pull next message; the consumer will be stopped (auto recovery disabled for consumer) | ConsumerName: {ConsumerName} KafkaExceptionNoAutoRecovery
2016 Trace Consuming canceled | ConsumerName: {ConsumerName} ConsumingCanceled
2017 Information No message consumed within the stall detection threshold ({StallDetectionThreshold}); the consumer will restart | ConsumerName: {ConsumerName} StaleConsumer
2022 Warning Message produced to {Topic}[{Partition}] but not acknowledged | ProducerName: {ProducerName} ProduceNotAcknowledged
2031 Information Assigned partition {Topic}[{Partition}]@{Offset} | ConsumerName: {ConsumerName} PartitionStaticallyAssigned
2032 Information Assigned partition {Topic}[{Partition}] | ConsumerName: {ConsumerName} PartitionAssigned
2033 Debug {Topic}[{Partition}] offset will be reset to {Offset} | ConsumerName: {ConsumerName} PartitionOffsetReset
2034 Information Revoked partition {Topic}[{Partition}]@{Offset} | ConsumerName: {ConsumerName} PartitionRevoked
2035 Debug Partition {Topic}[{Partition}] paused at offset {Offset} | ConsumerName: {ConsumerName} PartitionPaused
2036 Debug Partition {Topic}[{Partition}] resumed | ConsumerName: {ConsumerName} PartitionResumed
2037 Debug Successfully committed offset {Topic}[{Partition}]@{Offset} | ConsumerName: {ConsumerName} OffsetCommitted
2038 Error Error occurred committing offset {Topic}[{Partition}]@{Offset}: '{ErrorReason}' ({ErrorCode}) | ConsumerName: {ConsumerName} OffsetCommitError
2039 Warning Error in Kafka consumer: '{ErrorReason}' ({ErrorCode}) | ConsumerName: {ConsumerName} ConfluentConsumerError
2040 Error Fatal error in Kafka consumer: '{ErrorReason}' ({ErrorCode}) | ConsumerName: {ConsumerName} ConfluentConsumerFatalError
2041 Debug Kafka consumer statistics received: {Statistics} | ConsumerName: {ConsumerName} ConsumerStatisticsReceived
2042 Debug Kafka producer statistics received: {Statistics} | ProducerName: {ProducerName} ProducerStatisticsReceived
2043 Error Statistics JSON couldn't be deserialized StatisticsDeserializationError
2060 Warning {SysLogLevel} from Confluent.Kafka consumer: '{LogMessage}'; the consumer will try to recover | ConsumerName: {ConsumerName} PollTimeoutAutoRecovery
2061 Error {SysLogLevel} from Confluent.Kafka consumer: '{LogMessage}'; auto recovery disabled for consumer | ConsumerName: {ConsumerName} PollTimeoutNoAutoRecovery
2070 Trace Transactions initialized | ProducerName: {ProducerName}, TransactionalId: {TransactionalId} TransactionsInitialized
2071 Trace Transaction started | ProducerName: {ProducerName}, TransactionalId: {TransactionalId} TransactionStarted
2072 Information Transaction committed | ProducerName: {ProducerName}, TransactionalId: {TransactionalId} TransactionCommitted
2073 Information Transaction aborted | ProducerName: {ProducerName}, TransactionalId: {TransactionalId} TransactionAborted
2074 Debug Offset {Topic}[{Partition}]@{Offset} sent to transaction | ProducerName: {ProducerName}, TransactionalId: {TransactionalId} OffsetSentToTransaction
2201 Critical {SysLogLevel} from Confluent.Kafka producer: '{LogMessage}' | ProducerName: {ProducerName} ConfluentProducerLogCritical
2202 Error {SysLogLevel} from Confluent.Kafka producer: '{LogMessage}' | ProducerName: {ProducerName} ConfluentProducerLogError
2203 Warning {SysLogLevel} from Confluent.Kafka producer: '{LogMessage}' | ProducerName: {ProducerName} ConfluentProducerLogWarning
2204 Information {SysLogLevel} from Confluent.Kafka producer: '{LogMessage}' | ProducerName: {ProducerName} ConfluentProducerLogInformation
2205 Debug {SysLogLevel} from Confluent.Kafka producer: '{LogMessage}' | ProducerName: {ProducerName} ConfluentProducerLogDebug
2211 Critical {SysLogLevel} from Confluent.Kafka consumer: '{LogMessage}' | ConsumerName: {ConsumerName} ConfluentConsumerLogCritical
2212 Error {SysLogLevel} from Confluent.Kafka consumer: '{LogMessage}' | ConsumerName: {ConsumerName} ConfluentConsumerLogError
2213 Warning {SysLogLevel} from Confluent.Kafka consumer: '{LogMessage}' | ConsumerName: {ConsumerName} ConfluentConsumerLogWarning
2214 Information {SysLogLevel} from Confluent.Kafka consumer: '{LogMessage}' | ConsumerName: {ConsumerName} ConfluentConsumerLogInformation
2215 Debug {SysLogLevel} from Confluent.Kafka consumer: '{LogMessage}' | ConsumerName: {ConsumerName} ConfluentConsumerLogDebug
2301 Warning Error in Kafka admin client: '{ErrorReason}' ({ErrorCode}) ConfluentAdminClientError
2302 Error Fatal error in Kafka admin client: '{ErrorReason}' ({ErrorCode}) ConfluentAdminClientFatalError
2311 Critical {SysLogLevel} from Confluent.Kafka admin client: '{LogMessage}' ConfluentAdminClientLogCritical
2312 Error {SysLogLevel} from Confluent.Kafka admin client: '{LogMessage}' ConfluentAdminClientLogError
2313 Warning {SysLogLevel} from Confluent.Kafka admin client: '{LogMessage}' ConfluentAdminClientLogWarning
2314 Information {SysLogLevel} from Confluent.Kafka admin client: '{LogMessage}' ConfluentAdminClientLogInformation
2315 Debug {SysLogLevel} from Confluent.Kafka admin client: '{LogMessage}' ConfluentAdminClientLogDebug

MQTT

Id Level Message Reference
4011 Debug Consuming message {BrokerMessageId} from topic {Topic} | ConsumerName: {ConsumerName} ConsumingMessage
4012 Warning Failed to acknowledge message {BrokerMessageId} from topic {Topic} | ConsumerName: {ConsumerName} AcknowledgeFailed
4013 Warning No matching subscribers found for the produced message | Topic: {Topic} NoMatchingSubscribers
4021 Warning Error occurred connecting to MQTT | ClientName: {ClientName}, ClientId: {ClientId}, Broker: {Broker} ConnectError
4022 Debug Error occurred retrying to connect to MQTT | ClientName: {ClientName}, ClientId: {ClientId}, Broker: {Broker} ConnectRetryError
4023 Warning Connection to MQTT lost; the client will try to reconnect | ClientName: {ClientName}, ClientId: {ClientId}, Broker: {Broker} ConnectionLost
4024 Information Connection to MQTT reestablished | ClientName: {ClientName}, ClientId: {ClientId}, Broker: {Broker} Reconnected
4031 Debug Producer queue processing canceled | ClientName: {ClientName}, ClientId: {ClientId} ProducerQueueProcessingCanceled
4041 Information Consumer subscribed to {topicPattern} | ClientName: {ClientName}, ClientId: {ClientId}, ConsumerName: {ConsumerName} ConsumerSubscribed
4101 Error Error from MqttClient ({Source}): '{LogMessage}' MqttClientLogError
4102 Warning Warning from MqttClient ({Source}): '{LogMessage}' MqttClientLogWarning
4103 Information Information from MqttClient ({Source}): '{LogMessage}' MqttClientLogInformation
4104 Trace Verbose from MqttClient ({Source}): '{LogMessage}' MqttClientLogVerbose

Customizing the Log Levels

The level of each Silverback log event can be overridden using WithLogLevels. The level can be set statically or determined dynamically, for example depending on the exception type.

services
    .AddSilverback()
    .WithLogLevels(levels => levels
        .SetLogLevel(CoreLogEvents.LockAcquired.EventId, LogLevel.Debug)
        .SetLogLevel(
            IntegrationLogEvents.ProcessingConsumedMessageError.EventId,
            (exception, defaultLevel) => exception is TimeoutException ? LogLevel.Warning : defaultLevel));

The event ids are exposed by the CoreLogEvents, IntegrationLogEvents, KafkaLogEvents and MqttLogEvents static classes.

Tip

To reduce the noise, you can also use the standard logging filters to change the minimum level of the Silverback categories.

Tracing

An Activity is created:

  • in the Consumer when a message is received (initialized with the traceparent header, if submitted)
  • in the Producer when a message is being sent (submitting the Activity.Id in the traceparent header)
  • when a subscriber is being invoked (either internally or from a consumer)

This allows tracing the message handling and follow it across different services (distributed tracing).

The following table summarizes the activities and the information being tracked.

Id Description / Tags
Silverback.Integration.Produce A message is being produced to a message broker.

Tags:
  • messaging.message_id
  • messaging.destination
  • [messaging.kafka.message_key]
  • [messaging.kafka.partition]
Silverback.Integration.Consume A consumed message is being processed.

Tags:
  • messaging.message_id
  • messaging.destination
  • [messaging.sequence_id]
  • [messaging.kafka.message_key]
  • [messaging.kafka.partition]
Silverback.Core.Subscribers.InvokeSubscriber A subscriber is being invoked to process a message.

Tags:
  • SubscriberType
  • SubscriberMethod
Edit this page