Show / Hide Table of Contents

    Class KafkaConsumerEndpoint

    Represents a topic to consume from.

    Inheritance
    System.Object
    Endpoint
    ConsumerEndpoint
    KafkaConsumerEndpoint
    Implements
    IConsumerEndpoint
    IEndpoint
    System.IEquatable<KafkaConsumerEndpoint>
    Inherited Members
    ConsumerEndpoint.ErrorPolicy
    ConsumerEndpoint.ExactlyOnceStrategy
    ConsumerEndpoint.Batch
    ConsumerEndpoint.Sequence
    ConsumerEndpoint.NullMessageHandlingStrategy
    ConsumerEndpoint.ThrowIfUnhandled
    Endpoint.DefaultSerializer
    Endpoint.DisplayName
    Endpoint.Name
    Endpoint.FriendlyName
    Endpoint.Serializer
    Endpoint.Encryption
    Endpoint.MessageValidationMode
    Endpoint.BaseEquals(Endpoint)
    System.Object.Equals(System.Object, System.Object)
    System.Object.GetType()
    System.Object.MemberwiseClone()
    System.Object.ReferenceEquals(System.Object, System.Object)
    System.Object.ToString()
    Namespace: Silverback.Messaging
    Assembly: Silverback.Integration.Kafka.dll
    Syntax
    public sealed class KafkaConsumerEndpoint : ConsumerEndpoint, IConsumerEndpoint, IEndpoint, IEquatable<KafkaConsumerEndpoint>

    Constructors

    | Improve this doc View source

    KafkaConsumerEndpoint(TopicPartition[])

    Initializes a new instance of the KafkaConsumerEndpoint class.

    Declaration
    public KafkaConsumerEndpoint(params TopicPartition[] topicPartitions)
    Parameters
    Type Name Description
    Confluent.Kafka.TopicPartition[] topicPartitions

    The topics and partitions to be consumed.

    | Improve this doc View source

    KafkaConsumerEndpoint(TopicPartitionOffset[])

    Initializes a new instance of the KafkaConsumerEndpoint class.

    Declaration
    public KafkaConsumerEndpoint(params TopicPartitionOffset[] topicPartitions)
    Parameters
    Type Name Description
    Confluent.Kafka.TopicPartitionOffset[] topicPartitions

    The topics and partitions to be consumed and the starting offset.

    | Improve this doc View source

    KafkaConsumerEndpoint(IEnumerable<TopicPartition>, KafkaClientConfig)

    Initializes a new instance of the KafkaConsumerEndpoint class.

    Declaration
    public KafkaConsumerEndpoint(IEnumerable<TopicPartition> topicPartitions, KafkaClientConfig clientConfig = null)
    Parameters
    Type Name Description
    System.Collections.Generic.IEnumerable<Confluent.Kafka.TopicPartition> topicPartitions

    The topics and partitions to be consumed.

    KafkaClientConfig clientConfig

    The KafkaClientConfig to be used to initialize the KafkaConsumerConfig.

    | Improve this doc View source

    KafkaConsumerEndpoint(IEnumerable<TopicPartitionOffset>, KafkaClientConfig)

    Initializes a new instance of the KafkaConsumerEndpoint class.

    Declaration
    public KafkaConsumerEndpoint(IEnumerable<TopicPartitionOffset> topicPartitions, KafkaClientConfig clientConfig = null)
    Parameters
    Type Name Description
    System.Collections.Generic.IEnumerable<Confluent.Kafka.TopicPartitionOffset> topicPartitions

    The topics and partitions to be consumed.

    KafkaClientConfig clientConfig

    The KafkaClientConfig to be used to initialize the KafkaConsumerConfig.

    | Improve this doc View source

    KafkaConsumerEndpoint(String, KafkaClientConfig)

    Initializes a new instance of the KafkaConsumerEndpoint class.

    Declaration
    public KafkaConsumerEndpoint(string topicName, KafkaClientConfig clientConfig = null)
    Parameters
    Type Name Description
    System.String topicName

    The name of the topic.

    KafkaClientConfig clientConfig

    The KafkaClientConfig to be used to initialize the KafkaConsumerConfig.

    | Improve this doc View source

    KafkaConsumerEndpoint(String, Func<IReadOnlyCollection<TopicPartition>, IEnumerable<TopicPartition>>, KafkaClientConfig)

    Initializes a new instance of the KafkaConsumerEndpoint class.

    Declaration
    public KafkaConsumerEndpoint(string topicName, Func<IReadOnlyCollection<TopicPartition>, IEnumerable<TopicPartition>> topicPartitionsResolver, KafkaClientConfig clientConfig = null)
    Parameters
    Type Name Description
    System.String topicName

    The name of the topic.

    System.Func<System.Collections.Generic.IReadOnlyCollection<Confluent.Kafka.TopicPartition>, System.Collections.Generic.IEnumerable<Confluent.Kafka.TopicPartition>> topicPartitionsResolver

    The function that returns the topics and partitions to be consumed.

    KafkaClientConfig clientConfig

    The KafkaClientConfig to be used to initialize the KafkaConsumerConfig.

    | Improve this doc View source

    KafkaConsumerEndpoint(String, Func<IReadOnlyCollection<TopicPartition>, IEnumerable<TopicPartitionOffset>>, KafkaClientConfig)

    Initializes a new instance of the KafkaConsumerEndpoint class.

    Declaration
    public KafkaConsumerEndpoint(string topicName, Func<IReadOnlyCollection<TopicPartition>, IEnumerable<TopicPartitionOffset>> topicPartitionsResolver, KafkaClientConfig clientConfig = null)
    Parameters
    Type Name Description
    System.String topicName

    The name of the topic.

    System.Func<System.Collections.Generic.IReadOnlyCollection<Confluent.Kafka.TopicPartition>, System.Collections.Generic.IEnumerable<Confluent.Kafka.TopicPartitionOffset>> topicPartitionsResolver

    The function that returns the topics and partitions to be consumed.

    KafkaClientConfig clientConfig

    The KafkaClientConfig to be used to initialize the KafkaConsumerConfig.

    | Improve this doc View source

    KafkaConsumerEndpoint(String[])

    Initializes a new instance of the KafkaConsumerEndpoint class.

    Declaration
    public KafkaConsumerEndpoint(params string[] topicNames)
    Parameters
    Type Name Description
    System.String[] topicNames

    The name of the topics.

    | Improve this doc View source

    KafkaConsumerEndpoint(String[], KafkaClientConfig)

    Initializes a new instance of the KafkaConsumerEndpoint class.

    Declaration
    public KafkaConsumerEndpoint(string[] topicNames, KafkaClientConfig clientConfig = null)
    Parameters
    Type Name Description
    System.String[] topicNames

    The name of the topics.

    KafkaClientConfig clientConfig

    The KafkaClientConfig to be used to initialize the KafkaConsumerConfig.

    | Improve this doc View source

    KafkaConsumerEndpoint(String[], Func<IReadOnlyCollection<TopicPartition>, IEnumerable<TopicPartition>>, KafkaClientConfig)

    Initializes a new instance of the KafkaConsumerEndpoint class.

    Declaration
    public KafkaConsumerEndpoint(string[] topicNames, Func<IReadOnlyCollection<TopicPartition>, IEnumerable<TopicPartition>> topicPartitionsResolver, KafkaClientConfig clientConfig = null)
    Parameters
    Type Name Description
    System.String[] topicNames

    The name of the topics.

    System.Func<System.Collections.Generic.IReadOnlyCollection<Confluent.Kafka.TopicPartition>, System.Collections.Generic.IEnumerable<Confluent.Kafka.TopicPartition>> topicPartitionsResolver

    The function that returns the topics and partitions to be consumed.

    KafkaClientConfig clientConfig

    The KafkaClientConfig to be used to initialize the KafkaConsumerConfig.

    | Improve this doc View source

    KafkaConsumerEndpoint(String[], Func<IReadOnlyCollection<TopicPartition>, IEnumerable<TopicPartitionOffset>>, KafkaClientConfig)

    Initializes a new instance of the KafkaConsumerEndpoint class.

    Declaration
    public KafkaConsumerEndpoint(string[] topicNames, Func<IReadOnlyCollection<TopicPartition>, IEnumerable<TopicPartitionOffset>> topicPartitionsResolver, KafkaClientConfig clientConfig = null)
    Parameters
    Type Name Description
    System.String[] topicNames

    The name of the topics.

    System.Func<System.Collections.Generic.IReadOnlyCollection<Confluent.Kafka.TopicPartition>, System.Collections.Generic.IEnumerable<Confluent.Kafka.TopicPartitionOffset>> topicPartitionsResolver

    The function that returns the topics and partitions to be consumed.

    KafkaClientConfig clientConfig

    The KafkaClientConfig to be used to initialize the KafkaConsumerConfig.

    Properties

    | Improve this doc View source

    BackpressureLimit

    Gets or sets the maximum number of messages to be consumed and enqueued waiting to be processed. When ProcessPartitionsIndependently is set to true (default) the limit will be applied per partition. The default is 1.

    Declaration
    public int BackpressureLimit { get; set; }
    Property Value
    Type Description
    System.Int32
    | Improve this doc View source

    Configuration

    Gets or sets the Kafka client configuration. This is actually an extension of the configuration dictionary provided by the Confluent.Kafka library.

    Declaration
    public KafkaConsumerConfig Configuration { get; set; }
    Property Value
    Type Description
    KafkaConsumerConfig
    | Improve this doc View source

    MaxDegreeOfParallelism

    Gets or sets the maximum number of incoming message that can be processed concurrently. Up to a message per each subscribed partition can be processed in parallel. The default is 10.

    Declaration
    public int MaxDegreeOfParallelism { get; set; }
    Property Value
    Type Description
    System.Int32
    | Improve this doc View source

    Names

    Gets the name of the topics.

    Declaration
    public IReadOnlyCollection<string> Names { get; }
    Property Value
    Type Description
    System.Collections.Generic.IReadOnlyCollection<System.String>
    | Improve this doc View source

    ProcessPartitionsIndependently

    Gets or sets a value indicating whether the partitions must be processed independently. When true a stream will published per each partition and the sequences (ChunkSequence, BatchSequence, ...) cannot span across the partitions. The default is true.

    Declaration
    public bool ProcessPartitionsIndependently { get; set; }
    Property Value
    Type Description
    System.Boolean
    | Improve this doc View source

    TopicPartitions

    Gets the topics and partitions to be consumed.

    If both TopicPartitions and TopicPartitionsResolver are null the topics from the Names property will be subscribed and the partitions will be automatically assigned by the broker.

    If the collection is empty no partition will be consumed.

    Declaration
    public IReadOnlyCollection<TopicPartitionOffset> TopicPartitions { get; }
    Property Value
    Type Description
    System.Collections.Generic.IReadOnlyCollection<Confluent.Kafka.TopicPartitionOffset>
    | Improve this doc View source

    TopicPartitionsResolver

    Gets the function that returns the topics and partitions to be consumed.

    If both TopicPartitions and TopicPartitionsResolver are null the topics from the Names property will be subscribed and the partitions will be automatically assigned by the broker.

    If an empty collection is returned by the function no partition will be consumed.

    Declaration
    public Func<IReadOnlyCollection<TopicPartition>, IEnumerable<TopicPartitionOffset>> TopicPartitionsResolver { get; }
    Property Value
    Type Description
    System.Func<System.Collections.Generic.IReadOnlyCollection<Confluent.Kafka.TopicPartition>, System.Collections.Generic.IEnumerable<Confluent.Kafka.TopicPartitionOffset>>

    Methods

    | Improve this doc View source

    Equals(KafkaConsumerEndpoint)

    Declaration
    public bool Equals(KafkaConsumerEndpoint other)
    Parameters
    Type Name Description
    KafkaConsumerEndpoint other
    Returns
    Type Description
    System.Boolean
    | Improve this doc View source

    Equals(Object)

    Declaration
    public override bool Equals(object obj)
    Parameters
    Type Name Description
    System.Object obj
    Returns
    Type Description
    System.Boolean
    Overrides
    System.Object.Equals(System.Object)
    | Improve this doc View source

    GetHashCode()

    Declaration
    public override int GetHashCode()
    Returns
    Type Description
    System.Int32
    Overrides
    System.Object.GetHashCode()
    | Improve this doc View source

    GetUniqueConsumerGroupName()

    Gets a unique name for the consumer group (e.g. Kafka's consumer group id). This value (joint with the endpoint name) will be used for example to ensure the exactly-once delivery.

    Declaration
    public override string GetUniqueConsumerGroupName()
    Returns
    Type Description
    System.String

    Returns the unique name for the consumer group.

    Overrides
    ConsumerEndpoint.GetUniqueConsumerGroupName()
    Remarks

    It's not enough to use the endpoint name, since the same topic could be consumed by multiple consumer groups within the same process and/or using the same database to store the information needed to ensure the exactly-once delivery.

    | Improve this doc View source

    Validate()

    Validates the endpoint configuration and throws an EndpointConfigurationException if not valid.

    Declaration
    public override void Validate()
    Overrides
    ConsumerEndpoint.Validate()

    Implements

    IConsumerEndpoint
    IEndpoint
    System.IEquatable<T>
    • Improve this doc
    • View source
    GitHub E-Mail
    ↑ Back to top © 2020 Sergio Aquilini