Azure Event Hubs client library for .NET

Azure Event Hubs is a highly scalable publish-subscribe service that can ingest millions of events per second and stream them to multiple consumers. This lets you process and analyze the massive amounts of data produced by your connected devices and applications. Once Event Hubs has collected the data, you can retrieve, transform and store it by using any real-time analytics provider or with batching/storage adapters. If you would like to know more about Azure Event Hubs, you may wish to review: What is Event Hubs?

The Azure Event Hubs client library allows for publishing and consuming of Azure Event Hubs events and may be used to:

• Emit telemetry about your application for business intelligence and diagnostic purposes.

• Publish facts about the state of your application which interested parties may observe and use as a trigger for taking action.

• Observe interesting operations and interactions happening within your business or other ecosystem, allowing loosely coupled systems to interact without the need to bind them together.

• Receive events from one or more publishers, transform them to better meet the needs of your ecosystem, then publish the transformed events to a new stream for consumers to observe.

Source code | Package (NuGet) | API reference documentation | Product documentation

Getting started

Prerequisites

• Microsoft Azure Subscription: To use Azure services, including Azure Event Hubs, you'll need a subscription. If you do not have an existing Azure account, you may sign up for a free trial or use your MSDN subscriber benefits when you create an account.

• Event Hubs namespace with an Event Hub: To interact with Azure Event Hubs, you'll also need to have a namespace and Event Hub available. If you are not familiar with creating Azure resources, you may wish to follow the step-by-step guide for creating an Event Hub using the Azure portal. There, you can also find detailed instructions for using the Azure CLI, Azure PowerShell, or Azure Resource Manager (ARM) templates to create an Event Hub.

• C# 8.0: The Azure Event Hubs client library makes use of new features that were introduced in C# 8.0. You can still use the library with older versions of C#, but some of its functionality won't be available. In order to enable these features, you need to target .NET Core 3.0 or specify the language version you want to use (8.0 or above). If you are using Visual Studio, versions prior to Visual Studio 2019 are not compatible with the tools needed to build C# 8.0 projects. Visual Studio 2019, including the free Community edition, can be downloaded here.

  Important Note: The use of C# 8.0 is mandatory to run the examples and the samples without modification. You can still run the samples if you decide to tweak them.

To quickly create the needed Event Hubs resources in Azure and to receive a connection string for them, you can deploy our sample template by clicking:

If you'd like to run samples that use Azure.Identity, you'll also need a service principal with the correct roles. To make configuration for the identity samples easier, a PowerShell script script is available. Please see the Contributing Guide for more details about the script.

Install the package

Install the Azure Event Hubs client library for .NET with NuGet:

Install-Package Azure.Messaging.EventHubs -Version 5.0.0 

Obtain a connection string

For the Event Hubs client library to interact with an Event Hub, it will need to understand how to connect and authorize with it. The easiest means for doing so is to use a connection string, which is created automatically when creating an Event Hubs namespace. If you aren't familiar with shared access policies in Azure, you may wish to follow the step-by-step guide to get an Event Hubs connection string.

Key concepts

• An Event Hub client is the primary interface for developers interacting with the Event Hubs client library. There are several different Event Hub clients, each dedicated to a specific use of Event Hubs, such as publishing or consuming events.

• An Event Hub producer is a type of client that serves as a source of telemetry data, diagnostics information, usage logs, or other log data, as part of an embedded device solution, a mobile device application, a game title running on a console or other device, some client or server based business solution, or a web site.

• An Event Hub consumer is a type of client which reads information from the Event Hub and allows processing of it. Processing may involve aggregation, complex computation and filtering. Processing may also involve distribution or storage of the information in a raw or transformed fashion. Event Hub consumers are often robust and high-scale platform infrastructure parts with built-in analytics capabilities, like Azure Stream Analytics, Apache Spark, or Apache Storm.

• A partition is an ordered sequence of events that is held in an Event Hub. Partitions are a means of data organization associated with the parallelism required by event consumers. Azure Event Hubs provides message streaming through a partitioned consumer pattern in which each consumer only reads a specific subset, or partition, of the message stream. As newer events arrive, they are added to the end of this sequence. The number of partitions is specified at the time an Event Hub is created and cannot be changed.

• A consumer group is a view of an entire Event Hub. Consumer groups enable multiple consuming applications to each have a separate view of the event stream, and to read the stream independently at their own pace and from their own position. There can be at most 5 concurrent readers on a partition per consumer group; however it is recommended that there is only one active consumer for a given partition and consumer group pairing. Each active reader receives all of the events from its partition; if there are multiple readers on the same partition, then they will receive duplicate events.

For more concepts and deeper discussion, see: Event Hubs Features.

Examples

Inspect an Event Hub

Many Event Hub operations take place within the scope of a specific partition. Because partitions are owned by the Event Hub, their names are assigned at the time of creation. To understand what partitions are available, you query the Event Hub using one of the Event Hub clients.

var connectionString = "<< CONNECTION STRING FOR THE EVENT HUBS NAMESPACE >>"; var eventHubName = "<< NAME OF THE EVENT HUB >>"; await using (var client = new EventHubProducerClient(connectionString, eventHubName)) { string[] partitionIds = await client.GetPartitionIdsAsync(); } 

Publish events to an Event Hub

In order to publish events, you'll need to create an EventHubProducerClient. Producers publish events in batches and may request a specific partition, or allow the Event Hubs service to decide which partition events should be published to. It is recommended to use automatic routing when the publishing of events needs to be highly available or when event data should be distributed evenly among the partitions. Our example will take advantage of automatic routing.

var connectionString = "<< CONNECTION STRING FOR THE EVENT HUBS NAMESPACE >>"; var eventHubName = "<< NAME OF THE EVENT HUB >>"; await using (var producer = new EventHubProducerClient(connectionString, eventHubName)) { using EventDataBatch eventBatch = await producerClient.CreateBatchAsync(); eventBatch.TryAdd(new EventData(Encoding.UTF8.GetBytes("First"))); eventBatch.TryAdd(new EventData(Encoding.UTF8.GetBytes("Second"))); await producer.SendAsync(eventBatch); } 

Read events from an Event Hub

In order to read events from an Event Hub, you'll need to create an EventHubConsumerClient for a given consumer group. When an Event Hub is created, it provides a default consumer group that can be used to get started with exploring Event Hubs. In our example, we will focus on reading all events that have been published to the Event Hub using an iterator.

Note: It is important to note that this approach to consuming is intended to improve the experience of exploring the Event Hubs client library and prototyping. It is recommended that it not be used in production scenarios. For production use, we recommend using the Event Processor Client, as it provides a more robust and performant experience.

var connectionString = "<< CONNECTION STRING FOR THE EVENT HUBS NAMESPACE >>"; var eventHubName = "<< NAME OF THE EVENT HUB >>"; string consumerGroup = EventHubConsumerClient.DefaultConsumerGroupName; await using (var consumer = new EventHubConsumerClient(consumerGroup, connectionString, eventHubName)) { using var cancellationSource = new CancellationTokenSource(); cancellationSource.CancelAfter(TimeSpan.FromSeconds(45)); await foreach (PartitionEvent receivedEvent in consumer.ReadEvents(cancellationSource.Token)) { // At this point, the loop will wait for events to be available in the Event Hub. When an event // is available, the loop will iterate with the event that was received. Because we did not // specify a maximum wait time, the loop will wait forever unless cancellation is requested using // the cancellation token. } } 

Read events from an Event Hub partition

In order to read events for an Event Hub partition, you'll need to create an EventHubConsumerClient for a given consumer group. When an Event Hub is created, it provides a default consumer group that can be used to get started with exploring Event Hubs. To read from a specific partition, the consumer will also need to specify where in the event stream to begin receiving events; in our example, we will focus on reading all published events for the first partition of the Event Hub.

var connectionString = "<< CONNECTION STRING FOR THE EVENT HUBS NAMESPACE >>"; var eventHubName = "<< NAME OF THE EVENT HUB >>"; string consumerGroup = EventHubConsumerClient.DefaultConsumerGroupName; await using (var consumer = new EventHubConsumerClient(consumerGroup, connectionString, eventHubName)) { EventPosition startingPosition = EventPosition.Earliest; string partitionId = (await consumer.GetPartitionIdsAsync()).First(); using var cancellationSource = new CancellationTokenSource(); cancellationSource.CancelAfter(TimeSpan.FromSeconds(45)); await foreach (PartitionEvent receivedEvent in consumer.ReadEventsFromPartitionAsync(partitionId, startingPosition, cancellationSource.Token)) { // At this point, the loop will wait for events to be available in the partition. When an event // is available, the loop will iterate with the event that was received. Because we did not // specify a maximum wait time, the loop will wait forever unless cancellation is requested using // the cancellation token. } } 

Process events using an Event Processor client

For the majority of production scenarios, it is recommended that the Event Processor Client be used for reading and processing events. The processor is intended to provide a robust experience for processing events across all partitions of an Event Hub in a performant and fault tolerant manner while providing a means to persist its state. Event Processor clients are also capable of working cooperatively within the context of a consumer group for a given Event Hub, where they will automatically manage distribution and balancing of work as instances become available or unavailable for the group.

Since the EventProcessorClient has a dependency on Azure Storage blobs for persistence of its state, you'll need to provide a BlobContainerClient for the processor, which has been configured for the storage account and container that should be used.

var storageConnectionString = "<< CONNECTION STRING FOR THE STORAGE ACCOUNT >>"; var blobContainerName = "<< NAME OF THE BLOBS CONTAINER >>"; var eventHubsConnectionString = "<< CONNECTION STRING FOR THE EVENT HUBS NAMESPACE >>"; var eventHubName = "<< NAME OF THE EVENT HUB >>"; var consumerGroup = "<< NAME OF THE EVENT HUB CONSUMER GROUP >>"; BlobContainerClient storageClient = new BlobContainerClient(storageConnectionString, blobContainerName); EventProcessorClient processor = new EventProcessorClient ( storageClient, consumerGroup, eventHubsConnectionString, eventHubName ); 

More details can be found in the Event Processor Client README and the accompanying samples.

Using an Active Directory principal with the Event Hub clients

The Azure Identity library provides Azure Active Directory authentication support which can be used for the Azure client libraries, including Event Hubs.

To make use of an Active Directory principal, one of the available identity tokens from the Azure.Identity library is also provided when creating the Event Hub client. In addition, the fully qualified Event Hubs namespace and the name of desired Event Hub are supplied in lieu of the Event Hubs connection string.

var fullyQualifiedNamespace = "<< FULLY-QUALIFIED EVENT HUBS NAMESPACE (like something.servicebus.windows.net)>>" var eventHubName = "<< NAME OF THE EVENT HUB >>"; TokenCredential credential = new DefaultAzureIdentity(); await using (var producer = new EventHubProducerClient(fullyQualifiedNamespace, eventHubName, credential)) { // Publish events using the producer } 

When using Azure Active Directory, your principal must be assigned a role which allows access to Event Hubs, such as the Azure Event Hubs Data Owner role. For more information about using Azure Active Directory authorization with Event Hubs, please refer to the associated documentation.

Troubleshooting

Event Hubs Exception

An EventHubsException is triggered when an operation specific to Event Hubs has encountered an issue, including both errors within the service and specific to the client. The exception includes some contextual information to assist in understanding the context of the error and its relative severity. These are:

• IsTransient : This identifies whether or not the exception is considered recoverable. In the case where it was deemed transient, the appropriate retry policy has already been applied and retries were unsuccessful.

• Reason : Provides a set of well-known reasons for the failure that help to categorize and clarify the root cause. These are intended to allow for applying exception filtering and other logic where inspecting the text of an exception message wouldn't be ideal. Some key failure reasons are:

  • Client Closed : This occurs when an operation has been requested on an Event Hub client that has already been closed or disposed of. It is recommended to check the application code and ensure that objects from the Event Hubs client library are created and closed/disposed in the intended scope.

  • Service Timeout : This indicates that the Event Hubs service did not respond to an operation within the expected amount of time. This may have been caused by a transient network issue or service problem. The Event Hubs service may or may not have successfully completed the request; the status is not known. It is recommended to attempt to verify the current state and retry if necessary.

  • Quota Exceeded : This typically indicates that there are too many active read operations for a single consumer group. This limit depends on the tier of the Event Hubs namespace, and moving to a higher tier may be desired. An alternative would be to create additional consumer groups and ensure that the number of consumer client reads for any group is within the limit. Please see Azure Event Hubs quotas and limits for more information.

  • Message Size Exceeded : Event data as a maximum size allowed for both an individual event and a batch of events. This includes the data of the event, as well as any associated metadata and system overhead. The best approach for resolving this error is to reduce the number of events being sent in a batch or the size of data included in the message. Because size limits are subject to change, please refer to Azure Event Hubs quotas and limits for specifics.

  • Consumer Disconnected : A consumer client was disconnected by the Event Hub service from the Event Hub instance. This typically occurs when a consumer with a higher owner level asserts ownership over a partition and consumer group pairing.

  • Resource Not Found: An Event Hubs resource, such as an Event Hub, consumer group, or partition, could not be found by the Event Hubs service. This may indicate that it has been deleted from the service or that there is an issue with the Event Hubs service itself.

Reacting to a specific failure reason for the EventHubException can be accomplished in several ways, such as by applying an exception filter clause as part of the catch block:

try { // Read events using the consumer client } catch (EventHubsException ex) where (ex.Reason == EventHubsException.FailureReason.ConsumerDisconnected) { // Take action based on a consumer being disconnected } 

Other exceptions

For detailed information about the failures represented by the EventHubsException and other exceptions that may occur, please refer to Event Hubs messaging exceptions.

Next steps

Beyond the introductory scenarios discussed, the Azure Event Hubs client library offers support for additional scenarios to help take advantage of the full feature set of the Azure Event Hubs service. In order to help explore some of these scenarios, the Event Hubs client library offers a project of samples to serve as an illustration for common scenarios. Please see the samples README for details.

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.microsoft.com.

When you submit a pull request, a CLA-bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.

Please see our contributing guide for more information.

ncG1vNJzZmiZqqq%2Fpr%2FDpJuom6Njr627wWeaqKqVY8SqusOorqxmnprBcHDWnplonJ%2Bpu6bAjnqxrqqVY5qmv9KanqKml2OSt7HNrX%2BumqNkgm98jWlmoqaUmsVvtNOmow%3D%3D