Asynchronous Messaging & Eventing
Queues and events let parts of the system work independently and handle load. But they have rules that often catch out people new to them. Messages can arrive more than once, out of order, or fail again and again. Your handlers must be ready for all three. Design for at-least-once delivery and idempotency from the start.
Message brokers (for example, Azure Service Bus) separate producers from consumers. A producer sends a message, and a consumer processes it later. This gives resilience and scalability. But the contract is usually at-least-once delivery: the same message can be delivered twice (after a retry or crash), and order is not guaranteed unless you arrange it. So handlers must be idempotent: processing the same message twice must be safe.
Two other things matter. First, handle poison messages (ones that keep failing) with retries and a dead-letter queue, so they do not block the whole queue. Second, make sure a message and the database change that goes with it cannot get out of sync. The outbox pattern solves this.
Design handlers for reality
- AlwaysMake message handlers idempotent. Processing the same message twice gives the same result, not a duplicate effect. Use a dedupe or idempotency key (see Data Integrity & Transactions).
- DoAssume at-least-once delivery and possible reordering. Do not rely on messages arriving exactly once or in strict order unless the broker guarantees it (sessions or partitions).
- DoHandle failures with bounded retries plus a dead-letter queue, so one poison message does not block the queue or get re-processed forever.
- DoKeep messages small and versioned. Carry an id or reference, not large payloads. Tolerate unknown fields so producers and consumers can evolve (see Backward Compatibility).
- ConsiderAdding correlation ids to messages so a flow can be traced across services (see Observability & Logging Hygiene).
void Handle(PaymentRequested m) {
chargeCard(m.OrderId); // redelivery => charged twice
}
If the broker redelivers this message (which it can), the customer is charged again. At-least-once delivery makes this a real, frequent bug, not a rare one.
void Handle(PaymentRequested m) {
if (alreadyProcessed(m.MessageId)) return; // dedupe
chargeCard(m.OrderId);
markProcessed(m.MessageId);
}
A duplicate delivery is detected and ignored, so the charge happens exactly once even though the message may arrive twice.
Keep data and messages consistent
- DoUse the outbox pattern when a database change and a published message must both happen. Write the message to an outbox table in the same transaction, then publish it separately.
- DoAcknowledge or complete a message only after it is processed successfully. Then a crash during processing redelivers the message instead of losing it.
- AvoidPublishing a message and committing a database change as two separate steps. A crash between them leaves the system inconsistent (a message without data, or data without a message).
- AvoidPutting secrets or full sensitive payloads on the bus. Send a reference instead, and protect the broker as a trust boundary.
- NeverProcess a non-idempotent message that changes money or state without a guard against duplicate delivery.
Self-review checklist
- AskIf this message is delivered twice, is the result still correct?
- AskWhat happens to a message that keeps failing — bounded retry then dead-letter, or does it block the queue?
- AskIf a database change must go with a published message, can they get out of sync?
- AskAm I assuming ordering or exactly-once delivery that the broker does not actually guarantee?