The idempotent consumer pattern: designing for at-least-once delivery

Every durable messaging system you will integrate with - JMS, Kafka, SQS, Anypoint MQ - hands you the same uncomfortable contract: a message may arrive more than once. Not as a rare fault, but as a guaranteed, designed-in behaviour. If your consumer charges a card, creates an order, or sends an email on each message, redelivery means doing that twice. The fix is not to wish the broker were more reliable. It is to make your consumer idempotent: processing the same event twice produces the same effect as processing it once.

This guide builds that idea from first principles. The core move is dedup-by-event-id: give every event a stable identifier, record the ids you have already processed in a store, and short-circuit anything you have seen before. We will keep it vendor-neutral, then ground it in the concrete MuleSoft tools - the Idempotent Message Validator and Object Store - so you can both reason about the pattern and implement it.

Why "exactly-once" delivery is mostly a myth

The phrase "exactly-once delivery" sounds like the obvious thing to want, and brokers love to put it on a slide. The trouble is that it cannot be guaranteed at the delivery layer, for a simple reason rooted in distributed systems: the two generals problem. A consumer can process a message and then crash before its acknowledgement reaches the broker. The broker, having heard nothing, must either redeliver (risking a duplicate) or drop the message (risking a loss). There is no third option that survives arbitrary network and process failures.

So a broker can offer exactly two honest delivery semantics:

• At-most-once: deliver, do not redeliver on doubt. You never see duplicates, but you can lose messages.
• At-least-once: redeliver until acknowledged. You never lose messages, but you can see duplicates.

The crucial distinction the marketing blurs is delivery versus effect. What you actually care about is exactly-once effect - the card is charged once, the order exists once. And exactly-once effect is achievable, but not by the broker alone. It is the sum of two parts:

exactly-once effect = at-least-once delivery + idempotent processing

When a system advertises "exactly-once" - Kafka's EOS is the famous example - it is delivering at-least-once under the hood and adding deduplication so the result is exactly-once. The consumer-side dedup is not optional plumbing you can skip because the broker is good. It is the exactly-once guarantee. Once you internalise that, the rest of this pattern is just engineering.

At-least-once delivery: duplicates are guaranteed, not rare

If you assume at-least-once is the contract (and you almost always should), it helps to know why the duplicate shows up, because the causes are mundane and frequent:

• Lost acknowledgement. The consumer finished, but the ack was lost or arrived after the broker's visibility timeout. The broker assumes failure and redelivers.
• Consumer crash mid-flight. The process died after the side effect but before the ack. On restart, the message is still "in flight" and comes back.
• Rebalance or failover. A partition or queue is reassigned to another consumer while a message was being handled; the new owner re-reads it.
• Producer retries. The producer itself sent the message twice because its ack was lost - so the duplicate exists before any consumer touches it.

Notice the last cause: even a flawless consumer cannot prevent a duplicate that the producer created. This is why dedup has to live at the consumer boundary, keyed on something stable, rather than relying on "I'll just make sure I ack reliably." You cannot reliably ack your way out of a problem that can originate upstream of you. Treat every consumer as if the next message might be one it has already seen, because eventually it will be.

Dedup-by-event-id: the idempotent consumer pattern

The idempotent consumer pattern is small and durable. For each incoming event:

1. Extract a stable event id that is identical across redeliveries of the same logical event.
2. Check a dedup store: have I already recorded this id?
3. If new, do the work, then record the id (atomically with the work where possible).
4. If seen, skip the side effect and acknowledge - redelivery becomes a no-op.

on event(e):
  if store.contains(e.id):        # already processed
    ack()                         # redelivery is safe; do nothing
    return
  process(e)                      # the real side effect
  store.put(e.id)                 # remember it
  ack()

The subtlety lives in step 3. If you do the work and then record the id, a crash in between leaves the id unrecorded and the next redelivery reprocesses. If you record the id then do the work, a crash leaves the id recorded but the work undone, and redelivery skips it - a lost effect. The robust answer is to make the side effect and the id-record part of the same transaction when your sink supports it (for example, an upsert into a database table that also holds the processed id). When the sink cannot share a transaction - a third-party API call - you accept a tiny window and lean on the side effect itself being idempotent, which the next two sections address.

There is a second flavour worth naming. Sometimes you do not need a separate dedup store because the operation is naturally idempotent: a PUT that sets a record to a fixed state, a "set status = SHIPPED", an upsert keyed on a business id. Running it twice changes nothing the second time. When you can shape the operation that way, prefer it - it is simpler than maintaining a store. The explicit dedup store is for when the operation is not naturally idempotent, like "append a row" or "charge a card."

Choosing the dedup key and the store that holds it

The pattern is only as good as the key. A bad key either lets duplicates through or rejects legitimately distinct events.

The key must be stable and unique per logical event. Good choices, in rough order of preference:

Key source	When to use it	Risk
Producer-assigned event id	The producer stamps a UUID at creation and reuses it on retries	Best option; depends on a disciplined producer
Business idempotency key	Caller supplies an Idempotency-Key header for a request	Caller must generate and reuse it correctly
Broker message id	A JMSMessageID or Kafka offset is to hand	May change across redelivery or producer retries - verify before trusting
Content hash (SHA-256 of payload)	No id exists; payload is deterministic	Two genuinely distinct events with identical bodies collide as "duplicates"

A content hash is the fallback of last resort: it is convenient but it cannot distinguish "the same event redelivered" from "two real events that happen to look identical." Reach for a producer-assigned id whenever you can influence the producer.

The store has hard requirements too. It must be:

• Persistent, so processed ids survive a restart - an in-memory map forgets everything on crash, exactly when you need it most.
• Shared across every worker and cluster node, so a redelivery routed to a different instance still sees the id. A per-node store silently fails to dedupe under horizontal scaling.
• Time-bounded with a TTL, so it does not grow without limit. Set the TTL to at least the broker's maximum redelivery window - if a broker can redeliver up to 7 days later, a 1-hour TTL lets that duplicate through.

In MuleSoft, the Idempotent Message Validator implements exactly this check, backed by an Object Store. Its default idExpression is #[correlationId], and on a repeat it raises MULE:DUPLICATE_MESSAGE, which you handle to stop the flow cleanly. One sharp edge to know: the validator's default Object Store is in-memory, non-persistent, per-node, with a 5-minute entry TTL. That default silently violates two of the three requirements above. For real deduplication, point it at a persistent, shared Object Store (Object Store v2 on CloudHub is shared across workers) and set entryTtl / entryTtlUnit to cover your redelivery window. The component is correct; the default config is for demos.

Making POST-as-create safe under redelivery

HTTP gives you a useful lens. PUT and DELETE are idempotent by definition - "set this resource to X" or "ensure it is gone" yields the same state no matter how many times you call it. POST is not: "create a new order" run twice creates two orders. That is precisely the operation redelivery turns toxic, so it needs the idempotency-key treatment.

The standard design, popularised by Stripe and now common across payment and commerce APIs:

1. The caller generates an idempotency key (a UUID) and sends it, typically as an Idempotency-Key header, reusing the same key on any retry of the same intent.
2. On the first request, the server creates the resource, stores the response against that key, and returns 201 Created.
3. On a replay with the same key and the same payload, the server does not create a second resource. It returns the stored original response - the same 201 and body - so the retry is transparent and safe.
4. On the same key with a different payload, the server returns 409 Conflict - the key is being reused to mean something else, which is a client bug worth surfacing.

The behaviour that trips people up: a successful replay returns the original success response, not an error. The client cannot tell whether it hit the real creation or a replay, and that is the point. 409 is reserved strictly for the contradictory case of same-key-different-body. Internally this is the same dedup-by-id store from the previous section, except the stored value is the full response, not just a marker, so you can replay the exact original answer.

How the same pattern maps to Kafka, SQS, and Anypoint MQ

The vocabulary differs per broker, but the shape is identical: at-least-once delivery at your application boundary, plus consumer-side dedup for exactly-once effect.

Broker	Default semantics	Built-in dedup	Where you still dedupe
Kafka	At-least-once	EOS (idempotent producer + transactions) is scoped within Kafka	Any write to an external DB or API is at-least-once and must dedupe
SQS Standard	At-least-once	None	Always, in the consumer
SQS FIFO	Exactly-once processing	MessageDeduplicationId, within a 5-minute window	Beyond the 5-minute window, or for non-FIFO sinks
Anypoint MQ	At-least-once	None; redelivery on missed ack	In the consumer, e.g. Idempotent Message Validator + Object Store

The honest reading of this table: every one of these delivers at-least-once at the boundary where your code does real work. Kafka's exactly-once is genuine but scoped to consume-transform-produce inside Kafka; the moment your consumer writes to an external system, you are back to at-least-once and owe a dedup step. SQS FIFO's guarantee is real but time-bounded to a 5-minute window - a redelivery outside it can duplicate. Anypoint MQ makes no exactly-once claim at all and puts dedup squarely on the consumer, which is where the MuleSoft validator earns its keep.

So the portable mental model is one rule, not four: assume at-least-once, give every event a stable id, and dedupe at the consumer. Whether the store is an Object Store, a database table, or a Redis set is an implementation detail. The pattern survives the broker swap. If you want to drill the surrounding concepts - delivery semantics, retries, correlation ids - the free demo covers them with practice questions that reward understanding the model over memorising vendor trivia.