Circuit breaker vs retry: when to retry, when to stop trying

Retry and circuit breaker are the two resilience patterns engineers reach for when a call to a downstream system fails, and they are constantly confused. The short version: retry is for a brief, self-correcting hiccup; a circuit breaker is for a sustained outage. Use the wrong one and you do not just fail to help, you actively make the incident worse by hammering a service that is already on its knees.

This guide builds both patterns from first principles, shows why retry-without-a-breaker can turn your own clients into a denial-of-service attack on your dependency, and gives you a clear decision frame for combining them. The Mule anchors are real: Until Successful and reconnection strategies ship in the runtime, while the circuit breaker is an Anypoint API Gateway policy or a pattern you build yourself.

Transient hiccup vs sustained outage: two different failures

Before picking a pattern, classify the failure. There are two fundamentally different kinds, and the whole decision hinges on which one you are facing.

A transient fault is brief and self-correcting. The dependency is fundamentally healthy, but this one request lost. Examples: a connection reset, a single dropped packet, a brief lock contention in a database, a load balancer that was mid-failover for 200ms, or an HTTP 503 returned during a rolling deploy. The defining property is that the next attempt has a genuinely good chance of succeeding because nothing is fundamentally broken. Retrying is the right move.

A sustained outage is the opposite. The dependency is down, overloaded, or returning errors for everything, and it will stay that way for seconds, minutes, or longer. A database has run out of connections, a downstream service has crashed, a network partition has cut you off, or a dependency is melting under load. Here, the next attempt has roughly the same near-zero chance of succeeding as the last one - and worse, each attempt adds load to a system that is already failing. Retrying is exactly the wrong move.

The trap is that both failures look identical at the moment they happen: you get an exception or an error status code. You cannot tell them apart from a single response. Retry assumes the transient case; a circuit breaker is what notices when you have crossed into the sustained case and changes strategy.

The retry pattern: backoff, jitter, and bounded attempts

Retry is simple to state - if a call fails, try it again - but doing it correctly has three non-negotiable parts.

Bounded attempts

You must cap the number of retries. Infinite retry is never correct for a synchronous request: it ties up a thread, holds the caller waiting, and guarantees that a real outage becomes a thread-pool exhaustion incident. Pick a small bound - often 3 to 5 attempts - and define what happens when you exhaust it (return an error, fall back to a default, or route the message to a dead-letter queue for later).

Backoff

Retrying immediately is counterproductive. If a service is briefly overloaded, an instant retry just piles on more load at the worst moment. Exponential backoff spaces attempts out by growing the delay each time - for example 200ms, then 400ms, then 800ms, then 1600ms. This gives a struggling dependency room to recover between your attempts instead of being hit again the instant it stumbles.

Jitter

Backoff alone has a hidden failure mode. If 500 clients all hit the same outage at the same instant and all use the same backoff schedule, they retry in synchronised waves - 500 simultaneous requests at 200ms, then 500 more at 400ms. These "thundering herd" spikes can keep a recovering service down. Jitter adds randomness to each delay (say, the computed backoff plus or minus a random fraction) so attempts spread out smoothly instead of arriving in lockstep. Backoff plus jitter is the standard, battle-tested combination.

What retry only handles transient faults

Notice what retry does and does not solve. It handles the transient case beautifully and does nothing useful for a sustained outage except waste time and add load. Retry has no concept of "this dependency is down, stop trying." That gap is exactly what a circuit breaker fills.

In Mule, the retry primitive is the Until Successful scope, which retries the processors inside it up to maxRetries times, waiting at least millisBetweenRetries between attempts (default 60000). When it gives up it raises a MULE:RETRY_EXHAUSTED error you can catch. Note one honest limitation: the native scope spaces attempts at a roughly fixed minimum interval, not true exponential backoff with jitter - if you need that curve, you compute the delay yourself. Separately, reconnection strategies (reconnect with frequency and count, or reconnect-forever) handle re-establishing a connector's connection, which is a different layer from retrying a message.

The circuit breaker: closed, open, and half-open states

A circuit breaker is a small state machine that sits in front of a dependency and tracks its health. Named after the electrical device that trips to protect a circuit, it has exactly three states.

Closed is the normal, healthy state. Requests flow straight through to the dependency. The breaker counts failures as they happen. As long as the failure rate (or consecutive failure count) stays below a configured threshold, it stays closed and does nothing visible.

Open is the tripped state. Once failures cross the threshold, the breaker trips open and stops sending requests through entirely. Every call fails fast - it returns an error (or a fallback) immediately, without touching the dependency. This is the whole point: a downstream that is already overwhelmed gets a complete break from your traffic, and your callers get an instant answer instead of waiting on a doomed call to time out.

Half-open is the recovery probe. After a configured cool-down timeout, the breaker moves from open to half-open and lets a single trial request (or a small number) through. If that probe succeeds, the dependency looks healthy again, so the breaker resets to closed and normal traffic resumes. If the probe fails, the dependency is still down, so the breaker snaps back to open and waits out another cool-down before probing again.

State	What happens to requests	How it exits
Closed	Pass through; failures counted	Failure threshold crossed -> Open
Open	Fail fast immediately, no call made	Cool-down elapses -> Half-open
Half-open	One trial request allowed	Probe succeeds -> Closed; probe fails -> Open

The half-open state is what makes a breaker self-healing: it recovers automatically without a human, but it spends just one request to check instead of flooding a fragile service the moment the timer expires.

In Mule, there is no native circuit-breaker scope in the core runtime the way Until Successful exists for retry. The breaker is available as an Anypoint API Gateway policy (configurable as a global or private policy, with the same three states), or you implement the pattern yourself - holding the failure count and trip timestamp in an Object Store so the state survives across requests. Treat the breaker as a pattern you apply, not a drop-in component.

Why retry without a breaker becomes a self-inflicted DoS

This is the failure mode that turns a small outage into a major incident, and it is worth understanding precisely.

Imagine a service A that calls a downstream B, with retry configured (3 attempts) but no circuit breaker. Under normal load, A handles 1,000 requests per second, each making one call to B. Now B goes down.

Every one of those 1,000 requests per second now fails - and retries. With 3 attempts each, A is suddenly generating 3,000 requests per second against a dependency that is already down. You have tripled the load on B at the exact moment it can least handle it. If B was struggling rather than fully dead, your retries guarantee it stays down. If B shares infrastructure with other services, your retry storm can take them down too.

It gets worse upstream. Each request in A is now taking three times as long (waiting through three failed attempts and their backoff delays) before it gives up. Those requests hold threads and connections the whole time. A's thread pool fills with requests stuck retrying a dead dependency, and A stops being able to serve requests that have nothing to do with B. The outage in B has now spread into A - a cascading failure - purely because of well-intentioned retries.

This is the self-inflicted DoS: your own retry logic, multiplied across every client, becomes a denial-of-service attack on a dependency that is already down, and the back-pressure cascades up into your own service. A circuit breaker stops it cold: once B's failures trip the breaker open, A stops calling B entirely, fails those requests fast, frees its threads immediately, and gives B the breathing room to recover.

Combining retry and circuit breaker safely

Retry and circuit breaker are not alternatives - they are complementary, and production systems use both. They handle different failure durations, so the right design layers them. The mental model: retry handles the individual transient blip; the breaker handles the question of whether the dependency is healthy enough to be worth calling at all.

The standard composition is circuit breaker on the outside, retry on the inside:

1. A request arrives and checks the breaker first. If it is open, fail fast immediately - do not retry, do not call the dependency.
2. If the breaker is closed (or half-open and this is the probe), proceed to the call.
3. The call goes through the retry logic, which handles transient faults with bounded attempts, backoff, and jitter.
4. The outcome feeds the breaker: a final success keeps the failure count low; exhausting all retries counts as a failure toward the trip threshold.

This ordering is what keeps retry safe. The breaker gates whether retrying is even allowed, so during a real outage the breaker is open and no retries happen at all - which is precisely what prevents the self-inflicted DoS from the previous section. Retry still does its job for genuine transient faults while the dependency is healthy, but it can never amplify a sustained outage, because the breaker has already cut it off.

Get the order wrong - retry on the outside, breaker on the inside - and you reintroduce the problem: you would retry the breaker's fail-fast errors, which is pointless churn, and you lose the protection the layering was meant to provide.

Fail fast: when stopping is the correct behaviour

The hardest instinct to override is that stopping is sometimes the most helpful thing you can do. Engineers are trained to make things work, so failing fast - returning an error in milliseconds without even attempting the call - feels like giving up. It is the opposite.

Failing fast protects three different parties at once. It protects the downstream, by removing load so it can recover. It protects your own service, by freeing threads and connections instead of letting them pile up against a dead dependency. And it protects the caller, by returning an immediate, honest error instead of making them wait through a long timeout for a failure that was inevitable. A 5ms "circuit open" response is far kinder to a user than a 30-second hang that fails anyway.

The decision frame, in one line: retry when the next attempt has a real chance of working; fail fast when it does not. Transient faults clear, so retrying them helps. Sustained outages do not clear on the timescale of a retry, so retrying them only adds harm. The circuit breaker is the machinery that detects the switch from one regime to the other and changes behaviour automatically.

If you want to internalise where these patterns sit in a real Mule integration, the free 10-question demo walks through resilience and error-handling scenarios in the same style. The takeaway for both the exam and the 3 a.m. incident is the same: know which failure you are facing, retry the transient ones with backoff and jitter, and let a breaker stop you from trying when stopping is the only thing that helps.