Correlation IDs done right: tracing a request across every MuleSoft API

When a request fails somewhere across three or four MuleSoft APIs, the only thing that makes the logs usable is a single identifier that appears on every line the request touched. Without it you are grepping by timestamp and guessing which System API log entry belongs to which Experience API call. With it, you paste one value into your log search and the entire chain - Experience to Process to System and back - lines up in order.

That identifier is the correlation ID. Mule has first-class support for it, and the discipline around it is small: read an incoming ID, generate one only when none exists, and propagate the same value on every outbound hop. This guide teaches that honor-or-generate rule, shows how MDC logging puts the ID on every line for free, and clears up where a correlation ID sits relative to OpenTelemetry trace and span IDs.

Why one request needs one ID that survives every hop

A typical request in an API-led system is not one call. A mobile app hits an Experience API, which calls a Process API to orchestrate, which calls two or three System APIs to read and write records. That is one logical request and five or six Mule flows, often running on different workers, writing to different log files.

If each flow logs with its own internal event id, a production incident becomes an archaeology project. You know the customer saw an error at 14:32, but you cannot tell which System API call failed without manually correlating timestamps across logs - and timestamps lie when calls run in parallel or queues add delay.

A correlation ID solves this by being stable across the whole chain. It is assigned once, at the edge, and carried unchanged through every downstream call and every log line. One search returns the complete, ordered story of that request. This is the single most valuable logging practice in a distributed integration, and it costs almost nothing once the rule is in place. You can see the same idea applied to the demo questions at /demo if you want to test your understanding interactively.

Honor-or-generate: read it, create it only when absent, always propagate

The rule has three parts, and getting all three right is what separates a traceable system from a noisy one.

Honor an incoming ID. When a request arrives with an x-correlation-id header, the HTTP Listener does not invent a new value - it adopts that header as the event's correlation ID. This is the default Mule behavior, and it is what lets a caller (or an upstream API) set the ID and have it respected. The same applies to messaging sources: a JMS or Anypoint MQ message that carries a correlation ID seeds the Mule event with it.

Generate only when absent. If no correlation ID arrives, Mule's runtime generates one automatically when it creates the event. You do not need a flow component to mint an ID for the happy path - every Mule event already has a correlationId. You only write generation logic when you want a specific format (for example, prefixing with a system name) via the correlationIdGeneratorExpression on the configuration, or when a non-Mule entry point hands you nothing usable.

Always propagate. This is the part teams get wrong. When a flow calls a downstream HTTP service, you want the same ID to travel as x-correlation-id. The HTTP Request operation controls this with the sendCorrelationId attribute:

sendCorrelationId	Behavior
AUTO (default)	Sends the current event's correlation ID as x-correlation-id unless overridden by app config
ALWAYS	Always sends the current correlation ID on the outbound request
NEVER	Suppresses the header - the downstream API will generate its own ID and the chain breaks

The trap is NEVER, or manually overwriting the header with a fresh value. Either one starts a new trace at that hop, and your single-search guarantee is gone. The correct default is to let AUTO/ALWAYS carry the inherited ID forward untouched.

MDC logging so every line carries the ID without string-building

The naive way to get the correlation ID into logs is to concatenate it into every Logger message: "orderId=... correlationId=" ++ correlationId. That is repetitive, easy to forget, and pollutes the message text. Mule does it properly with the MDC (Mapped Diagnostic Context), which in Log4j 2 is the per-thread ThreadContext map.

Mule automatically places the event's correlation ID into the MDC under the key correlationId (and the processor location under processorPath). That means you configure the ID into your log pattern once, in log4j2.xml, and every single log line - from the runtime, from connectors, from your own Loggers - carries it without you touching the message.

The default Mule app pattern already includes it:

<PatternLayout pattern="%-5p %d [%t] [processor: %X{processorPath}; event: %X{correlationId}] %c: %m%n"/>

The %X{correlationId} token pulls the value straight from the MDC. If you want a cleaner, JSON-friendly setup, switch to %MDC to emit the whole context map, or template %X{correlationId} into a JSON layout so your log aggregator can index it as a real field. The key insight: logging the correlation ID is a configuration concern, not a coding concern. You should not see correlationId concatenated into Logger messages anywhere in a well-built app.

If you need to add your own business keys (an order id, a tenant id) to every line the same way, the Tracing module lets you set custom logging variables that land in the MDC alongside correlationId, so they appear on every subsequent log line without manual string-building.

Tracing a request across System, Process, and Experience APIs

Put the rule together across the api-led layers and the payoff is concrete.

A request enters the Experience API. If the client sent x-correlation-id, that value is honored; if not, the runtime generates one. From here on, nothing regenerates it:

1. The Experience API logs with the ID in the MDC and calls the Process API over HTTP with sendCorrelationId left at AUTO, so x-correlation-id carries the same value.
2. The Process API's HTTP Listener honors that incoming header as its event's correlation ID. Its own log lines now show the same ID, and it fans out to the System APIs - again propagating the header unchanged.
3. Each System API honors the header, logs with it, and returns. The response path adds nothing new; the ID was assigned once and only read thereafter.

The result: one value, set at the edge, appears on every log line in every app for that request. During an incident you grab the ID from the failing response (surface it in an error response header or body so support can read it), search all log streams for it, and read the chain top to bottom. This is the difference between a five-minute diagnosis and a five-hour one.

It is worth being explicit that this only works if every hop is HTTP-or-messaging that Mule's correlation handling understands. A hop through a system that strips headers, or a custom HTTP call that hardcodes its own ID, is a break point - find those and fix them first.

Correlation ID vs trace and span IDs (OpenTelemetry)

A correlation ID is not the same thing as an OpenTelemetry trace ID and span ID, and conflating them causes confusion when teams adopt distributed tracing tooling.

Concept	What it identifies	Lifetime
Correlation ID	One logical business request	Constant across the whole request, every hop
Trace ID	One distributed trace (the whole operation) in OpenTelemetry	Constant across the whole trace
Span ID	One unit of work within the trace (a single call/operation)	New for every span - each hop gets its own

The mental model: a trace ID is conceptually close to a correlation ID - both are one-per-request and constant. A span ID is the opposite of a correlation ID in lifetime - it is meant to change at every hop, because each span is a distinct segment of the trace with its own parent-child relationship.

In practice they are complementary, not competing. The correlation ID is your human-readable, log-grep handle that you can put in error messages and support tickets. The trace/span IDs power a tracing backend (Jaeger, Zipkin, an APM) that visualizes timing and the call tree. MuleSoft can export OpenTelemetry traces, and a common pattern is to carry the correlation ID as an attribute on the trace so a support engineer with a correlation ID can jump straight to the matching trace. You want both: the correlation ID for logs and tickets, the trace and span IDs for the waterfall view.

Pitfalls: regenerating per hop and losing the ID across queues

Two failure modes account for nearly every broken trace.

Regenerating per hop. The most common mistake is creating a new ID inside a downstream flow - either by setting sendCorrelationId="NEVER" (so the listener generates a fresh one), by overwriting the x-correlation-id header with a new value before the HTTP Request, or by using the Tracing module's with-correlation-id scope without understanding it replaces the ID for everything inside it. Each of these silently starts a new trace. The symptom is logs that look fine in isolation but cannot be stitched together. The rule is blunt: generate at most once, at the edge, and never again.

Losing the ID across queues. Synchronous HTTP propagation is well-trodden, but asynchronous boundaries are where IDs quietly vanish. When you publish to an Anypoint MQ queue or use a VM/async scope, the consuming flow runs in a different event context, and the correlation ID does not automatically ride along unless you carry it explicitly. The fix is to put the correlation ID on the message when you publish - Anypoint MQ lets you set user properties, so write the current correlationId as a property (or set the message's correlation id). On the consumer side, read that property back and restore it (the Tracing module's with-correlation-id scope is the right tool here, scoping the inherited ID over the processing). Skip this and every asynchronous job becomes an untraceable island.

The same caution applies to batch jobs and scheduled flows: a scheduler-triggered flow has no inbound request, so it generates a fresh ID per run. That is correct - each run is its own logical operation - but if that run publishes work that should correlate back to an original request, you must thread the original ID through yourself.

Get these two pitfalls right and the rest of the model holds. One ID, honored or generated once at the edge, propagated on every hop, sitting in the MDC so it lands on every log line. That is the whole discipline, and it turns a distributed system from opaque into searchable.