A trace that looks complete at the API boundary can still lose the most important part of the work. The request returns, a message enters a queue, and processing continues seconds or minutes later—possibly in another service, region, or provider. If trace context is missing or modeled incorrectly, the user journey becomes a collection of unrelated fragments.
This matters anywhere asynchronous processing carries business state: payment settlement, mobile-money callbacks, identity verification, order fulfillment, notification delivery, data synchronization, or offline-first workflows. The systems may be working as designed, yet an operator cannot answer a basic question: what happened after the request left the synchronous path?
The solution is not to force every operation into one long parent-child tree. OpenTelemetry span links provide a more accurate way to represent causal relationships when work fans out, batches together, or begins in a different active context.
Parent-child relationships have a limit
In OpenTelemetry, a span has zero or one parent. That structure is ideal for a straightforward call chain: an HTTP request invokes a service, which calls a database, which returns a result.
Messaging systems are different. One message may be consumed by multiple services. One consumer operation may process a batch containing messages from multiple producers. A worker may already be handling another request when a message arrives. There is no single parent that accurately expresses all of those relationships.
A span link records a relationship to another span context without making that span the parent. The OpenTelemetry tracing API allows a span to contain zero or more links, including links to contexts from the same trace or another trace.
Propagation and links solve different problems
Context propagation carries trace identity across a boundary. The W3C Trace Context Recommendation standardizes the traceparent and tracestate fields used to exchange that context between services.
For a message, the producer should attach a creation context that the consumer can extract. Without that context, the consumer cannot reliably correlate its work with the producer.
A link then describes how the consumer span relates to that extracted context. Propagation transports the evidence; the link models the causal relationship. Using one without understanding the other often produces broken or misleading traces.
Why messaging defaults to span links
The current OpenTelemetry messaging semantic conventions use links as the default mechanism for correlating producers and consumers. The specification gives three practical reasons:
- Messaging systems have many delivery models, so a single parent-child structure cannot be guaranteed.
- A batch can contain messages created by multiple spans, while a consumer span can have only one parent.
- Message processing may occur inside another active context, such as an HTTP server span.
For a single-message scenario, the message creation context may be used as the parent of the process span. But for fan-out, batching, retries, and mixed ambient contexts, links usually preserve causality more honestly.
Model four common asynchronous patterns
1. One producer, one consumer
Inject the message creation context when publishing. At consumption, extract that context and create a process span that links to it. If your instrumentation intentionally uses the creation context as the parent for a single message, document that choice and keep it consistent.
2. One message, multiple consumers
Each consumer creates its own process span and links back to the message creation context. This prevents one consumer from appearing to be the parent of another and makes the fan-out explicit.
3. Batch consumption
A batch-processing span should link to the creation context of each message it represents. Choosing one message as the parent would hide the other causal inputs. The number of links can become large, so teams should test SDK and backend limits and consider whether a smaller span per message is operationally justified.
4. Retries and dead-letter processing
Keep the original message context available, but also capture bounded attributes and events that explain the delivery attempt, retry count, settlement outcome, and dead-letter destination. Do not place sensitive payloads, credentials, personal data, or unrestricted message bodies into span attributes.
Add links at span creation when possible
The stable OpenTelemetry tracing API recommends adding known links when the span is created rather than calling AddLink later. Head-sampling decisions can consider only information that exists when the span starts.
This becomes important when a sampler keeps traces based on linked context or attributes. A link added after span creation may still be recorded, but it cannot influence a head-sampling decision that already happened.
Instrument the queue boundary deliberately
A practical implementation review should verify all of the following:
- The producer creates or obtains a message creation context.
- The context is injected into supported message metadata rather than the business payload.
- Intermediaries preserve the context fields.
- The consumer extracts the context before creating its processing span.
- The consumer uses a link or documented parent relationship appropriate to the delivery model.
- Batch operations retain the relationship to every represented message where feasible.
- Retries, acknowledgements, negative acknowledgements, and dead-letter outcomes are observable.
Test this across the real infrastructure—not only between two services on a laptop. Brokers, gateways, serverless triggers, protocol bridges, and managed integrations may rename, filter, or omit metadata.
Measure the asynchronous journey, not just the handler
A well-linked trace should help answer:
- How long did the message wait before processing began?
- Which producer event initiated this consumer work?
- Did processing fan out to multiple consumers?
- Was the message retried, settled, or sent to a dead-letter destination?
- Did the final user or business outcome complete within its expected window?
This complements our guidance on measuring completed payment journeys. API availability is only the first step when a transaction continues asynchronously. The reliability objective should reflect the time and correctness of the full journey.
Protect privacy and control telemetry growth
Correlation does not require copying message payloads into telemetry. Propagate only the context necessary for tracing, and use controlled attributes for operational analysis. Message IDs may be useful for correlation, but they are high-cardinality and must be governed according to retention, query, and access requirements.
The same discipline described in our metric cardinality budget applies here: keep bounded dimensions in aggregate signals and use traces for request-level investigation. Avoid turning every useful identifier into a metric label.
What to do this week
- Select one customer-critical workflow that crosses a queue or event bus.
- Verify that creation context survives the real broker and consumer path.
- Compare the trace structure with the actual delivery model.
- Add span links for fan-out or batch relationships that a single parent cannot represent.
- Test sampling, retries, and dead-letter handling.
- Confirm that no sensitive payload data is being exported.
A trace is useful only when its structure tells the truth about the system. For asynchronous services, that truth is often a graph of causal relationships—not a single uninterrupted tree.
Organizations interested in supporting vendor-neutral education on distributed tracing, messaging reliability, and resilient digital infrastructure can explore sponsorship or workshop collaboration with Observability Africa. Sponsors do not control editorial conclusions.