Trace the Message, Not Just the Request: Observability Across Queues

A request that enters an API and returns a response is relatively easy to follow. A message that crosses a queue is different. The producer may finish before the consumer starts. Several consumers may process the same event. A broker may redeliver a message. A batch may contain work from several upstream requests.

If trace context stops at the queue, responders see two disconnected stories. The producer appears successful, while the consumer failure looks unrelated. During an incident, that missing relationship can turn a focused investigation into a search across logs, dashboards, and teams.

OpenTelemetry provides the building blocks to preserve the relationship without pretending that asynchronous work behaves like a normal synchronous call.

Why asynchronous traces become fragmented

In synchronous HTTP or RPC communication, the downstream operation usually begins while the upstream operation is still active. The parent and child relationship is direct.

Messaging systems introduce time and delivery boundaries. A producer creates and sends a message. A broker stores or forwards it. One or more consumers receive and process it later. Processing may happen after a retry, inside a batch, or under another active context.

The operational questions also change:

• Did the producer successfully hand the message to the broker?
• How long did the message wait before processing began?
• Which consumer group processed it?
• Was it retried or redelivered?
• Did one message create several downstream operations?
• Was the message accepted but never settled?

A useful trace must preserve enough context to answer these questions without placing sensitive message content into telemetry.

Propagate context with the message

OpenTelemetry’s messaging conventions state that the producer should attach a message creation context so consumer traces can be correlated with the producer. The exact carrier depends on the messaging system, but the principle is consistent: inject trace context into message metadata when producing, then extract it before consumer processing.

W3C Trace Context defines the standard traceparent and tracestate fields used to carry distributed tracing information between systems. OpenTelemetry propagators provide a common API for injecting and extracting this context.

Propagation should be treated as an interoperability contract. Confirm that every producer, broker integration, framework, and consumer preserves the required metadata. A single service that rebuilds a message without copying its trace metadata can break the journey.

Do not place context only in an application log field. Logs may help with manual correlation, but they do not give tracing libraries the context required to connect operations automatically.

Model the messaging operations clearly

OpenTelemetry separates messaging work into operations such as create, send, receive, process, and settle.

This distinction matters. A send span describes the producer handing one or more messages to an intermediary. A process span describes the consumer acting on the message. A settle span can describe acknowledgement or rejection when settlement is a separate operation.

Use stable, low-cardinality destination information. A real queue or topic name may be useful. A destination containing a customer ID or another unbounded value should be represented with a stable template where supported. This keeps traces useful without creating uncontrolled dimensions in metrics derived from span data. See also how to set a metric cardinality budget.

Record errors using predictable classifications. Avoid turning full exception messages, message bodies, or unique identifiers into attributes used for aggregation.

Use span links when parentage is not honest

A single parent is not always enough for messaging.

A batch consumer may process messages created by many different traces. One message may be delivered to several consumers. Processing may begin inside an HTTP request or another active operation. Forcing one creation context to become the parent can hide other valid relationships.

OpenTelemetry recommends span links as the default correlation mechanism for many producer and consumer scenarios. A process span can link to the creation context of each message it handles. This preserves the relationship without inventing a false tree.

For a single-message workflow, instrumentation may use the creation context as the parent of the process span. Teams should understand what their library does and verify that the resulting trace structure matches the actual workflow.

Measure the waiting time, not only the processing time

A consumer can process messages quickly while users still wait because the queue is growing.

Trace data should be combined with messaging metrics that show:

• queue depth or backlog
• message age
• publish and consumption rate
• processing duration
• retry and redelivery count
• rejected or dead-lettered messages
• consumer lag where the platform exposes it
• settlement failures

The most important signal depends on the system. For a user-facing transaction, the age of the oldest pending message may reveal more than average handler duration. For a high-volume stream, lag by consumer group may be more useful.

Choose alerts around user or business consequences. A short backlog during a predictable peak may be normal. A message that exceeds the completion objective for a critical workflow may require action even when infrastructure remains healthy. Collector resilience also matters when the destination disappears, as explained in our guide to designing an OpenTelemetry Collector for outages.

Plan for retries and duplicate processing

Message delivery is often at least once. A consumer may finish its work but fail before acknowledgement, causing the broker to deliver the message again.

Tracing should make the retry visible without suggesting that every repeated delivery is a new user action. Preserve the message’s creation relationship, record the delivery or retry information exposed by the client, and use application-level idempotency where the workflow requires it.

Do not rely on a trace ID as an idempotency key. Trace context supports observation, not business correctness. The application should use a stable domain identifier designed for duplicate detection.

Keep sensitive data out of propagation

Trace context is metadata, not a place for customer details, credentials, or message content.

Validate incoming trace headers at trust boundaries. The W3C Trace Context recommendation notes that blindly accepting external sampling decisions can create security and cost risks. Apply normal authentication and authorization independently of tracing, and configure sampling controls appropriate to the environment.

Review message attributes before recording them. Conversation IDs and message IDs can help diagnosis, but their collection and retention should follow privacy, security, and cardinality rules. Our guide to redacting sensitive telemetry data provides a practical starting point.

A practical validation exercise

Choose one asynchronous workflow, such as payment confirmation, account activation, document processing, or notification delivery.

Run a controlled test and verify:

1. The producer span records the correct messaging operation and destination.
2. Trace context is injected into the message metadata.
3. The consumer extracts the context before processing.
4. The consumer span links to, or appropriately follows, the message creation context.
5. Queue waiting time can be distinguished from processing time.
6. A retry or redelivery is visible without creating misleading relationships.
7. No sensitive message content appears in span attributes.
8. The trace connects to the service metrics and logs responders use during incidents.

Then interrupt the consumer, allow a small backlog to form, restore processing, and confirm that telemetry still explains the complete journey.

Practical takeaways

• Carry trace context in message metadata, not only in logs.
• Treat propagation as a contract across every producer and consumer.
• Model create, send, receive, process, and settle operations accurately.
• Use span links when a simple parent-child tree would misrepresent the workflow.
• Monitor message age, lag, retries, and dead letters alongside processing time.
• Keep trace context separate from business idempotency.
• Minimize sensitive and high-cardinality message attributes.
• Test the trace during backlog, retry, and redelivery conditions.

Organizations that want an independent review of asynchronous workflows, telemetry coverage, and failure visibility can explore the Observability Africa audit and consulting service. Organizations may also support independent community education while Observability Africa retains editorial control.

Sources

• OpenTelemetry semantic conventions for messaging spans
• OpenTelemetry instrumentation libraries and propagators
• W3C Trace Context Recommendation

About Abe Apithy

Abe Apithy is the founder of Observability Africa, a platform dedicated to helping telecom, fintech, and energy organizations design and scale resilient, high-performance digital infrastructure. His work focuses on enabling real-time system visibility, operational reliability, and performance optimization in environments where downtime, latency, and inefficiency…