Will Your Telemetry Survive an Outage? Test Your OpenTelemetry Collector

A dashboard is only as trustworthy as the telemetry that reaches it. During a backend outage, a quiet chart can mean that the application is healthy—or that the evidence has stopped arriving. The operational question is not simply whether the OpenTelemetry Collector is running. It is how much data the pipeline can retain, for how long, and how quickly it can recover.

My recommendation is to treat telemetry delivery as a small reliability service with an explicit failure budget. Define what you can afford to lose, test the boundary, and make that boundary visible to the people responding to incidents. This article provides a planning exercise and a non-production test checklist, not a claim that any particular configuration guarantees delivery.

Separate buffering, retries, and durability

These mechanisms solve different problems. An exporter sending queue buffers work awaiting export. Retry handling attempts delivery again after eligible failures. Persistent storage can preserve queued data across a Collector restart. None of these removes every loss scenario: capacity exhaustion, failed storage, permanent export errors, and exhausted retries still require consideration. The official Collector resiliency guide explains those boundaries.

A useful review starts with three questions. If the backend disappears, where does the data wait? If the Collector restarts, does that waiting data survive? When the backend returns, is there enough spare throughput to clear the backlog while new data continues arriving? An answer to only the first question is not a recovery plan.

Size an outage window—not an arbitrary queue

Start by measuring arrivals at the specific exporter queue you intend to protect. Use the same units for arrival rate and queue capacity. Current exporter-helper documentation supports queue sizing in requests, items, or serialized bytes; check what your installed exporter and version actually support. A count of requests is not automatically a count of spans, and serialized bytes are not a measurement of total disk or memory consumption. See the exporter-helper configuration reference.

Here is a deliberately hypothetical planning example. Assume a queue receives a steady 20 requests per second, the downstream backend accepts nothing for ten minutes, and no queued requests expire or are discarded during that interval. The resulting backlog is 20 × 600 = 12,000 requests. Adding an illustrative 50% planning margin gives 18,000 requests. That margin is an assumption, not an OpenTelemetry recommendation.

This estimate is only a starting point. Repeat the calculation with observed burst traffic and request-size distributions. Include any backlog already present at the start of the outage. Measure actual storage growth with representative data before selecting a disk size. Avoid using an average workload as evidence that the system can withstand its busiest period.

Budget recovery throughput as carefully as storage

Continue the hypothetical example. After recovery, suppose the backend sustainably accepts 30 requests per second while 20 new requests per second keep arriving. Only the difference—10 requests per second—is available to clear the backlog. Draining 12,000 requests therefore takes about 1,200 seconds, or twenty minutes, under these simplified conditions.

If the sustainable export rate only equals incoming traffic, the backlog does not shrink. If it is lower, the backlog continues growing. This arithmetic is why a ten-minute outage can create a much longer period of delayed evidence. The useful target is a tested recovery window, not merely a large enough buffer.

Ask the backend owner about ingestion limits and test the recovery phase under realistic load. Increasing concurrency is not a substitute for downstream capacity. In the test report, separate the time when connectivity returned from the time when telemetry became current again.

Persistence needs a durable place to persist

The file_storage extension can back an exporter sending queue so that queued work is available after restart. The storage must remain accessible to the restarted Collector; an ephemeral path is not a durable recovery strategy. Storage failure or exhaustion can still cause loss. Follow the persistent-storage guidance, and validate permissions and lifecycle behavior in your actual deployment.

Document who owns disk-capacity alerts, who may inspect stored telemetry, and how the queue is recovered during replacement or maintenance. Treat local telemetry storage as part of the data-handling boundary. Keep test payloads synthetic, and review access and retention with the people responsible for your environment.

Do not mistake export failures for confirmed data loss

The Collector’s internal telemetry distinguishes several stages. Queue size and capacity show pressure. Enqueue-failure metrics identify data that failed to enter a queue. Send-failure metrics show unsuccessful export attempts, which may be retried; they do not by themselves establish permanent loss. Receiver and exporter counters also help describe flow. The internal telemetry documentation provides the metric definitions.

For a traces pipeline, inspect the availability and meaning of otelcol_exporter_queue_size, otelcol_exporter_queue_capacity, otelcol_exporter_enqueue_failed_spans, and otelcol_exporter_send_failed_spans in your release. Use the corresponding signal metrics for logs or metric points. Confirm exported names, units, and component support before writing alerts; not every component exposes every helper metric.

My operational preference is to observe the telemetry pipeline through a path that does not depend entirely on the backend being tested. Otherwise, the outage can also hide the evidence needed to diagnose the Collector. Preserve local test logs and use a separate check to record when the backend becomes reachable.

Run a controlled failure exercise

Use a staging environment or an isolated test Collector. Do not disconnect a live production telemetry path just to complete this checklist.

1. Generate synthetic telemetry with identifiable records and document any intentional filtering or sampling.
2. Measure steady-state input, output, queue occupancy, and storage usage.
3. Make the test backend unavailable for the target outage interval. Record queue growth, errors, and application-side effects.
4. In a separate persistence test, restart the Collector with queued data and the same durable storage. Check whether expected records resume delivery.
5. Restore the backend and continue generating traffic. Measure the time required to drain the backlog, not just the first successful export.
6. Reconcile expected records with records received, accounting for intentional processing and possible duplicates. Investigate gaps rather than equating successful retries with complete delivery.
7. Record the first limit reached and assign an owner and corrective action. Repeat after changes to batching, export configuration, storage, or backend limits.

Apply the method to the environment you actually operate

For teams in Africa, a relevant scenario may be an inter-site connection interruption, a remote backend dependency, or a constrained link shared with business traffic. These are possibilities to validate, not assumptions about every African deployment. Select outage durations and traffic patterns from your own operating history and service requirements.

Where link capacity is constrained, compare telemetry volume and recovery traffic with the capacity available to the application. Make deliberate decisions about which signals deserve retention during a prolonged interruption. The objective is not to collect everything indefinitely; it is to retain enough trustworthy evidence to support the decisions the service requires.

The decision to take back to your team

Ask for a one-page delivery statement: under a defined input load, the pipeline tolerates a stated backend interruption, survives a stated restart scenario, and catches up within a measured recovery window. Include the conditions under which data can still be lost. That statement is more useful during an incident than an undocumented queue setting.

The practical takeaways are straightforward: measure the input, match sizing units, validate persistence, reserve recovery capacity, and test loss boundaries. A Collector configuration is a hypothesis. A controlled failure exercise is evidence.

Related reading: Building Observability When Bandwidth Is Unreliable and OpenTelemetry Profiles: A Practical Adoption Guide.

Interested in supporting an open, vendor-neutral telemetry-resilience lab for African engineering communities? Contact Observability Africa. Any sponsorship would be disclosed and would not determine the findings.

About Abdoulaye Apithy

AB Apithy is the founder of Observability Africa, a platform dedicated to helping telecom, fintech, and energy organizations design and scale resilient, high-performance digital infrastructure. His work focuses on enabling real-time system visibility, operational reliability, and performance optimization in environments where downtime, latency, and inefficiency…