How to Build a Resilient OpenTelemetry Collector Pipeline

Abe Apithy

2 months ago

A quiet dashboard during an outage can mean the application is healthy, or it can mean the telemetry pipeline has stopped delivering evidence.

An OpenTelemetry Collector pipeline needs more than a running process. It needs a defined loss tolerance, enough buffer capacity for expected downstream failures, a recovery path, and monitoring that tells responders when the pipeline is falling behind.

This guide explains how to design and validate that pipeline. The examples are planning baselines, not delivery guarantees. Queue capacity, retry time, storage, and recovery throughput must be tested against the traffic and failure conditions of the real environment.

Start with a telemetry delivery objective

Before changing Collector settings, define what the organization can afford to lose. A practical objective should answer:

Which signals are most valuable during an incident?
How long may the downstream backend be unavailable?
How much telemetry loss is acceptable?
How quickly must the backlog clear after recovery?
What should happen when local capacity is exhausted?

Not every signal needs the same treatment. Security audit logs, payment traces, routine host metrics, and debug logs can have different loss tolerances. One oversized pipeline for every signal can waste scarce memory and storage. Separate pipelines or Collector tiers when the reliability requirements are materially different.

Understand what each resilience mechanism solves

Sending queue

An exporter sending queue holds telemetry while it waits for delivery. An in-memory queue can absorb short interruptions, but its contents disappear if the Collector process stops. If the queue fills, incoming telemetry is refused or dropped according to component behavior.

Retry handling

Retry handling attempts delivery again after eligible transient failures. It does not create unlimited durability. If the outage lasts longer than the configured retry period, or if the queue fills first, data can still be lost.

Persistent queue storage

The Collector can back a sending queue with the file_storage extension. This allows queued telemetry to survive a Collector restart. It does not protect against a failed or full disk, corrupted storage, incorrect permissions, or an outage that exceeds available capacity.

External message queue

A dedicated message queue can provide stronger separation between Collector tiers or between infrastructure and a backend. It also introduces another production system with its own capacity, security, replication, and operational requirements. Add it only when the delivery objective justifies the extra complexity.

Use a layered pipeline for important telemetry

A common design separates local collection from centralized processing:

Applications and hosts
        |
        v
Agent Collectors
        |
        v
Gateway Collectors
        |
        v
Observability backend

Agents keep collection close to the workload and can reduce dependence on one network path. Gateways centralize policy, transformation, sampling, and export. This separation also limits the work each tier must perform.

The gateway tier is not automatically resilient. It still needs multiple instances, appropriate load distribution, bounded queues, resource limits, and failure testing. Stateful processors introduce additional routing rules. For example, the tail-sampling processor requires all spans for a trace to reach the same sampling instance.

Configure a persistent sending queue

The following pattern combines file-backed queue storage with bounded retry behavior:

extensions:
  file_storage:
    directory: /var/lib/otelcol/storage

exporters:
  otlp:
    endpoint: telemetry.example:4317
    sending_queue:
      storage: file_storage
      queue_size: 5000
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
      max_elapsed_time: 10m

service:
  extensions: [file_storage]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp]

Do not copy these numbers into production without measurement. Queue units and behavior depend on the exporter and Collector version. Confirm the component documentation for the deployed release. Mount storage on a persistent volume with defined capacity, permissions, backup expectations, and cleanup behavior.

Size for outage duration and recovery throughput

A useful first estimate is:

required buffer bytes = incoming bytes per second × tolerated outage seconds × safety factor

This is only the start. Serialization, batching, compression, signal mix, queue representation, and traffic bursts can change actual storage use. Measure disk growth during a controlled export failure rather than relying only on calculated volume.

Recovery throughput is equally important. If telemetry arrives at 20 MB per second and the exporter can send only 20 MB per second after the backend returns, the queue never drains. The pipeline needs spare export capacity above the normal arrival rate.

backlog drain rate = available export throughput - current incoming throughput

Use that rate to estimate recovery time. A design that can buffer one hour of telemetry but needs eight hours to clear it may leave the team operating with delayed evidence for most of a working day.

Protect the Collector from its own backlog

Queues and retries consume memory, disk, connections, and CPU. The memory_limiter processor can help the Collector refuse new data before memory pressure causes an uncontrolled process failure. It does not create capacity, and it should be configured from the Collector container or host memory limit.

Place memory protection early in the pipeline. Use batching after processors that need individual or complete trace context, according to their documentation. Avoid adding processors simply because they are available. Every transformation adds work to a component that must remain dependable during outages.

For resource-constrained environments, separate critical and routine signals where possible. Preserve the evidence required for incident response first. Reduce low-value debug traffic at the source, use bounded metric dimensions, and apply deliberate sampling rather than depending on queue exhaustion as an accidental sampling policy.

Monitor the telemetry pipeline as production infrastructure

The Collector exposes internal telemetry. Build alerts and dashboards around the failure modes that matter to the delivery objective:

export failures and retry activity;
exporter queue size and queue capacity;
refused spans, metric points, and log records;
Collector memory and CPU;
persistent storage use and disk errors;
receiver accepted telemetry compared with exporter sent telemetry;
backlog age and recovery time where measurable;
configuration reloads, restarts, and unhealthy replicas.

Send critical pipeline alerts through a path that does not depend entirely on the failing pipeline. Otherwise, the monitoring system may be unable to report its own failure.

Use a consistent OpenTelemetry service naming standard for Collector instances and tiers. Clear identity makes it possible to distinguish an application incident from an agent, gateway, or backend delivery problem.

Test the failure modes that documentation cannot answer

A non-production resilience test should include:

Establish normal incoming and outgoing telemetry rates.
Disconnect or reject the downstream exporter endpoint.
Measure queue growth, retry behavior, memory, disk, and refused data.
Restart the Collector while telemetry remains queued.
Restore the backend and measure backlog drain time.
Repeat with a traffic burst above the normal rate.
Fill the persistent volume to the planned safety threshold.
Scale gateway replicas and confirm routing behavior for stateful processors.
Verify that alerts fire before loss becomes material.

Record what was sent, what was received, and what was intentionally discarded. Do not claim zero loss unless the test can prove it for the tested boundary. Collector success counters, backend counts, and controlled trace or record identifiers can help reconcile the result.

Common designs that create false confidence

Increasing queue size without measuring recovery: the outage is delayed, not solved.
Using only an in-memory queue: backend interruptions are buffered, but Collector restarts lose the backlog.
Adding persistent storage without disk alerts: a full disk becomes a silent new failure mode.
Setting retries to continue for a long time without capacity planning: old data competes with new data while the queue grows.
Running one gateway replica: queue durability does not remove the availability risk of a single process or host.
Using ordinary load balancing before tail sampling: spans from one trace may reach different samplers.
Testing only steady traffic: real outages often coincide with retries, error logs, and telemetry bursts.

A practical design review

For each signal and Collector tier, document the incoming rate, tolerated outage, queue type, retry limit, storage capacity, recovery throughput, failure alerts, and owner. Review the design whenever traffic, retention, backend, or sampling policies change.

Connect the review to the rest of the OpenTelemetry architecture. Control metric growth with a cardinality budget. Preserve causality through queues with Span Links. Validate that tail sampling and exporter queues do not compete for memory without a shared capacity plan.

A resilient Collector pipeline is not one that never loses data. It is one with a known failure boundary, visible pressure, controlled degradation, and a tested recovery path.

Teams that need a structured review of telemetry delivery, Collector capacity, and failure handling can request an independent observability audit. Observability Africa also welcomes support for vendor-neutral technical education. Sponsorship does not determine editorial conclusions.