When an OpenTelemetry Collector runs out of memory, the visible failure may look simple: a restart, an out-of-memory kill, or a sudden gap in telemetry. The real failure often began earlier. Data accumulated faster than the Collector could export it, queues held live data in memory, and backpressure reached components that were not prepared to retry.
A memory limiter can protect the Collector process, but it cannot guarantee that telemetry will survive. Teams need to understand the full pressure path, from receivers to processors, queues, exporters, and the destination.
Why Collector memory pressure is an operational risk
The OpenTelemetry Collector receives, processes, and exports traces, metrics, and logs. Its memory use depends on traffic volume, batch sizes, processors, exporter queues, and the behaviour of downstream systems.
A backend slowdown can therefore become a Collector problem. Exporters send more slowly, queues retain data longer, and memory consumption rises. If the process is killed, the team may lose the same telemetry needed to explain the incident.
This risk matters anywhere infrastructure capacity is constrained or connectivity is unreliable. It is especially important when Collector instances have small memory limits, traffic arrives in bursts, or a remote backend is reached through an unstable network path.
What the memory limiter actually does
The OpenTelemetry memory limiter documentation describes a soft limit and a hard limit.
- Above the soft limit, the processor refuses new telemetry and returns a non-permanent error to the preceding component.
- Receivers that support retries can resend the refused data or apply backpressure to their sources.
- Above the hard limit, the limiter can also force garbage collection.
- Normal operation resumes after memory usage falls below the soft limit.
This behaviour protects the process, but the documentation includes an important warning: data can be permanently lost when the component before the limiter does not retry correctly.
The memory limiter is therefore a safety control, not a storage system and not a complete reliability strategy.
Processor order matters
OpenTelemetry recommends placing memory_limiter first in the processor chain. This allows the Collector to shed load before later processors allocate more memory or hold data that cannot be flushed.
Processors that discard data, such as filters or samplers, should usually run early. Enrichment and transformation should happen only for data that will continue through the pipeline. Batching belongs later, after filtering and transformation.
processors:
memory_limiter:
check_interval: 1s
limit_percentage: 80
spike_limit_percentage: 20
batch: {}
service:
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [otlp]
This example shows the order, not a universal production value. Limits must be tested against the Collector’s container limit, traffic pattern, processor configuration, and expected bursts.
Backpressure needs an end-to-end design
Backpressure works only when every important boundary behaves as expected.
Ask these questions:
- Can the receiver return a retryable response?
- Will the telemetry producer retry, buffer, or drop the data?
- How long can exporter queues absorb a backend slowdown?
- Are queues held only in memory, or is persistent storage configured where appropriate?
- What happens when retry limits are exhausted?
- Can the team distinguish refused data, dropped data, queue growth, and exporter failure?
If these questions are unanswered, a memory limiter may prevent a crash while data loss continues elsewhere.
Set limits from the deployment boundary
For containerized Collectors, the OpenTelemetry project recommends coordinating the limiter with the container’s memory restriction. Percentage-based limits can be useful when the host environment exposes an appropriate cgroup memory limit. Fixed MiB values may be clearer for virtual machines or other environments with predictable capacity.
The project currently recommends configuring GOMEMLIMIT alongside the memory limiter. Its documented starting point is 80 percent of the Collector’s hard memory limit. The memory limiter should still be tested under realistic load because incoming data can consume memory before rejection takes effect.
Do not copy a limit from another environment without testing. A configuration that works for metrics may fail when high-cardinality attributes, large logs, or trace bursts enter the same Collector.
Monitor the Collector as a production service
A Collector should not be treated as invisible plumbing. Monitor it like any other critical service.
At minimum, teams should watch:
- process memory and container memory
- refused telemetry
- exporter send failures
- queue size and queue capacity
- dropped telemetry
- receiver acceptance rates
- Collector restarts and out-of-memory kills
Alerting should identify sustained pressure before the process reaches its hard limit. A rising queue with flat export throughput is often more useful than a late alert about memory alone.
For a broader failure model, see our guide to designing OpenTelemetry Collector resilience during outages. Teams should also control the data entering the pipeline through a practical metric cardinality budget.
A practical validation exercise
Test the pipeline under controlled pressure before relying on it during an incident.
- Apply a realistic memory limit to the Collector.
- Generate representative telemetry, including expected bursts.
- Slow or temporarily interrupt the backend.
- Observe queue growth, memory use, refusals, retries, and drops.
- Restore the backend and confirm that the pipeline recovers.
- Compare retained telemetry with the incident questions the team must answer.
The goal is not to prove that the Collector never refuses data. The goal is to understand how the system behaves when it does.
Practical takeaways
- Put the memory limiter first in the processor chain.
- Coordinate Collector limits with the actual deployment memory boundary.
- Treat refusals as a signal that upstream retry behaviour matters.
- Monitor queues and exporter health, not only process memory.
- Test downstream outages and burst traffic before production incidents occur.
- Remember that process survival and telemetry durability are different outcomes.
Organizations that want an independent review of Collector capacity, queue behaviour, signal volume, and failure paths can learn about the Observability Africa audit and consulting service. Sponsorship of community education and practical engineering resources is also welcome, with editorial independence preserved.