A metrics system can remain technically available while becoming operationally less trustworthy. One common cause is cardinality growth: every new combination of metric attributes creates another time series or metric point that the system must aggregate, transport, store, and query.
This is not only a platform-scale problem. It can begin with one seemingly useful attribute such as a customer ID, raw URL, session identifier, device identifier, or unbounded error message. In environments where teams must be deliberate about compute, bandwidth, and storage, uncontrolled cardinality turns observability from a diagnostic capability into a competing workload.
The right response is not to strip every dimension from metrics. It is to establish a cardinality budget: an explicit agreement about which dimensions are needed for operational decisions, how much growth is acceptable, and what the team will do when the budget is exceeded.
Why cardinality is multiplicative
Consider a request counter with four attributes:
service.name: 20 valueshttp.route: 40 valueshttp.request.method: 5 valueshttp.response.status_code: 10 values
The theoretical combination space is 20 × 40 × 5 × 10, or 40,000 series. Real traffic may not exercise every combination, but the example shows why reviewing attributes individually is not enough. A bounded attribute can still be expensive when multiplied by several others.
The Prometheus project warns that every unique label set creates a new time series and explicitly recommends against using unbounded values such as user IDs and email addresses as labels. OpenTelemetry uses the same underlying idea: metric cardinality is the number of unique attribute combinations tracked for a metric stream.
A limit protects memory, not necessarily meaning
The stable OpenTelemetry Metrics SDK specification says SDKs should support a configurable cardinality limit. If neither a matching View nor the MetricReader supplies one, the specified default is 2,000 metric points per collection cycle.
When a stream crosses its limit, additional combinations are represented by an overflow point identified by otel.metric.overflow=true. Measurements are not meant to disappear or be counted twice. However, their original attribute combinations are no longer independently available.
That distinction matters. The total can remain correct while a dashboard or SLO grouped by route, region, provider, or outcome becomes incomplete. A limit can prevent uncontrolled memory growth in the SDK, but it cannot decide whether the lost dimensions were essential to an operational question.
A team therefore needs two controls:
- A technical ceiling that keeps instrumentation safe.
- A semantic budget that protects the dimensions required for diagnosis, SLOs, and business-critical journeys.
Build the budget around decisions
Start with the questions the metric must answer. For an API, these might be:
- Which service or route is failing?
- Is the failure associated with a bounded dependency class?
- Which outcome category is affecting the SLO?
- Is a specific deployment version involved?
Then classify candidate attributes.
Keep bounded dimensions that drive action
Route templates, HTTP methods, controlled outcome categories, deployment environments, and stable service names are often useful because they support aggregation and ownership. The important word is controlled. A route template such as /payments/{payment_id} is bounded; a raw path containing every payment ID is not.
Move investigative identifiers to traces or logs
Request IDs, transaction references, session IDs, raw URLs, and detailed error text may be valuable during an investigation. They are usually poor metric attributes because their value set grows with traffic. Keep metrics suitable for aggregation, then use exemplars or trace correlation to move from a broad symptom to a specific request.
Challenge dimensions that are merely convenient
An attribute should not survive simply because a dashboard currently uses it. Ask whether it changes an engineering or operational decision. If two dimensions always lead to the same action, one may be redundant. If a field is collected only in case it becomes useful later, traces or sampled logs may be the more appropriate home.
Estimate before deployment
A lightweight review can expose most risks before instrumentation reaches production.
- List every metric attribute and its expected number of values.
- Multiply the expected values across dimensions for each metric stream.
- Repeat the estimate for credible worst cases, including autoscaling, new tenants, error storms, and route changes.
- Mark any dimension whose value set is driven directly by users, requests, database records, or arbitrary text.
- Compare the estimate with the configured SDK limit and the capacity of the backend.
This is not a forecast of the exact number of active series. It is a design review that makes hidden multipliers visible. Run it in code review for new instrumentation and again when services, routes, tenants, or deployment patterns change.
Detect overflow before dashboards mislead you
Do not wait for a storage bill or a slow query to reveal the problem. Monitor for the OpenTelemetry overflow attribute and alert when it appears on a metric used for an SLO or critical operational decision.
Also watch the telemetry pipeline itself:
- active series or metric-point growth by service and instrument;
- Collector memory, refused data, queue pressure, and export failures;
- ingestion volume and storage growth;
- query latency for the dashboards used during incidents;
- unexpected new attribute keys or rapidly expanding value sets.
This extends the principle from our guide to building an outage-resilient OpenTelemetry Collector: the telemetry path is production infrastructure and should be observable as such.
Reduce cardinality at the right layer
The best fix is usually at the instrumentation source. Replace raw paths with route templates, convert free-form errors into bounded categories, and remove identifiers that do not belong in metrics.
OpenTelemetry Views can filter attributes before metric aggregation and can define a stream-specific cardinality limit where supported. That is important because the specification calls for enforcing the limit after attribute filtering. Fixing the stream before aggregation preserves more useful capacity than dropping unwanted telemetry later.
The OpenTelemetry Collector remains valuable for governance across many services. Its processors can transform, filter, or enrich telemetry before export. Collector-side filtering can reduce downstream ingestion and storage, but it should not be treated as a substitute for unsafe SDK instrumentation: by the time an over-cardinal metric reaches the Collector, the emitting process may already have paid the memory and aggregation cost.
Make the operating rule explicit
A practical cardinality policy can fit on one page:
- No user IDs, email addresses, session IDs, transaction IDs, raw paths, or unbounded text in metric attributes.
- Every new attribute must name the decision it enables and its expected value count.
- Metrics supporting SLOs must have an overflow check.
- High-cardinality investigation moves through traces, exemplars, or appropriately governed logs.
- Cardinality growth is reviewed after major releases and during capacity planning.
- Owners are identified for high-volume instruments and Collector pipelines.
For teams operating with tighter infrastructure or bandwidth margins, this discipline is especially valuable. It preserves headroom for the signals that help engineers understand user impact. It also makes telemetry spend easier to explain because retention and dimensionality follow an operational purpose.
What to do this week
- Find the ten metrics with the highest active-series count.
- Identify which attribute combinations are responsible.
- Check critical streams for
otel.metric.overflow=true. - Remove or normalize one unbounded dimension at the source.
- Document a cardinality budget for one customer-critical journey.
Observability is not improved by collecting the largest possible dataset. It improves when teams retain enough context to make the right decision quickly—and can trust that the dimensions behind that decision are still intact.
Organizations interested in supporting independent technical education on reliable and cost-conscious telemetry can explore sponsorship or workshop collaboration with Observability Africa. Sponsorship does not determine our editorial conclusions.