How We Reduced Metrics Costs by 80% While Keeping High-Cardinality Data
Engineering

How We Reduced Metrics Costs by 80% While Keeping High-Cardinality Data

Pathum Fernando·Dec 15, 2024·8 min read

The Cost Problem No One Talks About

When your application crosses the threshold from "startup hobby project" to "production system serving real users," something unexpected happens to your observability bill. It doesn't grow linearly — it explodes.

At xScaler Labs, we've spoken to hundreds of engineering teams, and the pattern is always the same. You start instrumenting everything because that's what the docs tell you to do. Labels multiply. Cardinality climbs. And before you know it, your metrics bill is rivaling your compute spend.

We've seen teams spending $15,000/month just to store and query metrics data that, in many cases, they weren't even looking at.

Where the Money Goes

Most metrics backends charge based on one or more of these dimensions:

  • Active time series — the number of unique label combinations being ingested
  • Samples ingested per second — how fast data flows in
  • Query volume — how many times dashboards and alerts read data
  • Retention period — how long data is kept

The compounding effect is brutal. Add one new label with 100 possible values, and you've just 100x'd a portion of your series count. A cardinality explosion from a single bad label can double your bill overnight.

The xScaler Labs Approach

We took a fundamentally different approach to pricing and architecture:

1. Tiered Storage with Intelligent Downsampling

Raw data at full resolution is kept for the first 48 hours — the window where you're most likely debugging an active incident. After that, data is automatically downsampled to 5-minute intervals for the next 30 days, then 1-hour intervals for long-term retention.

The key insight: you almost never need per-second granularity when looking at data from three weeks ago. But you absolutely need it when debugging a production incident at 2 AM.

2. Label-Aware Compression

Not all labels are created equal. Our storage engine understands the structure of Prometheus labels and applies targeted compression that reduces storage by 10-15x compared to naive approaches.

High-cardinality labels like pod_name or request_id that change frequently are stored differently from low-cardinality labels like region or environment that rarely change.

3. Query-Time Aggregation

Instead of pre-computing every possible aggregation at ingest time (which is what drives costs up in traditional systems), we store the raw data efficiently and aggregate at query time. Modern hardware makes this fast enough that most dashboard queries complete in under 200ms.

Real Results

One of our early customers, a fintech company running 200+ microservices on Kubernetes, saw these results after migrating to xScaler Labs:

| Metric | Before (Datadog) | After (xScaler Labs) | Change | |--------|------------------|------------------|--------| | Monthly cost | $18,400 | $3,200 | -82% | | Active series | 2.1M | 2.1M | No change | | Query latency (p95) | 1.2s | 340ms | -72% | | Retention | 30 days | 1 year | +12x |

The critical point: they didn't lose any data. Every series, every label, every dimension was preserved. The savings came entirely from architectural efficiency, not from throwing data away.

Getting Started

If you're spending more than $2,000/month on metrics infrastructure and want to explore what xScaler Labs can do for you, start a free trial. Migration takes about 15 minutes — we support Prometheus remote write out of the box via https://euw1-01.m.xscalerlabs.com/api/v1/push, so there's zero re-instrumentation required.

Your existing Grafana dashboards, alerts, and recording rules continue to work unchanged — just point your Grafana data source to https://euw1-01.m.xscalerlabs.com/prometheus. We're a drop-in replacement for the backend, not a rip-and-replace of your entire stack.