Understanding Prometheus Remote Write at Scale

What is Prometheus Remote Write?

Prometheus remote write is the standard protocol for sending metrics from a local Prometheus instance to a remote, long-term storage backend. It's the bridge that lets you keep Prometheus's pull-based collection model while gaining the durability and scalability of a managed backend.

When remote write is enabled, Prometheus maintains a Write-Ahead Log (WAL) and periodically sends batches of samples to the configured endpoint over HTTP. If the remote endpoint is unavailable, data is buffered locally and retried.

The Architecture

Here's how the data flows:

1. Prometheus scrapes targets at configured intervals (typically 15s or 60s)
2. Samples are written to the local TSDB and the WAL simultaneously
3. The remote write sender reads from the WAL and batches samples
4. Batches are compressed (snappy compression) and sent via HTTP POST
5. The remote backend acknowledges receipt and stores the data

Tuning for High Throughput

When you're pushing millions of samples per second, the default remote write configuration won't cut it. Here are the key parameters to tune:

Batch Size and Capacity

remote_write:
  - url: "https://euw1-01.m.xscalerlabs.com/api/v1/push"
    queue_config:
      capacity: 10000
      max_shards: 200
      min_shards: 10
      max_samples_per_send: 5000
      batch_send_deadline: 5s

capacity controls the in-memory buffer size. For high-throughput setups, increase this to prevent dropped samples.

max_shards determines parallelism. Each shard is an independent sender goroutine. More shards = more parallel HTTP connections to the remote backend.

max_samples_per_send controls batch size. Larger batches are more efficient but increase latency. We recommend 2000-5000 for most setups.

Compression and Retry

remote_write:
  - url: "https://euw1-01.m.xscalerlabs.com/api/v1/push"
    remote_timeout: 30s
    queue_config:
      min_backoff: 30ms
      max_backoff: 5s
      retry_on_http_429: true

Enable retry on 429 (rate limiting) to handle temporary back-pressure gracefully. The exponential backoff prevents thundering herd issues during recovery.

Common Pitfalls

1. WAL Corruption

If Prometheus crashes ungracefully, the WAL can become corrupted. This leads to gaps in remote-written data. Enable WAL compression and ensure your storage is fast enough:

storage:
  tsdb:
    wal_compression: true

2. Label Explosion

Adding a label with unbounded cardinality (like user_id or trace_id) will cause your series count to explode. Use relabeling to drop high-cardinality labels before remote write:

remote_write:
  - url: "https://euw1-01.m.xscalerlabs.com/api/v1/push"
    write_relabel_configs:
      - source_labels: [__name__]
        regex: "go_.*"
        action: drop

3. Clock Skew

Remote write is sensitive to clock skew between the Prometheus instance and the remote backend. Ensure NTP is properly configured on all hosts. xScaler Labs accepts samples with up to 5 minutes of clock drift, but beyond that, data may be rejected.

Monitoring Remote Write

Prometheus exposes metrics about its own remote write performance:

• prometheus_remote_storage_samples_total — total samples sent
• prometheus_remote_storage_samples_failed_total — failed samples
• prometheus_remote_storage_samples_pending — buffered samples waiting to send
• prometheus_remote_storage_shard_capacity — current shard allocation

Set up alerts on the failed and pending metrics to catch issues early:

- alert: RemoteWriteLagging
  expr: prometheus_remote_storage_samples_pending > 10000
  for: 5m
  labels:
    severity: warning

xScaler Labs Optimizations

When writing to xScaler Labs, you get additional benefits:

• Automatic shard negotiation — we tell Prometheus how many shards to use based on current load
• Deduplication — if you run HA Prometheus pairs, we deduplicate at ingest time
• Schema-on-write validation — malformed series are rejected with clear error messages
• Real-time ingestion metrics — visible in your xScaler Labs dashboard within seconds

xScaler Labs exposes three primary endpoints per region:

• Push: https://euw1-01.m.xscalerlabs.com/api/v1/push — for remote write and OTLP ingestion
• Read: https://euw1-01.m.xscalerlabs.com/prometheus/api/v1/read — for Prometheus remote read
• Query: https://euw1-01.m.xscalerlabs.com/prometheus — as a Grafana Prometheus data source

The region prefix (euw1-01) ensures your data stays within your chosen region for compliance.