Mimir architecture: write and read paths

Grafana Mimir separates metrics ingestion, querying and background maintenance. Ingest storage, stable and preferred since Mimir 3.0, places Kafka or a compatible backend between distributors and ingesters. Each ingester consumes one partition. Write acknowledgement, query visibility and long-term storage are separate boundaries, with different failure and recovery conditions.

The useful starting question is what a successful write proves. In the classic architecture, distributors send samples directly to replicated ingesters. With ingest storage, Kafka accepts the write before an ingester necessarily makes it queryable. That distinction changes how to investigate a dashboard that is stale while ingestion still looks healthy.

Grafana's ingest-storage architecture guide explains the design and its consistency model. This operating guide follows those boundaries into partition planning, retained data, query readiness and scale-in. The classic architecture remains a separate deployment model; its documentation is not an obsolete description of every Mimir cluster.

Which components own each part of the path?

Long-term blocks live in object storage. Hash rings support sharding and service discovery, with ring state shared through a configurable key-value store. The components reference and hash-ring documentation describe these responsibilities; memberlist is a configuration choice, not a requirement implied by the word ring.

Where does a write become durable?

In classic Mimir, distributors replicate each series to ingesters. With the default replication factor of three, success requires a quorum, normally two ingesters. Those ingesters maintain recent samples in memory and a local write-ahead log. Persistent storage is necessary to recover local state after an abrupt failure.

In ingest storage, distributors shard writes across Kafka partitions and return success after Kafka confirms that the records have been committed. Ingester consumption happens asynchronously. A successful acknowledgement does not mean that the samples are already visible to a query or uploaded to object storage. Durability depends on the chosen backend's replication, acknowledgement, failure and retention configuration.

An ingester can recover local state from its WAL and resume from its committed offset. That recovery still requires the relevant log records and state to be available. Finite retention, storage loss and deliberate changes to the starting offset prevent a general promise that every acknowledged sample is recoverable indefinitely.

Putting Kafka between distributors and ingesters removes direct distributor-to-ingester writes and reduces their coupling. Ingester restarts no longer participate directly in the distributor's acknowledgement path, but can still affect reads and freshness. A Kafka outage prevents new writes through this path from succeeding. Upstream buffering and retries are bounded by the client's configuration; the remote-write guide covers that separate recovery boundary.

The Mimir 3.2.2 producer record-data limit defaults to 15,983,616 bytes. The Kafka backend guide recommends accommodating Mimir's larger records with a broker message.max.bytes of at least 16000000, or topic-level max.message.bytes=16000000. The broker and topic property names differ. Check the selected backend's configuration rather than applying an Apache Kafka setting blindly to another implementation.

How do ingesters map to partitions?

The documented ingester assignment uses the numeric suffix of its instance ID, matched by -([0-9]+)$. An instance named ingester-zone-a-13 consumes partition 13. In another zone, ingester-zone-b-13 can consume the same partition for read availability, using a separate consumer group to track its own progress.

The instance ID defaults to the hostname. A fully qualified name such as ingester-zone-a-0.ingester.mimir.svc.cluster.local fails that suffix rule and prevents startup. Set -ingester.ring.instance-id explicitly to an appropriate short ID when the hostname does not match. Preserve the correspondence between identity, local state and partition ownership when replacing instances.

The topic must have at least as many partitions as there are ingesters in one zone in the documented ordinal-based layout. Also check that the actual assigned IDs exist; a count alone cannot validate a custom layout with gaps. Plan per-zone ingester growth and Kafka topic capacity together.

A partition existing in Kafka is different from an Active partition in Mimir's partitions ring. The ring controls where distributors write. Pending partitions do not serve reads or writes; Active partitions serve both; Inactive partitions remain readable during retirement. Adding spare topic partitions alone does not create ingester throughput.

The Jsonnet ingest-storage configuration documents an auto-created topic default of 1000 partitions. That is a deployment default, not the binary's default: Mimir 3.2.2's auto_create_topic_default_partitions defaults to -1, using the broker default. Topic creation settings do not automatically resize an existing topic. Choose headroom deliberately and check the broker's operating limits.

What does replication protect now?

Classic Mimir replicates series from distributors to ingesters. In ingest storage, the Kafka-compatible backend provides write-path replication, while ingesters in multiple zones independently consume partition data for recent-data query availability. Broker replication and ingester zones protect different parts of the path. They are not interchangeable replica settings.

Do not infer a total resource reduction from a smaller ingester tier. Compare distributors, ingesters, the Kafka-compatible backend, cross-zone transfer, object storage, caches, compaction and operating work for the same workload and recovery requirements. The self-hosted incident-capacity guide shows why recovery requirements belong in that comparison.

How does a query find its data?

A query enters the query-frontend, which can split requests and use cached results. Remaining work is queued in the frontend or optional query-scheduler. Queriers execute PromQL over recent data held by ingesters and historical block data served through store-gateways. Ingester data includes the TSDB head and retained local blocks, not only samples still in memory.

The default ingester block-creation interval is two hours. After upload, local retention gives the read path time to discover the new object-storage blocks. Treat upload, discovery and query availability as separate observations when investigating a gap around a block boundary.

Ingest-storage queries use eventual consistency by default. If consumption falls behind, a dashboard can return older data even though writes have succeeded. Normal-operation latency expectations in documentation are not a workload benchmark or a freshness guarantee.

For a query that requires read-after-write consistency, a client can send X-Read-Consistency: strong. The query-frontend obtains offsets for active partitions and propagates them to ingesters, which wait until they have consumed through the required offsets. The documented guarantee covers samples written to Kafka before the query-frontend received the query. Obtaining the offsets and reaching them can add latency or fail.

In Mimir 3.2.2, -ingest-storage.kafka.wait-strong-read-consistency-timeout defaults to 20 seconds. This is an ingester wait limit, not the complete query deadline or a promise that data will appear within that interval. The current documentation also says that the ruler automatically requests strong consistency for rules depending on earlier rules in the same group. The Jsonnet ingest-storage guide requires remote ruler evaluation. Check the deployed evaluation path instead of assuming all consumers use the same consistency settings.

Why do store-gateway rollouts affect queries?

The store-gateway keeps index-headers on local disk and fetches the index and chunk portions needed for a query. Lazy loading concerns loading headers into memory. It does not remove the initial work of discovering blocks and downloading missing headers before readiness.

Persistent disks avoid unnecessary header downloads after a restart. Cold memory and rebuilding local headers can still affect readiness and query latency. Measure both during rollouts, and distinguish a slow initial synchronisation from a slow first query. Preserving a volume does not guarantee that every restart will be fast.

Historical lazy-loading and readiness discussions are useful diagnostic context. They do not establish a universal latency figure for current Mimir or prove that every header must be loaded into memory before readiness. Use the current component documentation and your deployment's metrics to distinguish those cases.

What does compaction still have to do?

Each ingester uploads its own blocks. The compactor combines blocks and removes replicated samples through vertical compaction. Split-and-merge compaction can produce a block per shard for a compacted range, so the result is not necessarily one block per tenant.

The compactor also maintains bucket indexes and enforces long-term retention when configured. Block retention is disabled by default. Kafka log retention and object-storage block retention are separate controls: changing one does not set the other. Compaction backlog matters to retained blocks and query work, and duplicate blocks can coexist until compaction and delayed deletion finish.

How should you size Kafka retention and storage?

Start with how long an ingester may be unable to progress, then allow for restart or replay, catch-up and an operating margin. Check the backend's effective time and size limits. With Apache Kafka's delete policy, size limits can cause older segments to expire before the intended time window. Review the semantics of the specific backend and version you run.

The maintainer's retention discussion explains why normal consumption delay is a poor basis for retention: longer retention covers unhealthy or lagging consumers, and encoded write requests repeat series labels. It does not establish a safe minimum for every workload. Measure encoded log growth instead of estimating it from compressed TSDB bytes per sample.

1. Measure representative stored log growth before replication, or document precisely what your byte-rate metric includes. Include realistic labels, batching and backend compression.
2. Set the required recovery window from outage, replay and catch-up scenarios. Check time and size-based expiry together.
3. Estimate retained log bytes, then account for actual replication, segment/index overhead, partition skew, replica placement, growth and maintenance capacity.
4. Exercise consumer recovery before shortening retention. Monitor lag, query freshness and remaining replay margin as well as disk utilisation.

Assume a constant 4 MB/s of stored log bytes before replication, six hours of retention and three physical copies on disk-backed brokers. These inputs are invented solely to show the arithmetic; they are not recommended settings. MB and GB below are decimal. The estimate assumes the measured rate already reflects the chosen compression and is not a network traffic counter.

Do not multiply by replication again if the source metric already counts physical replica bytes. This disk-backed model also needs adaptation for tiered storage or Kafka-compatible services that place durable data elsewhere. The backend's own capacity model controls what must be provisioned.

If a required offset has expired, that consumer cannot replay it from Kafka. Whether the result is a permanent data gap depends on surviving ingester state, other replicas and blocks already uploaded to object storage. A recovery runbook must identify those alternatives rather than treating an acknowledged write as an unlimited replay entitlement.

How do you scale ingesters safely?

Scale-out adds ingesters with the intended IDs and corresponding topic partitions, then moves their ring partitions through the documented lifecycle. An ingester is not an arbitrary worker that can take over any missing ordinal through an ordinary shared consumer-group rebalance.

Scale-in changes read availability as well as write routing. The scaling guide describes coordination with the rollout-operator. Its process inactivates a partition, waits for the relevant data to become queryable from long-term storage, prepares the ingester for shutdown and removes ownership before terminating it. A replica-count edit alone does not establish those conditions.

When should you keep the classic architecture?

Ingest storage is the preferred architecture, but migration still needs a workload-specific reason. A modest classic deployment with acceptable failure behaviour and mature runbooks may be better left in place while the team establishes how it would operate a Kafka-compatible backend. Adding that dependency without an owner can create more risk than it removes.

• Retain classic while the new backend's replication, storage and recovery ownership are unresolved.
• Validate consumers that depend on immediate read visibility. Review latency and failure behaviour under strong consistency rather than assuming the old contract carries over.
• Compare the full deployment and transition effort, not just one component's resource profile. Existing reliability pain should be explicit and measurable.
• Use the documented side-by-side migration procedure and retain a recovery path until the target behaviour has been accepted.

The Helm production guide explicitly marks its bundled single-node Kafka as demonstration-only. Use a production-grade Apache Kafka or compatible backend for production ingest storage. A managed service can change who operates it, but does not remove the need to understand retention and recovery.

What should you check before a version upgrade?

The release API returned Mimir 3.2.2 as the latest stable release on 9 October 2026. Its release notes include an Alertmanager local-file-loading security fix. This is separate from the architectural change introduced in 3.0; read the exact release notes and security guidance for the version being deployed.

• For 3.1, check that uploaded TSDB blocks use index format v2. Store-gateways no longer generate index-headers from v1 blocks, and the historical eager-loading-startup flag was removed.
• For 3.2, follow the required upgrade path from 3.1 and ensure all queriers are on 3.1 before beginning the transition. Internal querier/store-gateway gRPC changes and remote-execution defaults make mixed-version sequencing important.
• Recheck performance settings: 3.2 enables query sharding by default, reduces the default querier concurrency to eight and disables ingester request hedging by default. Those are release-specific defaults, not workload sizing recommendations.
• Review the versioned mixin alerts as well as the binary. An upstream alert change does not prove that your deployed rules have been updated.

Before changing architecture or versions, record a successful write, its first query-visible timestamp, a consumer recovery within the retained log window and a historical query after store-gateway restart. Exercise the queries and rule dependencies that actually matter to the service. Source documentation explains the design; these observations establish how the chosen deployment behaves.

Does Kafka replace Mimir's object storage?

No. In ingest storage, the Kafka-compatible backend retains the ingest log that ingesters consume. Ingesters still build TSDB blocks and upload them to object storage, where store-gateways and compactors participate in long-term querying and maintenance.

Can Mimir use a Kafka-compatible service instead of Apache Kafka?

The official Kafka backend guide documents several compatible options and their required settings. Mimir's protocol use is deliberately limited, but that is not a promise that every implementation works without configuration. Validate the selected backend's record limits, authentication, retention and failure semantics.

Can more Kafka brokers replace the need for more ingesters?

They address different limits. Broker capacity supports the ingest log; ingesters consume assigned partitions and provide recent-data queries. Increasing one tier does not prove that the other has sufficient consumption or query capacity. Measure the limiting path.

Will every query wait for Kafka consumption to catch up?

No. Queries use eventual consistency by default. A client can request strong read consistency, and the documented ruler path does so for dependent rules within a group. Review the actual query and ruler configuration rather than assuming a universal wait policy.