Scaling Prometheus with Grafana Agent and Remote Write to a Scalable TSDB

Prometheus is excellent for short-term, single-node monitoring, but when you need long-term retention, global querying, or to support hundreds of clusters, the standard single-server model starts to break down. A modern, battle-tested pattern is: keep Prometheus (or a lightweight agent) for scraping, use remote_write to ship metrics, and store/query them in a horizontally scalable Prometheus-compatible TSDB such as Grafana Mimir. This article explains the why, the architecture, configuration examples, and practical best practices for making that pattern work reliably. (grafana.com)

Why move beyond standalone Prometheus?

Standalone Prometheus is optimized for local scraping and fast queries over a limited retention window. At larger scale you run into operational pain points:

• High disk and CPU costs for long retention across many clusters.
• Difficulty querying across many Prometheus instances without federation.
• Operational complexity for high availability and global deduplication.

A remote_write architecture decouples scraping from long-term storage: Prometheus or a lightweight agent scrapes, batches samples, and sends them to a scalable backend that can horizontally scale, replicate, and index time-series for long-term queries. Grafana Mimir is an example of a modern, multi-tenant TSDB designed for that purpose. (grafana.com)

Grafana Agent’s role

Grafana Agent is a remote_write-focused Prometheus-compatible agent that sits between scrapers (or acts as a scraper) and the remote store. It offers features suited for production remote_write pipelines:

• A write-ahead log (WAL) to buffer samples during transient network outages and protect against data loss.
• Tunable batching and retry behavior for remote_write endpoints.
• Integrations to collect host metrics (node_exporter style) while staying lightweight. (grafana.com)

Using Grafana Agent (or Prometheus configured for remote_write) keeps local Prometheus instances slim: short local retention for fast debugging plus reliable forwarding to the central store.

High-level architecture

• Edge: Prometheus servers or Grafana Agents scrape targets in each cluster/region.
• Transport: The agent uses the Prometheus remote_write protocol to send compressed batches.
• Store: A horizontally scalable TSDB (Mimir, Thanos, Cortex, etc.) ingests, indexes, and stores data long-term.
• Visualize: Grafana connects to that store for unified dashboards and long-range queries.

This pattern gives you centralized querying and retention without forcing every Prometheus server to host years of data locally. Modern backends like Mimir also support OpenTelemetry metrics and are optimized for cost and query performance at scale. (grafana.com)

Minimal config examples

Prometheus (prometheus.yml) — remote_write stanza

global:
  scrape_interval: 15s

remote_write:
  - url: "https://mimir.example.com/api/v1/push"
    # add authentication headers or basic auth as required
    # remote_timeout, queue_config, and tls_config can be tuned
    queue_config:
      capacity: 10000
      max_shards: 200

Grafana Agent (agent.yaml) — prometheus.remote_write component

prometheus:
  wal_directory: /tmp/grafana-agent-wal
  remote_write:
    - url: "https://mimir.example.com/api/v1/push"
      send_batch_size: 1000
      batch_send_deadline: 5s
      timeout: 30s

These snippets illustrate the key pieces: the remote_write destination and a WAL-enabled agent for durability. Exact fields and names can vary between Grafana Agent and Prometheus; consult component docs for specifics. (grafana.com)

Practical tuning and best practices

• Use WAL buffering on the agent: The agent’s WAL helps absorb short outages without dropping samples. Monitor WAL size and health for early warning signs. (grafana.com)
• Tune batch size and deadlines: Larger batches improve throughput but increase memory and latency. A common pattern is a send_batch_size in hundreds-to-low-thousands and a batch_send_deadline in seconds. Test with your network and endpoint. (grafana.com)
• Watch prometheus_remote_storage_samples_pending: This metric indicates the queue backlog for remote_write. If it grows, check network, CPU, or destination performance. Prometheus docs recommend investigating CPU/network saturation when remotes fall behind. (prometheus.io)
• Reduce high-cardinality churn before sending: Relabeling and metric scrubbing at scrape-time (or via the agent) reduces cardinality and storage costs. Drop, labelmap, and relabeling rules are your friends.
• Respect remote_write semantics: Implementations should retry on 5xx and use backoff; the remote_write spec defines expected behavior for senders and receivers. Ensure your agent and backend are compatible. (next.prometheus.io)
• Consider local short-term retention: Keep a small local retention window in Prometheus for immediate debugging, but treat the remote store as the long-term source of truth.

Monitoring the pipeline

Treat the pipeline itself as an observability target:

• Agent metrics: monitor WAL health, buffer sizes, send/receive errors.
• Network: watch for spikes in latency or packet loss between agents and store.
• Backend ingestion metrics: track ingestion rate, commit latencies, and index sizes on the TSDB.

These operational metrics let you detect problems early (for example, a consistently growing samples_pending is a clear sign the remote endpoint is overloaded). (prometheus.io)

Querying and dashboards

Once metrics are centralized, dashboards and alerting can move from per-cluster Prometheus to a single Grafana connected to the scalable TSDB. This simplifies long-range queries (e.g., 3–12 months) and cross-cluster comparisons, and it centralizes alert routing and templating.

Remember: running large-range queries is heavier than short-term ones. Use downsampling (if supported by the backend), query time windows, and dashboard panels that avoid asking for massive ranges on every refresh.

Caveats and trade-offs

• Network cost and latency: shipping all raw samples increases network usage. Consider sampling or lowering scrape frequency for low-value metrics.
• Storage cost vs. retention: A scalable backend offers flexibility, but long retention costs money. Set SLO-driven retention policies.
• Complexity: Operating a multi-component pipeline (agents, remote store, object storage, query frontend) is more complex than single-node Prometheus. Use managed services if you prefer fewer operational responsibilities.

Why this pattern is relevant now

Recent iterations of Grafana Mimir and agent tooling have focused on reliability and scale, making remote_write pipelines easier to run in production. Modern Mimir architectures emphasize performance, multi-tenancy, and native support for both Prometheus and OpenTelemetry metrics, while Grafana Agent provides WAL-backed reliably batching for remote_write use cases. Together, they make centralized, long-term Prometheus-compatible monitoring practical for large environments. (grafana.com)

Conclusion

For teams that outgrow single-node Prometheus, adopting a remote_write pipeline with Grafana Agent and a scalable TSDB delivers centralization, long retention, and global querying without sacrificing local scrape performance. The pattern pairs a low-footprint scraper/agent with a purpose-built backend that can scale horizontally, and it is supported by recent improvements in both agent behavior and backend architectures. Treat the pipeline as an observable system, tune buffering and batching to your environment, and control cardinality early to keep costs and query performance manageable. (prometheus.io)