Written by Nelson Koning
on 

Managing observability dashboards with GitOps: an intro to observability-as-code

Dashboards are the backstage console of modern systems — they’re where SREs, developers, and product owners tune into how an app is performing. Yet many teams still edit dashboards by hand, paste JSON into a UI, or keep a single editable “gold” dashboard in production. Observability-as-code brings the same benefits we expect from infrastructure-as-code to dashboards: versioning, code review, repeatable deployments, and safe rollbacks.

This article walks through why GitOps is a natural fit for dashboards, the recent tooling choices that make it practical today, a small example of what a GitOps layout looks like, and common gotchas to watch for.

Why treat dashboards like code?

Think of dashboards like a music playlist. You want every listener (environment) to have the same curated tracks in the same order, but you also want the option to test a new cut in a private playlist before promoting it to everyone.

Recent developments that matter

Grafana — still the most widely used dashboard platform in many shops — has been pushing hard on “observability as code” primitives. Recent releases introduced a Git Sync workflow (where Grafana can synchronize dashboard changes with a Git repo and open PRs directly), a new resource-oriented API model, and a redesigned dashboard schema that is friendlier for diffs and composability. These changes make both in-Grafana Git workflows and external GitOps pipelines more practical. (grafana.com)

If you run Grafana on Kubernetes, the Grafana Operator exposes dashboards as Kubernetes custom resources (GrafanaDashboard). That makes it straightforward to manage dashboards with GitOps tools like Argo CD or Flux by storing the manifests in Git and letting the GitOps controller reconcile the cluster to the repo state. The Grafana docs and operator examples contain concrete CRD manifests you can drop into a repository. (grafana.com)

Finally, there are complementary approaches: Terraform and SDKs (e.g., Grafana Foundation SDK) let you generate or manage dashboards from code and wire them into CI/CD pipelines for automated provisioning. These are useful when dashboards are generated from service metadata or when you want stronger type-safety. (grafana.com)

Patterns for managing dashboards with GitOps

There are three common patterns you’ll see in the wild — choose the one that fits your team’s workflow and risk tolerance.

1) In-Grafana Git workflows (Git Sync)

2) Kubernetes-native GitOps (Grafana Operator + Argo CD / Flux)

3) CI/CD pipeline provisioning (Foundation SDK, Terraform, or GitHub Actions)

A minimal GitOps layout (example)

Here’s a tiny, realistic repository layout for the Kubernetes-native approach:

A sample GrafanaDashboard CR (simplified) looks like this (the operator examples use apiVersion grafana.integreatly.org/v1beta1):

apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDashboard
metadata:
  name: app1-overview
  namespace: observability
spec:
  instanceSelector:
    matchLabels:
      dashboards: "grafana"
  json: >
    {
      "id": null,
      "title": "App1 Overview",
      "panels": [],
      "time": {"from":"now-1h","to":"now"}
    }

Argo CD (or Flux) will sync the cluster folder to apply the manifest; the grafana-operator reconciles the CR into a dashboard inside Grafana. The operator supports several content sources: inline JSON, gzip-compressed JSON, remote URLs, or Jsonnet projects — which gives flexibility for generated dashboards. (grafana.github.io)

Best practices and gotchas

When to pick which pattern

A pragmatic closing note

Treating dashboards as code is about bringing discipline and reproducibility to a part of observability that often lives outside the rest of your engineering governance. Newer features and APIs from popular tooling are lowering the friction to adopt GitOps for dashboards, but every team will need to balance editability, review, and safety. Use automation where it helps (linting, previews, PR checks), and keep a single source of truth so your visualizations remain reliable and consistent across environments. (grafana.com)

References (selected)

Enjoy the rhythm of observability-as-code — like a well-arranged playlist, good dashboards should play nicely everywhere.