Grafana as an Ops Pulse, Not a Metrics Museum
- infrastructure
- grafana
- monitoring
- observability
- ci
Grafana is easy to turn into a museum. One more dashboard for every service, one more exporter for every process, a Pushgateway full of CI job gauges that nobody opens except when something is already on fire. On a laptop that cost is invisible. On a roughly 2 GiB VPS that also serves Learn Persian with Sepideh, the company hub, WordPress, TTS and cold apps, a museum is an outage waiting for a scrape interval.
This post is how we use Grafana at Norditco: as a human-readable ops pulse that wakes when someone needs it, sits on top of a deliberately thin Prometheus core, and fits into a larger follow-up loop that starts with unit tests in GitHub Actions and ends with Telegram when the host or a hot app is actually wrong. It is not a claim that Grafana is where we run test suites. It is a claim about where each kind of signal belongs.
If you want the host topology this sits on, see edge routing, cold apps and footprint. If you want the capacity decision that kept this stack on Compose, see Compose versus k3s.
Two clocks for software quality
Norditco follows repositories with two clocks that should not be collapsed into one dashboard.
Pre-merge and pre-deploy live in GitHub Actions on each product repo. Learn Persian with Sepideh, for example, runs lint, unit tests (Vitest), end-to-end checks, dependency audit and a production build on every push to master. The Norditco hub and other apps have their own Actions workflows. A red unit test stops the merge story before the VPS ever sees a new image. That is the right place for “did the code still mean what we thought.”
Post-deploy and always-on live on the VPS. An honest health gate runs after deploy. Prometheus scrapes host and a few app signals every minute. Alertmanager turns a small rule set into Telegram messages with an [alert] prefix. The morning secretary brief summarizes health classes and nginx hits. Grafana is the interactive pane on top of Prometheus (and optionally Loki) when a human needs graphs instead of a one-line alert.
Unit test failures are a developer problem in CI. Missing learnpersian_web containers or root disk under 8% free are an ops problem on the box. Mixing those into one Grafana folder would either wake the monitoring stack for every flaky test or bury real host alerts in CI noise. We keep the clocks separate on purpose.
What Grafana is for here
The public entry is monitor.norditco.com. Behind nginx and the on-demand wake layer, that hostname brings up Grafana only when someone opens it. Idle timeout is short (on the order of fifteen minutes). Prometheus, Alertmanager and node-exporter stay up without the UI.
The provisioned home dashboard is Norditco - Ops overview. It is one screen, not twenty. At a glance it shows:
- RAM available and root disk free
- One-minute load and nginx requests per minute
- Count of firing alerts and whether scrapers are up
- Time series for memory and CPU
- Whether core Docker containers we care about are present, and how much memory they use
- Nginx request rate and connections
- A plain-language table of what is wrong right now
That is enough to answer “is the box sick?” and “which app is eating RAM?” without a tour of unused panels. Datasources are provisioned from git: Prometheus always, Loki when the logs profile is running.
The lean core underneath
Grafana without a philosophy for what stays warm is just a pretty client on an expensive server. Our monitoring compose file splits three postures.
| Posture | What runs | Why |
|---|---|---|
| Always on | Prometheus, Alertmanager, node-exporter | Host pulse and alerting must work while nobody has the UI open |
| On demand | Grafana UI | Humans need graphs rarely; RAM is scarce |
| Off by default | Loki/Promtail, cAdvisor, Pushgateway | Useful sometimes; too expensive to scrape forever on this host |
Scrape and rule evaluation run every 60 seconds. Prometheus retention is short (days, not months) and capped in size. Nginx metrics arrive through a host textfile that node-exporter reads, not through a second always-on exporter process if we can avoid it.
Pushgateway exists in the compose file under a heavy profile. We could push CI job results or batch gauges into it. We deliberately do not keep it warm. On a 2 GiB machine, “every pipeline posts a metric” is how you recreate the museum. When we want CI history we open the Actions tab on the repo. When we want host history we open Grafana.
Alerts: few rules, human words
Alertmanager is allowed to interrupt people. That privilege is expensive, so the rule file is short.
Host rules cover low available RAM, low and critical root disk, and sustained high load. Exporter rules cover a dead node or cAdvisor scrape when that job is enabled. App rules use absent(...) on named containers for the hot paths we refuse to discover by Twitter: Learn Persian, Midad, backend, WordPress, TTS, and nginx as reported by our textfile signal.
Annotations are written for a human in a hurry: “RAM available under 12%”, “Root disk under 8% free - urgent”, “learnpersian site container missing”. Resolutions are sent too. Grouping and inhibition exist so a cascade does not become a wall of duplicates. Alerts use a dedicated [alert] prefix so they do not drown in secretary chatter from the morning brief.
Grafana’s firing-alerts panel reads the same ALERTS series. The UI and Telegram agree on what “wrong” means because they share Prometheus rules, not because someone copy-pasted thresholds into a second system.
How this follows a change through the company
A concrete path for a Learn Persian change shows where Grafana sits.
- Edit and push. The agent or a human lands a commit on the product repo.
- Actions. CI runs unit tests and the rest of the gate. If Vitest fails, the story stops. Nobody should “check Grafana” for a red unit test.
- Deploy. A known pin is applied on the VPS. The health gate checks the contract for that app after a grace period. Sleeping on-demand apps are not treated as down.
- Runtime. Prometheus keeps scraping. If the Learn Persian container disappears, Alertmanager fires. If RAM collapses under load, the host rule fires.
- Investigation. Someone opens
monitor.norditco.com, waits for Grafana to wake, and uses the ops overview: memory curves, which containers are present, nginx rate, the alerts table. - Daily pulse. Even when nobody opens Grafana, the morning brief still carries health classes and hit counts so silence does not mean blindness.
Repos pipelines are therefore “followed” by CI status checks and deploy logs first. Grafana follows what the deployed system does to the machine after those pipelines succeed. Unit testing proves the change. Grafana and alerts prove the host still has room to run it.
What we refuse
Scraping every GitHub workflow into Prometheus. Actions already stores that history. Duplicating it on the customer VPS pays RAM for a second copy of a signal that is not operationally urgent at 3 a.m.
A dashboard per microservice aesthetic. We have few always-on services. One overview beats twelve empty graphs.
Always-on Grafana. The UI is a tax. On-demand wake keeps the tax near zero when nobody is looking, which matches how often we actually need clickable charts versus a Telegram line.
Alert spam. If a rule cannot be acted on, it does not belong in Alertmanager. Curiosity queries belong in Grafana when the UI is awake, or in a one-off promtool / HTTP query, not in a phone buzz.
Pretending Loki is free. Log aggregation is available as a profile. It is not part of the lean core. When we need logs we turn the profile on; we do not keep Promtail chewing the disk for week-old container noise by default.
Honest limits
The lean core does not give you year-long capacity planning charts. Retention is short. cAdvisor is not always scraping every container. Cold apps that are supposed to sleep will look “absent” unless the health model and the brief understand sleeping versus down (we taught the brief that distinction; naive absent alerts still need care for workloads that are allowed to be off).
Grafana also does not replace product analytics. Consent-gated GA4 on the hub and product sites answers product questions. Nginx hit digests answer coarse demand questions. Grafana answers machine and process questions.
Finally, opening monitor.norditco.com has a cold-start cost. That is acceptable. An ops UI that is always warm on a swapping host is a worse trade.
Why this still counts as embracing Grafana
We did not reject Grafana. We rejected a shape of Grafana that assumes infinite sidecar budget. The product we run is:
- a provisioned datasource and a single opinionated overview dashboard in git
- a Prometheus that stays up for alerts and briefs
- a UI that wakes like our other cold tools
- a clear boundary with GitHub Actions for unit tests and pipeline follow-up
That boundary is the design. Pipelines and unit tests earn their green check on the repo. The VPS earns quiet nights when RAM, disk, nginx and the hot app containers stay inside the few rules we bother to alert on. Grafana is the place we go when quiet ends and we need to see the curves.
