Ask the system a question
Observability
Metrics tell you that something changed, logs tell you why, and dashboards are only useful if they answer a question someone actually asks. An alert is a promise that a human will act.
Three signals, three jobs
Counters and rates (Prometheus) are cheap and good at trends; logs are expensive and good at detail; traces are the only signal that shows where time went across services. Reaching for logs to answer a trend question is how observability gets expensive.
Scrape configuration is where the story starts: a target that is not scraped has no metrics, and no amount of dashboarding will invent them.
Alerts need a duration
An alert without `for:` fires on a single scrape and wakes someone for a blip. An expression like `rate(http_requests_total{status="5xx"}[5m]) > 0.05` with `for: 2m` asks a real question: sustained error rate, not one bad sample.
Alert on the symptom users feel (errors, latency, saturation), not on the cause you happen to suspect — the cause changes, the promise does not.
Commands in this lesson
Every one of these runs in the sandbox — no real account, no real bill.
| Command | Why |
|---|---|
echo ' scrape_interval: 15s' >> /home/student/prometheus.yml | How often metrics are collected; every rate depends on it. |
echo ' - job_name: node' >> /home/student/prometheus.yml | Targets are grouped by job, so you can reason about a whole class. |
echo ' expr: rate(http_requests_total{status="5xx"}[5m]) > 0.05' >> /home/student/alerts.yml | A rate over a window, not a raw counter — counters only go up. |
echo ' for: 2m' >> /home/student/alerts.yml | Suppresses the single-sample blip that would otherwise page someone. |
Now do it
Reading the model makes the first attempt faster. Only the lab makes you able to do it.
- Open the lab
Observability: Read the Failure from the Logs
Intermediate · 25 min · members
- Open the lab
Observability: Prometheus Scrape Config and Alerting Rules
Intermediate · 25 min · members
- Open the lab
Observability: Create and Manage Grafana Dashboards
Intermediate · 25 min · members
Observability · OphirCloud Academy · all lessons