Prometheus Server: This component is responsible for scraping and storing time-series data. It collects metrics from monitored targets by periodically scraping HTTP endpoints, allowing for flexible querying and visualization.
Prometheus Client Libraries: These libraries are available for various programming languages, enabling instrumentation of application code to expose custom metrics. (Prometheus Exporter)
Storage: metrics are stored in a time-series database called the “TSDB” (Time Series Database). The TSDB is a core component of the Prometheus server. When Prometheus scrapes metrics from targets, it stores them locally in its TSDB.
The endpoint exposed for metrics /metrics
Metrics format
- Metrics entries: Type and HELP attributes
- Help → description of what the metrics is
- Type → Counter,Gauge,histogram
- Counter → how many times x happened
- Gauge → what is the current vaule of x now
- Histogram → how long?
PromQL: is the query language used in Prometheus for querying and analyzing time-series data
PromQL
Expression language data types
Scalar - a simple numeric floating point value
String - a simple string value; currently unused
Instant vector - a set of time series containing a single sample for each time series, all sharing the same timestamp
Example http_requests_total → will return elements for all time series that have this metric name. http_requests_total{job="prometheus",group="canary"} → with lables match
Range vector - a set of time series containing a range of data points over time for each time series
Example http_requests_total{job="prometheus"}[5m] → return recorded within the last 5 minutes for all time series
The Grafana Ecosystem — From First Principles
The core problem: you can’t fix what you can’t see
When you run software in production, things go wrong in ways that aren’t obvious. A voice agent starts responding slowly. A loan application API throws errors at 2am. A database runs out of connections. You need to answer three questions:
- What is happening right now? (metrics)
- Why did it happen? (logs)
- Where in the system did it happen? (traces)
These three questions map directly to the three pillars of observability. Grafana is the ecosystem built around collecting, storing, and visualizing all three.
Mental model: the three pillars
METRICS LOGS TRACES
───────── ──────── ────────
Numbers Text events Request journeys
over time over time across services
"Latency was "ERROR: timeout "Request took 2.3s
2.3s at 14:22" at 14:22:05" — 2.1s in DB query"
Each pillar tells a different story. They are most powerful when connected — you spot an anomaly in metrics, pivot to logs for context, then trace the exact request to find the culprit.
The Grafana ecosystem: bird’s eye view
┌─────────────────────────────────────────────────────────────┐
│ YOUR APPLICATION │
│ │
│ voice-agent loan-api db-proxy auth-service │
└────┬───────────────┬──────────────┬──────────────┬─────────┘
│ │ │ │
│ (metrics) │ (logs) │ (traces) │
▼ ▼ ▼ ▼
┌─────────┐ ┌───────────┐ ┌──────────┐ ┌──────────┐
│Prometheus│ │ Loki │ │ Tempo │ │ Mimir │
│(metrics │ │ (log │ │ (trace │ │(long-term│
│ store) │ │ store) │ │ store) │ │ metrics) │
└────┬────┘ └─────┬─────┘ └────┬─────┘ └────┬─────┘
│ │ │ │
└──────────────┴─────────────┴──────────────┘
│
▼
┌─────────────┐
│ GRAFANA │
│ │
│ Dashboards │
│ Explore │
│ Alerts │
└─────────────┘
Grafana itself is just the visualization and querying layer. It doesn’t store anything. Everything else — Prometheus, Loki, Tempo, Mimir — is a data source that Grafana reads from.
Prometheus — the metrics engine
What it is
Prometheus is a time-series database. It stores numbers indexed by time and labels. Every data point is a tuple of (metric_name, labels, timestamp, value).
http_request_duration_seconds{service="voice-agent", route="/call", status="200"} 0.234 1718000000
http_request_duration_seconds{service="voice-agent", route="/call", status="500"} 2.100 1718000000
Why it exists
Before Prometheus, you’d log metrics to flat files or push numbers to a central server. Both approaches have problems — files get large, push-based systems are brittle. Prometheus inverted this: it pulls metrics from your application on a schedule, typically every 15 seconds. This is called scraping.
How scraping works
Your application exposes a /metrics endpoint in a plain-text format:
# Your voice agent's /metrics endpoint
# HELP call_latency_seconds Time to respond to a call
# TYPE call_latency_seconds histogram
call_latency_seconds_bucket{le="0.5"} 1024
call_latency_seconds_bucket{le="1.0"} 1536
call_latency_seconds_bucket{le="2.0"} 1598
call_latency_seconds_bucket{le="+Inf"} 1600
call_latency_seconds_sum 843.2
call_latency_seconds_count 1600
Prometheus reads this every 15s and stores it. You never push — Prometheus comes to you.
The four metric types
Counter — only goes up. Total calls handled, total errors, total bytes sent.
voice_calls_total{result="answered"} 14203
voice_calls_total{result="dropped"} 47
Gauge — goes up and down. Current active calls, current memory usage.
voice_calls_active 12
memory_used_bytes 4.2e9
Histogram — tracks distribution of values. How many requests fell into each latency bucket.
call_latency_seconds_bucket{le="0.5"} 1024 ← 1024 calls under 500ms
call_latency_seconds_bucket{le="1.0"} 1536 ← 1536 calls under 1s
Summary — pre-computed percentiles (less flexible than histograms, avoid for new work).
PromQL — the query language
You query Prometheus using PromQL. Some examples:
# Current active calls
voice_calls_active
# Rate of errors per second (over last 5 minutes)
rate(voice_calls_total{result="error"}[5m])
# 95th percentile latency
histogram_quantile(0.95, rate(call_latency_seconds_bucket[5m]))
# Error rate as percentage
rate(voice_calls_total{result="error"}[5m])
/
rate(voice_calls_total[5m])
* 100What Prometheus doesn’t do
Prometheus keeps data for a short window (typically 15 days). It’s designed for current operational health, not long-term trend analysis. For that, you need Mimir.
Loki — the log aggregator
What it is
Loki is a log storage and querying system. Think of it as “Prometheus for logs.” It deliberately doesn’t index the content of logs — it only indexes labels. The log lines themselves are stored compressed. This makes it cheap to run at scale.
Why it exists
The traditional approach to logs was Elasticsearch (or ELK stack). Elasticsearch indexes every word in every log line, which makes full-text search fast but storage expensive. Loki trades some query flexibility for much lower cost — it’s designed for cloud-native environments where you’re generating gigabytes of logs per day.
How logs get into Loki
You run a small agent called Promtail (or the newer Grafana Alloy) on each machine. Promtail tails log files, attaches labels, and ships the logs to Loki.
[Your voice-agent process]
│
│ writes to stdout / file
▼
[Promtail agent]
│ reads logs, attaches labels:
│ {app="voice-agent", env="prod", region="ap-south-1"}
▼
[Loki]
│
│ stores: label index + compressed log chunks
LogQL — the query language
Loki’s query language is called LogQL. It has two parts: a stream selector (using labels) and an optional filter.
# All logs from the voice agent in prod
{app="voice-agent", env="prod"}
# Filter to just errors
{app="voice-agent", env="prod"} |= "ERROR"
# Filter to timeout errors specifically
{app="voice-agent", env="prod"} |= "ERROR" |= "timeout"
# Parse JSON logs and filter by field
{app="voice-agent"} | json | latency_ms > 2000
# Count errors per minute (metric query)
rate({app="voice-agent"} |= "ERROR" [1m])Correlation with Prometheus
The critical feature is that if Prometheus and Loki use the same labels (app, env, region), Grafana can link them. You’re looking at a latency spike in a metric graph → you click → you’re immediately looking at the logs from that service during that exact time window.
Tempo — the distributed tracing backend
What distributed tracing is
When a user makes a call to your voice agent, that single request might touch many services:
User dials in
│
▼
[SIP gateway] — 5ms
│
▼
[voice-agent] — 2300ms total
├──► [LLM API] — 1800ms ← the bottleneck
├──► [auth-service] — 45ms
└──► [DB: fetch script] — 420ms
A trace records this entire journey as a tree of spans. Each span has a start time, end time, service name, and metadata. Traces are connected by a trace_id — a random ID that gets passed from service to service in HTTP headers.
How tracing works in practice
Your application uses an OpenTelemetry SDK. It automatically instruments HTTP calls, database queries, and anything you manually instrument. Every outgoing HTTP request gets a traceparent header injected. Every service extracts this header and creates a child span.
trace_id: 4bf92f3577b34da6a3ce929d0e0e4736
Span 1: voice-agent.handle_call [0ms ────────────────── 2300ms]
Span 2: auth-service.verify [5ms ──── 50ms]
Span 3: db.fetch_script [55ms ──────────────── 475ms]
Span 4: llm-api.generate_response [480ms ──────────────────────── 2280ms]
Now you can see immediately that the LLM API took 1800ms of a 2300ms total — not the database, not auth.
Why Tempo specifically
Tempo is Grafana’s trace storage backend. Like Loki, it’s designed to be cheap — it stores traces in object storage (S3, GCS) rather than a database. You can store billions of spans at low cost. The tradeoff: Tempo doesn’t let you search by arbitrary span attributes without additional indexing (Tempo search uses a component called the TraceQL metrics generator for this).
TraceQL — the query language
# Find traces where total duration > 2 seconds
{ duration > 2s }
# Find traces with an error in the voice-agent
{ .service.name = "voice-agent" && status = error }
# Find traces where the LLM call took over 1 second
{ .service.name = "llm-api" && duration > 1s }Mimir — long-term metrics storage
What it is
Mimir is a horizontally scalable, long-term metrics storage system. It is API-compatible with Prometheus, meaning you query it with PromQL exactly as you would Prometheus. The key differences:
- Prometheus is typically a single process; Mimir is a distributed system that scales horizontally
- Prometheus retains data for days to weeks; Mimir retains for months to years
- Mimir stores data in object storage (S3, GCS); Prometheus stores locally on disk
The relationship between Prometheus and Mimir
The most common setup has Prometheus scraping metrics and Mimir acting as long-term remote storage. Prometheus’s remote_write config ships metrics to Mimir in real time:
Prometheus (scrapes every 15s)
│
│ remote_write
▼
Mimir (stores forever, object storage)
│
│ queried by Grafana using PromQL
▼
Grafana
Prometheus becomes just a scraper and short-term buffer. Mimir becomes the single source of truth for historical metrics.
For smaller deployments, you skip Mimir entirely and use Prometheus directly. For a startup running voice agents for fintech clients, you’d likely start with Prometheus and add Mimir when you need more than 30-90 days of retention or want to run multiple Prometheus instances without data silos.
Grafana itself — the visualization layer
What it is
Grafana is a web application that connects to data sources and renders dashboards. It stores nothing itself (except dashboard configurations, alert rules, and user settings — in a small SQLite or Postgres database). All the data stays in Prometheus, Loki, Tempo, or Mimir.
Data sources
You configure data sources in Grafana’s settings:
Grafana Data Sources:
├── Prometheus → http://prometheus:9090
├── Loki → http://loki:3100
├── Tempo → http://tempo:3200
└── Mimir → http://mimir:9009
Every panel in a dashboard is backed by a query against one of these data sources.
Dashboards
What they are
A dashboard is a collection of panels arranged on a grid. Each panel runs a query and renders the result as a graph, table, stat number, heatmap, or other visualization.
┌─────────────────────────────────────────────────────────┐
│ Voice Agent Dashboard │
│ │
│ ┌───────────┐ ┌───────────┐ ┌───────────┐ │
│ │ Active │ │ Error │ │ p95 │ │
│ │ Calls: 12 │ │ Rate: 0.2%│ │ Latency: │ │
│ │ │ │ │ │ 1.2s │ │
│ └───────────┘ └───────────┘ └───────────┘ │
│ │
│ ┌─────────────────────────────────────────────────┐ │
│ │ Call latency over time (line graph) │ │
│ │ 2s ┤ ╭──╮ │ │
│ │ 1s ┤──────────────╯ ╰────────────── │ │
│ │ 0.5s┤ │ │
│ └─────────────────────────────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────────┐ │
│ │ Log volume by severity (bar graph) │ │
│ └─────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────┘
Dashboards are defined in JSON and can be version-controlled. A dashboard for a voice agent might have:
- Stat panels: active calls, error rate, p95 latency — showing current state at a glance
- Time series graphs: latency trend, call volume, error rate over time
- Log panels: recent ERROR log lines from Loki
- Table panels: top slowest routes in the last hour
Template variables
Dashboards support variables that act as filters. You might have a $service variable so the same dashboard works for every service:
PromQL with variable:
rate(http_requests_total{service="$service"}[5m])
You’d select voice-agent from a dropdown and the entire dashboard filters to that service.
Explore
What it is
Explore is Grafana’s ad-hoc query interface — a scratchpad for investigation. You go here when you don’t have a pre-built dashboard for what you’re looking at, or when you’re chasing down an incident.
In Explore you can:
- Write arbitrary PromQL queries and see the results
- Write LogQL queries and see raw log lines
- Write TraceQL queries and see trace waterfalls
- Switch between data sources without navigating away
- Use the correlation feature to move between metrics, logs, and traces for the same service/time window
Explore is where most incident investigation actually happens. Dashboards give you the overview; Explore is where you dig.
Alerts
How alerts work
Grafana’s alert system evaluates queries on a schedule. When a query crosses a threshold, an alert fires and is sent to a notification channel (PagerDuty, Slack, email, etc.).
Alert rule:
Query: histogram_quantile(0.95, rate(call_latency_seconds_bucket[5m])) > 2
Condition: value > 2 for 5 minutes
Labels: severity=critical, team=voice
Notify: PagerDuty channel
Grafana has two alert evaluation engines:
Grafana-managed alerts — Grafana itself evaluates the rules and sends notifications. Good for small setups.
Prometheus Alertmanager — Prometheus evaluates alert rules using PromQL, then sends firing alerts to Alertmanager, which handles routing, deduplication, grouping, and silencing. More powerful for complex environments.
A typical alerting architecture:
Prometheus
│ evaluates rules every 15s
│ fires alert if condition met for >5m
▼
Alertmanager
│ groups related alerts
│ routes by label (team=voice → voice-slack-channel)
│ deduplicates (don't page 100x for same issue)
│ silences during maintenance windows
▼
PagerDuty / Slack / Email
How everything connects — the complete picture
┌──────────────────────────────────────────────────────────────────────┐
│ YOUR SERVICES │
│ │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ voice-agent │ │ llm-proxy │ │ auth-svc │ │
│ │ │ │ │ │ │ │
│ │ /metrics │ │ /metrics │ │ /metrics │ │
│ │ stdout logs │ │ stdout logs │ │ stdout logs │ │
│ │ OTel SDK │ │ OTel SDK │ │ OTel SDK │ │
│ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ │
└─────────┼─────────────────┼─────────────────┼────────────────────┘
│ │ │
metrics│ logs │ traces │
│ │ │
▼ ▼ ▼
┌──────────┐ ┌──────────────┐ ┌──────────┐
│Prometheus│ │ Promtail │ │ OTel │
│(scrapes │ │ (tails logs,│ │Collector │
│ /metrics)│ │ ships to │ │(receives │
└────┬─────┘ │ Loki) │ │ spans, │
│ └──────┬───────┘ │ ships to │
│ │ │ Tempo) │
│ remote_write │ └────┬─────┘
▼ ▼ ▼
┌──────────┐ ┌──────────┐ ┌──────────┐
│ Mimir │ │ Loki │ │ Tempo │
│(long-term│ │(log store│ │(trace │
│ metrics) │ │ │ │ store) │
└────┬─────┘ └────┬─────┘ └────┬─────┘
│ │ │
└───────────────┴───────────────┘
│
▼
┌─────────────┐
│ GRAFANA │
│ │
│ Dashboards │─────► Operations team
│ Explore │─────► On-call engineer
│ Alerts │─────► PagerDuty/Slack
└─────────────┘
Real-world walkthrough: debugging a slow voice agent
Let’s say you run a voice agent for The Federal Savings Bank. Callers are complaining that the agent takes too long to respond after they speak. You get a PagerDuty alert at 2:17am.
Step 1 — the alert fires
Alert: voice_agent_p95_latency > 2s
Value: 3.4s (threshold: 2s)
Duration: firing for 8 minutes
Labels: service=voice-agent, env=prod
You open Grafana. You go to the Voice Agent dashboard.
Step 2 — metrics give you the shape
The time series panel shows latency was normal (0.8s p95) until 02:09, then spiked to 3.4s. It’s still elevated. Error rate is slightly up (0.4% → 1.2%). Active calls are normal at 14.
You run a PromQL query to narrow it down:
histogram_quantile(0.95,
rate(call_latency_seconds_bucket{service="voice-agent"}[5m])
)
by (route)This breaks latency down by route. The result shows:
/call/respond → 3.4s ← this one
/call/start → 0.3s (normal)
/call/end → 0.1s (normal)
The slowness is specific to /call/respond — the route that calls the LLM to generate a response. You now have a hypothesis: the LLM call is slow.
Step 3 — logs give you context
You click the Loki correlation button on the metrics panel. Grafana automatically opens Explore, scoped to the same service and time window.
You run:
{service="voice-agent", env="prod"} |= "respond" | json
| __error__ = ""
| duration_ms > 2000You see entries like:
2026-06-12 02:09:43 WARN LLM call slow {duration_ms: 3100, model: "gpt-4o", tokens_in: 2840}
2026-06-12 02:09:55 WARN LLM call slow {duration_ms: 2900, model: "gpt-4o", tokens_in: 2790}
2026-06-12 02:10:12 WARN LLM call slow {duration_ms: 3400, model: "gpt-4o", tokens_in: 3100}
Two things stand out: all the slow calls are going to gpt-4o, and the tokens_in values are around 2800-3100 — higher than usual. You check earlier logs:
2026-06-12 01:40:00 INFO LLM call ok {duration_ms: 780, model: "gpt-4o", tokens_in: 1200}
Token count nearly tripled. That’s the change that happened around 02:09. Someone likely deployed a longer system prompt.
Step 4 — traces show you exactly where time went
You still want to confirm. You switch to Tempo in Explore:
{ .service.name = "voice-agent" && duration > 2s }You pick one of the returned traces and open the waterfall view:
voice-agent.handle_respond [0ms ──────────────────── 3380ms]
├── auth.verify_token [2ms ─── 18ms]
├── db.fetch_conversation_history [20ms ────── 95ms]
├── prompt.build [96ms ──── 130ms]
└── llm.generate [131ms ──────────────────── 3350ms]
├── tokens_in: 2840
├── tokens_out: 312
└── model: gpt-4o
The LLM call consumed 3219ms of a 3380ms total. Everything else is healthy. The trace even shows the token counts as span attributes — matching exactly what the logs said.
Step 5 — confirm and fix
You check your deployment history. At 02:08, a teammate deployed a prompt update that added a detailed “conversation history recap” section — explaining loan terms to borrowers in full at every turn. That tripled input tokens.
You roll back the prompt, and within 2 minutes the Grafana dashboard shows latency returning to 0.9s. You write a post-mortem and add a Prometheus alert for tokens_in > 1500 as an early warning.
The mental model, summarised
SIGNAL TOOL QUERY LANGUAGE GOOD FOR
────── ──── ────────────── ────────
Numbers Prometheus PromQL Is something wrong?
Mimir How wrong? Since when?
Text events Loki LogQL Why is it wrong?
What exactly happened?
Request paths Tempo TraceQL Where is the slowness?
Which service? Which call?
Grafana ties all three together. Dashboards give you the always-on view. Explore gives you the investigation scratchpad. Alerts wake you up when you need to look. The correlation between signals — clicking from a metric spike directly to the relevant logs, then from a log line to the full trace — is what makes the ecosystem more than the sum of its parts.
The design principle behind all of it: instrument once, query flexibly. You add OpenTelemetry to your services, run Promtail for logs, expose a /metrics endpoint — and then you can ask any question about your system’s behaviour without changing the application code again.
The Grafana Ecosystem — From First Principles
A complete mental model of observability: metrics, logs, traces, and how Grafana ties them together.
0. Why Does Any of This Exist?
You deploy a voice agent. A caller calls in. The agent responds slowly. Maybe it times out. Maybe it halluccinates. Maybe it drops the call entirely.
You get a Slack message: “Something’s broken.”
Now what?
The problem is that a running software system is a black box. You can’t see inside it. You can only observe it from the outside — through the signals it emits. Those signals are:
| Signal | What it tells you | Example |
|---|---|---|
| Metrics | How much / how often / how fast | p99 latency = 3.2s, error_rate = 4% |
| Logs | What happened, literally | ERROR: LLM provider timeout after 5000ms |
| Traces | Where time was spent, end-to-end | Span tree showing ASR → LLM → TTS breakdown |
Grafana is the unified observation platform that lets you collect, store, query, and visualize all three — and then alert on them.
1. The Big Picture
┌─────────────────────────────────────────────────────────────────────┐
│ YOUR APPLICATION │
│ │
│ Voice Agent ──► emits ──► Metrics (counters, gauges, timers) │
│ ──► Logs (structured JSON lines) │
│ ──► Traces (spans with timing + context) │
└─────────────────┬───────────────────┬──────────────────┬───────────┘
│ │ │
▼ ▼ ▼
┌──────────┐ ┌──────────┐ ┌──────────┐
│Prometheus│ │ Loki │ │ Tempo │
│(or Mimir)│ │ │ │ │
│ METRICS │ │ LOGS │ │ TRACES │
└────┬─────┘ └────┬─────┘ └────┬─────┘
│ │ │
└───────────────────┴──────────────────┘
│
▼
┌──────────────────┐
│ GRAFANA │
│ (Visualization, │
│ Alerting, Explore│
└──────────────────┘
Each column (metrics / logs / traces) has its own collection, storage, and query story. Grafana is the single UI that talks to all of them.
2. Metrics — The Heartbeat of Your System
What is a metric?
A metric is a number measured over time. That’s it. Numbers are cheap to store and fast to query. You can have millions of them.
# At time 14:03:01
voice_agent_call_duration_seconds{agent="tfsb-inbound", status="completed"} = 18.4
# At time 14:03:11
voice_agent_call_duration_seconds{agent="tfsb-inbound", status="completed"} = 22.1
The curly-brace part {agent="tfsb-inbound", status="completed"} is called a label (or tag). Labels are how you slice and filter metrics.
The four metric types
| Type | Meaning | Example |
|---|---|---|
| Counter | Goes up only, never down | Total calls handled |
| Gauge | Goes up and down | Active concurrent calls right now |
| Histogram | Buckets of observations (for percentiles) | Call duration distribution |
| Summary | Pre-computed percentiles | p50/p95/p99 latency |
For voice agents, you’d typically have:
# Counter: how many calls have been handled
voice_calls_total{agent="tfsb", result="completed"} 1402
voice_calls_total{agent="tfsb", result="abandoned"} 38
voice_calls_total{agent="tfsb", result="error"} 12
# Gauge: live concurrency
voice_calls_active{agent="tfsb"} 7
# Histogram: latency distribution
voice_llm_latency_seconds_bucket{le="0.5"} 310
voice_llm_latency_seconds_bucket{le="1.0"} 890
voice_llm_latency_seconds_bucket{le="2.0"} 1350
voice_llm_latency_seconds_bucket{le="+Inf"} 1402
3. Prometheus — Metrics Collection and Storage
What is Prometheus?
Prometheus is a time-series database and scraper. It:
- Scrapes your app’s
/metricsHTTP endpoint every N seconds (default: 15s) - Stores the data in a compressed on-disk format (TSDB)
- Evaluates alerting rules
- Exposes a query language called PromQL
How does scraping work?
┌──────────────────────────────────────────────────────┐
│ Prometheus (runs on its own server/container) │
│ │
│ Every 15s: │
│ GET http://voice-agent:8080/metrics │
│ GET http://another-service:9090/metrics │
│ GET http://node-exporter:9100/metrics │
└──────────────────────────────────────────────────────┘
▲ ▲ ▲
│ │ │
┌────────┴──┐ ┌───────┴──┐ ┌──────┴──────┐
│Voice Agent│ │Other Svc │ │Node Exporter│
│/metrics │ │/metrics │ │(host metrics│
└───────────┘ └──────────┘ │like CPU/RAM)│
└─────────────┘
Your app exposes a /metrics endpoint. Prometheus pulls from it. This pull model is important — Prometheus is in charge of the scrape interval. The app just needs to keep its current state ready.
What does /metrics look like?
# HELP voice_calls_total Total voice calls handled
# TYPE voice_calls_total counter
voice_calls_total{agent="tfsb",result="completed"} 1402
voice_calls_total{agent="tfsb",result="error"} 12
# HELP voice_llm_latency_seconds LLM response latency
# TYPE voice_llm_latency_seconds histogram
voice_llm_latency_seconds_bucket{le="0.5"} 310
voice_llm_latency_seconds_bucket{le="1.0"} 890
voice_llm_latency_seconds_sum 1893.2
voice_llm_latency_seconds_count 1402
In Node.js, you’d use the prom-client library. In Python, prometheus_client. Both expose the /metrics endpoint automatically.
PromQL — the query language
PromQL is how you ask questions of Prometheus. Examples:
# Error rate as a percentage
rate(voice_calls_total{result="error"}[5m])
/ rate(voice_calls_total[5m]) * 100
# 99th percentile latency
histogram_quantile(0.99, rate(voice_llm_latency_seconds_bucket[5m]))
# Active calls per agent
voice_calls_activerate() converts a counter into a per-second rate. [5m] is a lookback window. histogram_quantile() computes percentiles from bucket data.
4. Mimir — Prometheus at Scale
Prometheus stores data on disk, on the same machine it runs on. That works up to a point. Problems start when:
- You need to store months/years of data
- You have multiple Prometheus instances (one per data center)
- You need high availability (Prometheus goes down = gap in data)
Mimir solves all three. It’s a horizontally scalable, highly available, long-term storage backend for Prometheus metrics.
┌─────────────┐ ┌─────────────┐
│ Prometheus │─────►│ Mimir │◄── Grafana queries here
│ (region 1) │ │ (distributed│ instead of Prometheus
└─────────────┘ │ cluster) │
│ │
┌─────────────┐ │ - S3/GCS │
│ Prometheus │─────►│ backend │
│ (region 2) │ │ - months │
└─────────────┘ │ of data │
└─────────────┘
Mimir is Prometheus-compatible — it speaks the same query API (PromQL). If you have a single machine and short retention, Prometheus alone is fine. When you grow, you point Prometheus to remote-write to Mimir.
For most voice agent deployments at early stage: just use Prometheus.
5. Logs — The Narrative of What Happened
What is a log?
A log is a timestamped text line emitted by your application. Logs tell you what happened, in human-readable form.
2024-01-15T14:03:01.234Z INFO [call:abc123] Call started, caller=+91984...
2024-01-15T14:03:02.100Z INFO [call:abc123] ASR transcribed: "I want to apply for a home loan"
2024-01-15T14:03:02.150Z INFO [call:abc123] Intent detected: loan_inquiry
2024-01-15T14:03:04.891Z WARN [call:abc123] LLM response took 2741ms (threshold: 2000ms)
2024-01-15T14:03:04.900Z INFO [call:abc123] Agent said: "Sure, I'd be happy to help..."
2024-01-15T14:03:28.000Z ERROR [call:abc123] LLM provider timeout after 5000ms
2024-01-15T14:03:28.001Z INFO [call:abc123] Transferring to human agent
Structured logs (JSON) are better for querying:
{"ts":"2024-01-15T14:03:28.000Z","level":"error","call_id":"abc123","event":"llm_timeout","duration_ms":5000,"provider":"openai"}Loki — Log Storage and Querying
Loki is Grafana’s log aggregation system. Its design philosophy is the opposite of Elasticsearch: it does not index log content. Instead it:
- Indexes only the labels (like
{agent="tfsb", env="prod"}) - Stores log content compressed, unindexed
- Searches log content by scanning within a label-filtered stream
This makes Loki very cheap to operate but slower for full-text search than Elasticsearch.
┌────────────────────────────────────────────┐
│ Your App │
│ console.log(JSON.stringify({...})) │
└──────────────────────┬─────────────────────┘
│ stdout/stderr
▼
┌─────────────────┐
│ Promtail │ ← Log shipper (runs as sidecar/agent)
│ (or Alloy) │ Tails log files, adds labels,
└────────┬────────┘ ships to Loki
│
▼
┌─────────────────┐
│ Loki │ ← Stores & indexes logs
└─────────────────┘
│
▼
┌─────────────────┐
│ Grafana │ ← Query with LogQL
└─────────────────┘
LogQL — the log query language
LogQL has two parts: a log selector (label filter) and a pipeline (transformations).
# All error logs from the TFSB agent
{agent="tfsb", env="prod"} |= "ERROR"
# All logs for a specific call ID
{agent="tfsb"} | json | call_id="abc123"
# Count errors per minute (log metric)
sum by (agent) (
rate({env="prod"} |= "ERROR" [1m])
)
# Extract LLM latency from logs and compute p99
{agent="tfsb"} | json | unwrap duration_ms
| histogram_quantile(0.99, sum by (agent) (rate([5m])))The | json pipeline parses each line as JSON and makes fields queryable. |= is substring match. | line_format reformats the output.
6. Traces — The Thread Through a Request
What is a trace?
A trace answers: “Where did time go, end-to-end, for a single request?”
A trace is made of spans. A span = “this unit of work took X milliseconds.” Spans are nested in a tree:
Trace: call_id=abc123 (total: 8.4s)
│
├── [span] inbound_call_handler 0ms → 8400ms
│ │
│ ├── [span] ASR transcription 0ms → 400ms
│ │ └── [span] deepgram_api 10ms → 390ms
│ │
│ ├── [span] intent_detection 400ms → 550ms
│ │
│ ├── [span] llm_inference 550ms → 5800ms ← SLOW
│ │ ├── [span] prompt_build 550ms → 600ms
│ │ └── [span] openai_api 600ms → 5800ms ← THE CULPRIT
│ │
│ └── [span] tts_synthesis 5800ms → 8400ms
Immediately you see: the OpenAI API call took 5.2 seconds. That’s your problem.
How are traces collected?
Your app uses an OpenTelemetry SDK to create spans. OpenTelemetry (OTel) is the industry standard for instrumentation.
const { trace } = require('@opentelemetry/api');
const tracer = trace.getTracer('voice-agent');
async function handleLLMInference(prompt) {
const span = tracer.startSpan('llm_inference');
span.setAttribute('provider', 'openai');
span.setAttribute('model', 'gpt-4o');
try {
const result = await callOpenAI(prompt);
span.setStatus({ code: SpanStatusCode.OK });
return result;
} catch (err) {
span.recordException(err);
span.setStatus({ code: SpanStatusCode.ERROR });
throw err;
} finally {
span.end(); // This records the duration
}
}Tempo — Trace Storage
Tempo is Grafana’s distributed tracing backend. It:
- Receives spans from your app (via OTel Collector or directly)
- Stores traces in object storage (S3, GCS, local disk)
- Lets you query by
trace_id
Your App
│ (OTLP gRPC/HTTP)
▼
┌─────────────────────┐
│ OTel Collector │ ← Optional but recommended: batches, samples, routes
│ (gateway) │
└──────────┬──────────┘
│
▼
┌─────────────────────┐
│ Tempo │ ← Stores traces
└─────────────────────┘
│
▼
┌─────────────────────┐
│ Grafana │ ← Visualizes traces as span trees
└─────────────────────┘
Tempo is query-by-trace-ID only in its basic form. You don’t search Tempo like you search logs. Instead, you typically:
- Find a slow request in metrics (e.g., a spike in p99 latency)
- Find the specific call in logs (filter to that time window, get the
call_id) - Use the
trace_idfrom logs to look up the full trace in Tempo
This flow is called correlating signals — and Grafana makes it seamless.
7. Grafana Alloy — The Modern Collector
Alloy (formerly Grafana Agent) is a single binary that replaces Promtail + OTel Collector + Prometheus scraper in one. It:
- Scrapes Prometheus metrics and remote-writes to Prometheus/Mimir
- Tails logs and ships to Loki
- Receives/forwards traces to Tempo
- Runs as a sidecar or DaemonSet in Kubernetes
┌─────────────────────────────────────────────┐
│ Grafana Alloy │
│ │
│ prometheus.scrape → remote_write (Mimir) │
│ loki.source.file → loki.write │
│ otelcol.receiver → otelcol.exporter │
└─────────────────────────────────────────────┘
It’s configured in a declarative “River” language (now called Alloy config syntax). For most teams starting out, you can still use Promtail + a simple OTel Collector as separate components.
8. Grafana — The Visualization and Navigation Layer
Grafana is the UI. It connects to data sources (Prometheus, Loki, Tempo, etc.) and lets you build dashboards, run ad-hoc queries, and set up alerts.
Data Sources
Everything in Grafana starts with a data source. You configure each backend once:
Grafana Settings → Data Sources
├── Prometheus → http://prometheus:9090
├── Loki → http://loki:3100
├── Tempo → http://tempo:3200
└── (optional) PostgreSQL, Elasticsearch, CloudWatch, etc.
Once configured, every panel, Explore query, and alert rule uses a named data source.
Dashboards
A dashboard is a collection of panels. Each panel is a query + a visualization type.
┌────────────────────────────────────────────────────────────┐
│ Dashboard: TFSB Voice Agent Overview │
│ │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────────┐ │
│ │ Active Calls │ │ Error Rate │ │ LLM p99 Latency │ │
│ │ [gauge] │ │ [time series│ │ [time series] │ │
│ │ 7 │ │ 3.2% │ │ 2.1s │ │
│ └──────────────┘ └──────────────┘ └──────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ Call Volume (last 24h) │ │
│ │ [bar chart] │ │
│ └─────────────────────────────────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ Recent Errors (Loki log panel) │ │
│ │ 14:03:28 ERROR call=abc123 LLM timeout 5000ms │ │
│ │ 14:01:14 ERROR call=xyz789 TTS provider error │ │
│ └─────────────────────────────────────────────────────┘ │
└────────────────────────────────────────────────────────────┘
Dashboard panels are linked together with variables — for example, a dropdown to select agent filters every panel simultaneously.
The Explore Page
Explore is Grafana’s ad-hoc investigation page. Unlike dashboards (which are pre-built), Explore is where you dig into a problem in real time.
Explore page:
├── Select data source: Prometheus
│ Query: histogram_quantile(0.99, rate(voice_llm_latency_seconds_bucket[5m]))
│ → See the spike at 14:03
│
├── Switch to: Loki
│ Query: {agent="tfsb"} | json | level="error"
│ → Find the specific error log lines
│
└── Switch to: Tempo
Query: trace_id=<from log>
→ See the full span tree for that one call
You can also use Explore’s split view — Prometheus on the left, Loki on the right, same time window, so you can correlate without switching.
Correlations — Jumping Between Signals
Grafana supports data links and correlations that let you click through from one signal to another:
- In a Loki log line: click
trace_id→ jumps to Tempo trace - In a Tempo span: “View logs for this service” → jumps to Loki filtered by service + time
- In a Prometheus panel: click a spike → opens Loki Explore at that exact timestamp
This is the real power of the stack: you can follow a problem from “something is slow” (metric) → “which calls?” (logs) → “why?” (trace).
9. Alerting
Grafana’s unified alerting engine lets you define rules that fire when a condition is true.
Alert Rule Anatomy
Name: High LLM Latency
Data source: Prometheus
Query: histogram_quantile(0.99, rate(voice_llm_latency_seconds_bucket[5m]))
Condition: IS ABOVE 2.0
For: 2m ← Must be true for 2 consecutive minutes before firing
Labels:
severity: warning
team: voice-platform
Annotations:
summary: "LLM p99 latency above 2s for {{ $labels.agent }}"
runbook_url: https://wiki/runbooks/llm-latencyAlerts flow through:
Alert Rule (evaluates every 1m)
│
▼
Alert State: Normal → Pending (condition met) → Firing
│
▼
Contact Points:
├── Slack: #voice-alerts channel
├── PagerDuty: oncall rotation
└── Email: team@company.com
Notification Policies route alerts to the right contact point based on labels. E.g., severity=critical goes to PagerDuty, severity=warning goes to Slack.
10. The Voice Agent Investigation — Step by Step
You get a message: “The TFSB agent is slow and dropping calls. Callers are getting transferred without resolution.”
Here’s exactly how you’d investigate:
Step 1: Dashboard — Orient
Open the TFSB Voice Agent dashboard. You see:
LLM p99 Latency: ████████████ 5.8s ← was 0.9s an hour ago
Error Rate: ██ 8.2% ← was 0.4% an hour ago
Active Calls: ███ 12 ← normal
Call Abandonment: ████ 14.3% ← was 2.1%
Something broke around 14:00. Latency spiked, errors rose, callers are abandoning.
Step 2: Metrics — Narrow the Signal
On the Explore page, with Prometheus:
# Break down errors by type
rate(voice_calls_total{result="error"}[5m]) by (error_type)Output shows llm_timeout is the dominant error type — not ASR, not TTS, not telephony. The LLM provider is the culprit.
Check if it’s all agents or just TFSB:
histogram_quantile(0.99, rate(voice_llm_latency_seconds_bucket[5m])) by (agent)Only agent="tfsb" is slow. Interesting — so it’s not a provider-wide outage. Something specific to TFSB’s configuration or model.
Step 3: Logs — Find the Error Pattern
Switch to Loki:
{agent="tfsb", env="prod"} | json | level="error" | __error__!="JSONParserErr"You see a cluster of:
14:03:28 ERROR call=abc123 event=llm_timeout duration_ms=5000 model=gpt-4o prompt_tokens=3847
14:04:01 ERROR call=def456 event=llm_timeout duration_ms=5000 model=gpt-4o prompt_tokens=3901
14:04:15 ERROR call=ghi789 event=llm_timeout duration_ms=5000 model=gpt-4o prompt_tokens=3812
Pattern: prompt_tokens is in the 3800–3900 range. Normally it’s ~800. The prompt grew by ~5x. And they’re all hitting the 5-second timeout exactly — OpenAI is responding, just too slowly for long prompts.
You check a successful call from 13:50:
13:50:12 INFO call=jkl012 event=llm_response duration_ms=820 model=gpt-4o prompt_tokens=794
Confirmed: something changed in the prompt around 14:00 that made it ballooning. A deployment happened at 13:58.
Step 4: Traces — Confirm the Breakdown
Click the trace_id in one of the error log lines. Tempo shows:
Trace: call=abc123 (total: 23.1s, status: ERROR)
│
├── [span] inbound_call_handler 0ms → 23100ms
│ │
│ ├── [span] asr_transcription 0ms → 380ms ✓
│ ├── [span] intent_detection 380ms → 510ms ✓
│ │
│ ├── [span] prompt_build 510ms → 720ms
│ │ Attribute: template_version="v2.3"
│ │ Attribute: prompt_tokens=3847 ← huge
│ │
│ ├── [span] llm_inference 720ms → 5720ms ✗ TIMEOUT
│ │ Attribute: model=gpt-4o
│ │ Attribute: status=timeout
│ │
│ └── [span] transfer_to_human 5720ms → 23100ms
The trace makes it unambiguous: prompt_build is generating a 3847-token prompt, which causes llm_inference to time out.
Step 5: Root Cause and Fix
- Check the deployment at 13:58: a developer accidentally changed the
template_versionfromv2.1tov2.3. - v2.3 includes the full borrower loan history in the system prompt on every turn — meant for a different use case.
- Rolling back to
v2.1fixes the issue within minutes.
Step 6: Alert So You Find It Faster Next Time
Alert: prompt_tokens > 1500 for 3 consecutive requests
Alert: llm_timeout_rate > 2% for 2 minutes
11. Component Summary Table
| Component | Stores | Query Language | Best For |
|---|---|---|---|
| Prometheus | Metrics (short-term) | PromQL | Fast queries, alerting, active scraping |
| Mimir | Metrics (long-term, scaled) | PromQL | Multi-region, high-cardinality, retention |
| Loki | Logs | LogQL | Cheap log storage, structured JSON logs |
| Tempo | Traces | TraceQL / trace ID | Distributed trace visualization |
| Alloy | (collector, not storage) | — | Unified agent for all signal types |
| Grafana | (visualization, not storage) | — | Dashboards, Explore, alerts, correlation |
12. Running This Stack Locally (Docker Compose)
A minimal local setup for a voice agent:
# docker-compose.yml
services:
prometheus:
image: prom/prometheus:latest
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml
ports:
- "9090:9090"
loki:
image: grafana/loki:latest
ports:
- "3100:3100"
tempo:
image: grafana/tempo:latest
ports:
- "3200:3200"
- "4317:4317" # OTLP gRPC
promtail:
image: grafana/promtail:latest
volumes:
- /var/log:/var/log
- ./promtail.yml:/etc/promtail/config.yml
grafana:
image: grafana/grafana:latest
ports:
- "3000:3000"
environment:
- GF_AUTH_ANONYMOUS_ENABLED=true# prometheus.yml
scrape_configs:
- job_name: voice-agent
static_configs:
- targets: ['voice-agent:8080']
scrape_interval: 15sGrafana will be at http://localhost:3000. Add Prometheus, Loki, and Tempo as data sources, and you have a full local observability stack.
13. Mental Model Summary
┌─────────────────────────────────────────────────────────────────┐
│ THE OBSERVABILITY STACK │
│ │
│ QUESTION SIGNAL TOOL QUERY LANGUAGE │
│ ───────────────────────────────────────────────────────── │
│ "Is something Metrics Prometheus PromQL │
│ wrong / slow?" (numbers) or Mimir │
│ │
│ "What happened Logs Loki LogQL │
│ exactly?" (events) │
│ │
│ "Where did time Traces Tempo TraceQL │
│ go in this (spans) / trace_id │
│ specific call?" │
│ │
│ "Show me all three" ────────► Grafana (Explore / Dashboards) │
│ │
│ "Wake me up when ─────────► Grafana Alerting │
│ something breaks" │
└─────────────────────────────────────────────────────────────────┘
The philosophy is: metrics to detect, logs to understand, traces to pinpoint. Each signal has a different cost, granularity, and purpose. You use all three together.
When you extend this stack — say, adding a new LLM provider or a new call flow — the pattern is always the same:
- Instrument the new code with counters, structured logs, and OTel spans
- Expose
/metricsand configure Promtail/Alloy to pick up logs - Create a dashboard panel for the new signal
- Write an alert rule for the failure mode you’re afraid of
Once you’ve done it once for one component, the pattern repeats for every component. The stack scales horizontally — more agents, more services, more data — without changing the mental model.