Skip to content

What is database observability?

Last updated

View as Markdown

Quick answer: There is no settled definition. In practice, database observability means keeping enough query-level, engine-native evidence that you can ask a question you did not think of in advance. Monitoring alerts on thresholds you chose. Observability is the claim that the evidence survives the incident. Treat it as a capability claim and test it.

Why does nobody agree what the term means?

Section titled “Why does nobody agree what the term means?”

Because it arrived from somewhere else. “Observability” comes from distributed application systems, where the useful framing is metrics, logs and traces. That framing was then applied to databases, which emit none of those three as their primary signal.

Every major vendor now uses the word, and they do not mean the same thing by it. Some mean expanded metrics plus query analytics. Some mean plan capture and application-to-database trace correlation. Several of the oldest tools in the category have sold wait-based analysis, top SQL and historical baselines for years under the name “performance analyzer”, and that is most of what the newer products call observability.

So the word does not tell you what a product does. It tells you what year the marketing was written.

What there is broad agreement on, across vendor documentation and practice:

  • It must let you investigate behaviour nobody predicted, not just fire alerts on thresholds somebody set.
  • Host metrics alone are not enough. It has to be query-level and engine-native.
  • History matters. Plan changes, workload shifts and configuration changes cannot be understood from a snapshot of right now.

That is a direction, not a definition.

What is the actual difference from monitoring?

Section titled “What is the actual difference from monitoring?”

One sentence: monitoring answers questions you wrote down in advance, observability is the claim that you kept enough evidence to answer one you did not.

Monitoring is “alert me when replication lag passes 30 seconds”. You knew to care about replication lag. You set the number.

The question you actually get at 3am is “why did checkout get slow for one customer in one region, starting some time after the Tuesday deploy”. Nobody wrote an alert for that. Answering it needs evidence that was already being collected before anyone knew the question, at a grain fine enough to isolate one query and one customer.

The honest version of the distinction is not monitoring versus observability as opposing things. Monitoring detects and prioritises. Observability supplies the evidence to diagnose what monitoring flagged. A product with no alerting is not better, it is incomplete.

What does the database itself give you?

Section titled “What does the database itself give you?”

More than people expect, and it is free. The gap between the raw signals and a product is retention and correlation, not access.

Who is running, and what are they waiting on. This is the single most useful query in PostgreSQL and most teams never run it. wait_event_type is the difference between “the database is busy” and “the database is blocked on disk” or “blocked on a lock”:

SELECT pid,
       state,
       wait_event_type,
       wait_event,
       now() - query_start AS running_for,
       left(query, 60) AS query
FROM pg_stat_activity
WHERE state <> 'idle'
  AND pid <> pg_backend_pid()
ORDER BY query_start;

What is blocking what. A lock wait is the one incident shape that is genuinely impossible to reconstruct afterwards. Either you captured the graph while it was happening or the answer is gone:

SELECT blocked.pid AS blocked_pid,
       blocking.pid AS blocking_pid,
       blocked.wait_event_type,
       blocked.wait_event,
       left(blocked.query, 60) AS blocked_query
FROM pg_stat_activity AS blocked
JOIN LATERAL unnest(pg_blocking_pids(blocked.pid)) AS bp(pid) ON true
JOIN pg_stat_activity AS blocking ON blocking.pid = bp.pid;

The counters, and when they were last zeroed. This is where most homegrown dashboards go wrong. pg_stat_database is cumulative since stats_reset, not a rate. A cache hit ratio computed straight off these columns is an average over however long the server has been up, which can hide a bad afternoon completely:

SELECT datname,
       xact_commit,
       xact_rollback,
       blks_hit,
       blks_read,
       stats_reset
FROM pg_stat_database
WHERE datname = current_database();

To get a rate you have to sample it twice and subtract. That sampling loop, kept running and kept retained, is most of what a product is selling you.

MySQL exposes the equivalent through performance_schema, aggregated by normalised statement digest:

SELECT DIGEST_TEXT,
       COUNT_STAR,
       ROUND(SUM_TIMER_WAIT / 1000000000000, 3) AS total_seconds,
       ROUND(AVG_TIMER_WAIT / 1000000000, 3) AS avg_ms,
       SUM_ROWS_EXAMINED,
       SUM_ROWS_SENT
FROM performance_schema.events_statements_summary_by_digest
ORDER BY SUM_TIMER_WAIT DESC
LIMIT 10;

PostgreSQL’s equivalent is pg_stat_statements, which needs the extension loaded at server start:

SELECT calls,
       round(total_exec_time::numeric, 1) AS total_ms,
       round(mean_exec_time::numeric, 2) AS mean_ms,
       left(query, 60) AS query
FROM pg_stat_statements
ORDER BY total_exec_time DESC
LIMIT 10;

Sort by total time, not by mean. A query taking 40ms five million times is the problem. The 9-second query that runs nightly is not.

What does database observability miss?

Section titled “What does database observability miss?”

This is the part the category pages skip, and it is the part worth knowing before you buy anything.

The plan is usually gone. Statement statistics tell you a query got slower. They do not tell you the planner switched from an index scan to a sequential scan, because the plan was never stored. Unless auto_explain was already enabled with a threshold low enough to have caught it, you are reconstructing the plan on a database whose statistics have since changed, which means you may not be able to reproduce it at all.

Normalisation throws away the parameter values. Both pg_stat_statements and performance_schema group by a digest of the query shape, so WHERE customer_id = 42 and WHERE customer_id = 9999 are one row. That is what makes the view readable. It is also why it cannot tell you the problem is one specific customer with 400,000 orders, which is very often the problem.

Sampling has gaps by definition. pg_stat_activity is a snapshot of this instant. Sample it every 10 seconds and you see whatever was running at those moments. A lock storm that resolved in 4 seconds may leave no trace anywhere. PostgreSQL has no built-in active session history; that is an extension or a managed-service feature, not core.

None of it says anything before you deploy. Every signal above is retrospective. It describes a database that has already been hurt. The question “will this migration take a lock that stops writes” is not an observability question at all, and no amount of collected history answers it.

How do I test a vendor’s claim?

Section titled “How do I test a vendor’s claim?”

Since the label is unregulated, ignore it and ask what the product can actually connect. In rough order of difficulty:

  1. Can it get from a user-facing symptom to a specific normalised query, a single execution, the wait or lock it hit, and the plan it used?
  2. Can it compare that against a baseline from before, and tell you what changed: workload, plan, schema, configuration, or deploy?
  3. Does it keep database-native evidence, or only CPU, memory and connection counts dressed up?
  4. Does it correlate with the application without somebody matching timestamps by hand?
  5. Can it produce a specific, reviewable recommendation rather than a chart?

One to three is competent database monitoring, whatever it is called. Four and five is where the observability claim starts being worth the word. Anything that acts on the finding is a different category again, and should be evaluated on whether the action is reviewable and reversible rather than on whether it is automatic.

How do I stop needing it so often?

Section titled “How do I stop needing it so often?”

The signals above are how you find out you were already wrong. The cheaper move is to shorten the list of things that can go wrong unobserved:

  • Turn on pg_stat_statements before you need it. It requires a server restart, and the day you want it is never a day you want to restart.
  • Set auto_explain with a sane threshold so the slow plans are captured when they happen rather than guessed at later.
  • Record stats_reset anywhere you report a ratio, so nobody reads a server-lifetime average as a current measurement.
  • Check the plan for a query before it ships, not after it is the incident. That is the one question none of this tooling answers retrospectively.

FAQ

Section titled “FAQ”
  • What is the difference between database observability and database monitoring?
    Monitoring answers questions you wrote down in advance, such as alerting when replication lag passes 30 seconds. Observability is the claim that you kept enough evidence to answer a question nobody anticipated. The difference is what survives the incident, not which product you bought.
  • Is database observability a real category or vendor marketing?
    Both. There is broad agreement on the direction: query-level detail, engine-native signals, history, and correlation across layers. There is no agreed checklist, so treat the term as a capability claim to verify rather than proof of a distinct architecture.
  • Do the three pillars of metrics, logs and traces apply to databases?
    Only loosely. A database exposes its own primitives: wait events, normalised statement statistics, lock graphs and query plans. CPU and memory come from the host, not from the database, and they are the least useful evidence of the set. They are also what generic tooling collects best.
  • What does database observability usually fail to capture?
    The plan at the moment of the incident, the parameter values that normalisation strips out, anything that happened between samples, and anything at all about a change before you deploy it.
  • Can I get database observability without buying a product?
    Partly. PostgreSQL gives you pg_stat_activity, pg_locks and pg_stat_statements for free. MySQL gives you performance_schema. Retention, correlation and the sampling loop are not free. The first two views show only this instant and keep no history; the statement views are running totals since the last reset, not a time series.
Section titled “Related”