Observability

Observability is the ability to answer questions about a running system from the data it already emits, without shipping new code to investigate. In a distributed system the two questions that matter most are asked of different signals.

Metrics are aggregates. They answer questions about a population of requests: the median and 95th-percentile latency of an endpoint, the error rate of a consumer, the depth of a queue over the last hour. They are cheap to store because the detail is discarded on the way in, which is also why they cannot tell you what happened to one particular request.

Distributed tracing keeps the detail for individual exchanges. A trace follows one request across every service that handled it, recording a span per hop with its timing and attributes. It answers “why was this call slow” and “where did this call fail”, which no amount of aggregation will.

The two are complementary: metrics tell you something is wrong and roughly where, and traces tell you what happened.

Instrumenting once

Micrometer’s Observation API lets application code be instrumented once and emit metrics, trace spans and correlated log context from that single instrumentation, rather than being annotated separately for each backend. Spring’s Observability with Spring Boot 3 introduces the model: an ObservationRegistry with handlers that react to the start, stop, error and scope events of each observation.

Tags on an observation are split into low-cardinality — a bounded set of values, safe to use as a metric dimension — and high-cardinality, such as a user or order id, which belongs on a span but would blow up a metrics backend. Getting that split right is most of the work.

Propagating context

A trace only spans services if the identifiers travel with the request. W3C Trace Context, a Recommendation since 2021, defines the two headers that carry them: traceparent, holding the trace id, the parent span id and the sampling flag, and tracestate, holding vendor-specific state alongside it. Any service that receives a request and makes further calls must forward these, including across asynchronous hops — a message published to a broker has to carry the context in its headers, or the trace ends at the producer.

References

• Micrometer Tracing reference documentation
• W3C Trace Context
• AWS monitoring and observability for the managed services we run this on