[readonly] markdown buffer
Logs Are Not Observability
A service fails in production, so somebody opens the logs.
They find thousands of lines saying that functions started, requests completed and objects were loaded. None of them answer the useful question:
What happened to this customer's operation?
The system has logging. It is not observable.
- gatewayrequest completed
- ordersloaded order
- paymentsretrying request
- ordershandler finished
- which user?
- unknown
- same operation?
- unknown
- final outcome?
- unknown
Logs are output
A log is one kind of evidence emitted by a running system.
Observability is the ability to use available evidence to understand its state, including failures nobody predicted in advance.
More output does not guarantee more understanding. Ten services can each print request completed while losing the identity that connects those requests into one operation.
This is not fixed by changing the log level to debug.
Start with operational questions
Before adding another line, decide what somebody must be able to answer:
- did the operation succeed for the user?
- where did it spend its time?
- which dependency failed?
- was the failure transient or permanent?
- how many users are affected?
- did a retry eventually complete the work?
Those questions determine the evidence the system needs.
A useful event contains stable context:
logger.info("payment.completed", {
operationId,
orderId,
attempt,
provider,
durationMs,
outcome: "succeeded",
});
It describes an outcome, not merely a point reached in the code.
Give the operation an identity
Requests, jobs and messages often cross process boundaries. Give the business operation a stable identifier and propagate it.
A transport request ID identifies one HTTP request. It may not identify the payment, import or fulfilment operation that survives several requests and retries.
Both are useful:
- a trace or request ID connects one execution path
- an operation ID connects every attempt at the same business outcome
Without that distinction, a retry looks like unrelated work and duplicate execution is difficult to recognise.
- gatewayrequest accepted
- ordersoperation claimed
- paymentsattempt 1 · timeout
- paymentsattempt 2 · declined
- ordersterminally failed
- user outcome
- failed
- failed at
- payments
- attempts
- 2
- duration
- 1.885s
Use different evidence for different questions
Logs are useful for discrete events with explanatory context.
Metrics answer aggregate questions: error rate, latency distribution, queue depth and the number of exhausted retries.
Traces show where one execution travelled and where it waited.
No single signal replaces the others. A trace is poor at telling you whether failures doubled across the whole system. A metric can show that failures doubled without identifying the affected operation. A log may explain the failure once you know which operation to inspect.
Observability comes from connecting them.
payment_declinedRecord outcomes at boundaries
Function-entry logs create noise because internal structure is rarely the thing an operator cares about.
Prefer events where responsibility changes:
- an API accepted or rejected a request
- a message was published or consumed
- a dependency returned an outcome
- a retry was scheduled
- a business operation completed or became terminally failed
Include duration, attempt number and stable identifiers where they matter. Use consistent field names so tools and people can query them.
Do not put secrets, access tokens or unnecessary personal data into logs. Evidence that creates another incident is not useful evidence.
Design for the bad day
Observability is not something a logging library adds after the feature is finished.
It is part of the operation's design. The code must preserve identity, distinguish attempts from outcomes and emit evidence at the boundaries where failure matters.
The test is simple: choose a customer operation and ask whether you can reconstruct what happened without reading the source code and guessing.
If the answer is no, adding more log lines is unlikely to help.
Record less noise. Preserve more meaning.