From Dashboards to Decisions
Over the last decade, I’ve built monitoring platforms using CloudWatch, Prometheus, Grafana, OpenSearch and distributed tracing across consulting engagements, enterprise platforms and cloud-native applications.
Like many engineers early in my career, I believed observability meant building better dashboards.
So I kept saying Yes to every single request!
Service dashboards.
Infrastructure dashboards.
Database dashboards.
Application dashboards.
Business dashboards.
Custom dashboards for individual teams.
Executive dashboards for engineering leadership.
Every new requirement resulted in another dashboard.
Eventually, I reached a point where there were hundreds of dashboards and thousands of metrics available.
I genuinely believed I had built everything an engineering team would need to diagnose production issues.
Then a production incident proved me wrong.
The Incident that changed my thinking
A critical customer-facing API suddenly breached its latency SLO.
Grafana showed the 95th percentile latency increasing from around 250 ms to nearly 6 seconds within a few minutes.
Like many production incidents, the first assumption was immediate.
“Something is wrong with Kubernetes.”
My investigation started exactly where most engineers begin.
Infrastructure.
The worker nodes were healthy.
There was no CPU saturation.
No memory pressure.
No disk throttling.
No network packet loss.
Pods weren’t restarting.
The Horizontal Pod Autoscaler hadn’t scaled.
Infrastructure looked exactly as expected.
Next, I moved to the application dashboards.
Request throughput remained stable.
Error rates stayed below one percent.
Only latency had increased.
At that point, I knew something important.
The platform wasn’t failing.
Something inside the request path was consuming time.
Monitoring had successfully narrowed the investigation.
It hadn’t explained the problem.
Following the request
The next stop was application logs.
Requests were reaching the services successfully.
Authentication completed normally.
Business logic executed without exceptions.
There were no retries.
No stack traces.
No timeout errors.
Every request eventually completed successfully.
Just much later than expected.
The logs confirmed that nothing was crashing.
They also exposed something I hadn’t considered earlier.
The application wasn’t failing.
It was waiting.
The next question became obvious.
Waiting for what?
The missing piece
Distributed tracing answered that question almost immediately.
Instead of investigating individual services, I followed a single customer request through the entire execution path.
| |
The trace completely changed the direction of the investigation.
The API wasn’t slow.
The Order Service wasn’t slow.
Kubernetes wasn’t under pressure.
Almost 90% of the request execution time was spent waiting for a downstream dependency.
Without distributed tracing, I would almost certainly have continued investigating the wrong layer of the platform.
That incident completely changed how I thought about observability.
What changed
From that day onward, I stopped treating monitoring, logging and distributed tracing as separate tools.
I started treating them as different stages of the same investigation.
Monitoring tells me that something is wrong.
Logs tell me what happened.
Distributed tracing tells me where time was actually spent.
Only when those three signals are viewed together does the system begin to explain itself.
Designing Observability around questions
That experience also changed how I designed dashboards.
Instead of organising dashboards around technologies, I started designing them around engineering questions.
One example was an SQS monitoring dashboard.
Rather than exposing dozens of CloudWatch metrics, I deliberately combined four metrics into a single operational view.
- Messages Sent
- Messages Received
- Messages Visible
- Messages Deleted
Individually, those metrics were useful.
Together, they answered the question I actually cared about during an incident.
Are producers generating work faster than consumers can process it?
Instead of forcing engineers to correlate four independent graphs, the dashboard immediately explained the behaviour of the queue.
That became my design principle.
A dashboard shouldn’t display metrics.
A dashboard should reduce investigation time.
How I design bbservability today
Every observability platform I design today follows the same investigation workflow.
| |
Every layer removes uncertainty.
Every layer reduces assumptions.
Every layer shortens the path to the root cause.
Looking Back
The biggest lesson I learned wasn’t about Grafana, Prometheus or OpenTelemetry.
It was about how engineers solve problems.
During an incident, nobody asks for another dashboard.
They ask questions.
- Why is the request slow?
- Which dependency is responsible?
- Is this an infrastructure issue or an application issue?
- Which customers are affected?
- What changed?
Today, I measure the success of an observability platform differently.
Not by the number of dashboards it contains.
But by how quickly it helps an engineer answer those questions and confidently decide what to do next.