Clients were reporting timeouts.
But a timeout didn’t always mean the transaction had failed.
In some cases, processing continued after the client stopped waiting for a response.
That created a difficult situation.
Before anything could be retried or corrected, we had to determine what had actually happened.
A request could appear to have failed while further investigation showed that processing had continued successfully.
We developed a reconciliation procedure to deal with this.
But it was manual, and manual investigation introduced another risk: human error.
The better question wasn’t how to get faster at reconciliation.
It was why this was happening in the first place.
Our monitoring wasn’t explaining the problem
We already had monitoring.
Prometheus collected infrastructure metrics such as CPU and memory. We also had application performance monitoring in place.
Yet what we could see didn’t consistently match what clients were experiencing.
Sometimes resource utilisation increased when the problem occurred.
Other times, clients were experiencing timeouts while the infrastructure appeared relatively healthy.
We investigated database performance and found queries worth improving.
We increased server resources.
We reviewed IIS configuration, application pools and request handling.
All were reasonable things to investigate.
Some improved parts of the platform.
But the recurring problem remained.
Sometimes it lasted minutes. Sometimes much longer.
Restarting a server could make the problem disappear temporarily.
Then it would gradually return.
Then we found time we couldn’t explain
The breakthrough came from comparing timestamps at different points in the request path.
Some requests appeared in the IIS logs but only reached the database much later.
Sometimes the difference was close to a minute.
Once the request reached the database, processing could be quick.
So where had that time gone?
The database couldn’t explain it.
CPU and memory couldn’t explain it.
And our monitoring wasn’t showing us what was happening during that period.
A large number of applications were sharing the same IIS infrastructure.
We experimented with application-pool segregation and reviewed request and connection settings.
Nothing gave us a clean answer.
The configured limits appeared capable of supporting the concurrency we expected.
We never found a single IIS setting that we could point to and say:
That was the root cause.
Eventually, the architecture became the question
For years, increasing resources had been a reasonable response as the platform grew.
And it had worked.
But the evidence was changing.
Increasing resources wasn’t reliably resolving the problem.
Database performance wasn’t explaining all of the delay.
Requests were spending significant time somewhere between layers.
And many different workloads were sharing the same IIS environment.
We didn’t have a perfect root cause.
But we had enough evidence to make an engineering decision.
Instead of continuing to make the same servers larger, we changed the architecture.
We separated the workloads
Previously, the same broad application estate ran across a pair of IIS servers behind a load balancer.
We moved away from that model and separated workloads by function.
API workloads had their own infrastructure.
UI workloads were separated.
Other application workloads were separated as well.
We retained redundancy, but stopped asking the same IIS environment to carry every type of workload.
And the recurring problem stopped.
Not temporarily after a restart.
It stopped.
Healthy infrastructure doesn’t necessarily mean a healthy request
Looking back, our monitoring wasn’t necessarily wrong.
Prometheus was reporting the infrastructure metrics we had configured it to collect.
Our application monitoring was giving us another view.
The database was processing requests quickly once they reached it.
Each told us something useful.
But none initially explained what our clients were experiencing.
That distinction has stayed with us:
Monitoring can tell you that the components you’re measuring are healthy.
It cannot tell you that you’re measuring the right things.
There was another lesson.
Engineering investigations don’t always end with a perfect root cause.
Sometimes you accumulate enough evidence to realise that continuing with the existing approach no longer makes sense.
For years, vertical scaling had been sufficient.
Eventually, it wasn’t.
We couldn’t explain every detail of the behaviour.
But we could explain enough to justify evolving the architecture.
We didn’t have perfect certainty.
We had enough evidence to make the next engineering decision.