We were building a new platform and had reached the point where the development environment was operational.
APIs were responding. Deployments were progressing. The team was continuing to build.
UAT and production were still ahead of us, but the foundations of the platform were taking shape.
As we continued building out the platform’s engineering controls, nothing initially suggested that the tests weren’t executing as we expected.
We discovered it while improving the pipeline.
We Were Improving the Pipeline
We were working through the CI/CD pipelines and integrating SonarCloud across the platform.
The objective was to bring automated tests, code coverage and code analysis together so that every build gave us better evidence about the state of the software.
We started with the applications and then worked through the supporting packages.
As part of that work, we also simplified some of the pipelines. There were stages adding unnecessary complexity and slowing down development, so we removed what wasn’t giving us enough value and focused on introducing the controls we actually wanted.
Eventually, one of the components was giving us the flow we were looking for:
Tests → Code Coverage → Azure DevOps → SonarCloud
Then we moved onto another application.
Something was missing.
Where Were the Test Results?
The application wasn’t producing the test and code coverage information we expected.
As part of the SonarCloud integration, the pipeline needed the results generated during compilation and unit-test execution so that test and code coverage information could be analysed.
For this application, those expected results weren’t being generated.
At first, this looked like another pipeline configuration problem.
So we followed it.
The investigation eventually led us to an environment configuration issue.
A previous change intended to keep the platform’s environments separated had an unintended consequence: in development, part of the test process was no longer executing as expected.
We corrected the configuration.
But that raised a much more uncomfortable question.
If the expected test results weren’t being generated now, what had been happening before?
The answer was simple.
The tests hadn’t been running.
The Dangerous Part Wasn’t the Failure
There had been no dramatic outage.
The pipeline hadn’t suddenly stopped everyone from working.
Developers hadn’t become unable to deploy.
The new platform continued taking shape.
That was precisely the problem.
We had something that looked like an engineering control.
The tests existed.
The pipeline existed.
Development continued.
But something had changed underneath that process, and the expected evidence from the tests was no longer there.
That gap was what eventually exposed the problem.
Then We Turned It Back On
Once the environment configuration was corrected, we enabled the tests properly.
A unit test in one of the applications failed.
The specific failure wasn’t the important discovery.
What mattered was that the moment the control started operating as expected, it surfaced something that required investigation.
Until then, the absence of a failing test had given us no useful information.
The moment the control was operating again, it did.
That changed the question we were asking.
Instead of:
“Do we have automated tests?”
we needed to ask:
“What evidence do we have that our automated tests are running?”
Those are very different questions.
Controls Need Evidence
This lesson extends far beyond unit testing.
A backup policy tells us backups should exist.
It doesn’t tell us they can be restored.
A health check tells us an endpoint responded.
It doesn’t necessarily tell us the application can perform its job.
A security scanner can exist in a pipeline.
It doesn’t mean its results are being reviewed or acted upon.
A disaster recovery plan can be documented.
It doesn’t mean the organisation can recover.
And a test suite sitting in a repository doesn’t give us confidence simply because it is there.
A control we assume is operating can be more dangerous than a control we know we don’t have.
When we know something is missing, we can make a conscious decision about the risk.
When we believe something is working, we make decisions based on confidence we may not actually have.
What It Reinforced for Us
We already had checks around the platform and its delivery pipelines.
This incident didn’t tell us that controls needed to be checked. We knew that.
What it reinforced was how easily a control can appear to be in place while a configuration change affects part of the process in a way that isn’t immediately obvious.
The answer isn’t simply to add more checks.
It’s to make the outcome of the important ones visible.
Did the tests execute?
What passed?
What failed?
Was code coverage produced?
Did the security scan complete?
Did the deployment pass the checks that mattered?
That visibility gives us another layer of confidence. It makes it easier to notice when something that worked yesterday isn’t behaving the same way today.
No engineering process eliminates every possibility of something slipping through.
The objective is to make those failures easier to detect — and to learn from them when they happen.
The lesson from that pipeline wasn’t that automated testing matters.
We already knew that.
It was a reminder of something broader.
Engineering confidence comes from controls we can see working.