The process was reasonable: resolve the incident, investigate it, create follow-up work, then return to the roadmap.
But we were on a hamster wheel. We were getting better at dealing with incidents without necessarily getting out of the conditions that kept producing them.
So the question became: how do we get out of this cycle?
Start by measuring how much firefighting actually costs
When I looked at the data, I noticed that production problems were consuming a fairly predictable share of our engineering capacity. Think of it as roughly 10%.
But we were planning as if that capacity was available for roadmap work. Then incidents happened and took it away.
If firefighting consistently consumes part of your engineering capacity, you are already paying for it. The question is whether you want incidents to decide how that capacity gets spent.
We started reserving capacity for remediation in advance.
We were not necessarily investing more capacity. We were taking capacity that was already being consumed reactively and making it explicit.
That gave us a way to start breaking the cycle.
Stop treating every incident as a separate problem
When an engineer is pulled away from planned work because something broke, the natural goal is to restore service, make a reasonable fix, and get back to the original work.
That is exactly what you want during an incident.
But if you organize all remediation this way, you can keep fixing individual symptoms without making the system much healthier.
Dedicated remediation capacity gave us room to ask a different question.
Instead of only asking:
How do we prevent this incident from happening again?
we could ask:
What is it about this part of the system that keeps producing problems?
Several incidents may be different symptoms of the same underlying weakness. Looking at them together can lead to a very different engineering plan than treating each one as a separate follow-up item.
Give engineers room to understand the problem space
We had a queuing system that had become an important part of our product architecture.
It had not been designed for the throughput, reliability, recovery, observability, and scale we now needed.
When an individual job became stuck, we could usually investigate enough to restore service.
But we did not have enough observability to understand why the broader system behaved the way it did. Deeper investigation involved too much guessing.
With dedicated remediation capacity, we could go beyond fixing the latest symptom.
One of the first investments was observability.
Once engineers could see what was actually happening, they could identify root causes and create a deliberate technical plan.
The question was no longer just what to fix after the latest incident.
It became: what would make this problem space healthy?
We did not immediately stop having production incidents. There were other weaknesses in the system.
But fewer causes started repeating.
For me, that became an important measure of whether we were getting off the hamster wheel.
Make firefighting part of the capacity discussion
There is also a management problem behind recurring firefighting.
When remediation is not planned, every incident creates a new negotiation with the roadmap.
Who should stop their work?
How deeply should we investigate?
How much remediation is enough?
Should we fix this properly now or get back to roadmap delivery?
Once we reserved capacity explicitly, we could have that discussion before the next incident.
How much capacity are production problems actually consuming? Is that amount acceptable? Where should we invest it? Are the same causes still coming back?
This also made roadmap planning more realistic.
We were not necessarily doing more total work. We stopped planning with capacity that we knew production problems were likely to consume.
That meant fewer unexpected interruptions, clearer ownership of remediation, and more reliable roadmap estimates.
It also reduced frustration for engineers. They did not object to improving the system. They disliked repeatedly stopping their work, switching context, and then trying to reconstruct what they had been doing.
Getting out of firefighting mode is a management decision
You cannot prevent every incident.
And you cannot know which production problem will happen next.
But you can recognize when firefighting has become a recurring part of how your engineering organization operates.
When that happens, don't keep treating each incident as an isolated surprise.
Look at how much capacity firefighting is consuming.
Make that capacity visible in your plan.
Give remediation clear ownership and enough room to investigate beyond the immediate symptom.
Look across incidents for common causes.
And track whether those causes keep coming back.
That is how you start getting off the hamster wheel: not by responding faster every time it turns, but by removing the problems that keep turning it.