4 Steps for Handling High-Pressure Bugs

You know that nauseous feeling that forms in the pit of your stomach, when you realize something bad has or is happening?

Even worse, you are likely both the one that caused it and the one that has to fix it!

I remember the first time I encountered this situation. It wasn’t fun.

The scenario went something like this: A few similar support requests had been submitted and I became suspicious that we had a legitimate bug on our hands. Sure enough, after tracing the issue back to a recent release, I realized I had mistakenly applied the same identifier to all users who’d registered since the deployment. 😱

Once the severity of the bug hit me, I reached for the garbage can and almost lost my lunch. 🤢

Since then, I‘ve formalized a method I can fall back on when these high-stress situations arise. In this post I’ll share that strategy with you in hopes it helps if when you encounter a similar event.

Step 1 - Breathe

Some people (think professional athletes) thrive under pressure. They perform their best when the spotlights are the brightest.

I am not that kind of a person. Over my lifetime, I have come to an understanding of how much pressure or stress I can handle and when I sense the issue is larger than I am comfortable with, I will employ this step before doing anything else.

When I feel the adrenaline rush I intentionally tell myself to calm down, collect my breathing, and try to lower my heart rate. I remind myself that a bug in a web application, as serious as it feels at the moment, is not a life-or-death matter… at least not in the kind of work I do.

Once the initial shock has begun to subside, and I’ve found some internal equilibrium, I move into the second step: diagnosis.

Step 2 - Diagnose

OK, we’ve controlled our breathing, regained composure, and are ready to dig in. In the diagnosis step, the goal is to clearly identify and understand what is going on. At the end of this step, I must be able to answer “Why is (fill-in-the-blank) happening?”

The first of many debugging approaches I use in pressure situations is called “rubber-ducking,” which is nothing more than walking through the problem in plain English with another person.

Often, the process of explaining the symptoms of the issue to someone else illuminates the actual problem. Collecting my disjointed thoughts and stringing them into sentences brings clarity. Otherwise I tend to stew around in a soup of unformed theories.

The goal of the diagnosis step is not to solve the problem, but is to understand the problem. This is an important distinction. Jumping to patches or remediation without truly understanding why we are in this situation and how we got here has a high likelihood of making the problem worse.

Only when you’re confident you understand and can articulate the problem should you move on to the patching phase.

Step 3 - Patch

In this step, the goal is to stop the bleeding as quickly as possible… apply a tourniquet if necessary!

Does the job queue need to be paused? Do we need to temporarily disable a feature? Should we put the application into maintenance mode? All of these are on the table during the brainstorming portion of the patch phase.

A word of caution - Arriving in the patch phase should be handled carefully, as it is a dangerous mixture of stress/anxiety coupled with the exhilaration of clearly identifying the problem in the diagnosis phase.

Planning a patch should be carried out with the same intentionality we employed while uncovering the problem. If you used the pairing strategy or rubber-ducking with a co-worker, do the same thing here. Describe the potential solution in words or walk through the proposed fix while pairing.

Weigh the potential solution against immediately applying a patch to stop the problem versus taking more time to vet the solution while the problem continues. This is not the time to be careless!

All patches at this point should do two things:

1. Be targeted at preventing the problem from continuing
2. Will not adversely affect the situation

Similar to the diagnosis step, only when you are confident in your solution should you move to executing it.

Step 4 - Remediate

In the remediation step, we attempt to identify the wake the bug left on our app. Do we have data to cleanup, queued jobs to retry, or <gasp> customers to contact?

Admittedly, arriving at the remediation step always feels like a let-down for me. By this point the wash of emotions has gone from nausea, anxiety, and fear, to the thrill of identifying the problem, and finally to the satisfaction of deploying a successful fix. Now we are left with the clean up, which for me is a combination of shame (for letting the bug get to production) and annoyance (for having to spend time manually fixing a problem that should never have occurred in the first place).

I try to absorb as many of these lessons as I can so that I will (hopefully) remember them when I am writing code on the next application feature. On the flip side, I try not to internalize too much as it can have a negative effect on my psyche and performance.

Stuff happens

I like to sleep at night and I try hard to write code that allows me to do so. Unfortunately, bugs will always find a way into my applications, and the breathe, diagnose, patch, and remediate strategy has helped me on more than one occasion.