Wednesday April 15, 2020 will be remembered by people working in Information Technology for the degradation across the board of the online banking systems of virtually every US bank. Starting around 8am EDT, pretty much everyone on the Eastern seaboard who was expecting a stimulus check direct deposit started clicking on [INQUIRE BALANCE] in their bank’s app or web site. This then rolled westward until everyone in the country who was expecting a direct deposit was wondering why their online banking broke down on they very day they most wanted to use it.
My client these days (well, most days actually) is a large bank so I became aware of this when I was called into an incident resolution meeting first thing in the morning. It wasn’t long before the scope of the problem became apparent. Down Detector, a watchdog that monitors the availability of online services, showed all US banking systems in distress. The screen shot above is about 9am and the blip on the right side of the line graph shows the service degradation starting about 8am.
As with so many other things in the age of the pandemic, we had to adjust to a “new normal”. Where incident teams normally focus on finding and fixing root cause, in this case it was more about how to change the architecture permanently for a world in which all the old-school in-branch customers suddenly are forced to use the app or the web site and singular events like stimulation payment issuance cause mass concurrent usage patterns.
We worked relentlessly all day long. It was tough but it showcased my team and my profession at our best. By the end of the day we managed to stabilize the services and make them more bullet proof. To be sure we have a lot more work to do to adjust to the sudden and complete shift of service usage patterns, but on this one day we were rock stars.
When it was finally over, I staggered down the stairs, mentally and physically exhausted, and hugged my wife for a really long time.
“Tough day?” she asked.
“One of the toughest,” I responded. “You?”
“Awful,” she said. “Online banking has been down all day long.”