WFH Diary – Stimulus and response

Wednesday April 15, 2020 will be remembered by people working in Information Technology for the degradation across the board of the online banking systems of virtually every US bank. Starting around 8am EDT, pretty much everyone on the Eastern seaboard who was expecting a stimulus check direct deposit started clicking on [INQUIRE BALANCE] in their bank’s app or web site. This then rolled westward until everyone in the country who was expecting a direct deposit was wondering why their online banking broke down on they very day they most wanted to use it.

Down Detector screen shot on April 15, 2020 shows all US banks online systems in distress.

Down Detector screen shot on April 15, 2020 shows all US banking online systems in distress.

My client these days (well, most days actually) is a large bank so I became aware of this when I was called into an incident resolution meeting first thing in the morning. It wasn’t long before the scope of the problem became apparent. Down Detector, a watchdog that monitors the availability of online services, showed all US banking systems in distress. The screen shot above is about 9am and the blip on the right side of the line graph shows the service degradation starting about 8am.

As with so many other things in the age of the pandemic, we had to adjust to a “new normal”. Where incident teams normally focus on finding and fixing root cause, in this case it was more about how to change the architecture permanently for a world in which all the old-school in-branch customers suddenly are forced to use the app or the web site and singular events like stimulation payment issuance cause mass concurrent usage patterns.

We worked relentlessly all day long. It was tough but it showcased my team and my profession at our best. By the end of the day we managed to stabilize the services and make them more bullet proof. To be sure we have a lot more work to do to adjust to the sudden and complete shift of service usage patterns, but on this one day we were rock stars.

When it was finally over, I staggered down the stairs, mentally and physically exhausted, and hugged my wife for a really long time.

“Tough day?” she asked.

“One of the toughest,” I responded. “You?”

“Awful,” she said. “Online banking has been down all day long.”

About T.Rob

Computer security nerd. WebSphere MQ expert. Autist. Advocate. Author. Humanist. Text-based life form. Find me on Twitter or LinkedIn.
This entry was posted in Family, Global issues, Humor and tagged , . Bookmark the permalink.

4 Responses to WFH Diary – Stimulus and response

  1. danjodea says:

    A long buildup for a great line. What’s better is that it’s real, no joke, just the lighter side. Thanks, and hope all things are well.

    • T.Rob says:

      Thanks, Dan. I hope you are doing well too. However, I prefer to think of it as a brief after-action report that happens to end on a funny note. The actual after-action report was about 10 pages long.

  2. Hi Rob,
    Thank YOU for your IT service.
    Your article was most informative, but the last line created hysterical laughter!!!
    LOVE your sense of humor.
    Stay Safe,
    Dick

    • T.Rob says:

      Thanks, Dick! Many years ago I did a few open mic nights. This is better because I can’t see all the people sitting there silent and not clapping. Thanks to blogging, I am aware only of the ones who do. Thanks! Tip your wait staff. I’ll be here all pandemic long.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.