Labor Day has new meaning in our house. I have tremendous new respect for all people who earn their living from hard, sweaty, dirty work.

And yet, while I call this my personal disaster recovery, it really isn’t that personal or that significant of a disaster.

My family and I came home to Hurricane Ida’s wreckage after it dumped an estimated seven inches of rain in our town. Our garage and backyard had three feet of water, and our basement settled at about two feet. I am told that the floodwaters came up from the ground and lasted approximately four hours.

The disaster wasn’t personal as many others on our block, neighborhood, town, and county faced similar issues and many far worse. The hurricane has already claimed over fifty victims in the northeast.

It also wasn’t a complete disaster. Broken appliances, damaged art, water sogged furniture we were storing for no good reason hardly make this a disaster. We are day three into a very sizable cleanup, and I spent the last two days cutting wet sheetrock and pulling drenched insulation. The front of our house looks like a sad flea market stand meets a construction dumpster.

The Lessons Learned From Small Disasters

But work with me here because, like any other disaster recovery effort, there are some hard lessons learned. I’d like to share mine with an IT spin since a big part of the CIO’s job is to protect against risk, respond to major incidents, and prepare for disasters.

1. Avoid storing useless crap – Our basement was filled with items once important and no longer relevant, making the damage and cleanup much worse than it needed to be.  Lesson: Avoid storing very old data that serves no business purpose
2. Don’t delay your response to a disaster – We were away when Ida hit our town and chose to avoid cutting our vacation short to come home, size up the damage, and begin the cleanup. In hindsight, we should have returned as we underestimated the cleanup. Lesson: Respond quickly and bring all resources to bear when responding to “once in a lifetime” disasters.
3. Focus on safety – While we were away, a family member who scouted the area for us told us about a clicking sound coming from our furnace. We immediately shut its power, recognizing that the furnace was probably flooded and the pilot light out. Lesson: Think through dangerous situations and respond to these issues with the highest urgency.
4. Save the key assets – In our case, the bottom row of my wine collection was submerged. Lesson: Key assets should never be stored in high-risk/vulnerable locations, like servers in a closet.

 

1. Ask experts for advice – We had no idea where to start and what to do with the cleanup. A neighbor advised us to cut the sheetrock, and I reviewed YouTube videos on how to do it properly and safely. Lesson: Disaster recovery is a program with many projects and tasks that you may not perform regularly, so call in your vendors and suppliers for their expertise.
2. Recruit helping hands – We were loaned a shop vac, a neighbor picked us up supplies from the hardware store, and my parents swung by to entertain my younger kids. Lesson: Don’t be afraid to ask for help, and look to pay it back in the future.
3. Prioritize and finish the job – I told my wife that we needed a plan, and while it was easy to get distracted when new issues popped up, it was more important to finish the task at hand before starting a new one. Lesson: Have everyone on the team understand the game plan and make the best effort to stay on task.
4. Empathize with those less fortunate – No one in our family was hurt, and we had neighbors who took on more water that resulted in greater damages to their homes. Lesson: Chances are, the disaster you’re facing is impacting customers, supply chains, and partners. Some may be worse off than you.
5. Have emergency funds and insurance – As we move from cleanup to recovery, we’re now reviewing what our insurance covers and the costs to rebuild the basement, fix/purchase new appliances, and lockdown other risks. Lesson: Disaster recovery must be more than a paper plan to check off the auditor’s box. Real disasters need plans and funding for recovery.
6. Prepare for a worse disaster – Will the next hurricane come in a week, year, or decade? It doesn’t matter, as we’re already taking steps to reduce the risks and impacts. Lesson: Never waste a disaster, and conduct multiple postmortems on reducing risks and preparing for the future. 

Simple Lessons for IT Disaster Recovery

Here again, are the ten IT lessons: