By Lisa DuBrock, CPA, CBC
Small events can have big consequences and can also teach important lessons.
Take for example; a financial institution that experienced a fire at their headquarters cafeteria. While the fire did not spread, the smoke did travel from the cafeteria to the data center which was located next door. Although a small fire, the smoke and heat were enough to trip the sprinkler system into operation thereby shutting down the institutions computer operations.
The firm immediately invoked their business continuity plan which called for relocating their headquarters operations to a mobile recovery center. Having previously identified fire as a serious risk to their data center they made arrangements to mitigate the risk by subscribing to an electronically transmitted back-up of their system to an alternate location. This location is periodically seeded with a full system back-up while incremental back-ups are applied nightly. Additionally, transactional log files are transmitted to the offsite location 3 times a day. Because of this prior planning and disciplined procedures when the fire struck, the firm was prepared to respond. The staff was notified of the change of location and when the next business day began; they were up and running having lost only 2 hours of transactional data.
Heavy clean-up of the data center commenced. The clean-up was awarded to a clean-up company familiar not only with general restoration but one that specialized in rehabilitating damaged data centers. Once the facility clean-up was completed, attention turned to the computer and network infrastructure. The firm’s goal was to quickly restore its server farm using in-house staff and return to normal operations as quickly as possible. Unfortunately, several of the server’s disk drives had been permanently damaged by the combination of heat, smoke and water. The fatal errors to several of the drives necessitated them being replaced in order to complete restoration. The vendor was called and expedited delivery of replacement units was arranged. The IT team pulled together rebuilding the server and reformatted all the drives. Applications were reinstalled and the latest data was applied to the file structure at the beginning of a weekend. The IT department was given the balance of the weekend to thoroughly test the system before start of business on Monday.
This event showed the importance of performing a comprehensive risk assessment and the value of a business continuity plan as opposed to just having a disaster recovery plan. This comprehensive business continuity plan allowed the IT department to:
- Establish an alternative recovery site,
- Perform regular backups,
- Identify a vendor that could provide quick shipment of replacement components,
- Identify a qualified restoration company with experience in restoring data centers, and
- Adequately train the IT staff on both response and recovery processes.
If the institution had not previously taken these steps to mitigate the risk of fire they could have lost some if not all of their customer’s information, account balances and transaction history, most certainly putting them out of business.
However, weaknesses in the design of the fire suppression system which led to the unnecessary flooding of the data center plus the placement of the center near cafeteria are other decisions which merit further review. Disasters can teach valuable lessons. The real question is, will businesses learn from these experiences?