Business continuity and disaster recoveryMidwest floods threaten IT infrastructure

As historic floodwaters continue to hammer Cedar Falls, Iowa, local businesses are already assessing the environmental disaster’s impact on IT operations, and how their disaster recovery plans are faring. Computerworld’s Brian Fonseca writes that as of Tuesday, 100 blocks in the city’s downtown are underwater and 3,900 homes have been evacuated in Cedar Falls. Wade Arnold, CEO of T8Design, said his company had prepared disaster recover plans to deal with tornados or electric outages caused by human negligence, but executives never dreamed they would have to contend with a swollen Cedar River surpassing 500-year flood levels. “When you make [disaster recovery] plans, you never think it could possibly happen to you,” said Arnold. “Going through this experience is going to make me look at those plans as something other than just an IT checklist.” Once it became apparent that rising floodwaters could damage its IT operations, the maker of software development tools warned its customers of possible latency issues. No disruptions or degradation of service has occurred to date, he noted. Arnold credits his company’s decision to locate its data center, which houses 60 servers, source code repositories, and client project, in a third party facility 5.4 miles away from downtown Cedar Falls. Still, he said T8Design was prepared to failover its systems to a secondary facility outside of Chicago if the floodwaters did cause the 18,000 sq. foot data center to shut down or lose power for a significant period of time. He said the company now plans to revise its disaster recovery plans to take into account employees missing work to assist community efforts such as sand-bagging or dealing with family or personal issues caused by a disaster. “When something like this happens, you don’t ever expect you’re not going to get back to normal - but there’s some people you’ll never see in the same” location again, remarked Arnold.

Phantom EFX, Inc., a Cedar Falls-based maker of computer games, may be permanently displaced due to last week’s unprecedented floods. The firm’s president, Jim Thompson, said the floodwaters reached 6 feet in Phantom EFX’s administrative offices and 5.5 feet in an adjoining warehouse facility. The company is now borrowing workspace for its 65 employees in three separate facilities in the city. As floodwaters approached the company’s headquarters on 9 June, Thompson said employees worked furiously to salvage about 75 percent of products and property stored there. He estimated the flood caused about $250,000 in damage to inventory, including product packaging, CDs, and other items. “We just ran out of time,” said Thompson. Thompson said employees sealed top racks of shelves, never suspecting that the floodwaters would get “past a foot or two in the building.” The employees were able to save 120 PCs, 80 monitors, and eight servers from water damage. Three high-end color printers, however, could not be removed from the building in time. The company lost little or no data as it is backed up daily and stored offsite at a local data center and another in California. Thompson said he is unsure of when or even if the building can be occupied again. He does plan to use the experience to craft a revised disaster recovery plan. “When a river comes up six feet higher than it ever has before, it’s tough to have that foresight. But it’s probably going to happen again, so we’re going to put some plans in place,” he added.