In his latest Expert's Corner, Michael Keating outlines the steps manufacturers can take to ensure that essential data is quickly recovered after a disaster. This is Keating's first in a series on disaster planning and recovery. Future installments will include preserving the supply chain and social media tools.
The 2013 hurricane season is about half over, and the National Hurricane Center has reported only one hurricane, Humberto, so far. However, the National Oceanic and Atmospheric Administration is predicting between six and nine hurricanes to reach U.S. shores before the season ends Nov. 30. As many as three of those hurricanes are expected to be major storms, with winds greater than 110 mph.
Meanwhile, almost a year after Superstorm Sandy devastated the Northeast cleanup continues. Areas are still recovering from the estimated $65 billion in damages and economic loss.
Manufacturers take a hit in the wallet from weather-related disasters. "Anecdotally, we have consistently seen financial impacts range from $50 million to well over $200 million after a 72-hour disruption," said Damian Walch, director of Resilience Services at Deloitte, a professional services firm. "It is vital for manufacturers and other companies to be prepared and determine what their response and recovery capability is when faced with a major outage or disaster."
Companies around the globe acknowledge that "resilience" is a significant challenge, he said. "Lean manufacturing, loss of key skills, and retirement of workers are just a few things that have caused concern. The complex eco-system of business partners, outsourcers, and suppliers has added even greater challenges to responding to business and IT-related disruptions and catastrophic outages."
Data recovery is crucial for manufacturers after a disaster, agreed Jarrett Potts, director of marketingof Colorado-based STORServer. He advises manufacturers: "Prepare for recovery. Don't prepare for backup, because it's all about recovery and retrieval of data. You can spend a lot of money and design the greatest backup system in the world, but if you can't recover your company's data in the limited amount of time that you want it to come back on, then it's worthless."
After a disaster, manufacturers quickly work to recover only the data that they need to get back up and running, Potts said. The volume of this crucial data may exceed hundreds of terabytes, he said, and download speeds from the cloud may not be fast enough. In those instances, companies should do "a cloud-based backup to a secondary site, and that secondary site would be able to quickly bring up the client's systems as a hot site," Potts advised. The hot site is a duplicate of the original site of the organization, with full computer systems and a live backup of data.
The potential inability to connect to the Internet is another cloud limitation, said Bryan Rice, co-founder of Net3 Technology, a managed services provider in Greenville, S.C. He told IMT that there is a danger for manufacturing executives to believe that virtualization and cloud-based solutions will solve all their disaster recovery woes.
"Most manufacturing facilities rely on older ERP solutions and supply chain logistics software which can create compatibility issues," he said. "These systems connect with automated production machinery for shop floor data collection and monitoring, and are universally hard-wired for this communication. If those communication paths are disrupted, so goes the primary production lines and processes with rippling effects throughout the entire supply chain."
Consultant Lisa Anderson of LMA Consulting Group in Claremont, Calif., urged manufacturers to make sure that nightly backups are completed. "Having a second location to store or perform the nightly backup is preferred," she added. "For example, if you have facilities in two states or if you can partner with a customer/supplier or utilize outsourced IT services, you can send your data to the second location nightly, so that you can minimize downtime during disasters." Another option, she said, is for manufacturers to collaborate with supply chain partners on backups.
Adam Stern, CEO of Infinitely Virtual, a provider of cloud services in Santa Monica, Calif., recommends an "application-consistent backup" to safeguard data. "It is the most secure data restoration in the event of data loss," he said. "This means your cloud computing services or IT provider has taken point-in-time snapshots of your data, enabling fast, clean restoration, without tape. This method avoids data corruption, and is the fastest means of getting you up and running, while minimizing downtime."
Avoid using tape in backup efforts, suggested Janson Hoambrecker of EVault, a Seagate-owned recovery solution provider based in San Francisco. "Backup tapes can have failure rates as high as 40 percent if the data is read from a different drive than it was written on, so recovery networks for hardware-based backup have to be configured to match the original network configuration (i.e. VLANs, VPNs, DNA, and firewall rules)," he said.
A cloud-based disaster recovery plan eliminates the need for a duplicate IT infrastructure offsite, which can reduce IT resources and hardware costs, Hoambrecker said. "The usage-based cost of cloud services means that companies only pay for what they need."
But manufacturers shouldn't give up on the cloud for data backups. One key benefit is that the cloud provides scale, said Greg Kefer, vice president of corporate marketing at Oakland, Calif.-based GT Nexus, a cloud service provider. "Cloud IT providers have robust, redundant infrastructure that is typically many times what any single company would have on its own," he said. "In the event a disaster impacts an IT environment, backup data centers kick in to ensure continuity of services. The data itself is backed up constantly so there should never be significant losses."
When in recovery mode, companies should avoid allowing the entire workforce to log on at the same time, advised Jerry Irvine, a National Cyber Security Partnership member, responsible for advising federal decision-makers on data protection policy.
"Because disaster recovery solutions are not always exact replicas of the production environment, their utilization capacity may not be the same. Allowing access to all users and maintaining all processes could over-subscribe the systems and cause systems failures, data corruption or data loss," said Irvine, CIO of Chicago-based Prescient Solutions.
Maintaining security is essential post-disaster, Irvine added. "When disaster recovery facilities and systems are designed only as secondary solutions to be used in case of a systems failure, many organizations neglect to consider implementing security configurations. Disaster recovery facilities and systems require the same level of security implementation as production environments."
This series on disaster recovery for manufacturers will return in two weeks.
Read part two of this series, How Manufacturers Can Preserve the Supply Chain After a Disaster.
Read part three of this series, Social Media Tools for Manufacturers in Times of Disaster.
Read part four of this series, Manufacturers Join Consortiums to Help Each Other After a Disaster. Michael Keating is senior editor for Government Product News and a contributing editor for American City and County, both published by Penton Media. Read his mid-year 2013 government budget and spending forecast at the Government Product News site. Go here for his IMT 2012 report on how to land government business. His most recent item for IMT was about Mid-year 2013 Update on the Government Market for Manufacturers. Keating has written articles on the government market for more than 100 publications, including USA Today, Sanitary Maintenance, IndustryWeek, and the Costco Connection. Michael can be reached through his website, MikeKeat.net.