JTC Event – Panel on Disaster Recovery
This is part of a series of notes from the Alberta Education, School Technology Branch, semi-annual conference called the Jurisdiction Technology Contacts event. This particular one was held at Mount Royal, November 27-28, 2013. These notes are my own.
John Shutte – Calgary Catholic
Data Centre is located on the 7th floor of their building, 200 feet from the river. Flooding came incredibly fast. Downtown Calgary has close calls just about every year. Thought this might be one of them, until it was almost too late. Be careful of becoming complacent – it’s not going to be a big deal!;>…. Eye opening stuff: communications between departments in the school district and the public/media. Were getting 10,000 hits per minute on the Friday morning from people looking for information. Is it easy for your people to update the website from wherever they are. SuperNet and Demarc is three stories below ground. Generators were tested for capacity and load, but not how long they would last and how often they would need to be filled up, and how to get diesel to the generator when there is no power.
Disaster Recovery plan hadn’t been updated in 10 years, and the data centre had changed massively since then.
Completely caught off guard by people. All plans assumed people would be able to come into work. Many gaps happened in the teams with people missing.
Had eight feet of water in one school, completely lost all of the technology in the school. Didn’t have a plan for how to get technology to the school within a reasonable time.
Plans for the future. Distributing some of the services. considering moving some data centre infrastructure to some schools. Perhaps a rack or two. Considering some co-locate facilities. Talking with CBE and other school jurisdictions about spreading infrastructure out.
Mike Rinkel – CBE
Mike reviewed CBE’s experiences
Lessons Learned from Flood
Expectations / Capabilities
Practice / Incidence Response
Templates and Checklists
Jump Bag and contact lists
Disasters develop quickly!
Cross departmental contacts
Food / Services
Distribute your eggs
Web Services Capacity
Work from home
Regularly review priorities
Kevin Dyck – Canadian Rockies
No schools flooded, but Exshaw school was isolated completely – no highway traffic at all.
Lessons learned that the school would be the central part of the community. Do we have the capacity to house the people? Had open wireless, and the community used it and were exceedingly appreciative. Cell services were shutdown, even emergency services were disconnected. Exshaw school had the only communication method (Internet) available. Able to work from home to get things working. Having a wireless network, Netflix was critical to keep the community intact.
Concerns about flooding. Remote backup exists, but location was also in flooded areas. Some ideas are having more dependency on cloud services.
Eelco and James – Foothills
Flooding came so fast people couldn’t get out fast enough to use their own vehicles. All communication methods went down because of the flooding. Staff were loaded up into the back of pickup trucks to get out of town.
Disaster Recovery Centre
Minimize Contact – Focus on Imperatives – QR Code’s everywhere, so staff could enter their personal e-mail addresses and contact information. Having folks personal e-mail addresses.
Access Granted –
Eelco / Drew – dug everything out.
Many people in central office were still working, even though their own houses were flooding.
Data Centre – Silt, mud, sewage comes with the water. Items stored in storage centre was destroyed. Were 1.5 CM away from losing primary servers. Lost backup storage, backup space. Had everything back up in three days.
Central office staff were relocated to a school, and after three days came in and turned everything on and started working. Payroll run went as scheduled. Website was very busy.
All network passwords were stored in written form, in a non-water proof box.
Rebuilding their disaster recovery plan, noting to put people first. Synrevoice saved their bacon, with call-outs and texts.
Everything happened so fast. There was no normal “end of year” no closure in the schools. People trembling when returning to a school building in the Fall, or later. What are the effects of Post-traumatic stress?
Take care of people first was the resounding message from everyone on the panel.
Is there one piece of advice you would give everyone else?
- Start a disaster recovery plan, and try to review it – make changes now
- You can’t schedule a disaster, but we can practice for it. Take a weekend each year and simulate.
- Don’t assume that it can’t happen to you
- Offsite copy of primary data somewhere else – high and dry!
- Pursue Partnerships