Skip to content
Tags

JTC Event – Panel on Disaster Recovery

November 27, 2013

This is part of a series of notes from the Alberta Education, School Technology Branch, semi-annual conference called the Jurisdiction Technology Contacts event. This particular one was held at Mount Royal, November 27-28, 2013. These notes are my own.

John Shutte – Calgary Catholic
Data Centre is located on the 7th floor of their building, 200 feet from the river. Flooding came incredibly fast. Downtown Calgary has close calls just about every year. Thought this might be one of them, until it was almost too late. Be careful of becoming complacent – it’s not going to be a big deal!;>…. Eye opening stuff: communications between departments in the school district and the public/media. Were getting 10,000 hits per minute on the Friday morning from people looking for information. Is it easy for your people to update the website from wherever they are. SuperNet and Demarc is three stories below ground. Generators were tested for capacity and load, but not how long they would last and how often they would need to be filled up, and how to get diesel to the generator when there is no power.

Disaster Recovery plan hadn’t been updated in 10 years, and the data centre had changed massively since then.

Completely caught off guard by people. All plans assumed people would be able to come into work. Many gaps happened in the teams with people missing.

Had eight feet of water in one school, completely lost all of the technology in the school. Didn’t have a plan for how to get technology to the school within a reasonable time.

Plans for the future. Distributing some of the services. considering moving some data centre infrastructure to some schools. Perhaps a rack or two. Considering some co-locate facilities. Talking with CBE and other school jurisdictions about spreading infrastructure out.

Mike Rinkel – CBE
Mike reviewed CBE’s experiences
Lessons Learned from Flood
Preparation:

Expectations / Capabilities
Communication Plan
Response Teams
Practice / Incidence Response
Templates and Checklists
Jump Bag and contact lists
Recovery

Surprises:
Disasters develop quickly!
Staff availability
CEMA expectations
Cross departmental contacts
Transportation
Food / Services

Lessons Learned
Distribute your eggs
Supply Depots
Cash
Web Services Capacity
Cross Training
Work from home
Regularly review priorities

Kevin Dyck – Canadian Rockies
No schools flooded, but Exshaw school was isolated completely – no highway traffic at all.

Lessons learned that the school would be the central part of the community. Do we have the capacity to house the people? Had open wireless, and the community used it and were exceedingly appreciative. Cell services were shutdown, even emergency services were disconnected. Exshaw school had the only communication method (Internet) available. Able to work from home to get things working. Having a wireless network, Netflix was critical to keep the community intact.

Concerns about flooding. Remote backup exists, but location was also in flooded areas. Some ideas are having more dependency on cloud services.

Eelco and James – Foothills
Flooding came so fast people couldn’t get out fast enough to use their own vehicles. All communication methods went down because of the flooding. Staff were loaded up into the back of pickup trucks to get out of town.

Relationships
RCMP
Disaster Recovery Centre
Minimize Contact – Focus on Imperatives – QR Code’s everywhere, so staff could enter their personal e-mail addresses and contact information. Having folks personal e-mail addresses.
Access Granted –
Eelco / Drew – dug everything out.
Many people in central office were still working, even though their own houses were flooding.

Data Centre – Silt, mud, sewage comes with the water. Items stored in storage centre was destroyed. Were 1.5 CM away from losing primary servers. Lost backup storage, backup space. Had everything back up in three days.

Central office staff were relocated to a school, and after three days came in and turned everything on and started working. Payroll run went as scheduled. Website was very busy.

All network passwords were stored in written form, in a non-water proof box.

Rebuilding their disaster recovery plan, noting to put people first. Synrevoice saved their bacon, with call-outs and texts.

Everything happened so fast. There was no normal “end of year” no closure in the schools. People trembling when returning to a school building in the Fall, or later. What are the effects of Post-traumatic stress?

Take care of people first was the resounding message from everyone on the panel.

Is there one piece of advice you would give everyone else?

  • Start a disaster recovery plan, and try to review it – make changes now
  • You can’t schedule a disaster, but we can practice for it. Take a weekend each year and simulate.
  • Don’t assume that it can’t happen to you
  • Offsite copy of primary data somewhere else – high and dry!
  • Pursue Partnerships
Advertisements

From → ATLE

Leave a Comment

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: