Are you prepared to restore your operations after a disaster hits? You may have a fantastic backup platform, and that platform may even include off-site storage capabilities, but do you know how to use your platform for data center disaster recovery, and get your systems back up and running as quickly as possible?
Let's cut through the marketing hype and ask some tough questions. One blog post is not sufficient to provide all of the answers to every scenario - solutions will vary according to budgets, technical capabilities, and general preferences. My hope is that these questions will help you to think about the big picture and where your organization stands.
 |
| Image generated by DALL-E v3 (OpenAI/Microsoft Copilot) |
Key Considerations for Disaster Recovery
Key considerations for disaster recovery extend far beyond having a backup solution in place. Here, we highlight the essential elements that should be part of your disaster recovery plan, ensuring that your organization is prepared to respond effectively to any incident and minimize its impact on your operations.
- Do you have playbooks to help you deal with specific scenarios?
- Earthquake, fire, or flood at your data center (damaged equipment)
- Malware or ransomware encrypting your data
- Data loss due to equipment failure or user error
- Extended power outage due to construction or environmental issue
- Loss of WAN or internet connectivity due to damaged fiber
- Do you have defined thresholds regarding how long data should be kept?
- Does your organization have policies that define how long you need to hold on to records?
- Are there state or federal regulations that dictate how long you need to keep data?
- If someone loses a valuable file and asks to have it recovered, how long should you be expected to be able to recover it? One week? One month? One year?
- If you discover that a server was silently breached and monitored by intruders for several months, are you prepared to rebuild the server from scratch? Or will you need a clean backup of that server (archived before the breach occurred)?
- If your primary data center or WAN aggregation point becomes unusable due to natural disaster or fiber cut, are you prepared to restore you operations elsewhere?
- Do you have defined thresholds regarding how long your services can be down before school is cancelled?
- Do you have a secondary data center ready to take over operations?
- Does your secondary data center automatically take over when the primary fails, or are there manual restore operations that need to occur?
- Is data continuously synchronized between the primary and secondary data centers, or is there some amount of data that would be lost during a failover?
- Do your schools or remote sites have network connectivity to the secondary data center?
- Are there specific network configurations or operations that need to occur in order to access data and the secondary data center?
- Do you have a backup internet connection that can be used in case the primary internet circuit is no longer accessible?
- Do you have firewall and content filtering services protecting the backup internet connection?
- If you are using AWS/Azure/GCP to host your VMs for restore operations, are you aware of the processes and configurations required to fail over operations to those cloud-hosted VMs?
- IP addressing
- DNS updates
- VPN requirements for end users
Data Backup Strategies
With the increasing reliance on cloud-hosted and Software as a Services (SaaS) platforms, the importance of safeguarding data against loss or corruption has never been more critical. Our goal is to equip organizations with the knowledge to develop a comprehensive data backup strategy that aligns with their operational requirements and risk management policies, ensuring resilience in the face of adversity.
- Do you have backups of your cloud-hosted or SaaS platforms?
- Many cloud-hosted platforms will guarantee system uptime but will not provide any guarantees regarding data integrity. Are you prepared to restore your cloud-hosted data in the event that it is unintentionally or maliciously wiped or corrupted?
- Are you aware of the service level agreements (SLAs) provided by your cloud/SaaS platforms?
- Have you documented the process for restoring data to your cloud-hosted services in the event of data corruption?
- Are your backups secure?
- Is your data backed up in multiple locations – to prevent issues with one copy being stolen or corrupted?
- Are your backups 'immutable' – prevented from being changed or overwritten?
- Is your backed up data encrypted – so as to be unreadable if stolen?
- If you are using tapes or removable media, are they tracked and stored in a secure (fireproof) location? If there are offsite copies of the removable media, are those copies kept in a secured container or fireproof safe?
- Is the backup data encrypted as it is transmitted over the network?
- Is your team ready to handle restoration efforts in an emergency?
- Have you determined team members responsibilities for business continuity duties? Each team member should know what they are responsible for during an emergency.
- Have you mapped out alternative roles in the event that a key individual is incapacitated or otherwise unavailable during an emergency?
- Do you have readily-available contact information for ISPs, circuit providers, cloud service providers, etc.?
- Do you have documented procedures regarding how to restore operations for various scenarios?
- Have you determined team members responsibilities for business continuity duties? Each team member should know what they are responsible for during an emergency.
- Is there alignment between the expectations of cabinet vs. the budget provided for disaster recovery solutions?
- Are the expected recovery point objective (RPO) and recovery time objective (RTO) reasonable for the budget provided? Alternatively, is the budget sufficient for the RPO/RTO expectations of cabinet?
- Is there a documented process regarding how much information is provided to end users and/or outside organizations during extended outages? How often should updates be provided?
- Are there specific communications requirements regarding ransomware or breached systems and the potential exposure of personally identifiable information (PII)?
Overcoming Challenges in Disaster Recovery
Unfortunately, there is no 'one size fits all' solution when it comes to disaster recovery and business continuity. The best solutions are determined as a result of understanding your environment, the expectations of your cabinet and decision makers, the technology options available to you, and your budget.
Resources for K-12 School Districts in California
A great source of information for K-12 school districts in California is the Disaster Recovery Resources archive created by the California County Superintendents' Technology Services Committee.
The link below is via ACSA; Downloading this PDF will provide links that will take you to the documents within the TSC's Google Drive.
The California Department of Education has additional resources that expand beyond the technical aspects of data recovery.
As always, I'd be happy to offer advice or assistance to anyone in the California education space regarding this or any other IT-related topics.
Ready to elevate your disaster recovery and data backup strategies? Contact NIC Partners today to explore tailored solutions that ensure your organization's reilience and continuity. Our team of experts is here to guide you through the latest in data protection and recovery planning, helping you safeguard your operations against any scenario. Don't wait for a disaster to test your preparedness – reach out now and secure your peace of mind with NIC Partners.