Not only be prepared, but work in the right direction…
June 25th, 2008
Being prepared is not always enough, you also have to have the right mindset to be able to work in the right direction and to be able to overcome obstacles, cause no matter how prepared you may be, the high water may still wash out your home.
A few weeks ago we had a flood, not a literal flood, but we did manage to have the things we had worked on so hard for years get washed away. On Memorial day in the USA, there was a bit of a problem at one of the Data Centers where our sites were located, but that’s not where the water started rising, it started rising right here at home.
We’ve got a pretty good backup system in place, or so we thought, where we have backups saved on our server, plus a backup drive on our local machine, redundancy is important right? Having data in two places should guarantee that nothing should get washed away, right?
Well, in a word, no.
First we had a hard drive crash, our local backup drive crashed, and we were without backups. On the Monday morning we figured we’d simply connect to the server and download some new ones. But the server was acting up and we started looking at what was wrong. Finally the server went down, and we tried to salvage data there. Unfortunately, the server was a write off, and so was the remote backup drive. Yes, the water was getting higher.
So we started looking for alternatives and started working on building the sites, by getting the “shell” ready for a re-launch. We started re-building the sites on our local computer while our tech gurus got the server going again, but the last wave hadn’t rolled in yet….
The computer we had started the rebuilding on crashed. Yes, the water was finally over our heads and we were about to drown, for even an excellent swimmer gets tired while being pushed at by the waves and pulled at by the undertoe.
Fortunately we didn’t drown, although it felt like we did. We had to start from scratch, but we did manage to find a few things to help us along the way, we found ourselves some old design ideas that we were able to start with, we found some data here and there that were important to getting things back up again, we found our way, slowly but surely, to the surface and started again. We’re building on higher ground this time, and we’ve got not only life rafts, but life vests as well. Our backup strategy had taken quite a hit, so we started looking at that first, we’ve now got a quad redundant system in place that we hope will prevent anything from possibly going wrong again.
- Backups on the remote server.
- Backups on the local computer.
- Backups on removable USB drive
- Backups on dvd.
That should be enough, but we’re looking at other options as well. The server was down for a few weeks, well, at least for some, the sites were accessible intermittantly by different people, and we were stalled a bit. So what’s the next thing we can do to prevent future problems? We’re working on it, a second server, and perhaps having that on a second host too. A backup functioning server that will be ready to run at a moments notice, and a cooperative backup that will restore to this remote server as it gets backed up from the primary server.
But all of this costs money, needless to say, I think we’ll be busy for the next little while, refining our backup strategy, and getting it done better this time around.
Yes, be prepared, but be read yto go off in a different direction if you find you’re not going where you want to.
Ray