Three Months of a Small Site (4): Moving House, and a Two-Month-Old Backup
Previous: Red for Up, or Green for Up? | Next: One Letter, Two Return Addresses | 中文版:搬家,以及一份两个月前的备份
The site first lived on a small, old server, with all its data in a single file. For one user that was ideal: simple, easy to move, and if anything went wrong you just copied the file away.
Then more people came, along with sign-in, watchlists, portfolios and alerts, and the little house got crowded. So it moved: to a roomier new server, with the data in a sturdier kind of storage.
The move went smoothly. On the day the new home went live I told myself: don’t tear down the old one yet; keep it as a standby, in case we need to switch back.
That sounds prudent. That sentence was the problem.
The forgotten old house
Some time later I went to look at the old server because it seemed to be struggling: short on memory, and borrowing disk space to cope.
I assumed the site’s old program was hogging it. It turned out not to be the main culprit; other things on the same machine were. But along the way I noticed something else.
The old copy of the site had been working diligently the whole time. Every fifteen minutes it fetched market prices and wrote them to its own file. Now and then two jobs tried to write that file at once, blocked each other, failed, and simply tried again next time.
Yet nobody was visiting it. The address had long since pointed at the new home. The data it collected was read by no one and sent nowhere. It worked alone in an empty house for weeks, knocking on the data sources’ doors for nothing.
A standby is not a backup
The question that followed was the one that made me break into a cold sweat: if the new home died tomorrow, what could the old one save?
Almost nothing.
The old home’s data was market prices it had fetched for itself. People’s accounts, watchlists, portfolios and alert settings lived only in the new home. The so-called standby was an old car that still ran, with none of your luggage in it.
So I went looking for a real backup in the new home. I found one, made in passing on moving day. It was more than two months old.
In other words, if the new server’s disk had failed that day, every watchlist and portfolio entry from those two months would have vanished with it.
What happened next
None of it was hard; it just hadn’t been done:
- Retire the old home completely. First archive its programs, data and settings in one package and check the package is complete, then stop and remove everything. Archive before dismantling, so there is still something to look at if you ever need it.
- Back up the new home every day. An automatic backup every night, keeping the last two weeks, and after each one, a check that the backup can actually be read back, not merely that “a file was created”.
- Don’t keep the backup only at home. A backup on the same machine as the original can’t survive the machine itself failing. A copy has to go somewhere else.
Takeaways
The lesson fits in a few lines:
- Look in on things you are “keeping as a standby” and see what they are doing;
- A standby is not a backup; first ask “what could it actually restore?”;
- Backups must be scheduled, verified, and stored off the machine;
- Archive before retiring something, then retire it completely.
The next post ends the series, with a sign-in that needed debugging.
Comments