Disaster Recovery and backup#

Currently we support Source Control for all the projects and software and on bioinfdev and bioinf4 there are a number of cron jobs running. On bioinfdev a cron job runs that copies all the extant svn directories to /backup/. On bioinf4 a cron job runs that archives the old webserver jobs (we keep a history of 6 months), and also archives the configuration and user tables in the database.

bioinfdev

Backup locationContents
/backupA complete mirror of the /svndata directory on bioinfdev
/backup/testingThe staging area for the compliance testing and the automated build system

bioinf4

Backup locationContents
/backupArchive mysql dumps for the jobs, request_results, and job_config_overrides tables from the rails db
/backup/disaster_recoveryArchive mysql dumps for the configurations, configurations_server, configurations_entries and users tables from the rails db

We don't for now backup /webdata/data/ or /webdata/binaries on bioinf4. We probably should although it is truly massive so it's not clear where it could/should go. In lieu of that in /backup/backup/ there are files of the directory listing that should tell you what software needs to be replaced in the event of a catastrophic failure. /webdata on bioinfdev is non-critical and under a fair degree of constant tinkering, backing this up would be very time consuming and resource hungry

Previous Catastrophic failures#

From time to time the server might completely die. Here is some documentation on previous catastrophic fails and their solutions

mySQL database Corruption#

If/when this happens there may or may not be a lot you can do about it. mysqld will probably refuse to start and spew out a whole load of stuff into the mysqld.log (/var/log/), you may wish to read that. You may have to force startup on the database.

  • 1 - Go to /var/lib/mysql and delete the inno_log files and start mysqld, if these are corrupt that db won't come up and this is the simplest fix
  • 2 - Failing that try http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html
  • 3 - Failing that you may wish to delete the innodb file and the logs, delete the directories for the database and restart mysqld afresh.

If you end up doing 3 you will need to add a RoR and root users back into the mysql database instance. You will have to rake the rails database to regenerate it. Luckily you won't have to manually redo the application or NewPred user settings. On bioinf4 /backup/disaster_recovery you will find dumps of all the settings and user information you need to get the database back to a working state. You will just have to live with the loss of all the job data.

Add new attachment

Only authorized users are allowed to upload new attachments.
« This page (revision-1) was last changed on 11-Feb-2013 14:57 by UnknownAuthor