But what if you want to add one more line of defense – copying the backups locally to your machine from time to time. In this way even if in sequence of unlikely and strange events that corrupt your remote machines, you would have a local backup of your important information. But as in our case most of the time we forget to do a copy to a local machine. That’s why I would like to make it automatic.

What we are currently doing is copying a backup from remote to local every time your machine starts its Internet interface. And yes, this would would not scale for GB backups. But we are still not there.

This is the script:

#! /bin/sh

# Copies production_backup on this machine

# This script should be copied at /etc/network/if-up.d/. In this way everytime the network starts a backup would be made
export REPOSITORY=/home/kireto/robopartans/rbp.cm
export USER_WITH_ACCESS=kireto

 

echo “Starting script:” >> /tmp/production_backup_copy.log
set -e

 

# Don’t bother when lo is configured.
if [ "$IFACE" = lo ]; then
exit 0
fi

 

#~ # Only run from ifup.
if [ "$MODE" != start ]; then
exit 0
fi

 

FILE=”production_backup$(date +’%Y_%m_%d’).sql.gz”
LOCAL_PATH=”$REPOSITORY/$FILE”
REMOTE_PATH=”\”/var/jenkins/jobs/Production\\ backup/workspace/$FILE\”"

if [ -f $LOCAL_PATH ]; then
echo “File exists locally” >> /tmp/production_backup_copy.log
exit 0
fi

 

sudo su – $USER_WITH_ACCESS -c “cd $REPOSITORY; scp root@elvis:$REMOTE_PATH .”
echo “Pulling at: $(date +”%D %T”)” >> /tmp/production_backup_copy.log

exit 0

This script is then added at

/etc/network/if-up.d

Now we have a distributed backup system, where simultaneous fail of all the machines and processes is impossible (except if someone starts a earth big Electromagnetic impulse)

Our server runs on a machine with a mysql as a database. Since we are a number of people on the team we decided to do this marvelous, genius idea. We would backup our production database every day with a sql dump and commit this dump it into git. Then when each of us does his normal job and checkouts other code changes he would also checkout the database. And the best part is that Git can manage differences between files so we are not actually storing the whole file. We store only the differences and this difference most of the time is in the users table – who has logged in, when, what has he done.

As a result we would have the database and all of it history distributed on several machines and we would have the whole history of each and every day. Just genius.

Turns out this is not a good idea. It took git only 315 commits of our not that large 50 MB database to stop working. This “fast”, “supppper speed”, “one of a kind” version control system just stopped working after the 315 build . When we were trying to pull and this was the error occurring:

remote: Counting objects: 677, done.
error: pack-objects died of signal 9/461)
error: git upload-pack: git-pack-objects died with error.
fatal: git upload-pack: aborting due to possible repository corruption on the remote side.
remote: aborting due to possible repository corruption on the remote side.
fatal: protocol error: bad pack header

So we took another approach

• Delete the whole backup branch locally and remotely

git branch -D production_backup

git push origin –delete production_backup

(locally) – git gc

(remotely) – git gc

• Setup a new production backup process where you have daily, weekly and monthly backup

• Daily – runs every day and only the last 8 backups are saved

mysqldump –user=root –password=ThePassword –databases production | gzip > production_backup$(date +”%Y_%m_%d”).sql.gz

• Weekly – runs every week and only the last 5 backups are saved

mysqldump –user=root –password=ThePassword –databases production | gzip > production_backup$(date +”%Y_%m_%d”).sql.gz

• Monthly – runs every month and only the last 13 are saved.

mysqldump –user=root –password=ThePassword –databases production | gzip > production_backup$(date +”%Y_%m_%d”).sql.gz

Take a look at the gzip part. Since sql dumps more often than not, contain text information you should always gzip them. And the $(date +”%Y_%m_%d”) just puts the current date in the file name.

Now we just have to figure our when an how to copy this backups to another machine

We were working on the campaigns downloads in Retiffy. This means generating a PDF for every certificate and then concatenating them into a single PDF and sending the link to the user. Now the problem is that this is done in a delayed_job and if you change the production databases and code without restarting delayed_job, the job preformed at a delay are still using the old code.

We haven`t dig deep enough to understand the internals and whether we could fix it in another way. Currently we are searching for the best solution to restart delayed_job on building production.

This is the initial source of the Ubuntu service that we have come up with. Hope it could help you. We will continue working on it with for restart and stop.

#!/bin/bash
# Delayed job for Retiffy production
# acts as startup service.
# USAGE: start
case “$1″ in
start)
echo “Starting Delayed_job for Retiffy Production”
RAILS_ENV=production
export RAILS_ENV
start-stop-daemon –start –pidfile /var/run/delayed_job.pid –make-pidfile –background –chuid www-data –exec /production/script/delayed_job — restart -i 2 || return 2
;;

esac
exit 0