But what if you want to add one more line of defense – copying the backups locally to your machine from time to time. In this way even if in sequence of unlikely and strange events that corrupt your remote machines, you would have a local backup of your important information. But as in our case most of the time we forget to do a copy to a local machine. That’s why I would like to make it automatic.

What we are currently doing is copying a backup from remote to local every time your machine starts its Internet interface. And yes, this would would not scale for GB backups. But we are still not there.

This is the script:

#! /bin/sh

# Copies production_backup on this machine

# This script should be copied at /etc/network/if-up.d/. In this way everytime the network starts a backup would be made
export REPOSITORY=/home/kireto/robopartans/rbp.cm
export USER_WITH_ACCESS=kireto

 

echo “Starting script:” >> /tmp/production_backup_copy.log
set -e

 

# Don’t bother when lo is configured.
if [ "$IFACE" = lo ]; then
exit 0
fi

 

#~ # Only run from ifup.
if [ "$MODE" != start ]; then
exit 0
fi

 

FILE=”production_backup$(date +’%Y_%m_%d’).sql.gz”
LOCAL_PATH=”$REPOSITORY/$FILE”
REMOTE_PATH=”\”/var/jenkins/jobs/Production\\ backup/workspace/$FILE\”"

if [ -f $LOCAL_PATH ]; then
echo “File exists locally” >> /tmp/production_backup_copy.log
exit 0
fi

 

sudo su – $USER_WITH_ACCESS -c “cd $REPOSITORY; scp root@elvis:$REMOTE_PATH .”
echo “Pulling at: $(date +”%D %T”)” >> /tmp/production_backup_copy.log

exit 0

This script is then added at

/etc/network/if-up.d

Now we have a distributed backup system, where simultaneous fail of all the machines and processes is impossible (except if someone starts a earth big Electromagnetic impulse)

We were working on the campaigns downloads in Retiffy. This means generating a PDF for every certificate and then concatenating them into a single PDF and sending the link to the user. Now the problem is that this is done in a delayed_job and if you change the production databases and code without restarting delayed_job, the job preformed at a delay are still using the old code.

We haven`t dig deep enough to understand the internals and whether we could fix it in another way. Currently we are searching for the best solution to restart delayed_job on building production.

This is the initial source of the Ubuntu service that we have come up with. Hope it could help you. We will continue working on it with for restart and stop.

#!/bin/bash
# Delayed job for Retiffy production
# acts as startup service.
# USAGE: start
case “$1″ in
start)
echo “Starting Delayed_job for Retiffy Production”
RAILS_ENV=production
export RAILS_ENV
start-stop-daemon –start –pidfile /var/run/delayed_job.pid –make-pidfile –background –chuid www-data –exec /production/script/delayed_job — restart -i 2 || return 2
;;

esac
exit 0