[Vserver] linux-vserver as a "poor man's warm standby solution"

From: Marcel Gsteiger <Marcel.Gsteiger_at_milprog.ch>
Date: Sun 19 Mar 2006 - 15:42:22 GMT
Message-Id: <s41d8a11.013@milprog1.milprog.ch>

Hi all

I successfully installed two similar boxes running FC4 w/linux-vserver, implementing a hot-standby of each vserver on the other box. All VServers are simply installed on both boxes in the same directory structure; half of the servers are active on box A and standby on box B, the other half is active on box B and standby on box A. This is what I call a "poor man's warm standby solution", i.e. without the need for a SAN infrastructure. This setup does "load balancing" by having the active vserver on the CPU of my choice statically. On both boxes a cronjob runs once every hour (of course not both at the same time) that takes a LVM snapshot of the vserver partition and then uses rsync to mirror the active server(s) to the standby box. This setup works very well now for several weeks. Depending on the data changing and overall load, the hourly script needs about 1-5 minutes to sync 8 vservers in my case, together occupying about 23 GB of disk space (running on a x86_64 2.8GHz Dual Xeon box w/software RAID 1 SATA disks).

For anyone interested, find my scripts below that do the work of syncing the servers and migrate a vserver from one mirror to the other (switchover).. As a prerequisite, each of the two boxes should be able to connect to the other one via SSH without using passwords, i.e. by public key authentication. The same scripts are installed on both servers, they automatically find out where they are running and what to do. I also have a .lan domain on my DNS to avoid having IP addresses in the script, but of course this can be adapted easily. In my own setup, I have a few additional site-specific scripts that take care of networking issues (firewalling, VLAN setup, static routes etc), this is necessary in my case since my vservers have RFC1918 IPs and are accessible from the outside ony via NAT.

Syncing the /etc/vserver directory has to be done manually, my script just rsyncs the /vservers partition.

Of course, the scripts come without any warranty whatsoever, all bugs have been added by me ;)

Regards
--Marcel

<file rsync_vservers>
#!/bin/bash
# synchronizes all running vservers to mirror and checks the status.
# Does not automatically launch vservers; this has to be done manually.
# stdout can typically be piped into logger from within a cronjob, e.g.
# /etc/rsync_vservers | /usr/bin/logger -t rsync_vservers
# 3-MAR-2006 milprog/mg
# 10-MAR-2006 milprog/mg : adapted to retry snapshot creation and wait 1 second before mounting

if [ "$HOSTNAME" = "milprogmail.ch" ]
then
  THISHOST=milprogmail.milnet.lan
else
  THISHOST=$HOSTNAME
fi

MIRROR1=lepus.milnet.lan
MIRROR2=milprogmail.milnet.lan
# maximum 1 MB/s bandwidth please during rsync
BWLIMIT=1024
SNAPSHOTSIZE=4096M

VSERVERVG=/dev/vg00
VSERVERLV=lv00_vservers
VSERVERSNAPSHOT=vservers_snapshot
SNAPSHOTMOUNT=/mnt/vservers-snapshot
VSERVERBASE=/vservers

# no need to change anything below (hopefully)

VSERVERCMD=/usr/sbin/vserver
RSYNCCMD=/usr/bin/rsync
LVCREATECMD=/usr/sbin/lvcreate
LVREMOVECMD=/usr/sbin/lvremove
MOUNTCMD=/bin/mount
UMOUNTCMD=/bin/umount
SSHCMD=/usr/bin/ssh
LVSCMD=/usr/sbin/lvs
AWKCMD=/bin/awk
BASENAMECMD=/bin/basename
DATECMD=/bin/date
SYNCCMD=/bin/sync
SLEEPCMD=/bin/sleep

if [ "$THISHOST" = "$MIRROR1" ]
then
   MIRROR=$MIRROR2
fi
if [ "$THISHOST" = "$MIRROR2" ]
then
   MIRROR=$MIRROR1
fi
if [ "$MIRROR" = "" ]
then
 echo "ERROR: VServer resync must be installed either on $MIRROR1 or $MIRROR2, not here ($THISHOST)"
 exit 1
fi
TIMESTART=`$DATECMD +%s`
RETRIES=0
MAXRETRIES=5
$SYNCCMD
if $LVSCMD --noheadings ${VSERVERVG}/${VSERVERSNAPSHOT} >/dev/null 2>&1
then
      echo "ERROR: Snapshot overlap; cannot rsync from $THISHOST to $MIRROR"
else
# echo "creating LV..."
# $LVSCMD
 if $LVCREATECMD -L${SNAPSHOTSIZE} -s --chunksize 4k -n ${VSERVERVG}/${VSERVERSNAPSHOT} ${VSERVERVG}/${VSERVERLV} >/dev/null 2>&1 && $MOUNTCMD -t ext2 -ni -o noatime,ro ${VSERVERVG}/${VSERVERSNAPSHOT} ${SNAPSHOTMOUNT} >/dev/null 2>&1
 then
  :
 else
# echo "retrying to create snapshot..."
  while [ $RETRIES -lt "$MAXRETRIES" ]
  do
    RETRIES=`expr $RETRIES + 1`
    $SYNCCMD
    $SLEEPCMD 1
    $UMOUNTCMD -ni $SNAPSHOTMOUNT
    $SLEEPCMD 1
    $LVREMOVECMD -f ${VSERVERVG}/${VSERVERSNAPSHOT} >/dev/null 2>&1
    $SLEEPCMD 1
    $SYNCCMD
    $LVREMOVECMD -f ${VSERVERVG}/${VSERVERSNAPSHOT} >/dev/null 2>&1
    $SLEEPCMD 1
    if $LVCREATECMD -L${SNAPSHOTSIZE} -s --chunksize 4k -n ${VSERVERVG}/${VSERVERSNAPSHOT} ${VSERVERVG}/${VSERVERLV} >/dev/null 2>&1 && $MOUNTCMD -t ext2 -ni -o noatime,ro ${VSERVERVG}/${VSERVERSNAPSHOT} ${SNAPSHOTMOUNT} >/dev/null 2>&1
    then
     break
    fi
  done
  if [ $RETRIES -eq "$MAXRETRIES"]
  then
      echo "ERROR: cannot create snapshot after `expr $MAXRETRIES-1` retries"
      $LVREMOVECMD -f ${VSERVERVG}/${VSERVERSNAPSHOT} >/dev/null 2>&1
      exit 2
  fi
 fi
   NACTIVE=0
   NSTANDBY=0
   NINACTIVE=0
   NERROR=0
   for i in /etc/vservers/*; do
    VSERVER=`$BASENAMECMD $i`
    if $VSERVERCMD $VSERVER running
    then
# echo "vserver $VSERVER running on $THISHOST"
     if $SSHCMD $MIRROR $VSERVERCMD $VSERVER running
     then
      NERROR=`expr $NERROR + 1`
      echo "ERROR: vserver $VSERVER running both on $THISHOST and $MIRROR"
     else
      NACTIVE=`expr $NACTIVE + 1`
# echo "rsyncing vserver $VSERVER from $THISHOST to $MIRROR..."
      if $RSYNCCMD -aqzpotSx4 --numeric-ids --bwlimit $BWLIMIT --recursive --safe-links --delete --force -e ssh ${SNAPSHOTMOUNT}/${VSERVER}/ $MIRROR:${VSERVERBASE}/${VSERVER}/
      then
# echo "rsync vserver $VSERVER from $THISHOST to $MIRROR done"
       :
      else
       echo "ERROR: cannot rsync $VSERVER from $THISHOST to $MIRROR"
      fi
    fi
    else
     if $SSHCMD $MIRROR $VSERVERCMD $VSERVER running
     then
# echo "vserver $VSERVER running on $MIRROR"
      NSTANDBY=`expr $NSTANDBY + 1`
     else
      NINACTIVE=`expr $NSTANDBY + 1`
      echo "WARNING: vserver $VSERVER inactive both on $THISHOST and $MIRROR"
     fi
    fi;
   done
   $SYNCCMD
   if $UMOUNTCMD -ni $SNAPSHOTMOUNT
   then
# echo "unmount of $SNAPSHOTMOUNT done"
    :
   else
    echo "ERROR: cannot unmount LV snapshot"
   fi
## else
## echo -n `$DATECMD +"%x %X" `
## echo " ERROR: cannot mount LV snapshot"
## fi
  SNAPPERCENT=`$LVSCMD --noheadings ${VSERVERVG}/${VSERVERSNAPSHOT} | $AWKCMD '{print $6}'`
  $SYNCCMD
  if $LVREMOVECMD -f ${VSERVERVG}/${VSERVERSNAPSHOT} >/dev/null 2>&1
  then
   TIMEEND=`$DATECMD +%s`
   echo "NOTICE: rsync_vservers complete; elapsedsecs=`expr $TIMEEND - $TIMESTART`; snapshotsize=$SNAPSHOTSIZE; snap%=$SNAPPERCENT; active=$NACTIVE; standby=$NSTANDBY; inactive=$NINACTIVE; error=$NERROR; snapshotretries=$RETRIES"
# $LVSCMD
  else
   echo "ERROR: cannot remove LV snapshot"
  fi
fi
</file>

<file migrate_vserver>
#!/bin/bash
# gets 2 parameters: vserver_name source_server
# ensures that vserver_name is running on source_server and standby on the corresponding mirror
# stops the vserver on source_server, does rsync of the remaining data and restarts on mirror
# if only the vserver_name is given, the script assumes that the server it's running on should become the new active server
#
# 19-MAR-2006 milprog/mg

if [ "$HOSTNAME" = "milprogmail.ch" ]
then
  THISHOST=milprogmail.milnet.lan
else
  THISHOST=$HOSTNAME
fi

MIRROR1=lepus.milnet.lan
MIRROR2=milprogmail.milnet.lan
SNAPSHOTSIZE=4096M
VSERVERVG=/dev/vg00
VSERVERLV=lv00_vservers
VSERVERSNAPSHOT=vservers_snapshot
SNAPSHOTMOUNT=/mnt/vservers-snapshot
# parameters for taking the snapshot: number of retries, interval in seconds
MAXSNAPSHOTRETRIES=3
RETRYINTERVAL=250

# no need to change anything below (hopefully)

LVCREATECMD=/usr/sbin/lvcreate
LVREMOVECMD=/usr/sbin/lvremove
LVSCMD=/usr/sbin/lvs
VSERVERCMD=/usr/sbin/vserver
VSERVERROOT=/vservers
RSYNCCMD=/usr/bin/rsync
SSHCMD=/usr/bin/ssh
AWKCMD=/bin/awk
BASENAMECMD=/bin/basename
DATECMD=/bin/date
SYNCCMD=/bin/sync
SLEEPCMD=/bin/sleep

if [ "$THISHOST" = "$MIRROR1" ]
then
   MIRROR=$MIRROR2
fi
if [ "$THISHOST" = "$MIRROR2" ]
then
   MIRROR=$MIRROR1
fi
if [ "$MIRROR" = "" ]
then
 echo "ERROR: $0 must be installed either on $MIRROR1 or $MIRROR2, not here ($THISHOST)"
 exit 1
fi

VSERVER=$1
if [ "$VSERVER" == "" ]
then
 echo "ERROR: VServer name missing"
 exit 2
fi
MIGRATEFROM=$2
if [ "$MIGRATEFROM" == "" ]
then
 MIGRATEFROM=$MIRROR
 MIGRATETO=$THISHOST
else
 if [ $MIGRATEFROM == "$THISHOST" ]
 then
  MIGRATETO=$MIRROR
 else
  if [ $MIGRATEFROM == "$MIRROR" ]
  then
   MIGRATETO=$THISHOST
  else
   echo "ERROR: $MIGRATEFROM is not a valid host for this script (use $THISHOST or $MIRROR)"
   exit 3
  fi
 fi
fi
if [ $MIGRATEFROM == "$THISHOST" ]
then
 RUNFROM=""
 RUNTO="$SSHCMD $MIGRATETO "
else
 RUNFROM="$SSHCMD $MIGRATEFROM "
 RUNTO=""
fi

if $RUNFROM $VSERVERCMD $VSERVER running >/dev/null 2>&1
then
  if $RUNTO $VSERVERCMD $VSERVER running >/dev/null 2>&1
  then
   echo "ERROR: vserver $VSERVER active on both $MIGRATEFROM and $MIGRATETO"
   exit 5
  else
   if [ $? -gt 1 ]
   then
    echo "ERROR: vserver $VSERVER unknown on $MIGRATETO as a standby server"
    exit 6
   else
    if $RUNFROM $LVCREATECMD -L${SNAPSHOTSIZE} -s --chunksize 4k -n ${VSERVERVG}/${VSERVERSNAPSHOT} ${VSERVERVG}/${VSERVERLV} >/dev/null 2>&1
    then
     TIMESTART=`$DATECMD +%s`
     if $RUNFROM $VSERVERCMD $VSERVER stop >/dev/null 2>&1
     then
      if $RUNFROM $RSYNCCMD -aqzpotSx4 --numeric-ids --recursive --safe-links --delete --force -e ssh ${VSERVERROOT}/${VSERVER}/ $MIGRATETO:${VSERVERROOT}/${VSERVER}/
      then
       if $RUNTO $VSERVERCMD $VSERVER start >/dev/null 2>1
       then
        TIMEEND=`$DATECMD +%s`
        echo "NOTICE: vserver $VSERVER migrated from $MIGRATEFROM to $MIGRATETO; elapsedsecs=`expr $TIMEEND - $TIMESTART`"
       else
        if $RUNFROM $VSERVERCMD $VSERVER start >/dev/null 2>&1
        then
         echo "ERROR: cannot start $VSERVER after migration from $MIGRATEFROM to $MIGRATETO; server was restarted again on $MIGRATEFROM"
         exit 8
        else
         echo "ERROR: cannot start $VSERVER after migration from $MIGRATEFROM to $MIGRATETO; server could not be restarted again on $MIGRATEFROM and is now dead on both mirrors"
         exit 9
        fi
       fi
      else
        if $RUNFROM $VSERVERCMD $VSERVER start >/dev/null 2>&1
        then
         echo "ERROR: cannot rsync $VSERVER for migration from $MIGRATEFROM to $MIGRATETO; server was restarted again on $MIGRATEFROM"
         exit 10
        else
         echo "ERROR: cannot rsync $VSERVER for migration from $MIGRATEFROM to $MIGRATETO; server could not be restarted again on $MIGRATEFROM and is now dead on both mirrors"
         exit 11
        fi
      fi
     else
      echo "ERROR: vserver $VSERVER cannot be stopped on $MIGRATEFROM"
      exit 7
     fi
     if $RUNFROM $LVREMOVECMD -f ${VSERVERVG}/${VSERVERSNAPSHOT} >/dev/null 2>&1
     then
      :
     else
       echo "ERROR: snapshot ${VSERVERVG}/${VSERVERSNAPSHOT} cannot be removed on $MIGRATEFROM after successful migration"
       exit 13
     fi
    else
     echo "ERROR: cannot migrate $VSERVER from $MIGRATEFROM to $MIGRATETO because snapshot ${VSERVERVG}/${VSERVERSNAPSHOT} cannot be established (possible overlap with backup/cronjob)"
     exit 12
    fi
   fi
  fi
else
  echo "ERROR: vserver $VSERVER not running on $MIGRATEFROM"
  exit 4
fi
</file>

_______________________________________________
Vserver mailing list
Vserver@list.linux-vserver.org
http://list.linux-vserver.org/mailman/listinfo/vserver
Received on Sun Mar 19 15:44:02 2006

[Next/Previous Months] [Main vserver Project Homepage] [Howto Subscribe/Unsubscribe] [Paul Sladen's vserver stuff]
Generated on Sun 19 Mar 2006 - 15:44:06 GMT by hypermail 2.1.8