Hi all
I successfully installed two similar boxes running FC4 w/linux-vserver, implementing a hot-standby of each vserver on the other box. All VServers are simply installed on both boxes in the same directory structure; half of the servers are active on box A and standby on box B, the other half is active on box B and standby on box A. This is what I call a "poor man's warm standby solution", i.e. without the need for a SAN infrastructure. This setup does "load balancing" by having the active vserver on the CPU of my choice statically. On both boxes a cronjob runs once every hour (of course not both at the same time) that takes a LVM snapshot of the vserver partition and then uses rsync to mirror the active server(s) to the standby box. This setup works very well now for several weeks. Depending on the data changing and overall load, the hourly script needs about 1-5 minutes to sync 8 vservers in my case, together occupying about 23 GB of disk space (running on a x86_64 2.8GHz Dual Xeon box w/software RAID 1 SATA disks).
For anyone interested, find my scripts below that do the work of syncing the servers and migrate a vserver from one mirror to the other (switchover).. As a prerequisite, each of the two boxes should be able to connect to the other one via SSH without using passwords, i.e. by public key authentication. The same scripts are installed on both servers, they automatically find out where they are running and what to do. I also have a .lan domain on my DNS to avoid having IP addresses in the script, but of course this can be adapted easily. In my own setup, I have a few additional site-specific scripts that take care of networking issues (firewalling, VLAN setup, static routes etc), this is necessary in my case since my vservers have RFC1918 IPs and are accessible from the outside ony via NAT.
Syncing the /etc/vserver directory has to be done manually, my script just rsyncs the /vservers partition.
Of course, the scripts come without any warranty whatsoever, all bugs have been added by me ;)
Regards
--Marcel
<file rsync_vservers>
#!/bin/bash
# synchronizes all running vservers to mirror and checks the status.
# Does not automatically launch vservers; this has to be done manually.
# stdout can typically be piped into logger from within a cronjob, e.g.
# /etc/rsync_vservers | /usr/bin/logger -t rsync_vservers
# 3-MAR-2006 milprog/mg
# 10-MAR-2006 milprog/mg : adapted to retry snapshot creation and wait 1 second before mounting
if [ "$HOSTNAME" = "milprogmail.ch" ]
then
THISHOST=milprogmail.milnet.lan
else
THISHOST=$HOSTNAME
fi
MIRROR1=lepus.milnet.lan
MIRROR2=milprogmail.milnet.lan
# maximum 1 MB/s bandwidth please during rsync
BWLIMIT=1024
SNAPSHOTSIZE=4096M
VSERVERVG=/dev/vg00
VSERVERLV=lv00_vservers
VSERVERSNAPSHOT=vservers_snapshot
SNAPSHOTMOUNT=/mnt/vservers-snapshot
VSERVERBASE=/vservers
# no need to change anything below (hopefully)
VSERVERCMD=/usr/sbin/vserver
RSYNCCMD=/usr/bin/rsync
LVCREATECMD=/usr/sbin/lvcreate
LVREMOVECMD=/usr/sbin/lvremove
MOUNTCMD=/bin/mount
UMOUNTCMD=/bin/umount
SSHCMD=/usr/bin/ssh
LVSCMD=/usr/sbin/lvs
AWKCMD=/bin/awk
BASENAMECMD=/bin/basename
DATECMD=/bin/date
SYNCCMD=/bin/sync
SLEEPCMD=/bin/sleep
if [ "$THISHOST" = "$MIRROR1" ]
then
MIRROR=$MIRROR2
fi
if [ "$THISHOST" = "$MIRROR2" ]
then
MIRROR=$MIRROR1
fi
if [ "$MIRROR" = "" ]
then
echo "ERROR: VServer resync must be installed either on $MIRROR1 or $MIRROR2, not here ($THISHOST)"
exit 1
fi
TIMESTART=`$DATECMD +%s`
RETRIES=0
MAXRETRIES=5
$SYNCCMD
if $LVSCMD --noheadings ${VSERVERVG}/${VSERVERSNAPSHOT} >/dev/null 2>&1
then
echo "ERROR: Snapshot overlap; cannot rsync from $THISHOST to $MIRROR"
else
# echo "creating LV..."
# $LVSCMD
if $LVCREATECMD -L${SNAPSHOTSIZE} -s --chunksize 4k -n ${VSERVERVG}/${VSERVERSNAPSHOT} ${VSERVERVG}/${VSERVERLV} >/dev/null 2>&1 && $MOUNTCMD -t ext2 -ni -o noatime,ro ${VSERVERVG}/${VSERVERSNAPSHOT} ${SNAPSHOTMOUNT} >/dev/null 2>&1
then
:
else
# echo "retrying to create snapshot..."
while [ $RETRIES -lt "$MAXRETRIES" ]
do
RETRIES=`expr $RETRIES + 1`
$SYNCCMD
$SLEEPCMD 1
$UMOUNTCMD -ni $SNAPSHOTMOUNT
$SLEEPCMD 1
$LVREMOVECMD -f ${VSERVERVG}/${VSERVERSNAPSHOT} >/dev/null 2>&1
$SLEEPCMD 1
$SYNCCMD
$LVREMOVECMD -f ${VSERVERVG}/${VSERVERSNAPSHOT} >/dev/null 2>&1
$SLEEPCMD 1
if $LVCREATECMD -L${SNAPSHOTSIZE} -s --chunksize 4k -n ${VSERVERVG}/${VSERVERSNAPSHOT} ${VSERVERVG}/${VSERVERLV} >/dev/null 2>&1 && $MOUNTCMD -t ext2 -ni -o noatime,ro ${VSERVERVG}/${VSERVERSNAPSHOT} ${SNAPSHOTMOUNT} >/dev/null 2>&1
then
break
fi
done
if [ $RETRIES -eq "$MAXRETRIES"]
then
echo "ERROR: cannot create snapshot after `expr $MAXRETRIES-1` retries"
$LVREMOVECMD -f ${VSERVERVG}/${VSERVERSNAPSHOT} >/dev/null 2>&1
exit 2
fi
fi
NACTIVE=0
NSTANDBY=0
NINACTIVE=0
NERROR=0
for i in /etc/vservers/*; do
VSERVER=`$BASENAMECMD $i`
if $VSERVERCMD $VSERVER running
then
# echo "vserver $VSERVER running on $THISHOST"
if $SSHCMD $MIRROR $VSERVERCMD $VSERVER running
then
NERROR=`expr $NERROR + 1`
echo "ERROR: vserver $VSERVER running both on $THISHOST and $MIRROR"
else
NACTIVE=`expr $NACTIVE + 1`
# echo "rsyncing vserver $VSERVER from $THISHOST to $MIRROR..."
if $RSYNCCMD -aqzpotSx4 --numeric-ids --bwlimit $BWLIMIT --recursive --safe-links --delete --force -e ssh ${SNAPSHOTMOUNT}/${VSERVER}/ $MIRROR:${VSERVERBASE}/${VSERVER}/
then
# echo "rsync vserver $VSERVER from $THISHOST to $MIRROR done"
:
else
echo "ERROR: cannot rsync $VSERVER from $THISHOST to $MIRROR"
fi
fi
else
if $SSHCMD $MIRROR $VSERVERCMD $VSERVER running
then
# echo "vserver $VSERVER running on $MIRROR"
NSTANDBY=`expr $NSTANDBY + 1`
else
NINACTIVE=`expr $NSTANDBY + 1`
echo "WARNING: vserver $VSERVER inactive both on $THISHOST and $MIRROR"
fi
fi;
done
$SYNCCMD
if $UMOUNTCMD -ni $SNAPSHOTMOUNT
then
# echo "unmount of $SNAPSHOTMOUNT done"
:
else
echo "ERROR: cannot unmount LV snapshot"
fi
## else
## echo -n `$DATECMD +"%x %X" `
## echo " ERROR: cannot mount LV snapshot"
## fi
SNAPPERCENT=`$LVSCMD --noheadings ${VSERVERVG}/${VSERVERSNAPSHOT} | $AWKCMD '{print $6}'`
$SYNCCMD
if $LVREMOVECMD -f ${VSERVERVG}/${VSERVERSNAPSHOT} >/dev/null 2>&1
then
TIMEEND=`$DATECMD +%s`
echo "NOTICE: rsync_vservers complete; elapsedsecs=`expr $TIMEEND - $TIMESTART`; snapshotsize=$SNAPSHOTSIZE; snap%=$SNAPPERCENT; active=$NACTIVE; standby=$NSTANDBY; inactive=$NINACTIVE; error=$NERROR; snapshotretries=$RETRIES"
# $LVSCMD
else
echo "ERROR: cannot remove LV snapshot"
fi
fi
</file>
<file migrate_vserver>
#!/bin/bash
# gets 2 parameters: vserver_name source_server
# ensures that vserver_name is running on source_server and standby on the corresponding mirror
# stops the vserver on source_server, does rsync of the remaining data and restarts on mirror
# if only the vserver_name is given, the script assumes that the server it's running on should become the new active server
#
# 19-MAR-2006 milprog/mg
if [ "$HOSTNAME" = "milprogmail.ch" ]
then
THISHOST=milprogmail.milnet.lan
else
THISHOST=$HOSTNAME
fi
MIRROR1=lepus.milnet.lan
MIRROR2=milprogmail.milnet.lan
SNAPSHOTSIZE=4096M
VSERVERVG=/dev/vg00
VSERVERLV=lv00_vservers
VSERVERSNAPSHOT=vservers_snapshot
SNAPSHOTMOUNT=/mnt/vservers-snapshot
# parameters for taking the snapshot: number of retries, interval in seconds
MAXSNAPSHOTRETRIES=3
RETRYINTERVAL=250
# no need to change anything below (hopefully)
LVCREATECMD=/usr/sbin/lvcreate
LVREMOVECMD=/usr/sbin/lvremove
LVSCMD=/usr/sbin/lvs
VSERVERCMD=/usr/sbin/vserver
VSERVERROOT=/vservers
RSYNCCMD=/usr/bin/rsync
SSHCMD=/usr/bin/ssh
AWKCMD=/bin/awk
BASENAMECMD=/bin/basename
DATECMD=/bin/date
SYNCCMD=/bin/sync
SLEEPCMD=/bin/sleep
if [ "$THISHOST" = "$MIRROR1" ]
then
MIRROR=$MIRROR2
fi
if [ "$THISHOST" = "$MIRROR2" ]
then
MIRROR=$MIRROR1
fi
if [ "$MIRROR" = "" ]
then
echo "ERROR: $0 must be installed either on $MIRROR1 or $MIRROR2, not here ($THISHOST)"
exit 1
fi
VSERVER=$1
if [ "$VSERVER" == "" ]
then
echo "ERROR: VServer name missing"
exit 2
fi
MIGRATEFROM=$2
if [ "$MIGRATEFROM" == "" ]
then
MIGRATEFROM=$MIRROR
MIGRATETO=$THISHOST
else
if [ $MIGRATEFROM == "$THISHOST" ]
then
MIGRATETO=$MIRROR
else
if [ $MIGRATEFROM == "$MIRROR" ]
then
MIGRATETO=$THISHOST
else
echo "ERROR: $MIGRATEFROM is not a valid host for this script (use $THISHOST or $MIRROR)"
exit 3
fi
fi
fi
if [ $MIGRATEFROM == "$THISHOST" ]
then
RUNFROM=""
RUNTO="$SSHCMD $MIGRATETO "
else
RUNFROM="$SSHCMD $MIGRATEFROM "
RUNTO=""
fi
if $RUNFROM $VSERVERCMD $VSERVER running >/dev/null 2>&1
then
if $RUNTO $VSERVERCMD $VSERVER running >/dev/null 2>&1
then
echo "ERROR: vserver $VSERVER active on both $MIGRATEFROM and $MIGRATETO"
exit 5
else
if [ $? -gt 1 ]
then
echo "ERROR: vserver $VSERVER unknown on $MIGRATETO as a standby server"
exit 6
else
if $RUNFROM $LVCREATECMD -L${SNAPSHOTSIZE} -s --chunksize 4k -n ${VSERVERVG}/${VSERVERSNAPSHOT} ${VSERVERVG}/${VSERVERLV} >/dev/null 2>&1
then
TIMESTART=`$DATECMD +%s`
if $RUNFROM $VSERVERCMD $VSERVER stop >/dev/null 2>&1
then
if $RUNFROM $RSYNCCMD -aqzpotSx4 --numeric-ids --recursive --safe-links --delete --force -e ssh ${VSERVERROOT}/${VSERVER}/ $MIGRATETO:${VSERVERROOT}/${VSERVER}/
then
if $RUNTO $VSERVERCMD $VSERVER start >/dev/null 2>1
then
TIMEEND=`$DATECMD +%s`
echo "NOTICE: vserver $VSERVER migrated from $MIGRATEFROM to $MIGRATETO; elapsedsecs=`expr $TIMEEND - $TIMESTART`"
else
if $RUNFROM $VSERVERCMD $VSERVER start >/dev/null 2>&1
then
echo "ERROR: cannot start $VSERVER after migration from $MIGRATEFROM to $MIGRATETO; server was restarted again on $MIGRATEFROM"
exit 8
else
echo "ERROR: cannot start $VSERVER after migration from $MIGRATEFROM to $MIGRATETO; server could not be restarted again on $MIGRATEFROM and is now dead on both mirrors"
exit 9
fi
fi
else
if $RUNFROM $VSERVERCMD $VSERVER start >/dev/null 2>&1
then
echo "ERROR: cannot rsync $VSERVER for migration from $MIGRATEFROM to $MIGRATETO; server was restarted again on $MIGRATEFROM"
exit 10
else
echo "ERROR: cannot rsync $VSERVER for migration from $MIGRATEFROM to $MIGRATETO; server could not be restarted again on $MIGRATEFROM and is now dead on both mirrors"
exit 11
fi
fi
else
echo "ERROR: vserver $VSERVER cannot be stopped on $MIGRATEFROM"
exit 7
fi
if $RUNFROM $LVREMOVECMD -f ${VSERVERVG}/${VSERVERSNAPSHOT} >/dev/null 2>&1
then
:
else
echo "ERROR: snapshot ${VSERVERVG}/${VSERVERSNAPSHOT} cannot be removed on $MIGRATEFROM after successful migration"
exit 13
fi
else
echo "ERROR: cannot migrate $VSERVER from $MIGRATEFROM to $MIGRATETO because snapshot ${VSERVERVG}/${VSERVERSNAPSHOT} cannot be established (possible overlap with backup/cronjob)"
exit 12
fi
fi
fi
else
echo "ERROR: vserver $VSERVER not running on $MIGRATEFROM"
exit 4
fi
</file>
_______________________________________________
Vserver mailing list
Vserver@list.linux-vserver.org
http://list.linux-vserver.org/mailman/listinfo/vserver
Received on Sun Mar 19 15:44:02 2006