I just suffered the worst nightmare of any server-admin:
kernel panic! This is the way it happened:
I tried to update gcc on all my guests. Because they run gentoo
(as well as guest), updating takes rather long time (everything
is compilled from sources). After a couple of hours I found out,
upgrade process failed on some guests. After a little search
I discovered (with vdlimit) those guests were short of inodes:
I increased limits in /etc/vservers/<vs>/dlimits/root/inodes_total
and tried to stop & start all vservers:
Stopping default vservers...
(maybe I should have first stop vserver-guests, then increase
And nothing more. Network connection broken and I could not connect
to my server any more. So I rushed to server-room, attached screen
just to see "Kernel panic..."!
I did hard reset and everything seems to work for now. But I'm very
surprised, full inodes-limit in vserver-guest could bring the whole
host-server down! The only strange entry I found in messages is:
Jul 6 16:50:56 obel vxW: [xid #0] !!! limit: ffff8101373b7078[VM,9] = 33
Otherwise no trace. Kernel 2.6.22, vserver 18.104.22.168. All guest and
host are gentoo, stable and updated.
Disk space was definitelly not a problem; all guests had ~10GB free.
I also have "10" in /etc/vservers/<vs>/dlimits/root/reserved ,
but this is probably related only to disk-space, not inodes number
(can someone from dev confirm?). If it is so, there should definitelly
be some reserved-limit for inodes too...
I'm not going to test it again, but something like this should
really *never* happen...
-- _______________________________________________________________ This mailbox accepts e-mails only from selected mailing-lists! Everything else is considered to be spam and therefore deleted.Received on Mon Jul 6 18:46:45 2009