Herbert Poetzl wrote:
>>
>> I just suffered the worst nightmare of any server-admin:
>> kernel panic! This is the way it happened:
>>
>> I tried to update gcc on all my guests. Because they run gentoo
>> (as well as guest), updating takes rather long time (everything
>> is compilled from sources). After a couple of hours I found out,
>> upgrade process failed on some guests. After a little search
>> I discovered (with vdlimit) those guests were short of inodes:
>> ...
>> inodes_used=99996
>> inodes_total=100000
>> ...
>> I increased limits in /etc/vservers/<vs>/dlimits/root/inodes_total
>> and tried to stop & start all vservers:
>> /etc/init.d/vserver.default stop
>> Stopping default vservers...
>>
>> (maybe I should have first stop vserver-guests, then increase
>> inodes_total limits)
>>
>> And nothing more. Network connection broken and I could not connect
>> to my server any more. So I rushed to server-room, attached screen
>> just to see "Kernel panic..."!
>
> well, it would be necessary to know what the kernel
> paniced about, i.e. get the full stack trace which
> accompanies such a panic (usually) ... you might
> gather that one with a serial console ...
I do not know how to do it, I have never seen kernel panic until now.
And I do not want to see it again, as this is my server, not testing
lab...
>> I did hard reset and everything seems to work for now. But I'm very
>> surprised, full inodes-limit in vserver-guest could bring the whole
>> host-server down! The only strange entry I found in messages is:
>>
>> Jul 6 16:50:56 obel vxW: [xid #0] !!! limit: ffff8101373b7078[VM,9] = 33
>> on exit.
>
> that is unusual for the kernel you described below,
> maybe you could give the exact details, as there never
> was a 2.2.0.7 patch for 2.6.22 :)
OK, then it is 2.6.22.19. But in gentoo repository this kernel
is called "linux-2.6.22-vserver-2.2.0.7"
>> Otherwise no trace. Kernel 2.6.22, vserver 2.2.0.7. All guest and
>> host are gentoo, stable and updated.
>>
>> Disk space was definitelly not a problem; all guests had ~10GB free.
>> I also have "10" in /etc/vservers/<vs>/dlimits/root/reserved ,
>> but this is probably related only to disk-space, not inodes number
>> (can someone from dev confirm?). If it is so, there should definitelly
>> be some reserved-limit for inodes too...
>
> well, the root reserve is something relevant for
> users vs root account, it is not relevant for guest
> vs host, and while the dlimit will deny guest processes
> any creation beyond the limit, it should not cause the
> kernel to panic ... unless there is a bug somewhere
> (which brings us back to the kernel panic trace)
All I know is my server paniced right when I tried to shutdown
default-guests, while some of them were on inode-limits.
And I have never had any problems with this server before...
> well, kernel panics per se should really *never* happen,
> but they do, so is life, all we can do is narrow down
> what the actual issue is and make sure that it doesn't
> happen _again_ (presumed that it is in Linux-VServer code)
If it is really *that* easy to get kernel panic (by just filling
inode-limits on a vserver-guest) then there is definitelly
something wrong. Be it kernel, or vserver-code...
Jarry
-- _______________________________________________________________ This mailbox accepts e-mails only from selected mailing-lists! Everything else is considered to be spam and therefore deleted.Received on Tue Jul 7 18:07:37 2009