Re: [vserver] cgroups soon incompatible with Libvirt

From: Serge Victor <serge_at_random.re>
Date: Tue 31 Dec 2013 - 19:39:35 GMT
Message-ID: <CAPRDrAFzJj2t5-1v+8UKH3bee4cfh-Fts3-nsG+ucudAkAG4gw@mail.gmail.com>

On Tue, Dec 31, 2013 at 2:37 PM, Art -kwaak- van Breemen
<ard@telegraafnet.nl> wrote:
> Hi,
>
> On Tue, Dec 31, 2013 at 02:30:51AM +0000, Sergiusz Pawlowicz wrote:
>> I think we still have some issues with cgroups. My server on 3.10x
>> line started to freeze.
>>
>> After downgrading to 3.4x I am able to see one of vservers is rapidly
>> leaking memory. It seems that probably cgroups on 3.10x kernel are not
>> preventing from overfilling all RAM, without enforcing cgroup limits.
>> The funny thing that vserver inside shows that memory is limited, but
>> it seems it is not being enforced.
>
> My experience is that memory limiting is enforced, but also for
> kernel things like the ext4 filesystem and other things.
> A vserver running out of memory that shares it's /vserver/ mount
> with other vservers will effectively lock the ext4 system (making
> it wait for memory within that memory cgroup), bringing down the
> whole server until the vserver exceding it's memory usage gets
> some more memory. These problems also seems to affect fusion-mpt
> drivers. This behaviour is not easy to trigger: it takes a uwsgi
> server 30 minutes to 24 hours of steady OOMing until it triggers
> the ext4 lock-wait-for-memory at the right spot.
>
> We have this kind of problems since the start of using memory
> cgroups, but we never had any real problems with it. Usually the
> OOMs made sure there were no real problems until people here
> started to use python/uwsgi with memory leaks and such. The 3.10
> series seems to be much faster lockable.
>
> I think it is not vserver related, but just a bug in the vanilla
> kernel memory cgroup system.

It sounds interesting :-)

The thing is kernel 3.4.x is properly enforcing it, killing the
leaking process with OOM, but 3.10.x is probably doing something more,
as you have suggested.

I still really do not know how far does it go, though. If I do not
share the filesystem but one LVM2 VG, do you think it still may count?

How can we try to fix it? Because the whole idea of vserver cgroups is
ruined, if it freezes the whole machine, is not it?

cheers,
Serge
Received on Tue Dec 31 19:40:22 2013

[Next/Previous Months] [Main vserver Project Homepage] [Howto Subscribe/Unsubscribe] [Paul Sladen's vserver stuff]
Generated on Tue 31 Dec 2013 - 19:40:22 GMT by hypermail 2.1.8