[vserver] Guest resource limiting causing destablization of host context.

From: Michael S. Zick <mszick_at_morethan.org>
Date: Tue 23 Aug 2011 - 12:41:32 BST
Message-Id: <201108230641.35271.mszick@morethan.org>

List,

In particular, operators of Linux-VServer hosts running
large numbers of memory resource limited guest contexts.

The basic complaint is that the host context (kernel)
becomes "unstable" (no details given) when one or more
guest context(s) reach and/or operate at the defined
memory resource limits.

This does not seem to make any sense to me. The L-VS
patch should be isolating the host context from the
behavior of any/all guests.
Any other result is a path to a DoS attack on the host.

The operator's response to this situation (rather than
report the problem here) is to run userspace, host
context, scripting that disables the guest for 5 minutes.
During which time the "host stabilizes".

__That__ is without a doubt a path to a DoS attack on
any particular guest context. Push it a bit and you
will cause the host's scripting to shut it down for
5 minutes.

Question:
Have any other operators of hosts with a large number
of memory resource limited guests noticed any host
instability when the guest(s) reach those limits?

Only limited information is available on which kernel/patch
is being run.
So this question may have to be answered in general, based
on the design of the L-VS patch and the isolation provided.

I.E:
ps34532:~# uname -a
Linux ps34532 2.6.38.8-2-vs2.3.0.37-rc17-blkio #20 SMP Thu Aug 4 11:28:54 PDT 2011 x86_64 GNU/Linux

Note: The "#20" is being used to track the local kernel modification level.
So it should be presumed this is __not__ a "clean" kernel + L-VS patch.

The following operator/customer exchange gives a bit more detail on
the viewpoint of both sides of this question:

Quote:
>
> > The site was up for two months without being zapped by your resource limiter.
> >
> > You __really, really__ need to continue "tweaking" as you call it how that
> > limiter works.
> >
> > If nothing else, drop that 5 minute duration to something reasonable.
> > __OR__
> > Add a 5 minute duration re-direct to a static "Server Busy" page.
>
> Hello Michael, thanks for your recommendations, I'll pass them along.
> However, the issue remains that it is not an issue with the VPS but with
> the usage that is being caused by your sites (bad scripts, code,
> inefficient sites, high traffic, etc.) which is causing the VPS to reach
> its RAM limit and thus be killed.
>

Ah, but it is so easy to "blame the customer" and not take the time
to read the thread.

Wrong subject, the above is based on (the Readers Digest form):
Me:
> You where doing good, a few months ago with uptimes greater than 60 days.
>
> More recently, things have taken a turn for the worse, much worse.
> (see below - I only generated one of the entries)
>
> Any chance you have added a load combination that is triggering this bug?
> https://bugzilla.kernel.org/show_bug.cgi?id=27142
> (Also recently discussed on the Linux-VServer M.l.)
> Guest reboot date/times:
> Thu Mar 17 06:00:19 PDT 2011
> Sat Apr 23 02:23:57 PDT 2011
> Wed Jun 22 09:15:26 PDT 2011
> Wed Jul 13 22:34:52 PDT 2011
> And accelerating to multiple reboots per day.

Dreamhost:
> > We are still tweaking how we handle guest machines when they hit the
> > memory limit. We certainly don't WANT to freeze them 5 minutes at a time
> > when that happens but it has effectively eliminated the host instability
> > to the benefit of all the other quests.
>

Please note that the subject is:
"change ... effectively eliminated the the host instability"

But the point is:
In the situation where the kernel has to enforce the resource limits,
such actions by the kernel __should not__ cause "host instability".

The host instance __should always__ remain stable, regardless of the
actions of any one or any combination of actions of the guests.
That is a __design basic__ to the Linux-VServer patch.
Anything that effects the host context stability is a possible path
to a DoS attack on the host context and as such a DoS attack on all
supported guests.

Your userspace scripting that is trying to "stabilize the host for
the benefit of all the guests" should never be required for technical
reasons, although they may be required by a business decision to
drive all customers into paying for unneeded resources.

> Unfortunately, we do not troubleshoot
> your sites for you but we did provide you with links on how to
> troubleshoot for the high usage. The new changes that we have made
> better protect the vserver and isolate VPS's that are causing issues from
> other VPS guests. Your VPS was killed again today:
>
> 2011-08-22 10:16 - ps34532 exceeded memory ceiling, killed all processes
> in context
> 2011-08-22 18:55 - ps34532 exceeded memory ceiling, killed all processes
> in context
>

The above being the result of your userspace scripting attempting to
do the resource management chore that is more propertly done in the kernel.
 
> You need to increase the RAM allocated to your VPS and troubleshoot your
> sites on what is causing the high usage. If you have any other questions
> or concerns, please do not hesitate to contact us.
>

Since I have not seen any reports of host context instability caused by
guests reaching or operating at their resource limits in the four years
that I have been following the Linux-VServer development;

I think it is past time to report this problem to the people who design
and write the Linux-VServer code.
Which I will do on your behalf.

/Quote

Mike
Received on Tue Aug 23 12:41:58 2011

[Next/Previous Months] [Main vserver Project Homepage] [Howto Subscribe/Unsubscribe] [Paul Sladen's vserver stuff]
Generated on Tue 23 Aug 2011 - 12:41:59 BST by hypermail 2.1.8