On Mon, Nov 16, 2015 at 10:03:57AM -0700, Daniel Urist wrote:
> I've attached the debug trace.
> The line where the failure occurs is this:
> /usr/share/util-vserver/vserver.stop: line 100: 30935 Killed
>> "${IONICE_CMD[@]}" "${NICE_CMD[@]}" "${NETNS_CMD[@]}" "${CHBIND_\
>> CMD[@]}" "$_VSPACE" --enter "$S_CONTEXT" "${OPTS_VSPACE[@]}"
>> "${OPTS_VSPACE_SHARED[@]}" -- "$_VTAG" --migrate "${OPTS_VTAG_ENTER[@]}"
>> --silent\
>> -- $_VCONTEXT $SILENT_OPT --migrate $OPT_VCONTEXT_CHROOT --xid
>> "$S_CONTEXT" -- "${INITCMD_STOP[@]}"
| [info] Will now restart.
| ifdown: shutdown eth0: Operation not permitted
This suggests that the guest has not been properly
cleaned up (or some kind of update reverted the
changes) and it tries to mess with the hardware.
| /usr/share/util-vserver/vserver.stop: line 100: 30935 Killed
I presume the very same scripts kill the parent
at some point (immediately before a reboot) which
causes the stop script to exit.
I would say this is harmless, as it is unlikely
that the runlevel scripts kill off all processes
before trying to shut down the services, but if
you want to improve this, you have to dig into
the rc script _inside_ the guest and prevent it
from killing the process executing the runlevel
change.
HTH,
Herbert
> I took a snapshot of the process table on both the host and
> vserver before shutting it down, and at that time there was no
> process 30935, so I really don't understand where that's coming
> from.
> On Thu, Nov 12, 2015 at 11:49 PM, Herbert Poetzl <herbert@13thfloor.at>
> wrote:
>> On Thu, Nov 12, 2015 at 08:04:17AM -0700, Daniel Urist wrote:
>>> On Wed, Nov 11, 2015 at 4:40 PM, Jean Weisbuch <jean@phpnet.org> wrote:
>>>> The MySQL shutdown could be quite long, especially if you have
>>>> a big buffer_pool_size or simply if you hit a bug and the
>>>> process can be neverending.
>>>> Are you sure that its not the case you are hitting and that
>>>> "vserver stop" would timeout before MySQL would be stopped?
>>> I don't think it's the timeout; shutting down the mysql server
>>> by hand doesn't take more than a few seconds. I've also tried
>>> bumping up the sync-timeout to 60 seconds with no difference.
>>>> But if its really the "mysqladmin ping" that hangs its strange
>>>> indeed, does it seems to hang before it crash or it directly
>>>> crash when it tries to execute it?
>>> From what I can see, it crashes as soon as it executes
>>> "mysqladmin ping". Again, this only happens during vserver
>>> shutdown; I can execute "mysqladmin ping" from within the
>>> vserver or from the host with no issues while the vserver is
>>> running.
>> Let's see what a 'vserver --debug <name> stop' produces
>> Thanks,
>> Herbert
>>>> Le 11/11/2015 23:50, Daniel Urist a écrit :
>>>> I've narrowed down the issue to the init script for the Percona XtraDB
>>>> mysql server. It seems to be crashing the vserver shutdown when it calls
>>>> "mysqladmin ping" to see whether the database is alive. I can run
>>>> "mysqladmin ping" successfully from within the vserver, and also from
>> the
>>>> host with "vserver exec". I can also successfully run "/etc/init.d/mysql
>>>> stop" from the host via "vserver exec". Is there something special that
>>>> happens during vserver shutdown that would cause this to fail?
>>>> On Mon, Nov 9, 2015 at 11:51 AM, Herbert Poetzl <herbert@13thfloor.at>
>>>> wrote:
>>>>> On Mon, Nov 09, 2015 at 10:49:15AM -0700, Daniel Urist wrote:
>>>>>> I am in the process of upgrading from debian wheezy to jessie
>>>>>> for both hosts and guests. All hosts and guests are running
>>>>>> sysv-rc (no systemd!). I'm getting the following error on
>>>>>> shutdown with jessie guests running on jessie hosts:
>>>>> Hey Daniel,
>>>>>> /usr/share/util-vserver/vserver.stop: line 100: 3511 Killed
>>>>>> "${IONICE_CMD[@]}" "${NICE_CMD[@]}" "${NETNS_CMD[@]}"
>>>>> "${CHBIND_CMD[@]}"
>>>>>> "$_VSPACE" --enter "$S_CONTEXT" "${OPTS_VSPACE[@]}"
>>>>>> "${OPTS_VSPACE_SHARED[@]}" -- "$_VTAG" --migrate
>> "${OPTS_VTAG_ENTER[@]}"
>>>>>> --silent -- $_VCONTEXT $SILENT_OPT --migrate $OPT_VCONTEXT_CHROOT
>> --xid
>>>>>> "$S_CONTEXT" -- "${INITCMD_STOP[@]}"
>>>>> This looks like the rc scripts executed to stop the guest
>>>>> kill the util-vserver stop script at some point.
>>>>>> This doesn't happen with wheezy guests running on jessie hosts,
>>>>> This is kind of suspected, as it is most likely a minor
>>>>> "problem" with the guest side rc scripts.
>>>>>> or with jessie guests on wheezy hosts.
>>>>> This is very interesting, are you sure that the very
>>>>> same guest does not exhibit this behaviours regardless
>>>>> of the host system (given that util-vserver is the same
>>>>> version).
>>>>>> If I switch from init style sysv to plain, I don't see any
>>>>>> errors, but I'm assuming that just suppresses the message
>>>>>> since the vserver shuts down almost instantaneously;
>>>>> An almost instant shutdown is not a bad thing per se, after
>>>>> all container style virtualization is very fast. This
>>>>> setup will not call any scripts inside the guest directly,
>>>>> instead it will signal the init process inside the guest
>>>>> to shut down. If everything goes as expected, your guest
>>>>> should simply disappear (no more processes in the guest
>>>>> context)
>>>>>> I'm concerned that it's not going down cleanly.
>>>>> It might be a good idea to check what scripts are actually
>>>>> executed inside the guest and supplement them with debug
>>>>> output (or run them with -x for bash scripts).
>>>>>> The kernel version for jessie is 3.18.21-vs2.3.7.4-beng (from
>>>>>> repo.psand.net), util-vserver is 0.30.216-pre3120-jessie0.1-1.
>>>>>> Any hints on how to fix or at least debug this?
>>>>> You can also run util-vserver with the --debug option,
>>>>> which might shed some light on the issue.
>>>>> HTH,
>>>>> Herbert
Received on Mon Nov 16 18:23:38 2015