On Tue, Jan 05, 2016 at 02:45:56PM -0700, Daniel Urist wrote:
> Thanks very much for your help.
> I finally had a chance to further debug this-- the issue was
> debian's init script /etc/init.d/sendsigs, which kills off
> remaining processes on shutdown, and apparently doesn't wait
> very long. Since my vserver is running a database, I need it go
> down cleanly.
> I also found another minor issue with debian jessie shutdown
> under sysv-rc; there are no kill scripts for cron, dbus or ssh
> since the "Default-Stop" field in the init script headers for
> these are empty, so these also don't go down cleanly. Since I'm
> running my vservers in a pacemaker/corosync cluster, I fixed
> these as well.
Thanks for the diagnostic and feedback.
Other debian folks might be interested in your modifications,
so please consider sharing them somehow.
> I'm definitely not a fan of systemd, but it seems pretty clear
> that the days of sysv-rc are numbered. Is there any chance
> vserver will work with systemd in the future?
We were in contact (back in the early days) with the systemd
developer but it became clear that he is not interested in
having systemd running inside a jail solution.
The main problem today with systemd is that it assumes to
have full hardware access, all priviledges and sole control
over cgroups, all which is not given (for good reasons) in-
side a Linux-VServer guest.
As far as I know there are attempts (like uselessd) to cut
the many heads of the hydra in a way which leaves an init
system which also works inside containers and chroots for
example.
We have no plans to virtualize hundreds of things, including
the cgroup, namespace and device interface to the kernel
just to trick systemd into believing that it is controlling
the entire machine while it is not.
There is also no good reason to utilize a resource saving
isolation technology to then again waste those resources
on a huge number of "useless" processes trying to do what
a simple shell script can do as well :)
I.e. if you don't care about resources, kvm does a nice
job running all kinds of distros with systemd :)
Best,
Herbert
> On Mon, Nov 16, 2015 at 11:24 AM, Herbert Poetzl <herbert@13thfloor.at>
> wrote:
>> On Mon, Nov 16, 2015 at 10:03:57AM -0700, Daniel Urist wrote:
>>> I've attached the debug trace.
>>> The line where the failure occurs is this:
>>> /usr/share/util-vserver/vserver.stop: line 100: 30935 Killed
>>>> "${IONICE_CMD[@]}" "${NICE_CMD[@]}" "${NETNS_CMD[@]}" "${CHBIND_\
>>>> CMD[@]}" "$_VSPACE" --enter "$S_CONTEXT" "${OPTS_VSPACE[@]}"
>>>> "${OPTS_VSPACE_SHARED[@]}" -- "$_VTAG" --migrate "${OPTS_VTAG_ENTER[@]}"
>>>> --silent\
>>>> -- $_VCONTEXT $SILENT_OPT --migrate $OPT_VCONTEXT_CHROOT --xid
>>>> "$S_CONTEXT" -- "${INITCMD_STOP[@]}"
>> | [info] Will now restart.
>> | ifdown: shutdown eth0: Operation not permitted
>> This suggests that the guest has not been properly
>> cleaned up (or some kind of update reverted the
>> changes) and it tries to mess with the hardware.
>> | /usr/share/util-vserver/vserver.stop: line 100: 30935 Killed
>> I presume the very same scripts kill the parent
>> at some point (immediately before a reboot) which
>> causes the stop script to exit.
>> I would say this is harmless, as it is unlikely
>> that the runlevel scripts kill off all processes
>> before trying to shut down the services, but if
>> you want to improve this, you have to dig into
>> the rc script _inside_ the guest and prevent it
>> from killing the process executing the runlevel
>> change.
>> HTH,
>> Herbert
>>> I took a snapshot of the process table on both the host and
>>> vserver before shutting it down, and at that time there was no
>>> process 30935, so I really don't understand where that's coming
>>> from.
>>> On Thu, Nov 12, 2015 at 11:49 PM, Herbert Poetzl <herbert@13thfloor.at>
>>> wrote:
>>>> On Thu, Nov 12, 2015 at 08:04:17AM -0700, Daniel Urist wrote:
>>>>> On Wed, Nov 11, 2015 at 4:40 PM, Jean Weisbuch <jean@phpnet.org>
>> wrote:
>>>>>> The MySQL shutdown could be quite long, especially if you have
>>>>>> a big buffer_pool_size or simply if you hit a bug and the
>>>>>> process can be neverending.
>>>>>> Are you sure that its not the case you are hitting and that
>>>>>> "vserver stop" would timeout before MySQL would be stopped?
>>>>> I don't think it's the timeout; shutting down the mysql server
>>>>> by hand doesn't take more than a few seconds. I've also tried
>>>>> bumping up the sync-timeout to 60 seconds with no difference.
>>>>>> But if its really the "mysqladmin ping" that hangs its strange
>>>>>> indeed, does it seems to hang before it crash or it directly
>>>>>> crash when it tries to execute it?
>>>>> From what I can see, it crashes as soon as it executes
>>>>> "mysqladmin ping". Again, this only happens during vserver
>>>>> shutdown; I can execute "mysqladmin ping" from within the
>>>>> vserver or from the host with no issues while the vserver is
>>>>> running.
>>>> Let's see what a 'vserver --debug <name> stop' produces
>>>> Thanks,
>>>> Herbert
>>>>>> Le 11/11/2015 23:50, Daniel Urist a écrit :
>>>>>> I've narrowed down the issue to the init script for the Percona XtraDB
>>>>>> mysql server. It seems to be crashing the vserver shutdown when it
>> calls
>>>>>> "mysqladmin ping" to see whether the database is alive. I can run
>>>>>> "mysqladmin ping" successfully from within the vserver, and also from
>>>> the
>>>>>> host with "vserver exec". I can also successfully run
>> "/etc/init.d/mysql
>>>>>> stop" from the host via "vserver exec". Is there something special
>> that
>>>>>> happens during vserver shutdown that would cause this to fail?
>>>>>> On Mon, Nov 9, 2015 at 11:51 AM, Herbert Poetzl <herbert@13thfloor.at
>>>>>> wrote:
>>>>>>> On Mon, Nov 09, 2015 at 10:49:15AM -0700, Daniel Urist wrote:
>>>>>>>> I am in the process of upgrading from debian wheezy to jessie
>>>>>>>> for both hosts and guests. All hosts and guests are running
>>>>>>>> sysv-rc (no systemd!). I'm getting the following error on
>>>>>>>> shutdown with jessie guests running on jessie hosts:
>>>>>>> Hey Daniel,
>>>>>>>> /usr/share/util-vserver/vserver.stop: line 100: 3511 Killed
>>>>>>>> "${IONICE_CMD[@]}" "${NICE_CMD[@]}" "${NETNS_CMD[@]}"
>>>>>>> "${CHBIND_CMD[@]}"
>>>>>>>> "$_VSPACE" --enter "$S_CONTEXT" "${OPTS_VSPACE[@]}"
>>>>>>>> "${OPTS_VSPACE_SHARED[@]}" -- "$_VTAG" --migrate
>>>> "${OPTS_VTAG_ENTER[@]}"
>>>>>>>> --silent -- $_VCONTEXT $SILENT_OPT --migrate $OPT_VCONTEXT_CHROOT
>>>> --xid
>>>>>>>> "$S_CONTEXT" -- "${INITCMD_STOP[@]}"
>>>>>>> This looks like the rc scripts executed to stop the guest
>>>>>>> kill the util-vserver stop script at some point.
>>>>>>>> This doesn't happen with wheezy guests running on jessie hosts,
>>>>>>> This is kind of suspected, as it is most likely a minor
>>>>>>> "problem" with the guest side rc scripts.
>>>>>>>> or with jessie guests on wheezy hosts.
>>>>>>> This is very interesting, are you sure that the very
>>>>>>> same guest does not exhibit this behaviours regardless
>>>>>>> of the host system (given that util-vserver is the same
>>>>>>> version).
>>>>>>>> If I switch from init style sysv to plain, I don't see any
>>>>>>>> errors, but I'm assuming that just suppresses the message
>>>>>>>> since the vserver shuts down almost instantaneously;
>>>>>>> An almost instant shutdown is not a bad thing per se, after
>>>>>>> all container style virtualization is very fast. This
>>>>>>> setup will not call any scripts inside the guest directly,
>>>>>>> instead it will signal the init process inside the guest
>>>>>>> to shut down. If everything goes as expected, your guest
>>>>>>> should simply disappear (no more processes in the guest
>>>>>>> context)
>>>>>>>> I'm concerned that it's not going down cleanly.
>>>>>>> It might be a good idea to check what scripts are actually
>>>>>>> executed inside the guest and supplement them with debug
>>>>>>> output (or run them with -x for bash scripts).
>>>>>>>> The kernel version for jessie is 3.18.21-vs2.3.7.4-beng (from
>>>>>>>> repo.psand.net), util-vserver is 0.30.216-pre3120-jessie0.1-1.
>>>>>>>> Any hints on how to fix or at least debug this?
>>>>>>> You can also run util-vserver with the --debug option,
>>>>>>> which might shed some light on the issue.
>>>>>>> HTH,
>>>>>>> Herbert
Received on Wed Jan 6 01:40:03 2016