Thanks very much for your help.
I finally had a chance to further debug this-- the issue was debian's init
script /etc/init.d/sendsigs, which kills off remaining processes on
shutdown, and apparently doesn't wait very long. Since my vserver is
running a database, I need it go down cleanly.
I also found another minor issue with debian jessie shutdown under sysv-rc;
there are no kill scripts for cron, dbus or ssh since the "Default-Stop"
field in the init script headers for these are empty, so these also don't
go down cleanly. Since I'm running my vservers in a pacemaker/corosync
cluster, I fixed these as well.
I'm definitely not a fan of systemd, but it seems pretty clear that the
days of sysv-rc are numbered. Is there any chance vserver will work with
systemd in the future?
On Mon, Nov 16, 2015 at 11:24 AM, Herbert Poetzl <herbert@13thfloor.at>
wrote:
> On Mon, Nov 16, 2015 at 10:03:57AM -0700, Daniel Urist wrote:
> > I've attached the debug trace.
>
> > The line where the failure occurs is this:
>
> > /usr/share/util-vserver/vserver.stop: line 100: 30935 Killed
> >> "${IONICE_CMD[@]}" "${NICE_CMD[@]}" "${NETNS_CMD[@]}" "${CHBIND_\
> >> CMD[@]}" "$_VSPACE" --enter "$S_CONTEXT" "${OPTS_VSPACE[@]}"
> >> "${OPTS_VSPACE_SHARED[@]}" -- "$_VTAG" --migrate "${OPTS_VTAG_ENTER[@]}"
> >> --silent\
> >> -- $_VCONTEXT $SILENT_OPT --migrate $OPT_VCONTEXT_CHROOT --xid
> >> "$S_CONTEXT" -- "${INITCMD_STOP[@]}"
>
> | [info] Will now restart.
> | ifdown: shutdown eth0: Operation not permitted
>
> This suggests that the guest has not been properly
> cleaned up (or some kind of update reverted the
> changes) and it tries to mess with the hardware.
>
> | /usr/share/util-vserver/vserver.stop: line 100: 30935 Killed
>
> I presume the very same scripts kill the parent
> at some point (immediately before a reboot) which
> causes the stop script to exit.
>
> I would say this is harmless, as it is unlikely
> that the runlevel scripts kill off all processes
> before trying to shut down the services, but if
> you want to improve this, you have to dig into
> the rc script _inside_ the guest and prevent it
> from killing the process executing the runlevel
> change.
>
> HTH,
> Herbert
>
> > I took a snapshot of the process table on both the host and
> > vserver before shutting it down, and at that time there was no
> > process 30935, so I really don't understand where that's coming
> > from.
>
>
> > On Thu, Nov 12, 2015 at 11:49 PM, Herbert Poetzl <herbert@13thfloor.at>
> > wrote:
>
> >> On Thu, Nov 12, 2015 at 08:04:17AM -0700, Daniel Urist wrote:
> >>> On Wed, Nov 11, 2015 at 4:40 PM, Jean Weisbuch <jean@phpnet.org>
> wrote:
>
> >>>> The MySQL shutdown could be quite long, especially if you have
> >>>> a big buffer_pool_size or simply if you hit a bug and the
> >>>> process can be neverending.
>
> >>>> Are you sure that its not the case you are hitting and that
> >>>> "vserver stop" would timeout before MySQL would be stopped?
>
> >>> I don't think it's the timeout; shutting down the mysql server
> >>> by hand doesn't take more than a few seconds. I've also tried
> >>> bumping up the sync-timeout to 60 seconds with no difference.
>
> >>>> But if its really the "mysqladmin ping" that hangs its strange
> >>>> indeed, does it seems to hang before it crash or it directly
> >>>> crash when it tries to execute it?
>
> >>> From what I can see, it crashes as soon as it executes
> >>> "mysqladmin ping". Again, this only happens during vserver
> >>> shutdown; I can execute "mysqladmin ping" from within the
> >>> vserver or from the host with no issues while the vserver is
> >>> running.
>
> >> Let's see what a 'vserver --debug <name> stop' produces
>
> >> Thanks,
> >> Herbert
>
> >>>> Le 11/11/2015 23:50, Daniel Urist a écrit :
>
> >>>> I've narrowed down the issue to the init script for the Percona XtraDB
> >>>> mysql server. It seems to be crashing the vserver shutdown when it
> calls
> >>>> "mysqladmin ping" to see whether the database is alive. I can run
> >>>> "mysqladmin ping" successfully from within the vserver, and also from
> >> the
> >>>> host with "vserver exec". I can also successfully run
> "/etc/init.d/mysql
> >>>> stop" from the host via "vserver exec". Is there something special
> that
> >>>> happens during vserver shutdown that would cause this to fail?
>
> >>>> On Mon, Nov 9, 2015 at 11:51 AM, Herbert Poetzl <herbert@13thfloor.at
> >
> >>>> wrote:
>
> >>>>> On Mon, Nov 09, 2015 at 10:49:15AM -0700, Daniel Urist wrote:
> >>>>>> I am in the process of upgrading from debian wheezy to jessie
> >>>>>> for both hosts and guests. All hosts and guests are running
> >>>>>> sysv-rc (no systemd!). I'm getting the following error on
> >>>>>> shutdown with jessie guests running on jessie hosts:
>
> >>>>> Hey Daniel,
>
> >>>>>> /usr/share/util-vserver/vserver.stop: line 100: 3511 Killed
> >>>>>> "${IONICE_CMD[@]}" "${NICE_CMD[@]}" "${NETNS_CMD[@]}"
> >>>>> "${CHBIND_CMD[@]}"
> >>>>>> "$_VSPACE" --enter "$S_CONTEXT" "${OPTS_VSPACE[@]}"
> >>>>>> "${OPTS_VSPACE_SHARED[@]}" -- "$_VTAG" --migrate
> >> "${OPTS_VTAG_ENTER[@]}"
> >>>>>> --silent -- $_VCONTEXT $SILENT_OPT --migrate $OPT_VCONTEXT_CHROOT
> >> --xid
> >>>>>> "$S_CONTEXT" -- "${INITCMD_STOP[@]}"
>
> >>>>> This looks like the rc scripts executed to stop the guest
> >>>>> kill the util-vserver stop script at some point.
>
> >>>>>> This doesn't happen with wheezy guests running on jessie hosts,
>
> >>>>> This is kind of suspected, as it is most likely a minor
> >>>>> "problem" with the guest side rc scripts.
>
> >>>>>> or with jessie guests on wheezy hosts.
>
> >>>>> This is very interesting, are you sure that the very
> >>>>> same guest does not exhibit this behaviours regardless
> >>>>> of the host system (given that util-vserver is the same
> >>>>> version).
>
> >>>>>> If I switch from init style sysv to plain, I don't see any
> >>>>>> errors, but I'm assuming that just suppresses the message
> >>>>>> since the vserver shuts down almost instantaneously;
>
> >>>>> An almost instant shutdown is not a bad thing per se, after
> >>>>> all container style virtualization is very fast. This
> >>>>> setup will not call any scripts inside the guest directly,
> >>>>> instead it will signal the init process inside the guest
> >>>>> to shut down. If everything goes as expected, your guest
> >>>>> should simply disappear (no more processes in the guest
> >>>>> context)
>
> >>>>>> I'm concerned that it's not going down cleanly.
>
> >>>>> It might be a good idea to check what scripts are actually
> >>>>> executed inside the guest and supplement them with debug
> >>>>> output (or run them with -x for bash scripts).
>
> >>>>>> The kernel version for jessie is 3.18.21-vs2.3.7.4-beng (from
> >>>>>> repo.psand.net), util-vserver is 0.30.216-pre3120-jessie0.1-1.
>
> >>>>>> Any hints on how to fix or at least debug this?
>
> >>>>> You can also run util-vserver with the --debug option,
> >>>>> which might shed some light on the issue.
>
> >>>>> HTH,
> >>>>> Herbert
>
>
>
>
>
>
>
>
>
Received on Tue Jan 5 21:44:01 2016