[vserver] [resend] vs2.2.0.x and scheduler getting stuck

From: Grzegorz Nosek <grzegorz.nosek_at_gmail.com>
Date: Sat 17 May 2008 - 10:37:02 BST
Message-ID: <121a28810805170237r164a512cn216de9643c2550ef@mail.gmail.com>

Hi all,

(sorry if you receive this message twice)

I've been experiencing some weird hangs (boom and it's dead, no panic,
not even a softlockup or lockdep warning). It happens about once a month
(of course, in production only), usually under some load (I suspected
I/O but it might be simple CPU usage too). It happened on several very
different machines (2-way pentium3, 4-way opteron 270).

After enabling the nmi watchdog I could trace it back to schedule(), or
at least the nmi watchdog felt the need to kill the machine right in the
middle of a schedule() call.

I did not match up the assembly with C code yet, but the following
snippet raised my suspicions:

(kernel/sched.c, 2.6.22.19 + vs2.2.0.7 + drbd + some trivia)
3696 vx_set_rq_time(rq, jiffies);
3697 try_unhold:
3698 vx_try_unhold(rq, cpu);
3699 pick_next:
3700
3701 if (unlikely(!rq->nr_running)) {
3702 /* can we skip idle time? */
3703 if (vx_try_skip(rq, cpu))
3704 goto try_unhold;
3705
3706 idle_balance(cpu, rq);
3707 if (!rq->nr_running) {
3708 next = rq->idle;
3709 rq->expired_timestamp = 0;
3710 goto switch_tasks;
3711 }
3712 }

Unless proven otherwise, it has the potential of looping forever in line
3704. AIUI, the scheduler will loop when

list_empty(&rq->hold_queue) && !rq->nr_running

OTOH, the above line looks like an idle CPU to me (though I'm far from
being a scheduler expert), so I'm most probably missing something as
otherwise the scheduler would never run ;)

I think that the solution would be to get some feedback from
vx_try_unhold about whether any progress can be made at all (e.g. if the
hold_queue is empty, vx_try_unhold will never change anything and
vx_try_skip will just keep updating rq->idle_time).

There's another back-goto later in the code (goto pick_next) but that
one looks rather harmless (or maybe only the trigger).

Is this _the_ bug I'm looking for or am I way off?

Best regards,
 Grzegorz Nosek

PS. For reference, grep VSERVER .config:

# CONFIG_VSERVER_LEGACY is not set
# CONFIG_VSERVER_LEGACYNET is not set
CONFIG_VSERVER_REMAP_SADDR=y
CONFIG_VSERVER_COWBL=y
# CONFIG_VSERVER_VTIME is not set
CONFIG_VSERVER_PROC_SECURE=y
CONFIG_VSERVER_HARDCPU=y
CONFIG_VSERVER_IDLETIME=y
CONFIG_VSERVER_IDLELIMIT=y
# CONFIG_VSERVER_PRIVACY is not set
CONFIG_VSERVER_CONTEXTS=256
CONFIG_VSERVER_WARN=y
# CONFIG_VSERVER_DEBUG is not set
CONFIG_VSERVER=y
CONFIG_VSERVER_NGNET=y
Received on Sat May 17 10:37:18 2008

[Next/Previous Months] [Main vserver Project Homepage] [Howto Subscribe/Unsubscribe] [Paul Sladen's vserver stuff]
Generated on Sat 17 May 2008 - 10:37:28 BST by hypermail 2.1.8