[vserver] Kernel > 3.0 and rcu_sched stalls

From: cryptronic <mail_at_cryptronic.de>
Date: Fri 27 Apr 2012 - 17:15:44 BST
Message-ID: <4F9AC630.4070609@cryptronic.de>

Hi list,

we have currently problems with kernels greater 3.0.

The situation is the following.

Suddenly all we can not log in via ssh or attached serial console. The
server is still replying on pings but a ssh connect times out.

On serial console we got lines like the following:
INFO: rcu_sched detected stalls on CPUs/tasks: { x,y,z} (detected by a,
t=b jiffies)

Now we got the first time a complete stack trace and now I want to ask,
whether this is known to others or if there is a way to debug such
things. Because i found such issues with all kernels > 3.0 with the
vserver patch i talk first of all to the list.

Kernelversion: various > 3.0.18 but in this case: 3.3.3-vs2.3.3.3
util-vserver: 0.30.216-pre3034

any more information required?

[18052.397180] BUG: Bad page map in process apache2
pte:8000000f9a57a044 pmd:1133813067
[18052.412862] addr:00007f615ee01000 vm_flags:00100073
anon_vma:ffff880f4588e760 mapping: (null) index:7f615ee01
[18052.434232] Pid: 18850, comm: apache2 Tainted: G D
3.3.3-vs2.3.3.3 #1
[18052.449585] Call Trace:
[18052.454476] [<ffffffff810c3424>] ? print_bad_pte+0x1f5/0x20d
[18052.465944] [<ffffffff810c553a>] ? unmap_vmas+0x61f/0x839
[18052.476896] [<ffffffff810b5a17>] ? pagevec_lru_move_fn+0xb0/0xc6
[18052.489054] [<ffffffff810c959c>] ? exit_mmap+0x99/0xfb
[18052.499481] [<ffffffff8103f785>] ? mmput+0x4c/0xec
[18052.509214] [<ffffffff810435cf>] ? exit_mm+0x103/0x10e
[18052.519855] [<ffffffff8105bcf1>] ? hrtimer_try_to_cancel+0x6b/0x74
[18052.532364] [<ffffffff81044e8a>] ? do_exit+0x25b/0x7ee
[18052.542789] [<ffffffff810ca5f0>] ? do_munmap+0x2c9/0x2eb
[18052.553558] [<ffffffff81045497>] ? do_group_exit+0x7a/0xa4
[18052.564672] [<ffffffff810454d3>] ? sys_exit_group+0x12/0x16
[18052.575966] [<ffffffff814d3379>] ? system_call_fastpath+0x16/0x1b
[18140.774496] INFO: rcu_sched detected stall on CPU 8 (t=780128 jiffies)
[18140.776488] INFO: rcu_sched detected stalls on CPUs/tasks: { 8}
(detected by 7, t=780130 jiffies)
[18140.776495] Pid: 3465, comm: apache2 Tainted: G B D
3.3.3-vs2.3.3.3 #1
[18140.776497] Call Trace:
[18140.776499] <IRQ> [<ffffffff8109d4c0>] ? __rcu_pending+0x21a/0x336
[18140.776509] [<ffffffff8109d66e>] ? rcu_check_callbacks+0x92/0xc2
[18140.776513] [<ffffffff8104bf96>] ? update_process_times+0x31/0x63
[18140.776517] [<ffffffff81079ecc>] ? tick_sched_timer+0x65/0x8b
[18140.776522] [<ffffffff8105b58b>] ? __run_hrtimer+0x75/0xe9
[18140.776525] [<ffffffff8105b838>] ? hrtimer_interrupt+0xd8/0x1a6
[18140.776530] [<ffffffff81018f92>] ? smp_apic_timer_interrupt+0x80/0x93
[18140.776535] [<ffffffff814d3e1e>] ? apic_timer_interrupt+0x6e/0x80
[18140.776536] <EOI> [<ffffffff810e7c28>] ? css_tryget+0x1e/0x38
[18140.776544] [<ffffffff810eb3a0>] ? __mem_cgroup_try_charge+0x187/0x4c4
[18140.776548] [<ffffffff81101792>] ? __d_lookup_rcu+0x37/0x114
[18140.776552] [<ffffffff810ebb6d>] ? mem_cgroup_charge_common+0x80/0xae
[18140.776556] [<ffffffff810dac01>] ? alloc_pages_vma+0xf0/0x11d
[18140.776560] [<ffffffff810c6ac1>] ? handle_pte_fault+0x1d4/0x95c
[18140.776564] [<ffffffff810247f7>] ? do_page_fault+0x2c0/0x2e5
[18140.776567] [<ffffffff810dfabb>] ? virt_to_slab+0x9/0x3c
[18140.776570] [<ffffffff810dffa1>] ? __cache_free+0x7d/0x1d3
[18140.776572] [<ffffffff814d2e05>] ? page_fault+0x25/0x30
[18141.039094] Pid: 3754, comm: pop3-login Tainted: G B D
3.3.3-vs2.3.3.3 #1
[18141.054545] Call Trace:
[18141.059420] <IRQ> [<ffffffff8109d328>] ? __rcu_pending+0x82/0x336
[18141.071935] [<ffffffff8109d66e>] ? rcu_check_callbacks+0x92/0xc2
[18141.084084] [<ffffffff8104bf96>] ? update_process_times+0x31/0x63
[18141.096408] [<ffffffff81079ecc>] ? tick_sched_timer+0x65/0x8b
[18141.108039] [<ffffffff8105b58b>] ? __run_hrtimer+0x75/0xe9
[18141.119147] [<ffffffff8105b838>] ? hrtimer_interrupt+0xd8/0x1a6
[18141.131127] [<ffffffff81018f92>] ? smp_apic_timer_interrupt+0x80/0x93
[18141.144144] [<ffffffff814d3e1e>] ? apic_timer_interrupt+0x6e/0x80
[18141.156465] <EOI> [<ffffffff810e7c1f>] ? css_tryget+0x15/0x38
[18141.168282] [<ffffffff810eb3a0>] ? __mem_cgroup_try_charge+0x187/0x4c4
[18141.181469] [<ffffffff81001703>] ? __switch_to+0x238/0x35f
[18141.192577] [<ffffffff810ebb6d>] ? mem_cgroup_charge_common+0x80/0xae
[18141.205601] [<ffffffff810dac01>] ? alloc_pages_vma+0xf0/0x11d
[18141.217230] [<ffffffff810c6ac1>] ? handle_pte_fault+0x1d4/0x95c
[18141.229205] [<ffffffff8105bcf1>] ? hrtimer_try_to_cancel+0x6b/0x74
[18141.241703] [<ffffffff8111f190>] ? ep_read_events_proc+0x6c/0x6c
[18141.253853] [<ffffffff810247f7>] ? do_page_fault+0x2c0/0x2e5
[18141.265309] [<ffffffff810637e4>] ? try_to_wake_up+0x19b/0x19b
[18141.276938] [<ffffffff81008855>] ? read_tsc+0x5/0x16
[18141.287010] [<ffffffff81073cf2>] ? timekeeping_get_ns+0xd/0x2a
[18141.298812] [<ffffffff8107494f>] ? getnstimeofday+0x4d/0x80
[18141.310094] [<ffffffff814d2e05>] ? page_fault+0x25/0x30
Received on Fri Apr 27 17:16:01 2012

[Next/Previous Months] [Main vserver Project Homepage] [Howto Subscribe/Unsubscribe] [Paul Sladen's vserver stuff]
Generated on Fri 27 Apr 2012 - 17:16:01 BST by hypermail 2.1.8