I had kernel panic on different servers under 3.0.24-vs2.3.2.3-beng
lately, these seems to be network related.
There is a recurrent message on the kern.log on these before the crashes.
Here is what i got on one of the servers :
Apr 29 21:42:33 localhost kernel: [823558.569637] ------------[ cut here
]------------
Apr 29 21:42:33 localhost kernel: [823558.569647] WARNING: at
net/ipv4/tcp_input.c:3375 tcp_ack+0x8c4/0x19b8()
Apr 29 21:42:33 localhost kernel: [823558.569656] Hardware name: X9SCL/X9SCM
Apr 29 21:42:33 localhost kernel: [823558.569657] Modules linked in:
nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack xt_tcpudp
iptable_filter ip_tables x_tables nfs lockd auth_rpcgss nfs_acl sunrpc
acpi_cpufreq mperf cpufreq_conservative cpufreq_userspace
cpufreq_powersave cpufreq_stats ext3 jbd coretemp ipmi_si
ipmi_msghandler loop snd_pcm snd_timer snd soundcore intel_agp tpm_tis
tpm snd_page_alloc i2c_i801 i2c_core button joydev ghes processor
intel_gtt tpm_bios hed pcspkr evdev thermal_sys psmouse serio_raw ext4
mbcache jbd2 crc16 usbhid hid sd_mod crc_t10dif ahci libahci libata
scsi_mod ehci_hcd usbcore e1000e [last unloaded: scsi_wait_scan]
Apr 29 21:42:33 localhost kernel: [823558.569686] Pid: 20900, comm:
nodeusage.sh Tainted: G W 3.0.24-vs2.3.2.3-beng #1
Apr 29 21:42:33 localhost kernel: [823558.569687] Call Trace:
Apr 29 21:42:33 localhost kernel: [823558.569688] <IRQ>
[<ffffffff8103f4ca>] ? warn_slowpath_common+0x78/0x8c
Apr 29 21:42:33 localhost kernel: [823558.569694] [<ffffffff812b1133>]
? tcp_ack+0x8c4/0x19b8
Apr 29 21:42:33 localhost kernel: [823558.569697] [<ffffffff812b22f4>]
? tcp_rcv_established+0xcd/0x686
Apr 29 21:42:33 localhost kernel: [823558.569699] [<ffffffff812b96ce>]
? tcp_v4_do_rcv+0x1a9/0x3b4
Apr 29 21:42:33 localhost kernel: [823558.569701] [<ffffffff812bab29>]
? T.1236+0x37/0x69
Apr 29 21:42:33 localhost kernel: [823558.569703] [<ffffffff812bb445>]
? tcp_v4_rcv+0x449/0x6e4
Apr 29 21:42:33 localhost kernel: [823558.569705] [<ffffffff8129e6cf>]
? ip_local_deliver_finish+0x13d/0x1aa
Apr 29 21:42:33 localhost kernel: [823558.569708] [<ffffffff81278556>]
? __netif_receive_skb+0x466/0x499
Apr 29 21:42:33 localhost kernel: [823558.569710] [<ffffffff812787c6>]
? netif_receive_skb+0x77/0x7d
Apr 29 21:42:33 localhost kernel: [823558.569712] [<ffffffff81278d2d>]
? napi_gro_receive+0x1f/0x2d
Apr 29 21:42:33 localhost kernel: [823558.569714] [<ffffffff8127889b>]
? napi_skb_finish+0x1c/0x31
Apr 29 21:42:33 localhost kernel: [823558.569720] [<ffffffffa0010941>]
? e1000_clean_rx_irq+0x1f6/0x28c [e1000e]
Apr 29 21:42:33 localhost kernel: [823558.569724] [<ffffffffa0010dde>]
? e1000_clean+0x71/0x229 [e1000e]
Apr 29 21:42:33 localhost kernel: [823558.569726] [<ffffffff81278e5a>]
? net_rx_action+0xa8/0x206
Apr 29 21:42:33 localhost kernel: [823558.569728] [<ffffffff81044c1b>]
? __do_softirq+0xc4/0x1a0
Apr 29 21:42:33 localhost kernel: [823558.569731] [<ffffffff81092690>]
? handle_irq_event_percpu+0x166/0x184
Apr 29 21:42:33 localhost kernel: [823558.569734] [<ffffffff8133655c>]
? call_softirq+0x1c/0x30
Apr 29 21:42:33 localhost kernel: [823558.569736] [<ffffffff8100408b>]
? do_softirq+0x3f/0x79
Apr 29 21:42:33 localhost kernel: [823558.569738] [<ffffffff810449eb>]
? irq_exit+0x44/0xb5
Apr 29 21:42:33 localhost kernel: [823558.569740] [<ffffffff810039d5>]
? do_IRQ+0x94/0xaa
Apr 29 21:42:33 localhost kernel: [823558.569742] [<ffffffff81334c93>]
? common_interrupt+0x13/0x13
Apr 29 21:42:33 localhost kernel: [823558.569743] <EOI>
[<ffffffff810ea2b7>] ? kmem_cache_free+0x27/0x62
Apr 29 21:42:33 localhost kernel: [823558.569749] [<ffffffff810d5ebe>]
? remove_vma+0x64/0x6c
Apr 29 21:42:33 localhost kernel: [823558.569751] [<ffffffff810d5fa0>]
? exit_mmap+0xda/0xfb
Apr 29 21:42:33 localhost kernel: [823558.569753] [<ffffffff8103d496>]
? mmput+0x36/0xd5
Apr 29 21:42:33 localhost kernel: [823558.569756] [<ffffffff810fde0b>]
? flush_old_exec+0x52b/0x60d
Apr 29 21:42:33 localhost kernel: [823558.569759] [<ffffffff81135830>]
? load_elf_binary+0x2af/0x1733
Apr 29 21:42:33 localhost kernel: [823558.569762] [<ffffffff810fd3f2>]
? get_arg_page+0x4b/0xd9
Apr 29 21:42:33 localhost kernel: [823558.569764] [<ffffffff810fd760>]
? copy_strings+0x1fe/0x245
Apr 29 21:42:33 localhost kernel: [823558.569766] [<ffffffff810feafd>]
? search_binary_handler+0xa0/0x243
Apr 29 21:42:33 localhost kernel: [823558.569767] [<ffffffff810fee36>]
? do_execve_common+0x196/0x2ca
Apr 29 21:42:33 localhost kernel: [823558.569771] [<ffffffff8102a991>]
? sys32_execve+0x3f/0x54
Apr 29 21:42:33 localhost kernel: [823558.569773] [<ffffffff81336c05>]
? ia32_ptregs_common+0x25/0x4b
Apr 29 21:42:33 localhost kernel: [823558.569775] ---[ end trace
1cf81a8e50efa328 ]---
And on another :
May 1 12:19:23 localhost kernel: [95833.147842] ------------[ cut here
]------------
May 1 12:19:23 localhost kernel: [95833.147844] WARNING: at
net/ipv4/tcp_input.c:2999 tcp_ack+0xe42/0x19b8()
May 1 12:19:23 localhost kernel: [95833.147845] Hardware name: X8STi
May 1 12:19:23 localhost kernel: [95833.147846] Modules linked in:
acpi_cpufreq mperf cpufreq_userspace cpufreq_stats cpufreq_conservative
cpufreq_powersave bonding ipt_LOG xt_limit xt_multiport iptable_filter
ip_tables x_tables ext3 jbd w83627ehf hwmon_vid coretemp ipmi_si
ipmi_msghandler loop snd_pcm snd_timer snd soundcore i2c_i801 i2c_core
tpm_tis tpm ioatdma tpm_bios snd_page_alloc i7core_edac evdev dca
edac_core button processor thermal_sys psmouse pcspkr serio_raw ext4
mbcache jbd2 crc16 dm_mod raid1 raid0 md_mod sd_mod crc_t10dif usbhid
hid uhci_hcd ahci libahci libata scsi_mod ehci_hcd usbcore e1000e [last
unloaded: scsi_wait_scan]
May 1 12:19:23 localhost kernel: [95833.147866] Pid: 1916, comm:
php5-cgi Tainted: G W 3.0.24-vs2.3.2.3-beng #1
May 1 12:19:23 localhost kernel: [95833.147867] Call Trace:
May 1 12:19:23 localhost kernel: [95833.147868] <IRQ>
[<ffffffff8103f4ca>] ? warn_slowpath_common+0x78/0x8c
May 1 12:19:23 localhost kernel: [95833.147871] [<ffffffff812b16b1>] ?
tcp_ack+0xe42/0x19b8
May 1 12:19:23 localhost kernel: [95833.147873] [<ffffffff812b02d4>] ?
tcp_validate_incoming+0x68/0x255
May 1 12:19:23 localhost kernel: [95833.147876] [<ffffffff812b27dc>] ?
tcp_rcv_established+0x5b5/0x686
May 1 12:19:23 localhost kernel: [95833.147878] [<ffffffff812b96ce>] ?
tcp_v4_do_rcv+0x1a9/0x3b4
May 1 12:19:23 localhost kernel: [95833.147880] [<ffffffff812bab29>] ?
T.1236+0x37/0x69
May 1 12:19:23 localhost kernel: [95833.147881] [<ffffffff812bb445>] ?
tcp_v4_rcv+0x449/0x6e4
May 1 12:19:23 localhost kernel: [95833.147884] [<ffffffffa02a6a19>] ?
bond_alb_initialize+0x159/0x159 [bonding]
May 1 12:19:23 localhost kernel: [95833.147886] [<ffffffff8129e6cf>] ?
ip_local_deliver_finish+0x13d/0x1aa
May 1 12:19:23 localhost kernel: [95833.147888] [<ffffffff81278556>] ?
__netif_receive_skb+0x466/0x499
May 1 12:19:23 localhost kernel: [95833.147890] [<ffffffff812787c6>] ?
netif_receive_skb+0x77/0x7d
May 1 12:19:23 localhost kernel: [95833.147892] [<ffffffff81278d2d>] ?
napi_gro_receive+0x1f/0x2d
May 1 12:19:23 localhost kernel: [95833.147894] [<ffffffff8127889b>] ?
napi_skb_finish+0x1c/0x31
May 1 12:19:23 localhost kernel: [95833.147898] [<ffffffffa0010941>] ?
e1000_clean_rx_irq+0x1f6/0x28c [e1000e]
May 1 12:19:23 localhost kernel: [95833.147902] [<ffffffffa0010dde>] ?
e1000_clean+0x71/0x229 [e1000e]
May 1 12:19:23 localhost kernel: [95833.147904] [<ffffffff81278e5a>] ?
net_rx_action+0xa8/0x206
May 1 12:19:23 localhost kernel: [95833.147906] [<ffffffff81044c1b>] ?
__do_softirq+0xc4/0x1a0
May 1 12:19:23 localhost kernel: [95833.147908] [<ffffffff81092690>] ?
handle_irq_event_percpu+0x166/0x184
May 1 12:19:23 localhost kernel: [95833.147910] [<ffffffff81044c94>] ?
__do_softirq+0x13d/0x1a0
May 1 12:19:23 localhost kernel: [95833.147912] [<ffffffff8133655c>] ?
call_softirq+0x1c/0x30
May 1 12:19:23 localhost kernel: [95833.147913] [<ffffffff8100408b>] ?
do_softirq+0x3f/0x79
May 1 12:19:23 localhost kernel: [95833.147915] [<ffffffff810449eb>] ?
irq_exit+0x44/0xb5
May 1 12:19:23 localhost kernel: [95833.147917] [<ffffffff810039d5>] ?
do_IRQ+0x94/0xaa
May 1 12:19:23 localhost kernel: [95833.147919] [<ffffffff81334c93>] ?
common_interrupt+0x13/0x13
May 1 12:19:23 localhost kernel: [95833.147920] <EOI>
[<ffffffff81336a83>] ? ia32_do_call+0x13/0x13
May 1 12:19:23 localhost kernel: [95833.147922] ---[ end trace
43efaa2da94bb3b9 ]---
It seems to be this bug : http://patchwork.ozlabs.org/patch/144844/
I installed linux-image-3.2.15-vs2.3.2.8-beng on the servers that had
this kernel hoping it wont happen again on these.
Greetings.
Received on Thu May 3 03:08:52 2012