[vserver] Problems with 2.6.33.2 memcg

From: Cryptronic <mail_at_cryptronic.de>
Date: Mon 12 Apr 2010 - 20:19:57 BST
Message-ID: <4BC3725D.80602@cryptronic.de>

Hi,

the cgroup thing is a really nice function, but I experienced that if
one cgroup gets into an oom situation the whole host system is nearly
unusable. load raises to 300+ only because of the oom killer. Can the
oom killer be configuered to kill the next best process instead of
using the whole oom killer featureset.

This problem makes nearly impossible to use memcg, because all guests
on the host do not react while the cg is in oom. Is there maybe a
possible tuning?

Values:

# head /etc/vservers/testcg/cgroup/memory.*
==> /etc/vservers/testcg/cgroup/memory.limit_in_bytes <==
536870912

==> /etc/vservers/testcg/cgroup/memory.memsw.limit_in_bytes <==
1610612736

example dmest output after oom occured:

[253637.756988] php5-cgi invoked oom-killer: gfp_mask=0xd0, order=0,
oom_adj=0
[253637.757020] php5-cgi cpuset=testcg mems_allowed=0
[253637.757049] Pid: 13023, comm: php5-cgi Tainted: G W
2.6.33.2-vs2.3.0.36.30.4 #1
[253637.757097] Call Trace:
[253637.757123] [<ffffffff810b385a>] ? dump_header+0x64/0x184
[253637.757153] [<ffffffff811e4f71>] ? ___ratelimit+0xc5/0xe0
[253637.757182] [<ffffffff810b39b9>] ? oom_kill_process+0x3f/0x134
[253637.757211] [<ffffffff810b41a4>] ? mem_cgroup_out_of_memory+0x67/0x7f
[253637.757242] [<ffffffff810e75d0>] ?
res_counter_check_under_limit+0x2b/0x33
[253637.757274] [<ffffffff810e8317>] ?
__mem_cgroup_try_charge+0x1c4/0x269
[253637.757327] [<ffffffff810e96d2>] ?
mem_cgroup_try_charge_swapin+0x5b/0xb5
[253637.757360] [<ffffffff810c9efd>] ? handle_mm_fault+0x5bb/0x8c3
[253637.757392] [<ffffffff810cfbb9>] ? mmap_region+0x3e8/0x51d
[253637.757424] [<ffffffff810248a7>] ? do_page_fault+0x25f/0x27b
[253637.757457] [<ffffffff81359805>] ? page_fault+0x25/0x30
[253637.757486] Task in /testcg killed as a result of limit of /testcg
[253637.757537] memory: usage 524164kB, limit 524288kB, failcnt 2682457
[253637.757566] memory+swap: usage 1565848kB, limit 1572864kB, failcnt
3718537
[253637.757596] Mem-Info:
[253637.757617] Node 0 DMA per-cpu:
[253637.757643] CPU 0: hi: 0, btch: 1 usd: 0
[253637.757669] CPU 1: hi: 0, btch: 1 usd: 0
[253637.757695] CPU 2: hi: 0, btch: 1 usd: 0
[253637.757722] CPU 3: hi: 0, btch: 1 usd: 0
[253637.757748] CPU 4: hi: 0, btch: 1 usd: 0
[253637.757774] CPU 5: hi: 0, btch: 1 usd: 0
[253637.757800] CPU 6: hi: 0, btch: 1 usd: 0
[253637.757826] CPU 7: hi: 0, btch: 1 usd: 0
[253637.757852] Node 0 DMA32 per-cpu:
[253637.757878] CPU 0: hi: 186, btch: 31 usd: 171
[253637.757905] CPU 1: hi: 186, btch: 31 usd: 170
[253637.757931] CPU 2: hi: 186, btch: 31 usd: 57
[253637.757957] CPU 3: hi: 186, btch: 31 usd: 179
[253637.757983] CPU 4: hi: 186, btch: 31 usd: 170
[253637.758009] CPU 5: hi: 186, btch: 31 usd: 166
[253637.758036] CPU 6: hi: 186, btch: 31 usd: 176
[253637.758062] CPU 7: hi: 186, btch: 31 usd: 166
[253637.758088] Node 0 Normal per-cpu:
[253637.758114] CPU 0: hi: 186, btch: 31 usd: 181
[253637.758140] CPU 1: hi: 186, btch: 31 usd: 153
[253637.758166] CPU 2: hi: 186, btch: 31 usd: 106
[253637.758192] CPU 3: hi: 186, btch: 31 usd: 107
[253637.758219] CPU 4: hi: 186, btch: 31 usd: 163
[253637.758245] CPU 5: hi: 186, btch: 31 usd: 173
[253637.758271] CPU 6: hi: 186, btch: 31 usd: 141
[253637.758297] CPU 7: hi: 186, btch: 31 usd: 116
[253637.758325] active_anon:455567 inactive_anon:345966 isolated_anon:0
[253637.758326] active_file:636964 inactive_file:860936 isolated_file:0
[253637.758327] unevictable:0 dirty:69 writeback:39265 unstable:0
[253637.758328] free:285568 slab_reclaimable:334023
slab_unreclaimable:86402
[253637.758329] mapped:125147 shmem:25748 pagetables:44064 bounce:0
[253637.758478] Node 0 DMA free:15828kB min:16kB low:20kB high:24kB
active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB
unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15244kB
mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB
slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB
pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB
pages_scanned:0 all_unreclaimable? yes
[253637.758646] lowmem_reserve[]: 0 2991 12081 12081
[253637.758686] Node 0 DMA32 free:298360kB min:3480kB low:4348kB
high:5220kB active_anon:378100kB inactive_anon:371588kB
active_file:197516kB inactive_file:1184520kB unevictable:0kB
isolated(anon):0kB isolated(file):0kB present:3063520kB mlocked:0kB
dirty:16kB writeback:60916kB mapped:15892kB shmem:17240kB
slab_reclaimable:353484kB slab_unreclaimable:53416kB
kernel_stack:3976kB pagetables:24644kB unstable:0kB bounce:0kB
writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[253637.758859] lowmem_reserve[]: 0 0 9090 9090
[253637.758895] Node 0 Normal free:829944kB min:10576kB low:13220kB
high:15864kB active_anon:1444168kB inactive_anon:1010528kB
active_file:2350340kB inactive_file:2259224kB unevictable:0kB
isolated(anon):0kB isolated(file):0kB present:9308160kB mlocked:0kB
dirty:260kB writeback:96144kB mapped:484696kB shmem:85752kB
slab_reclaimable:982608kB slab_unreclaimable:292192kB
kernel_stack:15384kB pagetables:151612kB unstable:0kB bounce:0kB
writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[253637.759086] lowmem_reserve[]: 0 0 0 0
[253637.759123] Node 0 DMA: 3*4kB 3*8kB 3*16kB 2*32kB 1*64kB 0*128kB
1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15828kB
[253637.759205] Node 0 DMA32: 18652*4kB 4161*8kB 1880*16kB 1885*32kB
877*64kB 233*128kB 29*256kB 3*512kB 0*1024kB 1*2048kB 1*4096kB = 299352kB
[253637.759291] Node 0 Normal: 153155*4kB 26609*8kB 92*16kB 0*32kB
0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 831060kB
[253637.759375] 1789422 total pagecache pages
[253637.759399] 265776 pages in swap cache
[253637.759424] Swap cache stats: add 17070423, delete 16804666, find
11968919/13733735
[253637.759469] Free swap = 21619328kB
[253637.759493] Total swap = 23446776kB
[253637.804817] 3145712 pages RAM
[253637.804842] 64004 pages reserved
[253637.804865] 1465339 pages shared
[253637.804889] 2061427 pages non-shared
[253637.804913] ------------[ cut here ]------------
[253637.804944] WARNING: at mm/oom_kill.c:411 __oom_kill_task+0x69/0x10b()
[253637.804974] Hardware name: MS-7522
[253637.804997] Modules linked in: cls_u32 sch_htb ifb xt_tcpudp
xt_state ipt_REJECT ipt_LOG xt_limit iptable_filter xt_DSCP xt_recent
nf_nat_ftp nf_nat nf_conntrack_ftp nf_conntrack_irc nf_conntrack_ipv4
nf_conntrack nf_defrag_ipv4 ip_tables x_tables nfs lockd nfs_acl
auth_rpcgss sunrpc pcspkr evdev tpm_tis tpm tpm_bios i2c_i801 i2c_core
wmi processor button dm_mirror dm_region_hash dm_log dm_snapshot
dm_mod raid1 md_mod sd_mod crc_t10dif ahci libata r8169 mii ehci_hcd
uhci_hcd usbcore nls_base thermal fan thermal_sys [last unloaded:
scsi_wait_scan]
[253637.805342] Pid: 13145, comm: php5-cgi Tainted: G W
2.6.33.2-vs2.3.0.36.30.4 #1
[253637.805390] Call Trace:
[253637.805415] [<ffffffff810b3629>] ? __oom_kill_task+0x69/0x10b
[253637.805445] [<ffffffff810b3629>] ? __oom_kill_task+0x69/0x10b
[253637.805476] [<ffffffff8103f4cb>] ? warn_slowpath_common+0x77/0xa3
[253637.805507] [<ffffffff810b3629>] ? __oom_kill_task+0x69/0x10b
[253637.805536] [<ffffffff810b3a50>] ? oom_kill_process+0xd6/0x134
[253637.805566] [<ffffffff810b41a4>] ? mem_cgroup_out_of_memory+0x67/0x7f
[253637.805598] [<ffffffff810e75d0>] ?
res_counter_check_under_limit+0x2b/0x33
[253637.805631] [<ffffffff810e8317>] ?
__mem_cgroup_try_charge+0x1c4/0x269
[253637.805662] [<ffffffff810e8f90>] ? mem_cgroup_charge_common+0x4d/0x7a
[253637.805695] [<ffffffff810c79e1>] ? __do_fault+0x101/0x457
[253637.805731] [<ffffffff810c9cfd>] ? handle_mm_fault+0x3bb/0x8c3
[253637.805764] [<ffffffff810248a7>] ? do_page_fault+0x25f/0x27b
[253637.805796] [<ffffffff81359805>] ? page_fault+0x25/0x30
[253637.805825] ---[ end trace 2de4787caa53e878 ]---
[253637.805852] tried to kill an mm-less task 8389 (php5-cgi)!
[253637.841020] 3145712 pages RAM
[253637.841046] 64004 pages reserved
[253637.841099] 1465351 pages shared
[253637.841123] 2061037 pages non-shared
[253637.841150] Memory cgroup out of memory: kill process
php5-cgi(8270:#7041) score 27164 or a child
[253637.841200] Killed process php5-cgi(8270:#7041) vsz:108656kB,
anon-rss:2872kB, file-rss:5912kB

another question:

Does vhashify also safe memory when using cgroups?

best regards & thanks
Received on Mon Apr 12 20:20:16 2010

[Next/Previous Months] [Main vserver Project Homepage] [Howto Subscribe/Unsubscribe] [Paul Sladen's vserver stuff]
Generated on Mon 12 Apr 2010 - 20:20:18 BST by hypermail 2.1.8