Re: [vserver] Problems with 2.6.33.2 memcg

From: Herbert Poetzl <herbert_at_13thfloor.at>
Date: Tue 13 Apr 2010 - 19:19:07 BST
Message-ID: <20100413181907.GD30628@MAIL.13thfloor.at>

On Mon, Apr 12, 2010 at 09:19:57PM +0200, Cryptronic wrote:
> Hi,

> the cgroup thing is a really nice function, but I experienced that if
> one cgroup gets into an oom situation the whole host system is nearly
> unusable. load raises to 300+ only because of the oom killer.

> Can the oom killer be configuered to kill the next best process
> instead of using the whole oom killer featureset.

yes, there is a sysfs attribute called oom_kill_allocating_task
which should do what you want, i.e. kill the task currently
allocating memory (i.e. the one active when the issue arises)

> This problem makes nearly impossible to use memcg, because all guests
> on the host do not react while the cg is in oom.

> Is there maybe a possible tuning?

tunings on the memcg subsystem should be done on mainline,
we are just providing the subsystem for Linux-VServer use

HTH,
Herbert

> Values:
>
> # head /etc/vservers/testcg/cgroup/memory.*
> ==> /etc/vservers/testcg/cgroup/memory.limit_in_bytes <==
> 536870912
>
> ==> /etc/vservers/testcg/cgroup/memory.memsw.limit_in_bytes <==
> 1610612736
>
>
> example dmest output after oom occured:
>
> [253637.756988] php5-cgi invoked oom-killer: gfp_mask=0xd0, order=0,
> oom_adj=0
> [253637.757020] php5-cgi cpuset=testcg mems_allowed=0
> [253637.757049] Pid: 13023, comm: php5-cgi Tainted: G W
> 2.6.33.2-vs2.3.0.36.30.4 #1
> [253637.757097] Call Trace:
> [253637.757123] [<ffffffff810b385a>] ? dump_header+0x64/0x184
> [253637.757153] [<ffffffff811e4f71>] ? ___ratelimit+0xc5/0xe0
> [253637.757182] [<ffffffff810b39b9>] ? oom_kill_process+0x3f/0x134
> [253637.757211] [<ffffffff810b41a4>] ? mem_cgroup_out_of_memory+0x67/0x7f
> [253637.757242] [<ffffffff810e75d0>] ?
> res_counter_check_under_limit+0x2b/0x33
> [253637.757274] [<ffffffff810e8317>] ?
> __mem_cgroup_try_charge+0x1c4/0x269
> [253637.757327] [<ffffffff810e96d2>] ?
> mem_cgroup_try_charge_swapin+0x5b/0xb5
> [253637.757360] [<ffffffff810c9efd>] ? handle_mm_fault+0x5bb/0x8c3
> [253637.757392] [<ffffffff810cfbb9>] ? mmap_region+0x3e8/0x51d
> [253637.757424] [<ffffffff810248a7>] ? do_page_fault+0x25f/0x27b
> [253637.757457] [<ffffffff81359805>] ? page_fault+0x25/0x30
> [253637.757486] Task in /testcg killed as a result of limit of /testcg
> [253637.757537] memory: usage 524164kB, limit 524288kB, failcnt 2682457
> [253637.757566] memory+swap: usage 1565848kB, limit 1572864kB, failcnt
> 3718537
> [253637.757596] Mem-Info:
> [253637.757617] Node 0 DMA per-cpu:
> [253637.757643] CPU 0: hi: 0, btch: 1 usd: 0
> [253637.757669] CPU 1: hi: 0, btch: 1 usd: 0
> [253637.757695] CPU 2: hi: 0, btch: 1 usd: 0
> [253637.757722] CPU 3: hi: 0, btch: 1 usd: 0
> [253637.757748] CPU 4: hi: 0, btch: 1 usd: 0
> [253637.757774] CPU 5: hi: 0, btch: 1 usd: 0
> [253637.757800] CPU 6: hi: 0, btch: 1 usd: 0
> [253637.757826] CPU 7: hi: 0, btch: 1 usd: 0
> [253637.757852] Node 0 DMA32 per-cpu:
> [253637.757878] CPU 0: hi: 186, btch: 31 usd: 171
> [253637.757905] CPU 1: hi: 186, btch: 31 usd: 170
> [253637.757931] CPU 2: hi: 186, btch: 31 usd: 57
> [253637.757957] CPU 3: hi: 186, btch: 31 usd: 179
> [253637.757983] CPU 4: hi: 186, btch: 31 usd: 170
> [253637.758009] CPU 5: hi: 186, btch: 31 usd: 166
> [253637.758036] CPU 6: hi: 186, btch: 31 usd: 176
> [253637.758062] CPU 7: hi: 186, btch: 31 usd: 166
> [253637.758088] Node 0 Normal per-cpu:
> [253637.758114] CPU 0: hi: 186, btch: 31 usd: 181
> [253637.758140] CPU 1: hi: 186, btch: 31 usd: 153
> [253637.758166] CPU 2: hi: 186, btch: 31 usd: 106
> [253637.758192] CPU 3: hi: 186, btch: 31 usd: 107
> [253637.758219] CPU 4: hi: 186, btch: 31 usd: 163
> [253637.758245] CPU 5: hi: 186, btch: 31 usd: 173
> [253637.758271] CPU 6: hi: 186, btch: 31 usd: 141
> [253637.758297] CPU 7: hi: 186, btch: 31 usd: 116
> [253637.758325] active_anon:455567 inactive_anon:345966 isolated_anon:0
> [253637.758326] active_file:636964 inactive_file:860936 isolated_file:0
> [253637.758327] unevictable:0 dirty:69 writeback:39265 unstable:0
> [253637.758328] free:285568 slab_reclaimable:334023
> slab_unreclaimable:86402
> [253637.758329] mapped:125147 shmem:25748 pagetables:44064 bounce:0
> [253637.758478] Node 0 DMA free:15828kB min:16kB low:20kB high:24kB
> active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB
> unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15244kB
> mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB
> slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB
> pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB
> pages_scanned:0 all_unreclaimable? yes
> [253637.758646] lowmem_reserve[]: 0 2991 12081 12081
> [253637.758686] Node 0 DMA32 free:298360kB min:3480kB low:4348kB
> high:5220kB active_anon:378100kB inactive_anon:371588kB
> active_file:197516kB inactive_file:1184520kB unevictable:0kB
> isolated(anon):0kB isolated(file):0kB present:3063520kB mlocked:0kB
> dirty:16kB writeback:60916kB mapped:15892kB shmem:17240kB
> slab_reclaimable:353484kB slab_unreclaimable:53416kB
> kernel_stack:3976kB pagetables:24644kB unstable:0kB bounce:0kB
> writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
> [253637.758859] lowmem_reserve[]: 0 0 9090 9090
> [253637.758895] Node 0 Normal free:829944kB min:10576kB low:13220kB
> high:15864kB active_anon:1444168kB inactive_anon:1010528kB
> active_file:2350340kB inactive_file:2259224kB unevictable:0kB
> isolated(anon):0kB isolated(file):0kB present:9308160kB mlocked:0kB
> dirty:260kB writeback:96144kB mapped:484696kB shmem:85752kB
> slab_reclaimable:982608kB slab_unreclaimable:292192kB
> kernel_stack:15384kB pagetables:151612kB unstable:0kB bounce:0kB
> writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
> [253637.759086] lowmem_reserve[]: 0 0 0 0
> [253637.759123] Node 0 DMA: 3*4kB 3*8kB 3*16kB 2*32kB 1*64kB 0*128kB
> 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15828kB
> [253637.759205] Node 0 DMA32: 18652*4kB 4161*8kB 1880*16kB 1885*32kB
> 877*64kB 233*128kB 29*256kB 3*512kB 0*1024kB 1*2048kB 1*4096kB = 299352kB
> [253637.759291] Node 0 Normal: 153155*4kB 26609*8kB 92*16kB 0*32kB
> 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 831060kB
> [253637.759375] 1789422 total pagecache pages
> [253637.759399] 265776 pages in swap cache
> [253637.759424] Swap cache stats: add 17070423, delete 16804666, find
> 11968919/13733735
> [253637.759469] Free swap = 21619328kB
> [253637.759493] Total swap = 23446776kB
> [253637.804817] 3145712 pages RAM
> [253637.804842] 64004 pages reserved
> [253637.804865] 1465339 pages shared
> [253637.804889] 2061427 pages non-shared
> [253637.804913] ------------[ cut here ]------------
> [253637.804944] WARNING: at mm/oom_kill.c:411 __oom_kill_task+0x69/0x10b()
> [253637.804974] Hardware name: MS-7522
> [253637.804997] Modules linked in: cls_u32 sch_htb ifb xt_tcpudp
> xt_state ipt_REJECT ipt_LOG xt_limit iptable_filter xt_DSCP xt_recent
> nf_nat_ftp nf_nat nf_conntrack_ftp nf_conntrack_irc nf_conntrack_ipv4
> nf_conntrack nf_defrag_ipv4 ip_tables x_tables nfs lockd nfs_acl
> auth_rpcgss sunrpc pcspkr evdev tpm_tis tpm tpm_bios i2c_i801 i2c_core
> wmi processor button dm_mirror dm_region_hash dm_log dm_snapshot
> dm_mod raid1 md_mod sd_mod crc_t10dif ahci libata r8169 mii ehci_hcd
> uhci_hcd usbcore nls_base thermal fan thermal_sys [last unloaded:
> scsi_wait_scan]
> [253637.805342] Pid: 13145, comm: php5-cgi Tainted: G W
> 2.6.33.2-vs2.3.0.36.30.4 #1
> [253637.805390] Call Trace:
> [253637.805415] [<ffffffff810b3629>] ? __oom_kill_task+0x69/0x10b
> [253637.805445] [<ffffffff810b3629>] ? __oom_kill_task+0x69/0x10b
> [253637.805476] [<ffffffff8103f4cb>] ? warn_slowpath_common+0x77/0xa3
> [253637.805507] [<ffffffff810b3629>] ? __oom_kill_task+0x69/0x10b
> [253637.805536] [<ffffffff810b3a50>] ? oom_kill_process+0xd6/0x134
> [253637.805566] [<ffffffff810b41a4>] ? mem_cgroup_out_of_memory+0x67/0x7f
> [253637.805598] [<ffffffff810e75d0>] ?
> res_counter_check_under_limit+0x2b/0x33
> [253637.805631] [<ffffffff810e8317>] ?
> __mem_cgroup_try_charge+0x1c4/0x269
> [253637.805662] [<ffffffff810e8f90>] ? mem_cgroup_charge_common+0x4d/0x7a
> [253637.805695] [<ffffffff810c79e1>] ? __do_fault+0x101/0x457
> [253637.805731] [<ffffffff810c9cfd>] ? handle_mm_fault+0x3bb/0x8c3
> [253637.805764] [<ffffffff810248a7>] ? do_page_fault+0x25f/0x27b
> [253637.805796] [<ffffffff81359805>] ? page_fault+0x25/0x30
> [253637.805825] ---[ end trace 2de4787caa53e878 ]---
> [253637.805852] tried to kill an mm-less task 8389 (php5-cgi)!
> [253637.841020] 3145712 pages RAM
> [253637.841046] 64004 pages reserved
> [253637.841099] 1465351 pages shared
> [253637.841123] 2061037 pages non-shared
> [253637.841150] Memory cgroup out of memory: kill process
> php5-cgi(8270:#7041) score 27164 or a child
> [253637.841200] Killed process php5-cgi(8270:#7041) vsz:108656kB,
> anon-rss:2872kB, file-rss:5912kB
>
> another question:
>
> Does vhashify also safe memory when using cgroups?
>
> best regards & thanks
Received on Tue Apr 13 20:18:51 2010

[Next/Previous Months] [Main vserver Project Homepage] [Howto Subscribe/Unsubscribe] [Paul Sladen's vserver stuff]
Generated on Tue 13 Apr 2010 - 20:18:53 BST by hypermail 2.1.8