Re: [vserver] Exploding Load in v2.3

From: Cryptronic <mail_at_cryptronic.de>
Date: Mon 18 Jan 2010 - 21:35:38 GMT
Message-ID: <4B54D42A.1050708@cryptronic.de>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hello Nik,

yes indeed i have 74 GB of RAM in that machine. VServers 50+.

I use a shared partition with ext3 or ext4.

remote syslog is not enabled yet, will do that in the next days. And
in syslog i could not find anything.

In messages i found the following which must be appeared in between
the load explosion, but not at the beginning:

before:
Jan 18 21:00:02 host kernel: [20304486.773634] vxW:
[»landscape-sysin«,28639:#9113|9113|9113] did lookup hidden
proc:ffff8812034377f8[#0,4026531914] »/proc/acpi«.

then:
Jan 18 21:11:45 host kernel: [20304486.368940] apache2 invoked
oom-killer: gfp_mask=0x0, order=0,
oomkilladj=0
 
 
 

Jan 18 21:11:45 host kernel: [20304486.368944] apache2 cpuset=/
mems_allowed=0-1
 
 
 

Jan 18 21:11:45 host kernel: [20304486.368947] Pid: 5090, comm:
apache2 Not tainted 2.6.31.7-vs2.3.0.36.27
#1
 
 
 

Jan 18 21:11:45 host kernel: [20304486.368949] Call
Trace:
 
 
 

Jan 18 21:11:45 host kernel: [20304486.368959] [<ffffffff810ae4e4>] ?
oom_kill_process+0x8e/0x265
 
 
 

Jan 18 21:11:45 host kernel: [20304486.368962] [<ffffffff810ae9bf>] ?
select_bad_process+0xd2/0x12f
 
 
 

Jan 18 21:11:45 host kernel: [20304486.368965] [<ffffffff810aea9d>] ?
__out_of_memory+0x81/0x8e
 
 
 

Jan 18 21:11:45 host kernel: [20304486.368967] [<ffffffff810aec80>] ?
pagefault_out_of_memory+0x64/0x8c
 
 
 

Jan 18 21:11:45 host kernel: [20304486.368972] [<ffffffff8102b0ac>] ?
mm_fault_error+0x39/0xdf
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369034] [<ffffffff8105085a>] ?
do_sigaction+0x86/0x169
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369036] [<ffffffff8102b3b1>] ?
do_page_fault+0x25f/0x27b
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369041] [<ffffffff812c2915>] ?
page_fault+0x25/0x30
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369043]
Mem-Info:
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369045] Node 0 Normal
per-cpu:
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369047] CPU 0: hi: 186,
btch: 31 usd:
75
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369049] CPU 1: hi: 186,
btch: 31 usd:
14
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369051] CPU 2: hi: 186,
btch: 31 usd:
169
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369053] CPU 3: hi: 186,
btch: 31 usd:
0
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369055] CPU 4: hi: 186,
btch: 31 usd:
52
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369056] CPU 5: hi: 186,
btch: 31 usd:
0
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369058] CPU 6: hi: 186,
btch: 31 usd:
119
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369060] CPU 7: hi: 186,
btch: 31 usd:
1
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369061] CPU 8: hi: 186,
btch: 31 usd:
50
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369063] CPU 9: hi: 186,
btch: 31 usd:
159
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369065] CPU 10: hi: 186,
btch: 31 usd:
85
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369067] CPU 11: hi: 186,
btch: 31 usd:
184
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369068] CPU 12: hi: 186,
btch: 31 usd:
145
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369070] CPU 13: hi: 186,
btch: 31 usd:
156
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369071] CPU 14: hi: 186,
btch: 31 usd:
74
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369073] CPU 15: hi: 186,
btch: 31 usd:
183
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369075] Node 1 DMA
per-cpu:
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369077] CPU 0: hi: 0,
btch: 1 usd:
0
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369079] CPU 1: hi: 0,
btch: 1 usd:
0
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369080] CPU 2: hi: 0,
btch: 1 usd:
0
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369085] CPU 3: hi: 0,
btch: 1 usd:
0
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369086] CPU 4: hi: 0,
btch: 1 usd:
0
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369088] CPU 5: hi: 0,
btch: 1 usd:
0
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369089] CPU 6: hi: 0,
btch: 1 usd:
0
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369091] CPU 7: hi: 0,
btch: 1 usd:
0
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369092] CPU 8: hi: 0,
btch: 1 usd:
0
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369094] CPU 9: hi: 0,
btch: 1 usd:
0
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369095] CPU 10: hi: 0,
btch: 1 usd:
0
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369097] CPU 11: hi: 0,
btch: 1 usd:
0
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369099] CPU 12: hi: 0,
btch: 1 usd:
0
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369100] CPU 13: hi: 0,
btch: 1 usd:
0
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369102] CPU 14: hi: 0,
btch: 1 usd:
0
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369103] CPU 15: hi: 0,
btch: 1 usd:
0
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369104] Node 1 DMA32
per-cpu:
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369106] CPU 0: hi: 186,
btch: 31 usd:
165
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369108] CPU 1: hi: 186,
btch: 31 usd:
156
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369110] CPU 2: hi: 186,
btch: 31 usd:
157
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369111] CPU 3: hi: 186,
btch: 31 usd:
174
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369113] CPU 4: hi: 186,
btch: 31 usd:
147
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369114] CPU 5: hi: 186,
btch: 31 usd:
178
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369116] CPU 6: hi: 186,
btch: 31 usd:
57
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369118] CPU 7: hi: 186,
btch: 31 usd:
180
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369119] CPU 8: hi: 186,
btch: 31 usd:
50
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369121] CPU 9: hi: 186,
btch: 31 usd:
175
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369122] CPU 10: hi: 186,
btch: 31 usd:
94
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369124] CPU 11: hi: 186,
btch: 31 usd:
157
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369125] CPU 12: hi: 186,
btch: 31 usd:
17
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369127] CPU 13: hi: 186,
btch: 31 usd:
182
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369129] CPU 14: hi: 186,
btch: 31 usd:
113
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369130] CPU 15: hi: 186,
btch: 31 usd:
161
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369131] Node 1 Normal
per-cpu:
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369133] CPU 0: hi: 186,
btch: 31 usd:
164
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369135] CPU 1: hi: 186,
btch: 31 usd:
172
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369136] CPU 2: hi: 186,
btch: 31 usd:
165
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369138] CPU 3: hi: 186,
btch: 31 usd:
15
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369139] CPU 4: hi: 186,
btch: 31 usd:
173
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369141] CPU 5: hi: 186,
btch: 31 usd:
60
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369142] CPU 6: hi: 186,
btch: 31 usd:
183
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369144] CPU 7: hi: 186,
btch: 31 usd:
15
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369146] CPU 8: hi: 186,
btch: 31 usd:
166
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369147] CPU 9: hi: 186,
btch: 31 usd:
37
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369149] CPU 10: hi: 186,
btch: 31 usd:
179
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369150] CPU 11: hi: 186,
btch: 31 usd:
103
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369152] CPU 12: hi: 186,
btch: 31 usd:
156
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369153] CPU 13: hi: 186,
btch: 31 usd:
21
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369155] CPU 14: hi: 186,
btch: 31 usd:
169
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369156] CPU 15: hi: 186,
btch: 31 usd:
40
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369160] Active_anon:3764691
active_file:3669111
inactive_anon:287488
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369161] inactive_file:3477160
unevictable:2098 dirty:483 writeback:0
unstable:0
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369162] free:139331
slab:6942059 mapped:402654 pagetables:170875
bounce:0
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369164] Node 0 Normal
free:42472kB min:17260kB low:21572kB high:25888kB
active_anon:7050920kB inactive_anon:479312kB active_file:7335212kB
inactive_file:6451136kB unevictable:6276k
B present:37232640kB pages_scanned:0 all_unreclaimable?
no
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369169] lowmem_reserve[]: 0 0 0
0
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369171] Node 1 DMA free:15648kB
min:4kB low:4kB high:4kB active_anon:0kB inactive_anon:0kB
active_file:0kB inactive_file:0kB unevictable:0kB present:15084kB
pages_scanned:0 all_unr
eclaimable?
no
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369176] lowmem_reserve[]: 0
3246 36324
36324
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369178] Node 1 DMA32
free:136268kB min:1540kB low:1924kB high:2308kB active_anon:513180kB
inactive_anon:135772kB active_file:469384kB inactive_file:497316kB
unevictable:0kB present
:3324740kB pages_scanned:0 all_unreclaimable?
no
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369183] lowmem_reserve[]: 0 0
33077
33077
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369186] Node 1 Normal
free:362936kB min:15700kB low:19624kB high:23548kB
active_anon:7494664kB inactive_anon:534868kB active_file:6871848kB
inactive_file:6960188kB unevictable:2116
kB present:33871360kB pages_scanned:0 all_unreclaimable?
no
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369191] lowmem_reserve[]: 0 0 0
0
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369193] Node 0 Normal: 7756*4kB
917*8kB 1*16kB 1*32kB 0*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB
0*4096kB =
42376kB
Jan 18 21:11:45 host kernel: [20304486.369200] Node 1 DMA: 0*4kB 2*8kB
1*16kB 2*32kB 1*64kB 1*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB
3*4096kB =
15648kB
Jan 18 21:11:45 host kernel: [20304486.369265] Node 1 DMA32: 16357*4kB
8349*8kB 8*16kB 0*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB
1*2048kB 0*4096kB =
136380kB
Jan 18 21:11:45 host kernel: [20304486.369272] Node 1 Normal:
88607*4kB 567*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB
0*1024kB 0*2048kB 1*4096kB =
363060kB
Jan 18 21:11:45 host kernel: [20304486.369279] 7328314 total pagecache
pages
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369280] 76871 pages in swap
cache
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369282] Swap cache stats: add
5351866, delete 5274995, find
197725297/198123099
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369284] Free swap =
136605644kB
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369285] Total swap =
140624968kB
 
 
 

Jan 18 21:11:46 host kernel: [20304486.781057] 18874368 pages
RAM
 
 
 

Jan 18 21:11:46 host kernel: [20304486.781060] 280153 pages
reserved
 
 
 

Jan 18 21:11:46 host kernel: [20304486.781061] 6318038 pages
shared
 
 
 

Jan 18 21:11:46 host kernel: [20304486.781063] 15270861 pages
non-shared
 
 
 

after that: reset happend.
                                                                                                                                          
 
 
 

I could not get any guest process causing that issue. In vtop no
abnormal cpu heavy process could be found.

Best regards.

On 01/18/10 22:15, Nikolay Kichukov wrote:
> Hello cryptonic.
>
> Sorry to hear that. Are you really having 74Gigs of RAM on that
> machine? How many virtual machines are there? What file system do
> you use for the guest partitions? Do you have a certain
> guest/process causing this load as it seems CPU on the host is not
> so busy.
>
> Do you have remote syslog enabled? Any clues in the host
> /var/log/messages or /var/log/syslog?
>
> Cheers, -Nik
>
> Cryptronic wrote: Hi list,
>
> we had problems on different kernel versions since vserver patch
> 2.3.
>
> We had the problem, that load runs in a view seconds into great
> dimensions:
>
> top - 21:11:53 up 23 days, 5:55, 2 users, load average:
> 1092.35, 530.76, 212.88 Tasks: 5614 total, 3 running, 5531
> sleeping, 0 stopped, 0 zombie Cpu(s): 3.6%us, 8.2%sy,
> 0.0%ni, 83.2%id, 4.6%wa, 0.0%hi, 0.3%si, 0.0%st Mem: 74376860k
> total, 73832132k used, 544728k free, 4835704k buffers Swap:
> 140624968k total, 4019584k used, 136605384k free, 24147056k
> cached
>
> As you can see "wa" is not very high.
>
> At a load of 500 Apache and Co stop working and delivering
> webpages.
>
> Load before it explodes is around 3 - 5.
>
> On shell in host system you can work as at load levels like 0.50,
> but vserver operations aren't possible: vserver XXX stop does not
> take any affect.
>
> dmesg is also empty.
>
> We're using the following versions:
>
> # vserver-info Versions: Kernel: 2.6.31.7-vs2.3.0.36.27 VS-API:
> 0x00020305 util-vserver: 0.30.216-pre2864; Nov 29 2009, 00:39:14
>
> Features: CC: gcc, gcc (Debian 4.3.4-6) 4.3.4 CXX: g++, g++ (Debian
> 4.3.4-6) 4.3.4 CPPFLAGS: '' CFLAGS: '-Wall -g -O2 -std=c99 -Wall
> -pedantic -W -funit-at-a-time' CXXFLAGS: '-g -O2 -ansi -Wall
> -pedantic -W -fmessage-length=0 -funit-at-a-time' build/host:
> x86_64-pc-linux-gnu/x86_64-pc-linux-gnu Use dietlibc: yes Build C++
> programs: yes Build C99 programs: yes Available APIs:
> v13,net,v21,v22,v23,netv2 ext2fs Source: e2fsprogs syscall(2)
> invocation: alternative vserver(2) syscall#: 236/glibc crypto api:
> nss python bindings: no use library versioning: yes
>
> Paths: prefix: /usr sysconf-Directory: /etc cfg-Directory:
> /etc/vservers initrd-Directory: $(sysconfdir)/init.d
> pkgstate-Directory: /var/run/vservers vserver-Rootdir:
> /var/lib/vservers
>
> At Kernel 2.6.29.6-vs2.3.0.36.14 this also happend. The hardware
> is also not the thing, because this happens on different
> architectures.
>
> To get the server back online, you have to reset the hardware,
> reboot and co not doing anything.
>
> Best regards
>>

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.14 (GNU/Linux)

iEYEARECAAYFAktU1CoACgkQOBdlVlcPuhxdwQCgyR4o0ZEYcfW5TUmlyxYrzzYz
25wAoJT6mReugXt1iYV/ud3VFtI/rYoF
=TZyO
-----END PGP SIGNATURE-----
Received on Mon Jan 18 21:36:02 2010

[Next/Previous Months] [Main vserver Project Homepage] [Howto Subscribe/Unsubscribe] [Paul Sladen's vserver stuff]
Generated on Mon 18 Jan 2010 - 21:36:04 GMT by hypermail 2.1.8