On 10/18/2010 10:42 PM, John A. Sullivan III wrote:
> On Mon, 2010-10-18 at 21:55 +0100, Gordan Bobic wrote:
>> On 10/18/2010 01:53 AM, Herbert Poetzl wrote:
>>
>>> putting superstition aside, can you recreate the issue?
>>> i.e. is there a script or procedure which reliably
>>> produces the 'corruption'?
>>
>> I believe I have just successfully re-created this, and it seems to be
>> specifically related to hashify.
>>
>> I have two VMs, call them vm1 and vm2. Their contents are fine. Run
>> hashify on them and reboot. All the binaries will be broken in the same way.
>>
>> The content of the files seems to be downright random. e.g. in the
>> current instance, bash binary is zero-filled for the first few
>> screenfuls I bothered to look through, and /sbin/init contains what
>> looks like it might be a fragment of a RPM file. It is almost as if
>> hashifying points the blocks somewhere random, most likely unused blocks
>> since I haven't observed any file system level corruption or corruption
>> of files in the host.
>>
>> I'll re-try it again in a bit, but so far it's been consistent.
>>
>
> Strange - we've been using 2.6.28.7-vs2.3.0.36.7 and
> util-vserver-0.30.216-1.pre2793.el5.centos with ext4 for almost two
> years now - in fact, we really need to upgrade. We've rebooted the
> hosts many times and have not seen this problem once - John
Yeah, really weird. It's reproducible, too. Files are fine before being
hashified. It could be a weird bug that only exhibits itself when
running with ^has_journal and data=writeback. I know I reported a weird
bug in the ext4 in 2.6.32.x that RHEL6b2 ships with that made it never
issue TRIM commands when running in data=writeback mode (data=writeback
is implicit with ^has_journal). So it is plausbible there are other edge
cases like this triggered by running ext4 without the journal.
I'll try with the latest available kernel and see if that makes the
problem go away.
Gordan
Received on Mon Oct 18 22:54:48 2010