Re: [vserver] Very serious hashify mysql/innodb conflict causes major data loss

From: John Alberts <john.m.alberts_at_gmail.com>
Date: Tue 09 Jun 2009 - 13:28:51 BST
Message-ID: <a23b6f900906090528j5cf38c33kc87039583a5f43c1@mail.gmail.com>

I just wanted to mention I had the same issue with mysql/innodb and
hashify a couple of months ago. As Herbert suggested, just don't
hashify the mysql db directory. I stopped using vhashify altogether
and this fixed my problems; however, the default when you run vhashify
is that it just does everything. Unless you know to avoid running it
on the mysql directory, your going to have problems.

Regards,
John

On Tue, Jun 9, 2009 at 7:08 AM, Herbert Poetzl<herbert@13thfloor.at> wrote:
> On Tue, Jun 09, 2009 at 07:41:28AM -0400, John A. Sullivan III wrote:
>> Hello, all.  We have been struggling for several weeks with an issue of
>> severe data loss on our zimbra email vserver.  We have narrowed it down
>> with considerable certainty to a conflict between mysql/innodb and
>> hashify.  We are running 2.6.28.7-vs2.3.0.36.7, i.e., a custom patched
>> 2.6.28.7 kernel and util-vserver-0.30.216-1.pre2793.el5.centos.  We can
>> reproduce the issue at will.
>
> updating to a recent version is advised ...
>
>> Zimbra is running its own instance of mysql with innodb support version
>> 5.0.67.  Zimbra runs a zimbra database and then many other mysql
>> instances named mboxgroupX where X is a one or two digit number.
>
>> Whenever we run vserver <zimbra server name> hashify, we see odd
>> behaviors:
>>
>>       * The mysql and zimbra database directories are time stamped with
>>         the time hashify ran as if mysql was restarted but there is no
>>         record of mysql being restarted.
>>       * MOST IMPORTANTLY, the innodb log stops recording but it appears
>>         that mysql is still functioning.  Mail is displayed, events
>>         (e.g., calendar, tasks) are recorded and are fully functional
>
> there is no pint in hashifying mysql databases,
> so I would suggest to exclude those dirs ...
>
>> The disaster occurs when mysql is restarted, e.g., when restarting the
>> zimbra service.  Since the innodb log has recorded no data since hashify
>> ran, mysql thinks the system has crashed and backs out all data since
>> the last hashify! At least, I believe that's what is happening in my
>> MySQL ignorance.
>
> IMHO hashify should not change the time stamps of
> files, but it's okay to update directory timestamps
> if files have been unified there ...
>
>> Obviously, this kind of data loss is intolerable.  I'd rather solve the
>> problem than simply forgo hashify on our zimbra servers.  Any ideas
>> about what could cause this behavior and how to troubleshoot it?
>
> well, my guess would be that mysql either uses
> the timestamps which get changed when an otherwise
> harmless file is unified (in that dir) ... the
> proper solution IMHO is to exclude that dir from
> hashification ...
>
> HTH,
> Herbert
>
>> Thanks
>> - John
>> --
>> John A. Sullivan III
>> Open Source Development Corporation
>> +1 207-985-7880
>> jsullivan@opensourcedevel.com
>>
>> http://www.spiritualoutreach.com
>> Making Christianity intelligible to secular society
>

-- 
John Alberts
Received on Tue Jun 9 13:29:04 2009
[Next/Previous Months] [Main vserver Project Homepage] [Howto Subscribe/Unsubscribe] [Paul Sladen's vserver stuff]
Generated on Tue 09 Jun 2009 - 13:29:07 BST by hypermail 2.1.8