Re: [vserver] Very serious hashify mysql/innodb conflict causes major data loss

From: John A. Sullivan III <jsullivan_at_opensourcedevel.com>
Date: Tue 09 Jun 2009 - 13:56:25 BST
Message-Id: <1244552185.6384.45.camel@jaspav.missionsit.net.missionsit.net>

On Tue, 2009-06-09 at 14:38 +0200, Daniel Hokka Zakrisson wrote:
> John Alberts wrote:
> > I just wanted to mention I had the same issue with mysql/innodb and
> > hashify a couple of months ago. As Herbert suggested, just don't
> > hashify the mysql db directory. I stopped using vhashify altogether
> > and this fixed my problems; however, the default when you run vhashify
> > is that it just does everything. Unless you know to avoid running it
> > on the mysql directory, your going to have problems.
>
> No, the default exclude list excludes all of /var. So unless you're storing
> the databases elsewhere, or use a custom exclude list where /var is not
> present, they will not get hashified.
>
> Daniel
Thanks very, very much to all who replied to this thread. Indeed,
Zimbra places the database directories by default under /opt/zimbra. I
will take a look at how we exclude those directories. Thanks again -
John
>
> > Regards,
> > John
> >
> >
> > On Tue, Jun 9, 2009 at 7:08 AM, Herbert Poetzl<herbert@13thfloor.at> wrote:
> >> On Tue, Jun 09, 2009 at 07:41:28AM -0400, John A. Sullivan III wrote:
> >>> Hello, all. We have been struggling for several weeks with an issue of
> >>> severe data loss on our zimbra email vserver. We have narrowed it down
> >>> with considerable certainty to a conflict between mysql/innodb and
> >>> hashify. We are running 2.6.28.7-vs2.3.0.36.7, i.e., a custom patched
> >>> 2.6.28.7 kernel and util-vserver-0.30.216-1.pre2793.el5.centos. We can
> >>> reproduce the issue at will.
> >>
> >> updating to a recent version is advised ...
> >>
> >>> Zimbra is running its own instance of mysql with innodb support version
> >>> 5.0.67. Zimbra runs a zimbra database and then many other mysql
> >>> instances named mboxgroupX where X is a one or two digit number.
> >>
> >>> Whenever we run vserver <zimbra server name> hashify, we see odd
> >>> behaviors:
> >>>
> >>> * The mysql and zimbra database directories are time stamped with
> >>> the time hashify ran as if mysql was restarted but there is no
> >>> record of mysql being restarted.
> >>> * MOST IMPORTANTLY, the innodb log stops recording but it appears
> >>> that mysql is still functioning. Mail is displayed, events
> >>> (e.g., calendar, tasks) are recorded and are fully functional
> >>
> >> there is no pint in hashifying mysql databases,
> >> so I would suggest to exclude those dirs ...
> >>
> >>> The disaster occurs when mysql is restarted, e.g., when restarting the
> >>> zimbra service. Since the innodb log has recorded no data since hashify
> >>> ran, mysql thinks the system has crashed and backs out all data since
> >>> the last hashify! At least, I believe that's what is happening in my
> >>> MySQL ignorance.
> >>
> >> IMHO hashify should not change the time stamps of
> >> files, but it's okay to update directory timestamps
> >> if files have been unified there ...
> >>
> >>> Obviously, this kind of data loss is intolerable. I'd rather solve the
> >>> problem than simply forgo hashify on our zimbra servers. Any ideas
> >>> about what could cause this behavior and how to troubleshoot it?
> >>
> >> well, my guess would be that mysql either uses
> >> the timestamps which get changed when an otherwise
> >> harmless file is unified (in that dir) ... the
> >> proper solution IMHO is to exclude that dir from
> >> hashification ...
> >>
> >> HTH,
> >> Herbert
> >>
> >>> Thanks
> >>> - John
> >>> --
> >>> John A. Sullivan III
> >>> Open Source Development Corporation
> >>> +1 207-985-7880
> >>> jsullivan@opensourcedevel.com
> >>>
> >>> http://www.spiritualoutreach.com
> >>> Making Christianity intelligible to secular society
> >>
> >
> >
> >
> > --
> > John Alberts
> >
>
>

-- 
John A. Sullivan III
Open Source Development Corporation
+1 207-985-7880
jsullivan@opensourcedevel.com
http://www.spiritualoutreach.com
Making Christianity intelligible to secular society
Received on Tue Jun 9 13:55:09 2009
[Next/Previous Months] [Main vserver Project Homepage] [Howto Subscribe/Unsubscribe] [Paul Sladen's vserver stuff]
Generated on Tue 09 Jun 2009 - 13:55:12 BST by hypermail 2.1.8