[vserver] Very serious hashify mysql/innodb conflict causes major data loss

From: John A. Sullivan III <jsullivan_at_opensourcedevel.com>
Date: Tue 09 Jun 2009 - 12:41:28 BST
Message-Id: <1244547688.6384.44.camel@jaspav.missionsit.net.missionsit.net>

Hello, all. We have been struggling for several weeks with an issue of
severe data loss on our zimbra email vserver. We have narrowed it down
with considerable certainty to a conflict between mysql/innodb and
hashify. We are running 2.6.28.7-vs2.3.0.36.7, i.e., a custom patched
2.6.28.7 kernel and util-vserver-0.30.216-1.pre2793.el5.centos. We can
reproduce the issue at will.

Zimbra is running its own instance of mysql with innodb support version
5.0.67. Zimbra runs a zimbra database and then many other mysql
instances named mboxgroupX where X is a one or two digit number.

Whenever we run vserver <zimbra server name> hashify, we see odd
behaviors:

      * The mysql and zimbra database directories are time stamped with
        the time hashify ran as if mysql was restarted but there is no
        record of mysql being restarted.
      * MOST IMPORTANTLY, the innodb log stops recording but it appears
        that mysql is still functioning. Mail is displayed, events
        (e.g., calendar, tasks) are recorded and are fully functional

The disaster occurs when mysql is restarted, e.g., when restarting the
zimbra service. Since the innodb log has recorded no data since hashify
ran, mysql thinks the system has crashed and backs out all data since
the last hashify! At least, I believe that's what is happening in my
MySQL ignorance.

Obviously, this kind of data loss is intolerable. I'd rather solve the
problem than simply forgo hashify on our zimbra servers. Any ideas
about what could cause this behavior and how to troubleshoot it? Thanks
- John

-- 
John A. Sullivan III
Open Source Development Corporation
+1 207-985-7880
jsullivan@opensourcedevel.com
http://www.spiritualoutreach.com
Making Christianity intelligible to secular society
Received on Tue Jun 9 12:40:07 2009
[Next/Previous Months] [Main vserver Project Homepage] [Howto Subscribe/Unsubscribe] [Paul Sladen's vserver stuff]
Generated on Tue 09 Jun 2009 - 12:40:09 BST by hypermail 2.1.8