Re: [vserver] For a period of 2 months, the server file system has switched three times into the readonly mode

From: Laurent Spagnol <laurent.spagnol_at_univ-reims.fr>
Date: Thu 29 Oct 2009 - 11:17:06 GMT
Message-ID: <4AE979B2.8060209@univ-reims.fr>

Christoph Lukas a écrit :
> Hi Stephane,
>
>
>> I've a vserver installed on Debian Lenny.
>>
>> For a period of 2 months, the server file system has switched three
>> times into the readonly mode.
>>
>
> I have seen something similar on one of our servers. I have found an
> EXT3 fs error in syslog 4 weeks ago. As it happened on the weekend I did
> not notice it until monday. The 'journal abort' and switch to read only
> mode happened just a few hours later.
>
> The filesystem check moved thousands of files to lost+found and I had to
> restore the whole data from our backup.
>
>
>> What do you think ?
>> * is it a vserver issue ?
>> * is it a hardware issue ?
>>
Hi,

I think it is a hardware issue -> fs errors are caused by I/O errors on
the block device.
This problem happens when the harddrive can't realocate a bad block that
kernel is trying to use.
Linux-Vserver patch do nothing at this level.

The "standard" comportement of kernel is to remount R/O the filesystem
when I/O errors happens on a "ext" filesystems.
(XFS does not support this feature, and i don't know for others).

You can have some problems, even with software RAID1: kernel may remount
R/O the filesystem before the physical device be set to faulty.

It may be changed in two ways: add option "errors= ...." to fstab, or
use tune2fs to write option directly on the filesystem.

Possibles options are:
errors=continue (very bad idea ...)
errors=remount-ro (default)
errors=panic (usefull for node fencing in HA/LB environment)

errors=panic will cause a "kernel-panic"
If you add some kernel parameters (kernel.panic_on_oops=1 and
kernel.panic=xxx), the kernel-panic will cause a reboot after "xxx" seconds.
-> sysctl -w parameter=value
-> echo "parameter = value" >> /etc/sysctl.conf (Debian ?)

 ... and the best thing to do is to change the hard drive !

 

>> * is it a debian issue ?
>>
>
> I personally think this is an issue with the 2.6.26 debian kernel. If it
> is a mainline issue or just a debian one I do not know.
>
> On our server it happened a few hours after the monthly md data
> integrity check.
>
> [..]
>
>
>> Do you need other information ?
>>
>
> As you also got this error on an md device, could you check if it just
> happened a few hours after the md data integrity check? Then this
> problem might be md related.
>
> I have switched to using the kernel we have build for our ubuntu jaunty
> systems [1]. It is running without problems for the last 3 weeks.
>
> Cheers,
> Christoph
>
> [1]: http://www.linux-vserver.org/Installation_on_Ubuntu
>
>
>
>
>

-- 
Laurent Spagnol
Administrateur Linux
Université de Reims
Centre de Ressources Informatiques
Campus du Moulin de la Housse
BP 1039 - 51687 Reims cedex 2
Tel: 03.26.91.88.32
Fax: 03.26.91.31.87
Received on Thu Oct 29 11:17:10 2009
[Next/Previous Months] [Main vserver Project Homepage] [Howto Subscribe/Unsubscribe] [Paul Sladen's vserver stuff]
Generated on Thu 29 Oct 2009 - 11:17:15 GMT by hypermail 2.1.8