Whole filesystem just went Read-Only

Had to power cycle to fix it......

[trillianjedi]

Anyone had this before?

Not pleasant - couldn't even do a reboot in SSH so had to power cycle. I'm not sure what caused it. The disk system is a RAID array - any way that these things get messed up?

TJ

[zCat]

New to me. You might want to post what kind of filesystem(s) you use (ext3, reiser, xfs etc.). Does /var/log/messages contain anything interesting?

[trillianjedi]

Hi zCat,

It's EXT3.

Here's the extract from messages where it rebooted following my power cycle:-

Jul 6 14:47:02 MyMachine kernel: EXT3-fs: INFO: recovery required on readonly filesystem.
Jul 6 14:47:02 MyMachine kernel: EXT3-fs: write access will be enabled during recovery.
Jul 6 14:47:02 MyMachine kernel: kjournald starting. Commit interval 5 seconds
Jul 6 14:47:02 MyMachine kernel: EXT3-fs: sda1: orphan cleanup on readonly fs
Jul 6 14:47:02 MyMachine kernel: EXT3-fs: sda1: 8 orphan inodes deleted
Jul 6 14:47:02 MyMachine kernel: EXT3-fs: recovery complete.
Jul 6 14:47:02 MyMachine kernel: EXT3-fs: mounted filesystem with ordered data mode.

Other than that the usual port scanning and SSH hacking attempts.

I'll move SSH from port 22 and also to a new IP...

TJ

[zCat]

Oh yes, and what kind of RAID do you have (software / hardware, RAID level?)

Not that I'll probably be able to say anything of much use, my sysadmin skills are very amateur (i.e. learnt on the job when something breaks...).

If you want my opinion though I'd say you're having hardware rather than hacking problems, maybe something along these lines:
[webmasterworld.com...]

Try inputting "dmesg", that might get you a swathe of angst-causing messages.

[freeflight2]

ext3 switches to read-only automatically if it encounters too many write errors

[trillianjedi]

OK thanks guys. So looks like I'm looking at a pair of duff HD's?

I always thought that RAID was meant to prevent this kind of thing? Or have I been desperately unluckly and just got a pair of defective drives.

The box is about 4 years old....

[zCat]

I'd guess (and I am guessing, so I might be wrong) that one of the drives is going bad; if you take this out of the RAID configuration, the other drive should work in read/write mode again. How to tell which drive is playing up is another question, dmesg might help, or /var/log/warn, or maybe your raid setup has a log file.

RAID isn't a magic bullet unfortunately, particulary not software raid, which has caused me a fair bit of pain in the past.

[trillianjedi]

dmesg might help

Don't seem to have that.

/var/log/warn

Nothing in /var/log for the RAID disks.

or maybe your raid setup has a log file.

Can't find one....

It's hardware RAID, which I can't access unless I'm in front of the machine, so might need to take a trip down to the datacenter (it's a co-lo box).

Thanks for all the help anyway, I'll keep digging.

TJ