homepage Welcome to WebmasterWorld Guest from 23.22.128.96
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Hardware and OS Related Technologies / Linux, Unix, and *nix like Operating Systems
Forum Library, Charter, Moderators: bakedjake

Linux, Unix, and *nix like Operating Systems Forum

    
Whole filesystem just went Read-Only
Had to power cycle to fix it......
trillianjedi




msg:911704
 2:21 pm on Jul 6, 2006 (gmt 0)

Anyone had this before?

Not pleasant - couldn't even do a reboot in SSH so had to power cycle. I'm not sure what caused it. The disk system is a RAID array - any way that these things get messed up?

TJ

 

zCat




msg:911705
 2:34 pm on Jul 6, 2006 (gmt 0)

New to me. You might want to post what kind of filesystem(s) you use (ext3, reiser, xfs etc.). Does /var/log/messages contain anything interesting?

trillianjedi




msg:911706
 2:52 pm on Jul 6, 2006 (gmt 0)

Hi zCat,

It's EXT3.

Here's the extract from messages where it rebooted following my power cycle:-


Jul 6 14:47:02 MyMachine kernel: EXT3-fs: INFO: recovery required on readonly filesystem.
Jul 6 14:47:02 MyMachine kernel: EXT3-fs: write access will be enabled during recovery.
Jul 6 14:47:02 MyMachine kernel: kjournald starting. Commit interval 5 seconds
Jul 6 14:47:02 MyMachine kernel: EXT3-fs: sda1: orphan cleanup on readonly fs
Jul 6 14:47:02 MyMachine kernel: EXT3-fs: sda1: 8 orphan inodes deleted
Jul 6 14:47:02 MyMachine kernel: EXT3-fs: recovery complete.
Jul 6 14:47:02 MyMachine kernel: EXT3-fs: mounted filesystem with ordered data mode.

Other than that the usual port scanning and SSH hacking attempts.

I'll move SSH from port 22 and also to a new IP...

TJ

zCat




msg:911707
 8:22 pm on Jul 6, 2006 (gmt 0)

Oh yes, and what kind of RAID do you have (software / hardware, RAID level?)

Not that I'll probably be able to say anything of much use, my sysadmin skills are very amateur (i.e. learnt on the job when something breaks...).

If you want my opinion though I'd say you're having hardware rather than hacking problems, maybe something along these lines:
[webmasterworld.com...]

Try inputting "dmesg", that might get you a swathe of angst-causing messages.

freeflight2




msg:911708
 8:39 pm on Jul 6, 2006 (gmt 0)

ext3 switches to read-only automatically if it encounters too many write errors

trillianjedi




msg:911709
 8:57 pm on Jul 6, 2006 (gmt 0)

OK thanks guys. So looks like I'm looking at a pair of duff HD's?

I always thought that RAID was meant to prevent this kind of thing? Or have I been desperately unluckly and just got a pair of defective drives.

The box is about 4 years old....

zCat




msg:911710
 9:27 pm on Jul 6, 2006 (gmt 0)

I'd guess (and I am guessing, so I might be wrong) that one of the drives is going bad; if you take this out of the RAID configuration, the other drive should work in read/write mode again. How to tell which drive is playing up is another question, dmesg might help, or /var/log/warn, or maybe your raid setup has a log file.

RAID isn't a magic bullet unfortunately, particulary not software raid, which has caused me a fair bit of pain in the past.

trillianjedi




msg:911711
 10:04 am on Jul 7, 2006 (gmt 0)

dmesg might help

Don't seem to have that.

/var/log/warn

Nothing in /var/log for the RAID disks.

or maybe your raid setup has a log file.

Can't find one....

It's hardware RAID, which I can't access unless I'm in front of the machine, so might need to take a trip down to the datacenter (it's a co-lo box).

Thanks for all the help anyway, I'll keep digging.

TJ

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Hardware and OS Related Technologies / Linux, Unix, and *nix like Operating Systems
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved