The joys of corruption

Author
Discussion

Silent1

Original Poster:

19,761 posts

236 months

Friday 4th April 2008
quotequote all
Sadly i've not received a cheque for my part in the zimbabe vote rigging.

What i have received though is a simultaneous collapse of 1TB of drives in the most fantastic self-corrupting fashion.

The next trick is going to be using VMWare to boot it off another drive and see what i can recover. banghead

Just what i wanted on my last day before work frown

JamieBeeston

9,294 posts

266 months

Friday 4th April 2008
quotequote all
Two words

Use Raid

frown

Silent1

Original Poster:

19,761 posts

236 months

Friday 4th April 2008
quotequote all
JamieBeeston said:
Two words

Use Raid

frown
Fortunately i've got an offsite backup, but it'll be a PITA to get that before i'm at work tomorrow night smile

cyberface

12,214 posts

258 months

Friday 4th April 2008
quotequote all
JamieBeeston said:
Two words

Use Raid

frown
I find that chanting random phrases from Aleister Crowley's 'The Book Of Lies' whilst pogo-ing around naked covered in blue paint and gritting sticks of RAM between my teeth wards off the evil spirits of hard drive failure.

That, and keeping them cool.

Never trusted Crowley enough to go without a backup though, dodgy geezer he was, yes sirrreeee.

(I don't normally correct Jamie B because he knows more than me about hardware but if you're going to use Raid, make sure it's not the ant killing stuff first, and then make sure it's not Cyberface and LukeBird style 'suicide' RAID-0. But Silent1 old bean you knew all this anyway, except perhaps the bit about the hard drive black magic. Hmmmm.)

agent006

12,043 posts

265 months

Friday 4th April 2008
quotequote all
JamieBeeston said:
Two words

Use Raid

frown
Which is fine until the controller dies and wrecks both disks.

Silent1

Original Poster:

19,761 posts

236 months

Friday 4th April 2008
quotequote all
cyberface said:
JamieBeeston said:
Two words

Use Raid

frown
I find that chanting random phrases from Aleister Crowley's 'The Book Of Lies' whilst pogo-ing around naked covered in blue paint and gritting sticks of RAM between my teeth wards off the evil spirits of hard drive failure.

That, and keeping them cool.

Never trusted Crowley enough to go without a backup though, dodgy geezer he was, yes sirrreeee.

(I don't normally correct Jamie B because he knows more than me about hardware but if you're going to use Raid, make sure it's not the ant killing stuff first, and then make sure it's not Cyberface and LukeBird style 'suicide' RAID-0. But Silent1 old bean you knew all this anyway, except perhaps the bit about the hard drive black magic. Hmmmm.)
They're on Raid 0 here already hehe

JamieBeeston

9,294 posts

266 months

Friday 4th April 2008
quotequote all
agent006 said:
JamieBeeston said:
Two words

Use Raid

frown
Which is fine until the controller dies and wrecks both disks.
Never in all of my years with thousands of servers seen a controller trash hdds ever.

Obviosuly RAID should be your first, not last line of defence anyway.. but what would you advocate over RAID?

Globulator

13,841 posts

232 months

Friday 4th April 2008
quotequote all
Silent1 said:
Sadly i've not received a cheque for my part in the zimbabe vote rigging.

What i have received though is a simultaneous collapse of 1TB of drives in the most fantastic self-corrupting fashion.

The next trick is going to be using VMWare to boot it off another drive and see what i can recover. banghead

Just what i wanted on my last day before work frown
Before work?

Silent1

Original Poster:

19,761 posts

236 months

Saturday 5th April 2008
quotequote all
Globulator said:
Silent1 said:
Sadly i've not received a cheque for my part in the zimbabe vote rigging.

What i have received though is a simultaneous collapse of 1TB of drives in the most fantastic self-corrupting fashion.

The next trick is going to be using VMWare to boot it off another drive and see what i can recover. banghead

Just what i wanted on my last day before work frown
Before work?
Yeah same old, nights not doing a lot frown

agent006

12,043 posts

265 months

Saturday 5th April 2008
quotequote all
JamieBeeston said:
Never in all of my years with thousands of servers seen a controller trash hdds ever.

Obviosuly RAID should be your first, not last line of defence anyway.. but what would you advocate over RAID?
I've seen it happen twice. Faulty RAID card happily hoses both (or all) disks on it.

I'm not advocating anything over raid, just agreeing with your first line of defence theory.

Silent1

Original Poster:

19,761 posts

236 months

Saturday 5th April 2008
quotequote all
Recovered 98% of the data woohoo

Ash 996 GT2

3,836 posts

242 months

Sunday 6th April 2008
quotequote all
RAID 0 acheives nothing apart from performance!

51mes

1,500 posts

201 months

Sunday 6th April 2008
quotequote all
agent006 said:
JamieBeeston said:
Never in all of my years with thousands of servers seen a controller trash hdds ever.

Obviosuly RAID should be your first, not last line of defence anyway.. but what would you advocate over RAID?
I've seen it happen twice. Faulty RAID card happily hoses both (or all) disks on it.

I'm not advocating anything over raid, just agreeing with your first line of defence theory.
Thats nothing I've seen a dodgy PSU take out the motors of a whole tray of disks in an array, oh and both controllers... although this went as far as the fuses on the board...

the perfect defence mechanism is RAID on the server and then one of the new CDP (continuous) data protection engines that effectively journal all writes made by a server and allow you to recover to any point in time... Legato replistor is one we've played with in the labs actually transparently sniffing writes off the SAN, but it has a host based splitter as well..

You replicate to a remote location - and this then becomes your backup (so raid protect it) ...

Silent1

Original Poster:

19,761 posts

236 months

Sunday 6th April 2008
quotequote all
Ash 996 GT2 said:
RAID 0 acheives nothing apart from performance!
Exactly

Ash 996 GT2

3,836 posts

242 months

Sunday 6th April 2008
quotequote all
Silent1 said:
Ash 996 GT2 said:
RAID 0 acheives nothing apart from performance!
Exactly
Ah, now I understand smile

LukeBird

17,170 posts

210 months

Sunday 6th April 2008
quotequote all
cyberface said:
(I don't normally correct Jamie B because he knows more than me about hardware but if you're going to use Raid, make sure it's not the ant killing stuff first, and then make sure it's not Cyberface and LukeBird style 'suicide' RAID-0. But Silent1 old bean you knew all this anyway, except perhaps the bit about the hard drive black magic. Hmmmm.)
Aaah can't beat suicide RAID hehe
Ash 996 GT2 said:
RAID 0 acheives nothing apart from performance!
Indeed, but if performance is what you want (need wink) then it's great smile
I should add that's on my personal machine and the work machines I have control over (as in, not part of our work network) are RAID 1, which just isn't as 'nice'! hehe

neil_bolton

17,113 posts

265 months

Monday 7th April 2008
quotequote all
agent006 said:
JamieBeeston said:
Never in all of my years with thousands of servers seen a controller trash hdds ever.

Obviosuly RAID should be your first, not last line of defence anyway.. but what would you advocate over RAID?
I've seen it happen twice. Faulty RAID card happily hoses both (or all) disks on it.

I'm not advocating anything over raid, just agreeing with your first line of defence theory.
Ditto to both points - I've just suffered the perils of suicide Raid at home, and thus realised there was absolutely no point at all in doing it (apart from wasting hours).

Also, I've seen two hardware Raid controllers die in quick sucession - thus taking out my Unix box with Raid 10 frown

Bah.

p.s. the special Harddrive dance works.

Don

28,377 posts

285 months

Monday 7th April 2008
quotequote all
We just use disk mirroring. yes

And automatic nightly backups.

And "deep-store" off-site backups weekly(ish)...

JamieBeeston

9,294 posts

266 months

Monday 7th April 2008
quotequote all
neil_bolton said:
Ditto to both points - I've just suffered the perils of suicide Raid at home, and thus realised there was absolutely no point at all in doing it (apart from wasting hours).

Also, I've seen two hardware Raid controllers die in quick sucession - thus taking out my Unix box with Raid 10 frown

Bah.

p.s. the special Harddrive dance works.
I can only speak from experience here.

Software raid is worth exactly as much as you've paid for it.

Hardware raid on a cheap / integrated low end mobo again as such.

Pay for a decent card, and you're as safe as houses (as part of a calorie controlled diet!)

RAID Card failure is certainly something I'd stick in the <1% chance of happening on decent kit.. but as always, your stats are only as good as the quality of your kit.

agent006

12,043 posts

265 months

Monday 7th April 2008
quotequote all
JamieBeeston said:
but as always, your stats are only as good as the quality of your kit.
Sorry, forgot to mention that the two failures I've had were both on high end Compaq kit (back in the days when they were good Compaq, not HP).

You don't seem to realise that I'm agreeing with you, although I'm happy to have an argument about it (whetever it is) if that's what you're after.