Monumental work cockups

[tobinen]

During a round of redundancies a few years ago, a disgruntled employee decided it would be a good idea to delete the server base directory of a very large customer who/m we hosted. Carnage ensued. Back then they didn't march off the premises staff made redundant immediately. They do now. Not really a cock-up as a deliberate action, but it was always a possibility. Now hosted customers do not have the root password made available to all staff, but a chosen few.

[Tonsko]

tobinen said:

During a round of redundancies a few years ago, a disgruntled employee decided it would be a good idea to delete the server base directory of a very large customer who/m we hosted. Carnage ensued. Back then they didn't march off the premises staff made redundant immediately. They do now. Not really a cock-up as a deliberate action, but it was always a possibility. Now hosted customers do not have the root password made available to all staff, but a chosen few.

Christ. Have you investigated Thycotic Secret Server?

[hidetheelephants]

matchmaker said:

MRI Stonehaven?

¿Qué?

[eltax91]

Tonsko said:

tobinen said:

During a round of redundancies a few years ago, a disgruntled employee decided it would be a good idea to delete the server base directory of a very large customer who/m we hosted. Carnage ensued. Back then they didn't march off the premises staff made redundant immediately. They do now. Not really a cock-up as a deliberate action, but it was always a possibility. Now hosted customers do not have the root password made available to all staff, but a chosen few.

Christ. Have you investigated Thycotic Secret Server?

I know the clowns that run that show. Don't do it!

[Tonsko]

eltax91 said:

I know the clowns that run that show. Don't do it!

Thycotic?

[ChemicalChaos]

Tonsko said:

eltax91 said:

I know the clowns that run that show. Don't do it!

Thycotic?

[eltax91]

Tonsko said:

eltax91 said:

I know the clowns that run that show. Don't do it!

Thycotic?

Yes

[robm3]

I set up a warehouse for Ingram Micro back in the 90's and put all the pallet racking (about 3,000 or 8 runs) in the wrong direction. Another project manager came in a few months later and moved it into the correct positions where it worked much better.
I was very inexperienced but could bullsh$t so well nobody questioned it at the time.

The biggest cock up/white elephant I saw was at the 3COM factory in Dublin. Someone set up a massive automated pallet convey system to cope with 1,000 pallets per day throughput but actually got the volumes mixed up. Throughput was circa 100 and so the system never ever was needed (forklifts moved it all).

[Tonsko]

eltax91 said:

Yes

We buy their software, pm me if want

[AB]

telecat said:

lufbramatt said:

Eric Mc said:

eltax91 said:

Mine comes from when I was doing my placement year at a large gypsum products manufacturer.

I was testing the boss' PA's out of office as it wasn't working, and of course she dare not leave that Friday without the world knowing she was off on the Monday

Anyways, couldn't get it to work on the PC so went onto the server admin console. Being young and inexperienced, I didn't notice the check box in exchange that says 'send only one reply per recipient' and being even more dumb I decided the best way to test this first was to copy the rules to my profile too.

Then I sent her an email.... from my email address. Cue the email server sending reply after reply to our two mailboxes. This is around 2003/4, so everything ran on real tin and nobody had failover and the server fell over.

1200 people without email, in a business where the vast majority of ordering was over email. The call centre went into meltdown with building firms shouting the odds as they had not received or confirmations and for the first time in its history the call centre had to work Saturday morning calling all the builders that couldn't get through to take their orders the old fashioned way.

No my finest hour.

Sounds bad but I still haven't a clue what you are describing.

Out of office set up on both accounts, which then sent each other a reply each time they received an out of office email, this then made the email server crash and the company went into meltdown. Hth.

I came across a term for it. It's called a "Fatal Embrace".

I did a similar thing with a potential customer, we'd sent a quote out from our info@ email address which we rarely use/monitor and the guy we'd sent it to had gone on holiday for 2 weeks. His out of office came back to our info@ address and our info@ address sent the "Thanks for your email..." auto response which was then met with his out of office and vice-versa.

We didn't know what was happening until he got back from 2 weeks away to more than 20,000 emails and we had the same. Didn't cause such problems as above though.

[matchmaker]

hidetheelephants said:

matchmaker said:

MRI Stonehaven?

¿Qué?

Offshore safety training centre (now closed). Used to be known as RGIT and when I worked at Aberdeen Coastguard MRCC in the 1980's was a Declared SAR Facility.

[droopsnoot]

telecat said:

lufbramatt said:

Eric Mc said:

eltax91 said:

Mine comes from when I was doing my placement year at a large gypsum products manufacturer.

I was testing the boss' PA's out of office as it wasn't working, and of course she dare not leave that Friday without the world knowing she was off on the Monday

Anyways, couldn't get it to work on the PC so went onto the server admin console. Being young and inexperienced, I didn't notice the check box in exchange that says 'send only one reply per recipient' and being even more dumb I decided the best way to test this first was to copy the rules to my profile too.

Then I sent her an email.... from my email address. Cue the email server sending reply after reply to our two mailboxes. This is around 2003/4, so everything ran on real tin and nobody had failover and the server fell over.

1200 people without email, in a business where the vast majority of ordering was over email. The call centre went into meltdown with building firms shouting the odds as they had not received or confirmations and for the first time in its history the call centre had to work Saturday morning calling all the builders that couldn't get through to take their orders the old fashioned way.

No my finest hour.

Sounds bad but I still haven't a clue what you are describing.

Out of office set up on both accounts, which then sent each other a reply each time they received an out of office email, this then made the email server crash and the company went into meltdown. Hth.

I came across a term for it. It's called a "Fatal Embrace".

I thought it was a "Race condition". To me, a "deadly embrace" is where one process won't continue because it's waiting for a condition (a record unlock for example) that the locking process won't unlock until something in the first process occurs.

[Vaud]

droopsnoot said:

I thought it was a "Race condition".

No, a race condition is different -"A race condition is an undesirable situation that occurs when a device or system attempts to perform two or more operations at the same time, but because of the nature of the device or system, the operations must be done in the proper sequence in order to be done correctly."

I think the situation of emails responding to each other is "a rubbish configuration" or "poor code"

[21TonyK]

This one...

http://www.bbc.co.uk/news/uk-england-devon-2901208...

That was the situation 2 years ago.

Today the bulldozers started demolishing and clearing the site ready for a whole new school to be built.

£10m+

[eltax91]

Tonsko said:

eltax91 said:

Yes

We buy their software, pm me if want

YHM

[miniman]

21TonyK said:

This one...

http://www.bbc.co.uk/news/uk-england-devon-2901208...

That was the situation 2 years ago.

Today the bulldozers started demolishing and clearing the site ready for a whole new school to be built.

£10m+

Hopefully this time they will focus more on building a school and less on knitting an environmental paradise from yoghurt and despair.

[hidetheelephants]

matchmaker said:

hidetheelephants said:

matchmaker said:

MRI Stonehaven?

¿Qué?

Offshore safety training centre (now closed). Used to be known as RGIT and when I worked at Aberdeen Coastguard MRCC in the 1980's was a Declared SAR Facility.

Ah ha; before my time. I should have clarified that the incident took place within the last couple of years and that the boats were launched from a ship, which is implicit in the class society being present but probably only the 'in-crowd' would know this.

[OzzyR1]

"Three men founded Apple Computer on April 1, 1976

And while we're all familiar with Steve Jobs and Steve Wozniak, you probably have never heard of Ronald Wayne, Apple's third-co-founder, who bailed out early, and gave up his ten percent stake in Apple for $800.

Had he kept it, that stake would be worth $35 billion today

But he insists he has absolutely no regrets."


Of course he hasn't, none at all...

http://www.cnbc.com/id/44505957

https://en.wikipedia.org/wiki/Ronald_Wayne

[pingu393]

OzzyR1 said:

"Three men founded Apple Computer on April 1, 1976

And while we're all familiar with Steve Jobs and Steve Wozniak, you probably have never heard of Ronald Wayne, Apple's third-co-founder, who bailed out early, and gave up his ten percent stake in Apple for $800.

Had he kept it, that stake would be worth $35 billion today

But he insists he has absolutely no regrets."


Of course he hasn't, none at all...

http://www.cnbc.com/id/44505957

https://en.wikipedia.org/wiki/Ronald_Wayne

I just hope that Ron is on PH so he can post this himself. If he does, he is the sure-fire thread winner and we can all go back to costing ourselves mere millions .

[karona]

21TonyK said:

This one...

http://www.bbc.co.uk/news/uk-england-devon-2901208...

That was the situation 2 years ago.

Today the bulldozers started demolishing and clearing the site ready for a whole new school to be built.

£10m+

How about a 3 Billion pound cockup
http://www.bbc.com/news/uk-scotland-edinburgh-east...