Biggest F*-up you have gone through
Discussion
So being in maintenance you tend to experience many different things along the line of your work.
Im interested to hear about your biggest monumental feck-ups that you have had during your working life. Being not far from the big 50 I can honestly say that there have not been that many. However back on topic the biggest one has to be a routine maintenance procedure that quickly turned into something not very routine. And well it was quite something to go through.
The team turned up early doors and we were distributed through the site at various locations. the maintenance procedure being to run the site up from generators for a few hours, basically run the generators on load and make sure it all goes well. the site being a major office complex over a few floors and a data centre that controlled operations within the business at various locations through out the business. Nothing to worry about eh ? of course all the previous runs had gone well and it had turned into pretty much a routine activity every few months just to spin everything over and make sure it all worked.
Well this morning it all went as before, four substations powered the site and the substations were in one location. First substation had the highvoltage supply to it failed from the main switchboard. This woke up the generators and they started up. Power at the first substation swapped over from utility power to generator. Power was restored to the substation and it remained on generator as you had to switch it back over to uility manually. Saved it from doing this itself automatically in case the business needs required no supply interruptions.
Anyways substation 2 was failed it changed to generator and then substation 3 was done. It was when we got to the 4th one that things didn't go well at all. I was in the generator control room, and they failed the supply to the 4th substation. Nothing happened at the substation. So this had happened before, circuit breaker was showing open but it was holding off the change over. An engineer following procedure attempted to rack out the circuit breaker to free the interlocking and allow the alternative breaker to close. however there was a mechanical failure in the switchboard.
There was an explosion as the circuit breaker faulted. Right at that moment the generator circuit breaker got a close command and closed up, this paralleled the generators with the mains supply and caused a fault with tripped out the high voltage supply. This also affected the generators and they all faulted. No power at all. As said I was in the generator room, they made a couple of strange noises before they shut down. silence. very terrifying silence indeed. darkness all around and then the radio burst into life. We have a fire. all people to the switchroom.
And there started many hours of trying to recover the situation. We worked to try and find out the extend of the damage and if we could switch power about to try and get some of the services back into action. We had to get the utility power company to restore the HV to site and the generators needed to be reset to get them back into play. this took a few hours and of course the UPS units on site drained their batteries before they shut down and that was us a true dark site.
Quite a situation to go through indeed.
Im interested to hear about your biggest monumental feck-ups that you have had during your working life. Being not far from the big 50 I can honestly say that there have not been that many. However back on topic the biggest one has to be a routine maintenance procedure that quickly turned into something not very routine. And well it was quite something to go through.
The team turned up early doors and we were distributed through the site at various locations. the maintenance procedure being to run the site up from generators for a few hours, basically run the generators on load and make sure it all goes well. the site being a major office complex over a few floors and a data centre that controlled operations within the business at various locations through out the business. Nothing to worry about eh ? of course all the previous runs had gone well and it had turned into pretty much a routine activity every few months just to spin everything over and make sure it all worked.
Well this morning it all went as before, four substations powered the site and the substations were in one location. First substation had the highvoltage supply to it failed from the main switchboard. This woke up the generators and they started up. Power at the first substation swapped over from utility power to generator. Power was restored to the substation and it remained on generator as you had to switch it back over to uility manually. Saved it from doing this itself automatically in case the business needs required no supply interruptions.
Anyways substation 2 was failed it changed to generator and then substation 3 was done. It was when we got to the 4th one that things didn't go well at all. I was in the generator control room, and they failed the supply to the 4th substation. Nothing happened at the substation. So this had happened before, circuit breaker was showing open but it was holding off the change over. An engineer following procedure attempted to rack out the circuit breaker to free the interlocking and allow the alternative breaker to close. however there was a mechanical failure in the switchboard.
There was an explosion as the circuit breaker faulted. Right at that moment the generator circuit breaker got a close command and closed up, this paralleled the generators with the mains supply and caused a fault with tripped out the high voltage supply. This also affected the generators and they all faulted. No power at all. As said I was in the generator room, they made a couple of strange noises before they shut down. silence. very terrifying silence indeed. darkness all around and then the radio burst into life. We have a fire. all people to the switchroom.
And there started many hours of trying to recover the situation. We worked to try and find out the extend of the damage and if we could switch power about to try and get some of the services back into action. We had to get the utility power company to restore the HV to site and the generators needed to be reset to get them back into play. this took a few hours and of course the UPS units on site drained their batteries before they shut down and that was us a true dark site.
Quite a situation to go through indeed.
Decided (early in my IT career) to trust MS when Exchange told me that the master DB of all the company's email accounts needed to be rebuilt and that Exchange could do it automatically in but a moment!
Cue about 3 days of mad panic trying to get E-mails flowing again and restoring from various backups when all Exchange managed to do was corrupt the file completely and fail to offer any solutions.
Not my finest hour that one, taught me never to trust Microsoft though!!
Cue about 3 days of mad panic trying to get E-mails flowing again and restoring from various backups when all Exchange managed to do was corrupt the file completely and fail to offer any solutions.
Not my finest hour that one, taught me never to trust Microsoft though!!
Years ago, working in a lab with someone who was using something with similar properties to Gaviscon - turns solid-ish when in contact with acid (hence why Gaviscon is good for heartburn or whatever it treats).
Chap was done with many litres of this and poured it down the sink, someone later on was washing some (not too nasty) acid away...
This resulted in all of the drainage being blocked and the kicker...
The lab was on the 10th floor
Took a while for the drains people and maintence to clear out!
Chap was done with many litres of this and poured it down the sink, someone later on was washing some (not too nasty) acid away...
This resulted in all of the drainage being blocked and the kicker...
The lab was on the 10th floor
Took a while for the drains people and maintence to clear out!
Was selling Ford (and Iveco) commercials at a main dealer in Cornwall - Cornish Ford if you're interested, now defunct.
As usual, August 1st saw a lot of vehicles going out, so PDI were done some weeks before and the vehicles all shiny and number plated sat in our compound gathering July dust. Customer is due to pick up a new Transit panel van so, I thought I'd be clever and give it a quick wash on the gantry car wash. Apparently, and as I found out, Transits didn't fit. Massive gouge down the passenger door, through the door skin. Door a write off and customer due in 45 minutes.
Luckily, we had a matching blue chassis cab in the compound as yet unsold. A crate of beer for the truck workshop later and delaying the customer with handover forms and coffee and his new panel van turns up outside the showroom with a perfect (if not quite the same shade) set of doors. He never noticed, but I felt truly sick every time I thought about that for a long time afterwards.
Car sales Manager managed to write off a P100 pick up he was towing on a trailer to the Royal Cornwall Show. Brand new pick up, already signwritten a sold to local builder who was looking forward to seeing it on our stand, P100 didn't fit on the twin axle trailer, it overhung the back massively, and the CSM decided that a Granada estate was the ideal towcar. All going well until he got to a down hill and the very overweight trailer decided the Granada was in the way, so tried to overtake first one side (well caught) and then the other (not so well caught) a couple of 360 degree turns, a P100 on it's roof in the ditch, two flat tyres on the trailer and a Granada remarkably unscathed - with no MOT.
As usual, August 1st saw a lot of vehicles going out, so PDI were done some weeks before and the vehicles all shiny and number plated sat in our compound gathering July dust. Customer is due to pick up a new Transit panel van so, I thought I'd be clever and give it a quick wash on the gantry car wash. Apparently, and as I found out, Transits didn't fit. Massive gouge down the passenger door, through the door skin. Door a write off and customer due in 45 minutes.
Luckily, we had a matching blue chassis cab in the compound as yet unsold. A crate of beer for the truck workshop later and delaying the customer with handover forms and coffee and his new panel van turns up outside the showroom with a perfect (if not quite the same shade) set of doors. He never noticed, but I felt truly sick every time I thought about that for a long time afterwards.
Car sales Manager managed to write off a P100 pick up he was towing on a trailer to the Royal Cornwall Show. Brand new pick up, already signwritten a sold to local builder who was looking forward to seeing it on our stand, P100 didn't fit on the twin axle trailer, it overhung the back massively, and the CSM decided that a Granada estate was the ideal towcar. All going well until he got to a down hill and the very overweight trailer decided the Granada was in the way, so tried to overtake first one side (well caught) and then the other (not so well caught) a couple of 360 degree turns, a P100 on it's roof in the ditch, two flat tyres on the trailer and a Granada remarkably unscathed - with no MOT.
1 pretty big one but not very exciting. I Started out doing seasonal work in a Microbiology lab where everything had to be washed and disinfected regularly. I poured some items into the sink to wash, big glug of disinfectant and turn the hot tap on. Was in a rush to go and see the girlfriend so promptly tidied up and left.
When I came in Monday morning there was a lot of mopping activity down in the labs. I had left the tap on over the weekend. Thankfully the security man had done the rounds on the Sunday morning and noticed the water seeping out under the door and got it turned off, but not before I had flooded the basement and destroyed the bottom shelf of archived records kept down there.
When I came in Monday morning there was a lot of mopping activity down in the labs. I had left the tap on over the weekend. Thankfully the security man had done the rounds on the Sunday morning and noticed the water seeping out under the door and got it turned off, but not before I had flooded the basement and destroyed the bottom shelf of archived records kept down there.
Toaster Pilot said:
Countdown said:
Our BACS system failing on payroll day (or technically two days before payroll day)
Cue 1500 CHAPS payments......
I hope you paid less than the £20 most banks charge for them!!!Cue 1500 CHAPS payments......
Local petrol station was having some pumps refurbed. Somehow, I’m not even sure how this is possible, the installers connected one of the petrol pump to the diesel tank and diesel pump to the petrol tank.
I filled up my basically empty petrol with £70 of diesel and it conked out before I got off the forecourt. Thought I’d messed up but double checked the pump I’d used, double checked my recipe and I had definitely put fuel in from the petrol pump and paid for petrol. Quick sniff inside the filler and the distinct smell of diesel greated me.
It seems I was the first person to realise where the issue was occurring. The installer was still on site and he checked and confirmed my suspicions (his face was ghostly white when he realised).
Luckily they had a workshop on site so got mine drained down and fuel system cleaned within an hour. However, it seems a lot of people had filled half tanks and were only just finding out the issue. By the time I left they had 20 cars scattered around the area awaiting recovery and the phone was ringing every minute with more angry customers!
Dread to think what the total cars affected was. To this day I don’t understand how it was even possible to get it wrong, I wouldn’t believe it myself if I hadn’t gone through it!
I filled up my basically empty petrol with £70 of diesel and it conked out before I got off the forecourt. Thought I’d messed up but double checked the pump I’d used, double checked my recipe and I had definitely put fuel in from the petrol pump and paid for petrol. Quick sniff inside the filler and the distinct smell of diesel greated me.
It seems I was the first person to realise where the issue was occurring. The installer was still on site and he checked and confirmed my suspicions (his face was ghostly white when he realised).
Luckily they had a workshop on site so got mine drained down and fuel system cleaned within an hour. However, it seems a lot of people had filled half tanks and were only just finding out the issue. By the time I left they had 20 cars scattered around the area awaiting recovery and the phone was ringing every minute with more angry customers!
Dread to think what the total cars affected was. To this day I don’t understand how it was even possible to get it wrong, I wouldn’t believe it myself if I hadn’t gone through it!
Cupramax said:
Toaster Pilot said:
Countdown said:
Our BACS system failing on payroll day (or technically two days before payroll day)
Cue 1500 CHAPS payments......
I hope you paid less than the £20 most banks charge for them!!!Cue 1500 CHAPS payments......
t storm and we ended up duplicating the previous months payroll in the end as the system was completely borked.Toaster Pilot said:
Countdown said:
Our BACS system failing on payroll day (or technically two days before payroll day)
Cue 1500 CHAPS payments......
I hope you paid less than the £20 most banks charge for them!!!Cue 1500 CHAPS payments......
Gassing Station | The Lounge | Top of Page | What's New | My Stuff