Farking IT systems ...
Author
Discussion

zcacogp

Original Poster:

11,239 posts

267 months

Tuesday 15th February 2005
quotequote all
Having a VERY bad day today.

I have just discovered that a LARGE chunk of the work I have spent the last 3 weeks doing has disappeared, due to a cock-up on behalf of the IT service provider.

The neanderthal on the end of the "Telephone Support Line" has just had a 15-minute ear-bashing from yours truly. I don't think it made any impact though - I'd swear that he had more fingers than IQ points. In fairness, it wasn't his fault, but he wasn't a model of helpfulness in sorting my situation out. No, there wasn't a back-up, as the back-up tapes for the whole of the last 4 weeks are unreadable! Fan--tastic. "The best bet is to recreate the work you lost, sir. Had you lost alot of editing?"

So, back to the drawing board. For the record, I am a contractor, and the client in question is a Government Department. The IT supplier is a PFI provider, and this is "normal" (apparently.) I'll not post how much the IT contract they have is worth, but it is clear that they have been stitched-up more than Pudsy Bear after 10 rounds with a particularly vicious cat. And who pays? Yes chaps, wallets at the ready, Greedy Gordo's coming 'round again.

At moments like this, I seriously consider drink.


Oli.

viggen114

259 posts

276 months

Tuesday 15th February 2005
quotequote all
There will be a penalty clause in the Service Level Agreement
Find it and get someone in authority to exercise it, after all how much have you billed for the lost work.

[advisemode]whenever working on government/pfi/outsourced systems always make sure you got you own backups. Even if it is just mailing it to yourself[/advisemode]

zcacogp

Original Poster:

11,239 posts

267 months

Tuesday 15th February 2005
quotequote all
viggen114 said:
There will be a penalty clause in the Service Level Agreement
Find it and get someone in authority to exercise it, after all how much have you billed for the lost work.

[advisemode]whenever working on government/pfi/outsourced systems always make sure you got you own backups. Even if it is just mailing it to yourself[/advisemode]


Mr Viggen,

There should be a penalty clause in the agreement, but no matter how many people I ask (and apparently this has happened before, to a lesser extent, to other people), no-one seems to know anything about it.

TBH, it's no skin off my nose. I have billed for the lost work, and as I am on T&M, the end result is that they will have a considerably less comprehensive end result than had the problem not happened. (Although of course, this will reflect slightly badly on me, as it's my work.)

Should keep backups? Of course I should, but as the client in question is so hugger-mugger in love with the provider, keeping back-ups is frowned on. So, being a good boy, I toed the party line and relied upon the provider. And, in practice, keeping back-ups is quite difficult. USB ports are disabled so no memory sticks, no (easy) access to CD-Burners, files too large to fit on a floppy disk, and a mail gateway that spits out attachments larger than 1mb to external addresses. Local hard disks are locked down to prevent user access. (All done "For security reasons". Yeah right ... )

I'm just composing a long Email to the head of the unit I am working for, telling them, in detail, why their system is so uttly sh1t3 and why a group of half-pissed girl guides could do a better job.


Oli.

P.S. Sorry - this post probably came across as slightly abrasive - it is, but the bile is not intended to be directed at you Mr Viggen. Thanks for replying!

ETA:

>> Edited by zcacogp on Tuesday 15th February 12:50

JonRB

79,385 posts

295 months

Tuesday 15th February 2005
quotequote all
zcacogp said:
USB ports are disabled so no memory sticks, no (easy) access to CD-Burners, files too large to fit on a floppy disk, and a mail gateway that spits out attachments larger than 1mb to external addresses. Local hard disks are locked down to prevent user access.
Blimey. I think I'd find it hard to work under such circumstances.
I have a 2Gb USB Memory Stick on which I keep code snippets, help files, electronic reference books, off-line website content, etc., and of course code backups.

I'd feel like I was working with a hand tied behind my back without it (although I'm sure I'd cope if I needed to, because in truth I seldom need any of it).

Anyway, back to the topic in hand. If the client have made it impossible for you to keep a local backup and have lost all your work due to their negligence (and not yours) then it is not really your problem. Take this opportunity to write it all again but better than before - it isn't often one gets the opportunity to do this, especially whilst getting paid for it.

zcacogp

Original Poster:

11,239 posts

267 months

Tuesday 15th February 2005
quotequote all
JonRB said:

Blimey. I think I'd find it hard to work under such circumstances.

Too right. Difficult indeed. The answer is to sneak your own lappie into the building (OK when you are chummy with the security people) and look at everything on that.

Do it all again, but better? It's not actually code I was producing. It was a data gathering exercise, with assumptions made from the data gathered. I can't gather it better, per se, I can just gather it again.

Heigh ho, at least I am being paid for it. Must stop grumbling.


Oli.

ThatPhilBrettGuy

11,810 posts

263 months

Tuesday 15th February 2005
quotequote all
Just make sure that every up the line knows that 3 weeks have just been added to the time scales. That usually gets things going.

JonRB

79,385 posts

295 months

Tuesday 15th February 2005
quotequote all
zcacogp said:
Do it all again, but better? It's not actually code I was producing. It was a data gathering exercise, with assumptions made from the data gathered. I can't gather it better, per se, I can just gather it again.
Sorry. Bad assumption on my part.

zcacogp

Original Poster:

11,239 posts

267 months

Tuesday 15th February 2005
quotequote all
JonRB said:
Sorry. Bad assumption on my part.
Hey, don't be sorry chum. It was a perfectly reasonable assumption.

Apologies if I was brusque in what I said.


Oli.

JonRB

79,385 posts

295 months

Tuesday 15th February 2005
quotequote all
zcacogp said:
Apologies if I was brusque in what I said.
Don't worry - you were fine.

Anyway, as I said, so long as you don't have a contractual obligation to correct defective work at no additional cost (often advocated by the PCG as a good clause for IR35, but scares the out of me) and/or it isn't your fault, then really it is just an extreme annoyance and a waste of everyone's time and money. But, as you say, no actual skin off your nose.

Contracting - can't beat it.

Mr E

22,718 posts

282 months

Tuesday 15th February 2005
quotequote all
I think you're being quite restrained actually.

I'd be spitting blood, being paid or not.

JonRB

79,385 posts

295 months

Tuesday 15th February 2005
quotequote all
Mr E said:
I'd be spitting blood, being paid or not.

It's annoying, but nothing to get too upset about really.

I've worked on projects that have been underway for over a year - good code too - and the client had pulled the plug, not renewed any contractors' contracts and then laid off a load of staff. Has happened at least 3 times over the past 8 years in fact (although I wasn't on board for the whole project in each case). All you can do is just shrug and take the attitude that one's invoices were paid and that is that.

>> Edited by JonRB on Tuesday 15th February 13:54

zcacogp

Original Poster:

11,239 posts

267 months

Tuesday 15th February 2005
quotequote all
Mr E said:
I think you're being quite restrained actually.

I'd be spitting blood, being paid or not.

Thanks ... "Restrained". I'll see if I can incorporate it on my CV somewhere!

As Jon says, there isn't a fat lot you can do. Talking to the bunch of wet lettuces which pass as a help desk is a profoundly unrewarding activity, and I have been paid for the time spent. Sure, it's soul destroying, but at the moment I'm doing this contract for the money not the fulfillment. You don't go contracting for the job satisfaction, or the good working conditions. You take on a contractor as they are expensive, but you can treat them badly, as they are not subject to most employment law.

Jon, I AM risk-bearing, and am supposed to correct defective work at my own expense. But if push came to shove here, I'd simply point out that I have followed the requirements of the client, and the facilities offered by the client (namely, their IT system) have let me down. I took all steps suggested to prevent loss of information (they say it is un-necessary to do anything other than use their "secure" system). I doubt they would be daft enough to get nasty about it ...


Oli.

JonRB

79,385 posts

295 months

Tuesday 15th February 2005
quotequote all
zcacogp said:
Jon, I AM risk-bearing, and am supposed to correct defective work at my own expense. But if push came to shove here, I'd simply point out that I have followed the requirements of the client, and the facilities offered by the client (namely, their IT system) have let me down. I took all steps suggested to prevent loss of information (they say it is un-necessary to do anything other than use their "secure" system). I doubt they would be daft enough to get nasty about it ...
Indeed. Looks like you are covered on that one (hence my "and/or" when I mentioned such a clause). Besides, doubtless you have PI/EL/PL insurance as well so you'd be covered even if they were stupid enough to pursue.

I had a client who wanted to invoke the non-performance clause because their expectations far exceeded the services I supply (and had been up-front with throughout the hiring process). Without going into any details I indicated that I would vigorously defend the good name of both my company and myself if they did and they suggested a mutually agreed termination with all outstanding invoices settled. Not ideal, but better than getting legal.
But I digress.

zcacogp

Original Poster:

11,239 posts

267 months

Tuesday 15th February 2005
quotequote all
JonRB said:
Besides, doubtless you have PI/EL/PL insurance

I have to say, I never had bothered with such insurance, although I am sure I should. TBH, I only got into contracting as a short-term thing (6 months max), which sounds a bit daft 3 years on ... IN the event of problems I would pursue the idea of taking all the money out of the company and allow it to be bankrupted by the litigent. Not sure of the legalities of this, but I understand it to be one of the advantages of running a company as opposed to doing it in your own name.


Oli.

sadako

7,080 posts

261 months

Tuesday 15th February 2005
quotequote all
I work for an IT supplier who happen to sponsor touring cars. I have never been on a site that didnt have good backup policies, at least nightly.

I have heard of a place with a state of the art tape safe, EM free with highly secured access to only a few people, storing years of backups. They spent millions on it, making sure no stray magnetic fields affected the tapes. Unfortunatly it was so good that noone thought to do test restores.

New guy comes on site. Did a test restore and it failed, test restores the others and found out all of the tapes had been erased. There is a massive panic as idiotic middle management drones try to work out what happened whilst beginning proceedings against the supplier of their tape safe room.

New guy sits there working it out, notices the cleaner. Cleaner takes out his copy of the secure key, opens the tape safe room, starts waxing the floor with a floor buffer. Middle management, whilst not telling anybody else, gave him the key because they didnt like the dirty floor...

Not one of our sites though

tycho

12,127 posts

296 months

Tuesday 15th February 2005
quotequote all


they guy who was doing my job before me was sacked because there had been no backups done for a month.

Most companies take backups very, very seriously.

zcacogp

Original Poster:

11,239 posts

267 months

Tuesday 15th February 2005
quotequote all
tycho said:


they guy who was doing my job before me was sacked because there had been no backups done for a month.

Most companies take backups very, very seriously.

Quite.

I have the list of failiures of backups for the last 3 weeks in front of me (they sent it to me "To see", not because it is part of my job.)

It would seem that if a back-up fails, they don't bother to redo it. And a large amount of the back-ups done suffer from either "Media Failiure" (WTF?) or are "Unreadable".

Having said that, there are also a large number of gaps which they cannot explain.

Ah well, as agreed it's not my problem. (It's just a shame that I actually did anything for the last 3 weeks, as I could have spent the time at home doing sweet FA!)

The more worrying thing is the vast waste of money this represents, and who is paying for it? And if it has happened to me, how many other people have had the same problem? Government waste? Mad.


Oli.

LongQ

13,864 posts

256 months

Tuesday 15th February 2005
quotequote all
tycho said:


Most companies take backups very, very seriously.


Some even understand why.

slinky

15,704 posts

272 months

Tuesday 15th February 2005
quotequote all
So many people backup files so regularly, but so few actually attempt restoring them...

What is the point in a backup solution if you don't know whether it works???

One of the jobs that our ops team did at one of my old places was to restore the backups every morning to confirm that the data was good..

I know that doesn't help in this situation, but it's something that everyone should consider in their backup implimentation.

slinky

matt_t16

3,402 posts

272 months

Tuesday 15th February 2005
quotequote all
slinky said:
So many people backup files so regularly, but so few actually attempt restoring them...

What is the point in a backup solution if you don't know whether it works???

One of the jobs that our ops team did at one of my old places was to restore the backups every morning to confirm that the data was good..

I know that doesn't help in this situation, but it's something that everyone should consider in their backup implimentation.

slinky


Slinks, thee and me really need to start a consultancy - I've been trying to get 2 contractors upto speed today so they can seamlessly replace me when the time comes - now you would think 30quid an hour would buy some skills wouldn't you I fail to see why its difficult to understand why there are two backups on two sites. One site backs up itself and the remote site and vice versa that way theres two copies of both sites data in two different locations.

Matt