How do you permanantly store a LOT of data?

[Scraggles]

ubuntu pc developed the click of death, so another 1 TB drive in it's place and find that it is a fking hitachi crap drive, used a WD green

2 TB ext WD for about £80 from pc world, reserve online at web price, buy in store...

backed up all my photos, then onto music and currently steam backups

Got all the pictures I want to keep online, those not online are on branded DVD's and some gold CD's

in 5 years, would not be worried about 90% of the photos, the ISP hosts the website, currently 16 GB of files after being told not to worry about bandwidth or server space

brother has a pro smugmug account, think that is around 14 GB or so, if the OP leaves the PC on over night for a few days, I suspect it will all be ok

[ZesPak]

http://www.rawzor.com/

This could save you on about 25%-50% realistically on storage space, thus reducing any other cost by that much.

[spants]

I would do it this way:

http://www.pistonheads.com/gassing/topic.asp?h=0&a...

with unRAID Server Plus software (lime-technology.com)

Put 4x2TB disks in the trays - gives 6TB useable and 2tb parity
Move the 160gb drive to the cd slot and use it as a cache drive

The advantage over raid5 is that the data is stored per drive and not striped.
On both systems, lose one drive and you are ok
on raid5, lose 2 drives and you lose the lot
on UNRAID, lose 2 drives and you lose the data on one drive.

For proper redundancy, build 2 systems and install crashplan. Do a backup while machines are local and then move 2nd system offsite and replicate the changes

[ZesPak]

spants said:

...
For proper redundancy, build 2 systems and install crashplan. Do a backup while machines are local and then move 2nd system offsite and replicate the changes...

While all your points are valid, it comes down to one thing:

His friend doesn't sound like a real computer whiz (which is ok, of course), so first of all, he would have to "outsource" the server staging, second of all, he thinks 300-400£/year is expensive.

So there is actually no way out of this one with that kind of budget.

[stevieb]

ZesPak said:

spants said:

...
For proper redundancy, build 2 systems and install crashplan. Do a backup while machines are local and then move 2nd system offsite and replicate the changes...

While all your points are valid, it comes down to one thing:

His friend doesn't sound like a real computer whiz (which is ok, of course), so first of all, he would have to "outsource" the server staging, second of all, he thinks 300-400£/year is expensive.

So there is actually no way out of this one with that kind of budget.

If he thinks 400 a year is expensive he should look at how much money his work is bringing him... Is he happy poising all his historical work for what is a small amount of cash.

[DennisCooper]

mrmr96 said:

DennisCooper said:

For long term storage and of course off PC and site backups, I'd say it'd be best to invest in a Blu Ray recorder and burn off all images to Blu Ray Disks and store those off site

Two points:

1. Why Blu Ray and not Tape?

2. Don't make a single copy for offsite storage, make two and keep one off site and one on site. Maybe a fireproof safe too?

Hiya,

To successfully run Tape drives LTO5 i think is the latest generation (I may be wrong!) you'd need proper server level hardware and software and that's not cheap. Certainly from the tidbits of info the OP gave, it doesn't 'sound' like the photographer is an IT expert. So whilst tape drives do have a place in small businesses, I still think they are useful only when you have a dedicated IT manager amongst the staff of the business in order to be an effective use. Small office/Home Office for those business people who don't really know IT and it's complexities are not going to go for Servers and Tape drives and Data backup software at that level - they'll most likely not know what to do once Windows Server boots into it's desktop screen!

Many of the following responses are again, I suspect, way too complex for what the photographer actually needs. The OP hasn't posted back to provide further details - perhaps the photographer is indeed fully IT literate and was once a hardware specialist or infrastructure analyst and is now opening a side business? or perhaps he's one of those users that doesn't know much more about computers than perhaps surfing the web, sending emails and updating facebook etc! we just don't know!

Internet backups are a distince possibility as upload speeds get faster and general broadband links for the home become more robust and offer good average speeds throughout the day etc. I'm still under the impression that anything perhaps over and above 5GB of online storage from the 'cloud' based companies costs money? and if you stop a subscription you must download everything again or it gets deleted?

I'm waiting for even more convenience, when the latest generation of Secure Digital cards called 'SDXC' gets up to the 2TB theoretical capacity and at an affordable price! - currently at 128 GB and costs around £500 for one of those!

Also looking through some responses - is there such a thing as a SATA 4TB hard drive available at Consumer electronics level?

I'm again under the impression that Windows (perhaps the Apple OS as well) can't handle hard disks greater than 2TB directly ?

Cheers, Dennis!

[mrmr96]

ukwill said:

There are numerous packages available right now that provide unlimited storage for as litte as $5 per month.

I use mozy.com it's ace.

[Roop]

Why on earth are people suggesting online storage for a use case that surely is far better suited to offline...? Sure, keep recent stuff online, but for long term storage you are barmy keeping it online IMO. Double copies of BD-ROM is the way forward in this case, surely...? As suggested already. A copy on-site and a copy off-site.

- £80 for a BD-R DL capable drive.
- BD-R discs store 25 GB and cost around 75p. That's 3p per GB, about double HDD.
- Each disc takes up next to no storage space and will keep for 25+ years.
- Futureproof. (newer drives will read the discs you create for a long time to come).

In 15 years from now, I'd far more fancy my chances of reading a properly stored BD-R than an old HDD that's been cranking away for ever and a day in a NAS. Anyways, I can't see an HDD lasting that long TBH...

[mrmr96]

^ Why blu ray not tape?

[Roop]

mrmr96 said:

^ Why blu ray not tape?

Tape's good and would be my second choice, but my reason for choosing BD-R first is that the TCO is higher for tape than BD-R. There's other advantages as well:

- Reading a tape in 5 years is fine provided you are using the same drive (and I mean SAME - I had a DDS4 drive that died - replaced by same make but newer model and it wouldn't read the old tapes...!) This is not an issue for BD-R.
- Access to data on optical media is virtually instant compared to tape (although this might not be a requirement to be fair).
- BD-R is good for 50 years on the shelf vs. 25 for tape.
- You can use a BD-R for non-work stuff (watching movies etc).

[paddyhasneeds]

If the chap can get in the habit of archiving by the project to optical media then blu-ray, maybe even DVD should be fine given good quality media.

The problem comes when people don't do it "little and often" and end up with a 2-3tb mountain of "stuff" with simply no time, or time but no idea, of how best to deal with it.

[Murph7355]

Personally I'd avoid Blu-Ray. I reckon it'll be outmoded sooner than many of the other formats mentioned.

tbh, hard disks are so cheap that if you want to ship stuff off site, I'd do it on those.

[Roop]

Murph7355 said:

Personally I'd avoid Blu-Ray. I reckon it'll be outmoded sooner than many of the other formats mentioned.

tbh, hard disks are so cheap that if you want to ship stuff off site, I'd do it on those.

It's a good point but with Blu-Ray now thoroughly entrenched in home devices (domestic Blu-Ray players, recorders, PS3 etc) it'll be around for a while. The chances of optical media being readable 15+ years down the line with backwards compatibility is far higher than your mechanical SATA HDD firing up and working.

All IMHO, but in 15+ years, I have never seen a professional grade long term storage solution that uses HDD. It's mostly tape with some optical.

[lestag]

[quote=Roop - BD-R is good for 50 years on the shelf vs. 25 for tape.
- You can use a BD-R for non-work stuff (watching movies etc).
[/quote]

Wanna post a link for that stat - genuine interest

[CommanderJameson]

If the chap's business depends on it, tape is the rational choice. It's expensive, and it's not as straightforward as just slopping everything onto a set of big disks, but it's proven and reliable.

The latest iteration of LTO can fit 1.5GB onto a single tape.

[Roop]

lestag said:

[quote=Roop - BD-R is good for 50 years on the shelf vs. 25 for tape.
- You can use a BD-R for non-work stuff (watching movies etc).

Wanna post a link for that stat - genuine interest

http://www.microboards.com/uploadedfiles/Products/Microboards/TUV.pdf

http://buyblurayhd.com/the-advantages-of-optical-a...

http://en.wikipedia.org/wiki/Linear_Tape-Open

http://h10010.www1.hp.com/wwpc/us/en/sm/WF05a/1216...

[spants]

ZesPak said:

spants said:

...
For proper redundancy, build 2 systems and install crashplan. Do a backup while machines are local and then move 2nd system offsite and replicate the changes...

While all your points are valid, it comes down to one thing:

His friend doesn't sound like a real computer whiz (which is ok, of course), so first of all, he would have to "outsource" the server staging, second of all, he thinks 300-400£/year is expensive.

So there is actually no way out of this one with that kind of budget.

True, I was mentioning on how I would do it

I guess for him, probably Blueray or stick with external disks. He has to balance the cost to his business if things went wrong against the outlay.

[plasticpig]

CommanderJameson said:

If the chap's business depends on it, tape is the rational choice. It's expensive, and it's not as straightforward as just slopping everything onto a set of big disks, but it's proven and reliable.

The latest iteration of LTO can fit 1.5GB onto a single tape.

I think you mean 1.5TB not 1.5GB . Tape is a rational choice for server based backup where data restoration is an urgent requirement and the data is constantly changing. If there is no urgent requirement for the data to be restored then Blu-ray is a better proposition. I would view long term picture storage as an archive solution and not a backup solution as the data is not changing.

[dilbert]

lestag said:

paddyhasneeds said:

Proper backup isn't cheap.

+1 wot he said.

Firstly there is no permanent option. The most permanent is onto negatives for photos. as we have 100 years of experiance in managing that medium, and hudreeds , if not thousands of years managing paper and vellum.
Secondly the photographer needs to identify:
why they need to store the data
how long they need to store the data
how they are going to structure and store the data so it can be easily pruned once it meets its end of life. assuming it has a EOL

Tape is the longest reliable long term archive option, but even then its only 5 - 7 years IIRC and then needs to be restored and put on a new tape (or refresh the old one)

The issue with any digital medium is wether it will be able to be read correctly in 50 years time. will will the tapes be able to be read by a tape drive (try gettinga LS120 drive these days) in 30 years time as tape technology will of changed. ( just go back 30 years and look a the storage devices available and wether they can be read now and the software formats of the time and whether they can be read)

Raided storage with security permissions to add only and no delete by default, will help solve the fat fingered issue identified by paddyhasneeds and replicating the storage (ie synctoy in contribute mode) or backup to tape and kept off site will help DR

I have disk drives that I used back in the early 90's and I can still read them 20 years later. Who knows what the future holds.

Compression is your friend. LZW will give you pretty nearly 50% lossless on a RAW image. If you try, you can probably do better with little loss in image quality.

Tapes are notorious. You can chuck something on a tape, you can only assume you'll get it back once. Your assumption will sometimes fail. It's just life.

Probably the most effective way to keep costs down is to "refine". Only save the stuff you want to keep. As they say; "Time is money".

When you don't know what you want to keep, tape is great. It's probabilistic. If you save everything, the odds are better that the thing you want in future will still be there.

The thing is, if you have not the time to review the data now, will you ever in future?

Edited by dilbert on Saturday 1st January 13:18

[fluffnik]

plasticpig said:

If there is no urgent requirement for the data to be restored then Blu-ray is a better proposition. I would view long term picture storage as an archive solution and not a backup solution as the data is not changing.

I have one of these with a pair of mirrored 2TB disks.



It will do automated full or differential back-ups of any card or USB device you plug into the top pop-up bit, with date based directory names, and will burn optical disks including BD-R DL from a web interface or via iSCSI.

I archive my photos in DVD then BD sized chunks and stash duplicate BDs at my Mum's...

I'd expect 2TB would be heaps for anything required immediately online even if not enough for a complete archive, and it only takes ~20mins to dump a whole 25G BD to disk if needed.