How do you permanantly store a LOT of data?

[DennisCooper]

I have a sneaky suspicion that the aforementioned Photographer will not be an IT bod and thus having things like Tape drives and enterprise level large capacity storage servers will not quite what he'll be looking for! if £3/400 per year on drives is being mentioned as 'expensive' it shuts down any of those options!

As mentioned, depending on how the user wants to store and retrive and use his data is what will determine the best route to take. As he's a photographer, 'about to go professional' then yes, costs will be increased as he'll need to have access to pictures taken, in other words he cannot lose it. How to do this as cheaply as possible ?! I'd say the following could be a good route!

1. Get a new relatively powerful full tower case PC from the likes of Mesh, Dell, Chillblast etc etc - perhaps up to around £750 or so. These will have the capacity to take perhaps 3 or maybe even 4 2TB SATA hard disks. For response and speed, perhaps specify a 64GB or 128 GB SSD drive to have Windows installed on and plenty of space for applications such as Photoshop etc, of course, no photo's will be stored on this drive.

2. 2TB SATA hard drives can be had for aroundn £75 delivered from Ebuyer and the likes. So in the PC itself, with a 4 bay internal HDD enclosure, that'd be 6TB of storage rightaway and internally.

3. Purchase one of these for around £150 or £240. http://linitx.com/viewcategory.php?catid=146&p... 8 bay or 5 bay SATA storage case. The latest PC's I'd have thought would have at least 2 PCI Express slots. With 8 2TB hard drives, thats a further 16TB of storage available. With 2 of these 8 bay cases, and populated, 32 TB of storage giving 38TB of storage when including the main PC drives.

4. All the above is great for storage, very easily and quick retrieval, setup as just a bunch of disks or perhaps pooled to keep as many drive letters available to Windows. For long term storage and of course off PC and site backups, I'd say it'd be best to invest in a Blu Ray recorder and burn off all images to Blu Ray Disks and store those off site - friends or relatives house etc in case the above PC and Drive Cases are flooded, damaged by fire, stolen, drive failures etc at the address of the photographer.

All the above would be 'relatively' easy for a non IT person to look after themselves, shouldn't take up absolutley bags of time either and of course, won't cost an absolute fortune!

Now the potential there for around 38TB means absolutley masses of digital RAW quality images, most likely much more than would be required, so the costs and components can be reduced, perhaps go for the 3 disks in the PC first, get used to the routine, and add the 5 or 8 bay enclosures as needs arise. Burning all images to DVD-R will save some money as they're cheaper, it'll just mean much more disk burning as they hold perhaps 8.5 GB for a Dual layer disk. If so inclined, 2 or 3 optical disk copies could be made to keep at other locations as well, further reducing data loss risk.

You'll all have to forgive me on the RAID and data drive Pooling etc, it's been a little while since I worked in the Enterprise Data Storage area, even then it was for around 8 months so I'm not a 'master level' expert either. With SATA drives for home use now just so cheap, a 'big rig' like the above which is more than suitable for home businesses would be relatively affordable and importantly 'easy to use' for non IT guys/business owners. of course, said photographer may well get away with using perhaps a 4 bay NAS system from the likes of Drobo, Synology or make use of Windows Home Server etc etc to start off with as well!

The above is my plan for my large data storage needs, and I'm only up to about 7 TB of music and video data! soon to be around 9TB!
Hope this helps!
Cheers, Dennis!

Edited by DennisCooper on Thursday 30th December 23:41

[ukwill]

10mb upload speeds availble now. No doubt in time it will increase. If you're storing pics you can use upload sw to specify when/what to upload.

Edited by ukwill on Friday 31st December 00:12

[mrmr96]

DennisCooper said:

For long term storage and of course off PC and site backups, I'd say it'd be best to invest in a Blu Ray recorder and burn off all images to Blu Ray Disks and store those off site

Two points:

1. Why Blu Ray and not Tape?

2. Don't make a single copy for offsite storage, make two and keep one off site and one on site. Maybe a fireproof safe too?

[Police State]

If it's just images he needs to store and wants to use them for repro purposes, then he should think about saving them as compressed Tiffs. They are 'lossless' and save and open very quickly. He should save approx 75% file space for each 20-30mb image. It's a no cost 'upgrade' an no matter what else he does on the hardware side, he should at lease do this on the data/software side.

[stevieb]

a Basic Netgear 4 bay NAS is getting affordable now.

£210 for a bare device and then £63 for each 2TB Drive.

NAS
http://www.scan.co.uk/products/netgear-rnd4000-100...

HArdDrive - now only £63 for a 2TB drive
http://www.scan.co.uk/products/2tb-samsung-hd204ui...


This will only give you 8TB of storage in 1 device but would only cost £460.
I personally think you should ignore RAID and go for 2 devices with 4TB Each and upgrade when you need the extra space £336 Each so £672 for 4TB of locally backed up data.



At present i have 2 of these devices. each with 2x2TB Drives in (2 spare for upgrading in the spring). one is stored in the house. The second is stored in a detached Centrally heated garage. In addition to this i also have 2x4TB USB drives where one is onsite keeping a local copy for the month and the second is stored at my workplace in my desk draw. At the end of the month i swap them around. so i have 2 live copies (Onsite) and 1 monthly copy stored offsite.

Is there any chance he can setup a NAS device and also start doing a monthly offsite copy of his work.

I am no professional photographer, but it would be worth more then my life if i lost any of the kids photos. I have covered most things that could happen with the exceoption of if anything was to hit the town (Nuke/bomb) then the data would be all but gone.

Edited by stevieb on Friday 31st December 00:23

[Skier]

ukwill said:

Skier said:

ukwill said:

Whats wrong with Flickr pro?

Have you ever worked out how long it would take to upload 1.5TB of data on your typical 256kbps broadband upload speed? (I pick 1.5TB as the original poster mentioned 2 x 1TB drives almost being full and the requirements incraeasing rapidly). Just for fun work out how long it would take to upload a single 20MB file - a very conservative size for a single image in raw format.

Skier

Edited by Skier on Thursday 30th December 21:06

Edited by Skier on Thursday 30th December 21:06

If I were a semi-pro (or pro) tog I wouldn't be uploading anywhere near 1.5Tb of data per day. Furthermore I'd get an ISP that provides at least 1mb/s upload. There are plenty of togs who use online storage.

You're right, you wouldn't - 'cos you can't! I didn't say that anyone was looking to upload 1.5TB per day, simply that this is the (approx) quantity of data that was backed-up to the two external hard drives; uploading this would take an age.

Fast upload speeds are available but not to the majority of the population. For example, I live in a rural environment and BT is the only provider available: 2.3Mbps download, approx 300Mbps upload. If I could get faster fibre optic I would.

Skier

[HereBeMonsters]

Get it done externally. If I needed to RELY on data being backed up (Wedding Photos etc. for clients) then I would get them backed up in a proper data centre with a decent SLA from the host. They will have the hardware and the expertise, give it over to them.

[Mattt]

It could be an option though, get the initial upload done by someone with a fibre line - then maintaining & adding to the backup is easy on Home BB.

[ukwill]

Skier said:

ukwill said:

Skier said:

ukwill said:

Whats wrong with Flickr pro?

Have you ever worked out how long it would take to upload 1.5TB of data on your typical 256kbps broadband upload speed? (I pick 1.5TB as the original poster mentioned 2 x 1TB drives almost being full and the requirements incraeasing rapidly). Just for fun work out how long it would take to upload a single 20MB file - a very conservative size for a single image in raw format.

Skier

Edited by Skier on Thursday 30th December 21:06

Edited by Skier on Thursday 30th December 21:06

If I were a semi-pro (or pro) tog I wouldn't be uploading anywhere near 1.5Tb of data per day. Furthermore I'd get an ISP that provides at least 1mb/s upload. There are plenty of togs who use online storage.

You're right, you wouldn't - 'cos you can't! I didn't say that anyone was looking to upload 1.5TB per day, simply that this is the (approx) quantity of data that was backed-up to the two external hard drives; uploading this would take an age.

Fast upload speeds are available but not to the majority of the population. For example, I live in a rural environment and BT is the only provider available: 2.3Mbps download, approx 300Mbps upload. If I could get faster fibre optic I would.

Skier

According to BT/Virgin over 30 million homes will have the option to get a 100mb/s - 10mb/s service by 2012. There are numerous packages available right now that provide unlimited storage for as litte as $5 per month.

[stevieb]

Skier said:

ukwill said:

Skier said:

ukwill said:

Whats wrong with Flickr pro?

Have you ever worked out how long it would take to upload 1.5TB of data on your typical 256kbps broadband upload speed? (I pick 1.5TB as the original poster mentioned 2 x 1TB drives almost being full and the requirements incraeasing rapidly). Just for fun work out how long it would take to upload a single 20MB file - a very conservative size for a single image in raw format.

Skier

Edited by Skier on Thursday 30th December 21:06

Edited by Skier on Thursday 30th December 21:06

If I were a semi-pro (or pro) tog I wouldn't be uploading anywhere near 1.5Tb of data per day. Furthermore I'd get an ISP that provides at least 1mb/s upload. There are plenty of togs who use online storage.

You're right, you wouldn't - 'cos you can't! I didn't say that anyone was looking to upload 1.5TB per day, simply that this is the (approx) quantity of data that was backed-up to the two external hard drives; uploading this would take an age.

Fast upload speeds are available but not to the majority of the population. For example, I live in a rural environment and BT is the only provider available: 2.3Mbps download, approx 300Mbps upload. If I could get faster fibre optic I would.

Skier

If done externally into a datacentre you will not be transferring the amount of data you are quoting, as many of the remote backup solutions have some form of compression. depending on the file some RAW files can be compressed from 20Meg to 4Meg. Given the bulk of data that he currently has it could be reduiced to 500Gigs for Transfer using compression built into some of the client backup software.

[Mattt]

That number sounds slightly optimistic to me.

ETA: the BT/Virign estimate

Edited by Mattt on Friday 31st December 00:38

[HereBeMonsters]

Mattt said:

It could be an option though, get the initial upload done by someone with a fibre line - then maintaining & adding to the backup is easy on Home BB.

Or just take your disks to the data centre...

[robsa]

He could always look at something like the Thecus NAS's, such as the 4100/4200. I think they are about £275 plus the cost of HDD's of which they take up to 4x and can be RAIDed. I believe SCAN ( http://bit.ly/aVGypVk) are selling 1TB Samsung SATA drives for £38 each including vat. 4 of those will give him 3TB RAIDed! They hot swap too and use a 1000 base network adapter. So for less than £500 he could have a super fast, hot-swap, RAIDed NAS!


Nurd solution:
Alternatively he could buy a cheap server from EBAY which is always full of the things, install freeNAS the linux NAS OS on it and fill it with either SATA or SAS drives. OK, SAS drives are smaller and pricer but boy do those babies move!

Edited by robsa on Friday 31st December 00:48

[ukwill]

Mattt said:

That number sounds slightly optimistic to me.

ETA: the BT/Virign estimate

Edited by Mattt on Friday 31st December 00:38

Well it's the figures they're both quoting. There's more - apparently 2/3rds of the UK by 2015.

[theboss]

Mattt said:

That number sounds slightly optimistic to me.

ETA: the BT/Virign estimate

Edited by Mattt on Friday 31st December 00:38

Indeed, especially when you consider there are less than 30 million households in the whole country

http://www.statistics.gov.uk/census2001/profiles/c...

[ukwill]

theboss said:

Mattt said:

That number sounds slightly optimistic to me.

ETA: the BT/Virign estimate

Edited by Mattt on Friday 31st December 00:38

Indeed, especially when you consider there are less than 30 million households in the whole country

http://www.statistics.gov.uk/census2001/profiles/c...

Nice. A 2001 census.

Here some more up to date stats: http://www.statistics.gov.uk/pdfdir/iahi0810.pdf

Google the bt/virgin figures yourself. This article reckons 25million between the two by 2015. http://www.techwatch.co.uk/2010/10/08/virgin-media...

[Famous Graham]

Assuming 20MB for each RAW file, that's 78 1/2 thousand images.

I would suggest some housekeeping may be in order

[theboss]

ukwill said:

theboss said:

Mattt said:

That number sounds slightly optimistic to me.

ETA: the BT/Virign estimate

Edited by Mattt on Friday 31st December 00:38

Indeed, especially when you consider there are less than 30 million households in the whole country

http://www.statistics.gov.uk/census2001/profiles/c...

Nice. A 2001 census.

Here some more up to date stats: http://www.statistics.gov.uk/pdfdir/iahi0810.pdf

Google the bt/virgin figures yourself. This article reckons 25million between the two by 2015. http://www.techwatch.co.uk/2010/10/08/virgin-media...

Both of which confirm my point - that there are less than 30 million households in the whole country - so the figure quoted above is, as Mattt put it, 'slightly optimistic'.

From the second link:

BT has a target of hooking up two-thirds of households in the UK with fibre broadband before 2015 arrives, and Virgin is aiming to expand its coverage by a similar amount to around 15 to 16 million homes (there being around 25 million or so households in the UK according to ONS figures).

[MaximumJed]

Quite a few people are already using or intending to use one of these as a raid server:

http://pistonheads.co.uk/gassing/topic.asp?h=0&amp...

The £100 cashback offer has been extended to the end of January and I'll be picking one up in the next couple of weeks. It has 4 proper bays plus the CD bay can be used for another SATA (which has been pre-wired). That should allow for plenty of in-home space for a reasonable outlay. As for backing up a lot of TB of data, I haven't worked out a convenient way of doing that myself yet, I'm about to fill up 5TB and I need to sort something out!

[paddyhasneeds]

Keep in mind as well that with any NAS/SAN/DAS unit the manufacturer will only warranty the unit, not the data on it. Why does this matter? Well what happens if you buy a unit with a 3 year warranty and put 10tb of data on it and then the unit fails? Your drives may work perfectly, but the manufacturer may no longer make something that you can put them into and still be able to access your data, so they'll give you a new enclosure but sorry about your data.

Ditto outsourced cloud backup, read the small print and it usually boils down to "Sorry we lost everything you ever created, but here, have your subscription fees back".

This is the problem that advances in storage capacity (or more specifically the cost per gb/tb) has created - go out and spend £50 and you've got 70 times more storage in your home PC than we had in around £40k worth of server 10 years ago, albeit with zero redundancy, but it's got so crazy that £300-400 every couple of years is considered expensive, which I appreciate it is if you don't have the money, but it isn't actually expensive for what it is.

So far Pugwash hasn't put any importance on the data which makes it quite difficult to give a qualified answer. If he needs to keep the data because he's selling a service where, for example, people can come back to him for prints then I'd suggest the data is much more important than if he simply wants to keep every photograph he's ever taken "just in case".