Updating my ageing backup solution
Discussion
I'm a semi-professional film maker who generates a few TB a year in footage that I need to backup.
My current solution is a 2006 Mac Pro hacked to run El Capitan, a 80GB SSD boot drive, two 12TB, two 3TB and one 10TB data drive.
I currently use around 20TB of the 38TB of available storage and add 2-5TB a year. This is currently only used for backup and mirrored online with Backblaze.
The solution works OK, but at 14 years old it starting to show its age, takes up a bit of space and pumps out a lot of heat.
I'm looking for a more modern solution - thinking along the lines of a small direct attached storage unit that I can just transfer my existing drives over, is there anything stopping me using something like this?
https://www.amazon.co.uk/TerraMaster-D5-300C-Exter...
My current solution is a 2006 Mac Pro hacked to run El Capitan, a 80GB SSD boot drive, two 12TB, two 3TB and one 10TB data drive.
I currently use around 20TB of the 38TB of available storage and add 2-5TB a year. This is currently only used for backup and mirrored online with Backblaze.
The solution works OK, but at 14 years old it starting to show its age, takes up a bit of space and pumps out a lot of heat.
I'm looking for a more modern solution - thinking along the lines of a small direct attached storage unit that I can just transfer my existing drives over, is there anything stopping me using something like this?
https://www.amazon.co.uk/TerraMaster-D5-300C-Exter...
I'm not an expert by any means, but for that amount of data, I would be looking at a commercial solution rather than a hobby/home user nas drive.
A "proper" server with SAS drives for example.
Most NAS boxes run their own firmware, if the box goes pop, it's not easy to transfer the drives into a new box without reformatting.
A "proper" server with SAS drives for example.
Most NAS boxes run their own firmware, if the box goes pop, it's not easy to transfer the drives into a new box without reformatting.
I’ve also generated a few TB of video a year. Recent projects are all backed up to a couple of Synology NAS units. I’ve previously been archiving footage onto 50GB blu-Ray disks, where a pack of 25 disks will hold a bit over a TB. Very slow to burn, and bulky to store. More recently I’ve been copying footage to 1TB Sabrent NVMe SSD’s in an external thunderbolt enclosure, writing at 1500 MB/sec vs the optical disk’s 75 MB/sec
The QNAP NAS range can backup or sync with Backblaze. They do various models from home to pro servers. I have a slightly old 2 disk small business model and it runs fine, software updates stil coming regularly. I like that their OS (a skinned Linux like most NASes) is identical on all models - I remotely administer most of my family's.
https://www.backblaze.com/blog/qnap-nas-backup-to-...
Their semi pro models can get quite clever with enterprise features such as ssd caching if you want speed. You can also sync to your PC/laptop if you want to hold a local working copy of current stuff.
It's not always the easiest to configure as their documentation isn't perfectly written, but most features are not that hard, and they guides and forum have improved a lot in the last 10 year's.
https://www.backblaze.com/blog/qnap-nas-backup-to-...
Their semi pro models can get quite clever with enterprise features such as ssd caching if you want speed. You can also sync to your PC/laptop if you want to hold a local working copy of current stuff.
It's not always the easiest to configure as their documentation isn't perfectly written, but most features are not that hard, and they guides and forum have improved a lot in the last 10 year's.
Phunk said:
Cheers for the suggestions so far.
Unfortunately a NAS isn't workable with Backblaze, tape drives are a bit too much faff and SSD storage will cost a fortune for 40TB.
Ideally needs to be a DAS solution
The difficulty you face is as a small business/semi pro as you've discovered it's really easy to build up TB of data but pretty expensive to protect it properly against loss.Unfortunately a NAS isn't workable with Backblaze, tape drives are a bit too much faff and SSD storage will cost a fortune for 40TB.
Ideally needs to be a DAS solution
I'd consider a mix of DAS and perhaps some sort of NAS but consider how you'd protect either/both of those against either finger trouble or ransomware.
Tape still has its place and whilst a tape drive is a few grand once you have it you can store 10TB (ish) for about £50.
Cheers for the ideas so far!
Whilst I agree that a NAS would be a good solution, using one instead of a DAS would mean that my Backblaze annual bill would go up from £50 to £1238 + costs for any downloads I make.
Annoyingly it needs to be DAS - unless there is an alternative cloud mirror solution for a NAS that isn't mental money.
Whilst I agree that a NAS would be a good solution, using one instead of a DAS would mean that my Backblaze annual bill would go up from £50 to £1238 + costs for any downloads I make.
Annoyingly it needs to be DAS - unless there is an alternative cloud mirror solution for a NAS that isn't mental money.
Phunk said:
Cheers for the ideas so far!
Whilst I agree that a NAS would be a good solution, using one instead of a DAS would mean that my Backblaze annual bill would go up from £50 to £1238 + costs for any downloads I make.
Annoyingly it needs to be DAS - unless there is an alternative cloud mirror solution for a NAS that isn't mental money.
Look at Synology and Qnap and look at Amazon Glacier perhaps.Whilst I agree that a NAS would be a good solution, using one instead of a DAS would mean that my Backblaze annual bill would go up from £50 to £1238 + costs for any downloads I make.
Annoyingly it needs to be DAS - unless there is an alternative cloud mirror solution for a NAS that isn't mental money.
I don't know Backblaze but I wouldn't be surprised if there isn't some sort of "hack" that can make it do what you with a NAS/network location.
Phunk said:
Cheers for the ideas so far!
Whilst I agree that a NAS would be a good solution, using one instead of a DAS would mean that my Backblaze annual bill would go up from £50 to £1238 + costs for any downloads I make.
Annoyingly it needs to be DAS - unless there is an alternative cloud mirror solution for a NAS that isn't mental money.
For your volume of data, have you thought about a 2x tiered approach?Whilst I agree that a NAS would be a good solution, using one instead of a DAS would mean that my Backblaze annual bill would go up from £50 to £1238 + costs for any downloads I make.
Annoyingly it needs to be DAS - unless there is an alternative cloud mirror solution for a NAS that isn't mental money.
- DAS for the current projects. (Hot)
- NAS for the work you've just 'put-to-bed'. (Warm)
BTW - When it comes to the disks themselves, it's worth seeing if you can get them from different production batches, as it reduces the risk of them all failing at around the same time. It might mean purchasing a couple of sets, over a 6mth window & swapping around but it'll also mean you have spares ready to swap-out when necessary.
M
I'm not sure that Backblaze can tell the difference on their end if you're backing up from a PC versus a NAS. Unless the PC option uses a bespoke client application?
There may be two ways around this:
1. Have a NAS, but use your PC to run the backup from mapped drives on the PC (although if a client app it may detect that).
2. Run a VM on the NAS (QNAP do this, I think others too) and run the sync from inside that VM. The VMs run dog slow, but they work and for cloud backup they'd be fast enough.
There may be two ways around this:
1. Have a NAS, but use your PC to run the backup from mapped drives on the PC (although if a client app it may detect that).
2. Run a VM on the NAS (QNAP do this, I think others too) and run the sync from inside that VM. The VMs run dog slow, but they work and for cloud backup they'd be fast enough.
Your best option is to use a NAS internally but you need an off-site backup too, Backblaze will be too slow I'd think so AWS is ideal. https://aws.amazon.com/glacier/faqs/
You can create an account on AWS for free, enter and credit card and then only pay for what you use. Use two factor authentication. AWS is very secure - your storage is by default private now. You can also choose where it's stored, some regions are cheaper than others, but London is available and slightly more expensive than the US. I backup a lot to AWS and it's amazingly fast.
Create a S3 bucket to get the hang of it then glacier for long term, limited access storage, choose encrypted. Once you have the storage use something like https://s3browser.com/ to copy your data to it. There is a folder sync tool which is great for up to 500GB or so a day but for larger files, just copy them over to AWS manually. You can choose to encrypt them first as well so even if your files are stolen they are useless.
The volume of data you want to backup makes consumer systems less and less useful, it's an interesting situation as lots of people are generating so much data now, traditional free/consumer tools are becoming less useful.
You can also look at commercial software which will backup to cloud storage directly, there's plenty of choice but you need to buy it and the good ones aren't that cheap.
You can create an account on AWS for free, enter and credit card and then only pay for what you use. Use two factor authentication. AWS is very secure - your storage is by default private now. You can also choose where it's stored, some regions are cheaper than others, but London is available and slightly more expensive than the US. I backup a lot to AWS and it's amazingly fast.
Create a S3 bucket to get the hang of it then glacier for long term, limited access storage, choose encrypted. Once you have the storage use something like https://s3browser.com/ to copy your data to it. There is a folder sync tool which is great for up to 500GB or so a day but for larger files, just copy them over to AWS manually. You can choose to encrypt them first as well so even if your files are stolen they are useless.
The volume of data you want to backup makes consumer systems less and less useful, it's an interesting situation as lots of people are generating so much data now, traditional free/consumer tools are becoming less useful.
You can also look at commercial software which will backup to cloud storage directly, there's plenty of choice but you need to buy it and the good ones aren't that cheap.
Phunk said:
I'm looking for a more modern solution - thinking along the lines of a small direct attached storage unit that I can just transfer my existing drives over, is there anything stopping me using something like this?
https://www.amazon.co.uk/TerraMaster-D5-300C-Exter...
It's not what I would recommend for your scenario, however I picked up something along similar lines a while back: https://www.amazon.co.uk/Icybox-IB-3810u3-10-Bay-E...https://www.amazon.co.uk/TerraMaster-D5-300C-Exter...
The benefit being that I could just plonk all my drives in it with data in-situ and it would 'just work'. For the most part it was true; most of the disks were fine however some being shucked from desktop enclosures had some funky volume formatting done by the caddy. Fortunately I had space to stage the data elsewhere, format the drive and partition it how I wanted then move the data back again. Other upsides? It's fast, even with SMR drives and especially when moving data between drives (which just uses the backplane). Even over the USB to the PC I see sustained writes around 120Mb/s+. It's also handy to be able to power down individual drives when not needed.
Downsides? Well it's 'JBOD' so no redundancy in there. It is effectively just a giant USB hub (although that's a bit of a disservice to be fair). It runs cool enough but the fans were noisy, especially in the PSU. I swapped these out for Noctua versions of a similar size, chucking a PWM controller in as well. It's not silent but close enough and temps rarely go much above 30degC.
I am, however a classic example of the 'do as I say, not as I do' camp and agree with others here; you need a proper NAS with RAID and an offsite replica somewhere. How valuable is the data? If you lost it all tomorrow would it set you back much? If yes or even 'somewhat' then back it up properly...
Edited by Funk on Wednesday 18th November 19:24
seveb said:
Your best option is to use a NAS internally but you need an off-site backup too, Backblaze will be too slow I'd think so AWS is ideal. https://aws.amazon.com/glacier/faqs/
You can create an account on AWS for free, enter and credit card and then only pay for what you use. Use two factor authentication. AWS is very secure - your storage is by default private now. You can also choose where it's stored, some regions are cheaper than others, but London is available and slightly more expensive than the US. I backup a lot to AWS and it's amazingly fast.
Create a S3 bucket to get the hang of it then glacier for long term, limited access storage, choose encrypted. Once you have the storage use something like https://s3browser.com/ to copy your data to it. There is a folder sync tool which is great for up to 500GB or so a day but for larger files, just copy them over to AWS manually. You can choose to encrypt them first as well so even if your files are stolen they are useless.
The volume of data you want to backup makes consumer systems less and less useful, it's an interesting situation as lots of people are generating so much data now, traditional free/consumer tools are becoming less useful.
You can also look at commercial software which will backup to cloud storage directly, there's plenty of choice but you need to buy it and the good ones aren't that cheap.
How much does it cost to get 30TB of data back out of AWS & how long would it take to restore?You can create an account on AWS for free, enter and credit card and then only pay for what you use. Use two factor authentication. AWS is very secure - your storage is by default private now. You can also choose where it's stored, some regions are cheaper than others, but London is available and slightly more expensive than the US. I backup a lot to AWS and it's amazingly fast.
Create a S3 bucket to get the hang of it then glacier for long term, limited access storage, choose encrypted. Once you have the storage use something like https://s3browser.com/ to copy your data to it. There is a folder sync tool which is great for up to 500GB or so a day but for larger files, just copy them over to AWS manually. You can choose to encrypt them first as well so even if your files are stolen they are useless.
The volume of data you want to backup makes consumer systems less and less useful, it's an interesting situation as lots of people are generating so much data now, traditional free/consumer tools are becoming less useful.
You can also look at commercial software which will backup to cloud storage directly, there's plenty of choice but you need to buy it and the good ones aren't that cheap.
Phunk said:
I'm a semi-professional film maker who generates a few TB a year in footage that I need to backup.
My current solution is a 2006 Mac Pro hacked to run El Capitan, a 80GB SSD boot drive, two 12TB, two 3TB and one 10TB data drive
OP, I don't think you told us what machine you do your video editing on, so we know what connectivity it has? That's presumably not on a 2006 Mac Pro limited to SATA 2, USB 2, PCIe v1 ?My current solution is a 2006 Mac Pro hacked to run El Capitan, a 80GB SSD boot drive, two 12TB, two 3TB and one 10TB data drive
mikef said:
Phunk said:
I'm a semi-professional film maker who generates a few TB a year in footage that I need to backup.
My current solution is a 2006 Mac Pro hacked to run El Capitan, a 80GB SSD boot drive, two 12TB, two 3TB and one 10TB data drive
OP, I don't think you told us what machine you do your video editing on, so we know what connectivity it has? That's presumably not on a 2006 Mac Pro limited to SATA 2, USB 2, PCIe v1 ?My current solution is a 2006 Mac Pro hacked to run El Capitan, a 80GB SSD boot drive, two 12TB, two 3TB and one 10TB data drive
I’ve also upgraded the Mac Pro to USB 3.0 and can transfer data either by plugging a drive in or over Ethernet.
Gassing Station | Computers, Gadgets & Stuff | Top of Page | What's New | My Stuff