PDA

View Full Version : Long Term Data Storage - HDD & Online?



decryption
23rd October 2010, 09:51 PM
I've got an archive of all the MacTalk podcasts we've done and it adds up to approx 430GB of data. So far, it's been living on a single HDD with some other junk and I figure it's about time this stuff is archived properly. To achieve this it's probably best to have 1x 500GB HDD kept at home somewhere safe, plus also uploading it online somewhere and saving the password for it somewhere safe too. Not adding to the data really, just want to preserve this lot of data for the future :)

Got a couple of questions about storing it though:


What's the most reliable HDD out there?
How is best to store the HDD at home?
How often and how to maintain the HDD?
What online service is the most robust for storing 500GB of data?
Bigpond count uploads (gargh) and it will take approx 60 days to upload 500GB at 100kb/sec (max speed of most residential connections), so is there any way to get access to a faster connection, or somewhere in AUS I can send in a HDD for them to upload?

glacierdave
24th October 2010, 07:53 AM
Pick a hard drive, any hard drive. All the major manufacturers go through patches of reliability issues and you usually don't know they've entered one of these patches until drives start to fail - at which point, it's too late to decide you should have gone with a different brand.

If I was to get serious, I would have one copy on a 500GB hard drive (random brand) and then another copy on another 500GB hard drive (DIFFERENT but still random brand). i'd probably put each in a USB enclosure but buy my own enclosure so I get to choose the hard drive inside it. Both of these copies would go off-site somewhere.

I'd also keep a copy on "live storage" - in my case, I'd keep a copy on one of my Windows Home Server boxes - because it's "live storage" you know it still works and will get transferred to new hardware over time so it stays accessible.

Periodically (every month or three or six) I'd grab the externals (or take a laptop to the offsite location) and verify that they still work. Remember to keep the power supplies and all needed cables with the external - if you suddenly find you need them in a hurry you'll probably also need to have the power supply and USB cable handy too.

Oh, and if it looks like USB is about to get replaced with some new and incompatible standard, that's your cue to migrate your backups to that new interface/form factor/etc.

I'm not a huge fan of cloud storage for this much data and I've never really researched the choices available so can't really help there - my main concern would be to ensure the company doing the storage is likely to still be there tomorrow (or in five years) and that you can get them to send you the data in some form other than download if you need it in a hurry - you don't want to sit around waiting for 500GB to download.

The biggest issue anyone faces with long term archiving of digital information is being able to access it later. Between technology changes and reliability of media it's a hard ask - mostly these days I'm leaning towards ensuring my "live data" is in multiple locations AS LIVE DATA and migrate it as technologies change.

atcdev
24th October 2010, 08:15 AM
It's great to have the chance to help out the famous Decryption! I hope you find the advice useful.

I'm in a similar situation at the moment. I'm trying to find a backup solution for my iTunes library. It's just over 200 gig but growing fast, I've decided to rip the dvd collection after getting onboard the AppleTV bandwagon.

My experience is that no HDD is reliable in the long term so redundant arrays are the way to go. That gives you the option pulling dead drives and replacing them when they inevitably fail. More expensive in the short term, but now you have the option of just buying cheap drives. I'm using a Drobo for this side of the equation, there are less expensive options around but maybe not as flexible.

When it comes to offsite storage I'm taking the 'sneakernet' approach. I'm throwing everything onto one of a pair of 1 TB drives and taking it over to my brother's place. The second drive gets backed up onto a few weeks later, I go to my brother again and pick up the first drive and drop off the second. This also keeps the drives 'fresh', anecdotally I hear they like to be spun up at least every few months.

Depending on the sensitivity of the info you may want to consider putting it in a 'Knox (http://agilewebsolutions.com/knox)' Vault.

The online services for storage are pretty attractive but as you observed it will cost you a lot of your monthly bandwidth.

decryption
24th October 2010, 10:06 AM
Uploading it online looks like it will be a pain in the arse, and it will keep costing me money every month until I stop using the service!

So I think a 500GB HDD stashed at home, plus another 500GB HDD stashed at my parents place should do the trick.

Now to devise some sort of fire, dust, air, moisture, etc. proof container for them.

glacierdave
24th October 2010, 01:37 PM
There's an app for that. :)

Well, hardware anyway.

Check out ioSafe - Internal and External Hard Drives and Network Storage (NAS) Solutions | Fireproof | Waterproof (http://www.iosafe.com/) for serious protection - check out YouTube for some videos - there's some videos of these drives being charred, abused, etc.

decryption
24th October 2010, 02:10 PM
Any Australian sellers? It seems the iosafe website won't ship to Australia :(

glacierdave
24th October 2010, 07:44 PM
Any Australian sellers? It seems the iosafe website won't ship to Australia :(

I thought I got something from a local disti a little while back (that's how I knew about the product) - I'll see what I can find for ya.

kungfucamel
24th October 2010, 09:55 PM
Have a look at the Amazon S3 services - something like 15c a gb and they usually have a pretty good transfer allowance as well. With that amount of data you can also put it on a cheap external drive, ship it to them so they can put the initial data on the server without transferring and eating up your data limit. Amazon also just opened up a data warehouse in Singapore so the speeds should be decent.

BiRDBRAiN
25th October 2010, 05:45 AM
I agree Amazon S3 is the cheapest online storage solution.

Plus you can mount an S3 store in a few common Mac client apps such as Transmit.

Had a look at the Import/Export (by USB portable drive) option. Looks like they only do it in the US and EU at the moment:
AWS Import/Export (http://aws.amazon.com/importexport/)

decryption
25th October 2010, 07:33 AM
Have a look at the Amazon S3 services - something like 15c a gb and they usually have a pretty good transfer allowance as well. With that amount of data you can also put it on a cheap external drive, ship it to them so they can put the initial data on the server without transferring and eating up your data limit. Amazon also just opened up a data warehouse in Singapore so the speeds should be decent.

It's still $75/month :(

That's $900/year!

ajbeardy
25th October 2010, 08:14 AM
Maybe buy a MacMini and load it up and send to these people, still $35US per month though.

macminicolo: Mac Mini colocation | Mac mini hosting (http://www.macminicolo.net/)


"For an additional fee you can add an external hard drive to sit near your mini. Over the years, we've seen the least complex backup be the most stable. Personally, we use the internal drive as the daily drive. Then, a 500GB BUS-powered external drive partitioned. One partition is the same size as the internal drive and clones each night with SuperDuper. (This is useful incase we need to boot from it right away.) The rest of the drive is used with Time Machine and just runs on its own."

wseries
25th October 2010, 04:31 PM
I'm storing my spare external SATA drives with these cases:

ThinkGeek :: Betamax Style SATA HD Case (http://www.thinkgeek.com/computing/c64d/)

brettc
26th October 2010, 02:01 AM
Any Australian sellers? It seems the iosafe website won't ship to Australia :(
Yup!

Welcome to iosafe Australia (http://www.iosafe.com.au/Australia/)

Disclosure: I work for ioSafe.