Toward More Reliable Long-Term Data Storage

Always keep a backup.

That's usually the first thing a blog says just after your hard drive crashes and you're looking up options for recovering it. Usually, you're scrolling through on your phone, an old laptop, or on a old computer that inexplicably starts smoking the moment you turn it on.

As usual, you'll take the customary 3 to 10 days to recover, which should, of course, only take about 3 to 10 minutes, and start realizing you'll never really know what or how much data you just lost again, for the 25th year in a row.

Rummaging around, you're shocked to find out how much has survived. Much of it on the cloud you know you shouldn't be relying on.

But in addition, you'll start seriously considering a real upgrade to a more reliable long-term "hot storage", meaning storage that turned on and ready when you are, able to access it at a keystroke. Cold storage is the stack of DVDs that should back up important things for you, but in reality, have never been used to restore the merest speck of data in decades.

There Are No Good Options

When it comes to crashes, costs, and data reliability in the real world... Windows sucks. Apple sucks. Linux also sucks. But each for vastly different reasons.

In theory, you should be able to easily save all your user files from a typical Linux setup. In practice, the average user will never in 100 lifetimes succeed in doing so. Enter the external purpose-built device. External storage.

It's fine to have those little external USB storage drives. They're cheap. They're visible on your home network. You'll successfully limp along for years, just like you do on that tire that has a slow air leak. Must be gangstalking! (Nah. That would require a worldwide conspiracy of millions.)

An external USB drive is pretty good, because the operating system probably won't kill what's on there. I briefly considered tape. Briefly, but very, very seriously. Those tapes can be stored for a long time. In fact, as soon as I bought them, I stored them on my shelf and haven't opened them since.

What's stored on them will survive 10 Microsoft-caused crashes in a row, or user-caused crashes with Linux. The biggest problem with the system is nothing is stored on those tapes.

Welcome to the real world.

Scalable, Future-Proof Data Storage for the Common Man

Why do they only give you one hard drive in a computer? For any kind of data reliability, you need at least two. Preferably three or four.

The last time I tried to put together redundant drive system in a RAID configuration, I had three disk failures at roughly the same time. And quickly. They heated each other up, vibrated themselves apart, and melted each other down. They were name brand drives, but evidently I'd gotten the wrong kind.

They are now paperweights.

The results:

A total file system failure in less time than ever before.

In other words, the exact opposite of what I was trying to achieve. With a fully modern file system, the data would have detected the failure and migrated the data to the safest place. Even on cheap drives, the data probably would still be safe, completely intact years later, having surviving one or two simultaneous drive failures out of four.

With a more modern file system, I could toss a 4 terabyte drive into the pool to coexist with the smaller drives.

I mean, in the real world, that's what happens, isn't it? You had 500 gigabyte drives a while back. Then you upgraded to 1 terabyte drives. Then 2 terabyte, then 4. You shouldn't have to buy 4 identical new drives every time you upgrade your data pool.

How long have we lived in the real world now? Shouldn't the engineers have figured this out yet? That replacing every drive in a data center for every upgrade is a very stupid, wasteful idea?

With the right kind of file system, you should be able to chuck the big, new drive in there with the smaller ones and immediately see improvements in the reliability and size of your pooled data.

Speed and Convenience!

I don't know about you, but I'd rather install an operating system once, install a program once and have it forever. Not five times. This is horribly inefficient. It strains all the networks.

Once I finally have something set up the way I like it, it has no problems, no bugs, not bogged down, I don't want to have to rebuild it every couple of years for the rest of my life to get it to keep running smoothly.

If something breaks, I want do-overs. I want an undo button for my operating system.

And if I get a fast drive, I want my system to take full advantage of it and load the fast things fast and the slow things even faster. I want every Adobe product to load like it was stored on a RAM disk. I've got the memory. If I've got memory to spare, why isn't the file system using it to speed things up for me?

And if you've got a new technology, like the faster SSDs, your file system should be able to take advantage of its speed by caching frequently accessed data there for you. That fast new drive should be packed full of your most frequently accessed data.

The Hammer Test

The data storage should pass the hammer test. If my angry ex girlfriend hit any given drive or component with a hammer, would I instantly lose a significant chunk of my life's work? If yes, my data storage solution fails the hammer test.

I want to be able to recover and get back to work almost as fast as I can plug in a new hard drive.

Optics

CDs, then DVDs offered virtually unlimited storage! Which is true. But it's 212 DVDs per terabyte. These days, you might have 1 to 4 terabytes or more per drive, per device. With more data pouring in from your ultra high def smartphone cameras recording in slow motion at 60 frames per second, plus your "real" camera the a YouTube and Instagram era where daily uploads of video are becoming much more common.

If you're a data hoarder who backs up that kind of data, that's a pretty big stack of DVD backups.

Oh, and you've got to keep all three original camera angles with their original data rate in case you want to do color correction later or use AI to upscale the data to 8K in a few years.

YouTubers are frequently storing more than 100 terabytes of their original videos. And if they try to "back up" everything by uploading it all to YouTube, our enemy's itchy delete finger suddenly makes life pretty scary.

They keep changing the rules, and it's impossible for all your old videos to be in compliance.

Controversial dating guru Owen Cook from Real Social Dynamics lost about 10 years of videos that way because YouTube changed its rules about what content was acceptable. He didn't have the staff to go through every single (almost daily) video to ensure it was both compliant today and future-proof tomorrow when YouTube inevitably changes its rules again down the road.

His team flew around the world for a decade at a cost of thousands of dollars per month per instructor so these young men, chronicling their discoveries as they happened could produce useful dating advice for young men.

Now that they're all 10 years older and they're not exactly getting any younger, that entire decade from all those guys will probably be lost to history. For some of them, it's simply a matter of time.

But he's philosophical about it, partly because he's a full-time philosopher. "Whatever."

Every video on that site was a lead magnet. (A piece of content which draws new subscribers over time.) Now he'll lose the ongoing audience for those videos.

Maybe he could recoup some of those losses if he had a complete back-up of every RSD video ever produced. I'm sure members would pay a monthly subscription fee to peer into that vault of older videos that are now "too hot for YouTube." Which they are.

No vault? Too bad. Total loss. How do you download 1,000 terabytes of video with one week's notice? You don't. The network doesn't even move data that fast.

Best case scenario, to download a library of that size would take 104 days over a gigabit connection or...

It Would Take 6.4 years To Download Your Data Over a T3 Connection.

They've made our civilization highly dependent on their data centers. 90 days notice isn't nearly enough. So what do they do? Crash your hard drive with every 10th Windows update, apparently. Forced reboots while you're in the middle of something. You just wake up one morning and it's gone. All gone.

Thank you for updating Windows.

Not that I'm complaining. Before the printing press, you had to re-copy everything by hand if you wanted a backup.

And a couple thousand years ago, you had to be crucified before people would spread the word.

But many great video producers have died along the way with no idea that YouTube would scrap their all their videos for political reasons.

Their entire legacy, their life and work may only have ever publicly existed on YouTube and that channel is gone the second a social justice warrior flags it because they said, "this forge is as hot as a bitch in heat" in 2009 or "to hammer the slag off, you've got to beat it like a rented negro".

Today, "strang 'em up" could be considered hate speech, if a white person says it. Tomorrow, you're deleted because you didn't glamorize whoring for a living while killing white people in your music video, like Rihanna and Donald Glover do.

Good thing this will never have any affect on present-day Chaz Warlords in the autonomous zone

Toward More Bulletproof, Future-Proof File Storage.

At the time of writing, all factors weighed carefully, ZFS NAS is probably the best, most modern file system in the world, especially if quality components (ECC ram, WD Red drives).

It gives you scalability, speed, efficiency, caching, lets you choose how to balance redundancy with efficiency.

Isn't it RAM-hungry? Depends.

Some like to use more RAM for high-performance situations. Others get by with less. Is your NAS shared on a network with 4 video editors working at the same time? Then you probably want a gigabyte of RAM per terabyte of storage.

Others are perfectly satisfied with SSD-like performance over their gigabit network. The point is, if you want more performance later, you can upgrade it later. No need to overhaul everything. No need to change to another file system.

Halfway between fast ram and slow disk is SSD.

It's a file system good enough for your gigabit home network.

Bill Gates (windows) killed my laptop again with yet another one of its forced updates. When nothing else worked, the final solution wiped the drive clean. So frustrating to have the exact same problems 25 years in a row.

Maybe it's time to invest in a modern data storage appliance.