NOTE: This article has nothing to do with genealogy. If you are looking for genealogy-related information, I suggest you skip this article. In contrast, if you want to know about the latest technology of methods of safely and securely saving your backed-up files, read on.
Decentralized cloud file storage services, sometimes called cooperative storage clouds, is a new method of storing files in the cloud. Decentralized cloud file storage services have several advantages over the various cloud-based file storage services we have been using for several years (DropBox, BackBlaze, Google Drive, Microsoft OneDrive, iDrive, Amazon Web Services, SpiderOak, Mega, and dozens of other, similar services).
Is there a need for cloud-based file storage services? Absolutely!
Computer users around the world are creating huge quantities of latest information yearly. This year alone, humanity will produce more information than the previously 5,000 years combined! Having one copy of each piece of information is insufficient. Additional backup copies are needed. The demand for storage doesn’t appear to be slowing down.
Cloud storage services of most any sort are great for sharing files with friends and families, keeping documents in sync between all of your devices, restoring files that were deleted accidentally or by hardware malfunction, and so much more. Storing files in the cloud frequently is cheaper than purchasing additional hard drives. In addition, cloud storage protects against in-home and in-office disasters: fires, floods, burglaries, and similar threats.
More than once I have accidentally deleted important files from my desktop computer. Occasionally, a hardware malfunction has deleted a file or two or even an entire hard drive. Restoring files from the cloud has saved me from disaster on a number of occasions and I assume that thousands (or tens of thousands or more) other computer users have similar stories.
Yet, even today’s cloud-based file storage services are not perfect. To be sure, they are more reliable than my own computer(s) but they still are not 100% reliable. For instance, Amazon Drive just announced they are shutting down their cloud-based file storage service (luckily, they provided 17 months’ advanced notice to give users plenty of time to move their files to other services.) Move it or lose it. Several other cloud-based file storage services have suffered with system outages, sometimes for extended periods of time.
Most cloud storage providers use centralized architectures (all files stored in one place or in a number of places all owned by the same company), so data is also susceptible to a single point of failure, limited encryption, and minimum privacy policies—meaning your data’s security and privacy can be compromised.
The single point of failure is a significant weakness in traditional cloud-based file storage services. For instance, the largest Amazon Web Services data center in North America once experienced an extended power outage, resulting in website downtimes and permanently lost data due to hardware failures.
Another concern is data privacy. The most recent and impactful of these was the Equifax hack of 2017, in which almost 150 million customers – about half of the United States population – had their personal identification and credit information compromised in some way.
Of course, a court order from a U.S. Federal court will supply all the data you have stored online to anyone who can justify the reason from the request.
Right now, the majority of data making up the many websites we use every day sits in data warehouses owned by just three companies: Amazon Web Services, Microsoft Azure, and Google Cloud. All three of those companies MUST obey court orders from U.S. courts. We have also often seen these companies suffer blackouts, and swaths of the Web go down for hours — that’s the problem with having single points of failure.
Certainly, there must be a better alternative. Luckily, there is.
Decentralized Cloud Object Storage Networks
A move to decentralized cloud object storage networks appears to be replacing traditional centralized data centers. Not only are these decentralized storage services more reliable and more secure, they also have proven to be much cheaper to operate as well.
Decentralized cloud storage refers to the concept of breaking each file to be stored into smaller pieces, encrypting each piece separately, and then storing the individual pieces in different locations around the world. No one file is every stored completely in one location.
To protect against unplanned outages, each segment of each of every file is stored in multiple locations. If 5 or 10 copies of every encrypted piece of every file are stored in 5 or 10 different locations, the odds of all 5 or 10 becoming unavailable at the same time are remote indeed.
With decentralized cloud storage, a court order is useless. Even the owners and operators of the decentralized cloud storage services are unable to retrieve and read your files. Therefore, they cannot give the files to the courts or to anyone else. The only person who can retrieve and read your files is YOU, the one person who created, encrypted, and broke the file(s) up into small pieces before storing the result in different locations all over the world.
Most of the decentralized cloud storage services already in operation use crowd-sourced file storage spaces. That is, the decentralized cloud storage services themselves typically do not own any file storage space. Instead, they contract out with many different companies and individuals around the world who have extra available storage space available on their various hard drives. When a customer decides to save a file into a the decentralized cloud storage service, the file is first broken into many smaller pieces inside the user’s computer, each piece is encrypted while still inside the user’s computer, then each piece is copied to multiple locations to various locations around the world.
The end user typically is not aware of the location of each segment of every file and usually does not care. In short, “it just works.” (Some of today’s decentralized cloud storage services do have methods of displaying the various storage locations being used.)
Should the end user then later decide to retrieve a file for some reason, each piece is retrieved from the world-wide locations, is decrypted (inside the user’s computer) and then is presented to the user in the identical format that it started with.
IPFS
Most of today’s decentralized cloud storage services use a method of breaking the file(s) into smaller segments, encrypting the file(s) and then distributing them that is called the InterPlanetary File System (I love that name!) or IPFS for short.
The peer-to-peer IPFS retrieves pieces from multiple nodes at once, enabling substantial bandwidth savings. With up to 60% savings for video, IPFS makes it possible to efficiently distribute high volumes of data.
IPFS powers the creation of diversely resilient networks that enable persistent availability — with or without internet backbone connectivity. This means better connectivity for the developing world, during natural disasters, or just when you’re on a flaky coffee shop wi-fi connection.
The average lifespan of a web page is 100 days before it’s gone forever. The medium of our era shouldn’t be this fragile. IPFS makes it simple to set up resilient networks for mirroring data, and thanks to content addressing, files stored using IPFS are automatically versioned. Of course, web pages are not the only thing that should be stored in cloud-based file storage services.
IPFS is a distributed system for storing and accessing files, websites, applications, and data.
What does that mean, exactly? Let’s say you’re doing some research on aardvarks. You might start by visiting the Wikipedia page on aardvarks at: https://en.wikipedia.org/wiki/Aardvark.
When you put that URL in your browser’s address bar, your computer asks one of Wikipedia’s computers, which might be somewhere on the other side of the country (or even the planet), for the aardvark page.
However, that’s not the only option for meeting your aardvark needs! There’s a mirror of Wikipedia stored on IPFS, and you could use that instead. If you use IPFS, your computer asks to get the aardvark page like this: /ipfs/QmXoypizjW3WknFiJnKLwHCnL72vedxjQkDDP1mXWo6uco/wiki/Aardvark.htmlhttps://en.wikipedia.org/wiki/AardvarkTIP
The easiest way to view the above link is by opening it in your browser through an IPFS Gateway. Simply add https://ipfs.io to the start of the above link and you’ll be able to view the page →(opens new window)
IPFS knows how to find that sweet, sweet aardvark information by its contents. The IPFS-ified version of the aardvark info is represented by that string of numbers in the middle of the URL (QmXo…), and instead of asking one of Wikipedia’s computers for the page, your computer uses IPFS to ask lots of computers around the world to share the page with you. It can get your aardvark info from anyone who has it, not just Wikipedia.
And, when you use IPFS, you don’t just download files from someone else — your computer also helps distribute them. When your friend a few blocks away needs the same Wikipedia page, they might be as likely to get it from you as they would from your neighbor or anyone else using IPFS.
IPFS makes this possible for not only web pages but also any kind of file a computer might store, whether it’s a document, an email, a video, or even a database record.
Decentralization
Making it possible to download a file from many locations that aren’t managed by one organization:
Supports a resilient internet. If someone attacks Wikipedia’s web servers or an engineer at Wikipedia makes a big mistake that causes their servers to catch fire, you can still get the same webpages from somewhere else.
Makes it harder to censor content. Because files on IPFS can come from many places, it’s harder for anyone (whether they’re states, corporations, or someone else) to block things. I assume IPFS can help provide ways to circumvent actions like these when they happen.
Can speed up the web when you’re far away or disconnected. If you can retrieve a file from someone nearby instead of hundreds or thousands of miles away, you can often get it faster. This is especially valuable if your community is networked locally but doesn’t have a good connection to the wider internet. (Well-funded organizations with technical expertise do this today by using multiple data centers or CDNs — content distribution networks. IPFS hopes to make this possible for everyone.)
That last point is actually where IPFS gets its full name: the InterPlanetary File System. The goal is to build a system that works across places as disconnected or as far apart as planets. While that’s an idealistic goal, it keeps the IPFS developers working and thinking hard, and almost everything they create in pursuit of that goal is also useful here at home.
IPFS Services
So far, the traditional cloud-based file storage services (Dropbox, Google Drive, iDrive, BackBlaze, OneDrive, SpiderOak, and many others) have NOT switched to decentralized cloud file storage technology. However, a number of new startup services have been formed in recent years that have jumped onto the decentralized cloud file storage technology, including Filecoin, Storj, Barracuda, BitTorrent, MaidSafe and the Safe Network, Oberon, NFT.Storage, and other companies you probably have never heard of are dominating the market. I must admit I have not tried all of them so I cannot make specific recommendation as to which IPFS service is “the best.”
My Experience with Storj
Despite that disclaimer, I can report on one decentralized cloud file storage service. I signed up for an account on Storj.io. I decided to try Storj for three primary reasons:
1. Storj offers 150 gigabytes of FREE storage space to everyone. That is a huge amount of free storage space, much more than most of the company’s competitors.
2. Most of the decentralized cloud file storage services apparently do not have merchant accounts and therefore cannot accept credit card payments. Instead, most of them are paid only by proprietary crypto payments. However, Storj.io is an exception: it offers payments either by crypto currency or by credit card. I pulled out a credit card and signed up. (There is no charge until you exceed 150 gigabytes of FREE storage space. So far, I am still using free storage space although I expect I will soon go over 150 gigabytes of storage space.) Once you go over 150 gigabytes of FREE storage space, the fee is $4.00 (U.S. for every terabyte of stored data although you are only charged for whatever you use that is above and beyond 150 gigabytes.)
3. Storj advertises that they have an (Amazon) S3-compatible gateway. This means if you already have or will obtain software that communicates with Amazon S3, it can be reconfigured to work with Storj. There are many S3-compatible products in the marketplace. Admittedly, I have not use the S3-compatible capabilities. However, I have used Arq (a popular backup product that I already owned that works with Microsoft Windows and with Macintosh and now with Storj.) Arq makes it easy to automate backups of part or all of my Mac. It makes backups automatically in the middle of the night or at times that I am not at home. You can learn more about ARQ at https://www.arqbackup.com/
So far, I have backed up nearly 100 gigabytes of data and the software has worked flawlessly. I have nearly 2 gigabytes of data on my Mac so I expect to expand the size of my backups soon.
Summation
So far, I am very happy with Storj.io and its implementation of the InterPlanetary File System. It works well, automatically (even if I am not at home), and very securely. It installed quickly and easily. No surprises.
I don’t have any method of testing the security but I believe it is top-notch. I believe that not even the CIA or the Russian government can hack into my files. Nor can the biggest threat of all: Facebook. The fact that Storj is much cheaper than the other cloud-based file storage services I have tried simply is even more enticing.
If you would like to learn more about Storj or even download the product and try it yourself, go to https://www.storj.io/. Can you use 150 gigabytes of FREE file storage space?