NOTE: This article is not about any of the “normal” topics of this newsletter: genealogy, history, current affairs, DNA, and related topics. However, it is a method of using the latest technology, increasing the security of your information, and of saving money, so I decided to publish the information here.
What Is Storj and Why Should I Care?
Simple answer: Storj is an open source, decentralized file storage solution that is cloud-based, easy-to-use, and inexpensive. It stores files “off site” (and optionally makes them available to others) for its customers.
Answer from Wikipedia
Storj is a decentralized storage platform that uses a cooperative storage cloud to store data on many different nodes around the world. The company was founded in 2014.
Both of the above explanations are correct but do not properly explain what StorJ is nor why anyone would be tempted to use the company’s services.
A longer and more complete description
Cloud storage of files has become a very popular service in recent years, attracting companies like Dropbox, Google Drive, Apple iCloud Drive, Microsoft Azure, BackBlaze, SpiderOak, Sync, Mega, pCloud, and several dozen others. Prices vary widely and most of these services will provide a small amount of free file space as enough for you to test it.
Most of the cloud-based file storage services will store files originating in all the popular computer operating systems, including Windows, Macintosh, Linux, Chromebooks, Android, and iOS.
Almost all these services operate by building large data centers and filling them with hundreds or even thousands of file servers as places to store their customers’ files. Almost all these services (with a few exceptions) must purchase the data centers, the servers, and all the related hardware. For most of these companies, the expenses for all this is measured in the millions of U.S. dollars, an expense that must be passed on to their customers.
Of course, a disaster (fire, flood, earthquake, tornado, or wars (such as a data center in the middle of the Russia-Ukraine war zone)) in one of these data centers will have a huge impact to the company’s business. The bigger cloud-based file storage services solve this by building multiple data centers in multiple countries spread all over the world. The file servers in any one location will periodically copy their stored files to servers in a distant data centers located in another part of the world. Expenses in those cases may go into the billions of dollars.
In contrast, Storj has a different business model.
Storj (pronounced as “storage”) does not own the majority of file servers. In fact, Storj claims on it’s web site at https://www.storj.io/how-it-works: “We don’t own or operate a single data center.”
So how does a cloud file storage service store customers’ data in the cloud without owning “or operat(ing) a single data center?”
Instead of owning their own (expensive) data center, Storj contracts the file storage space out to both large corporations and to private individuals who have sufficient storage space available for use by Storj. The Storj company then compensates the storage space providers for the space used. Since Storj does not have to spend millions (or billions) of dollars to purchase the servers and data centers, the savings are passed on to end users. Storj’s fees are amongst the lowest in the industry.
Storj runs on the Interplanetary File System, usually abbreviated to IPFS. (You gotta love that name: Interplanetary File System!)
Instead of uploading your files to a single server, the IPFS (and including Storj) breaks each upload down into 80 or more distinct pieces (called “sharding”) that get copied to 80 or more distinct storage points throughout the world. These “distinct storage points throughout the world” are owned by other people and companies, not by Storj.
The InterPlanetary File System (IPFS) is a protocol, hypermedia and file sharing peer-to-peer network for storing (and optionally sharing) data in a distributed file system.
Your files are therefore first encrypted, then broken up into 80 or more hard-to-trace fragments and all the fragments of all the files are distributed across the globe, making for one of the safest and most secure content-upload suites available. Any potential hackers cannot find and decode your entire file(s) simply because every file is broken up into 80 or more distinct pieces and stored in different locations. One piece might be stored in Singapore, another piece in Poughkeepsie, one in Caribou, Maine, still another piece in Rio de Janeiro, and the remaining 76 or more pieces stored in still other locations.
Not only are your files safe from government spies, hackers, and others, even the employees of Storj are unable to decode any of your encrypted and distributed files. (There is a method of NOT encrypting your files, should you wish to not do so. However, that remains under the control of the owner of each file at all times.)
IPFS allows users to host and receive content in a manner similar to BitTorrent. As opposed to a centrally located server, IPFS is built around a decentralized system of user-operators who hold a portion of the overall data, creating a resilient system of file storage and sharing. Any user in the network can serve a file, and other peers in the network can find and request that content from any node who has it. If any one (or more) file servers in the network are unreachable, the “missing piece” of the files is quickly supplied from other servers in the network that have redundant copies of the file being retrieved.
All of this file splitting and multi-server file retrieval is invisible to the end user. For instance, he or she simply requests a normally-addressed “http” file in his or her web browser and it appears (often in less time) in his or her normal web browser. Of course, the user storing the file(s) is not limited to World Wide Web-formatted (http) files; each file can be any standard file format such as DOC, PDF, EXE, or any other file format.
You can read more about the Interplanetary File System (IPFS) at: https://ipfs.tech.
The use of IPFS (and therefore Storj) files has many advantages. The more obvious advantage is file redundancy: files can be retrieved even if a significant number of file servers are off-line and therefore inoperative.
Traditional cloud storage solutions, like Dropbox or Google Drive have limitations. While files are backed up redundantly, bandwidth from a traditional data center or unexpected outages can restrict access to your files. There’s also the issue of privacy. These companies have control over your files, including the ability to access them with or without your knowledge.
Another advantage is security. With each file in your Storj account broken up into 80 or more separate pieces (shards) and stored in 80 or more different servers located all over the world, a hacker cannot access your files as he or she will never discover all the 80 or more servers needed.
When you need to access a file that you earlier backed up on Storj, the file service only needs 29 of those pieces (“shards”) to reconstruct the file. However, it actually requests 39 pieces to be downloaded simultaneously, using only the fastest 29 pieces that reach you first.
The Storj project uses blockchain and peer-to-peer networks. It distributes the files so redundancy is well established. It also guarantees you’re the only one who can control access to your files: you can make access private (only you can retrieve your files) or make them public (anyone may retrieve your file).
File sharding and redundancy also has numerous advantages when a traditional server becomes overloaded. Normal file servers, such as HTTP, downloads files from one server at a time — but peer-to-peer IPFS retrieves pieces from multiple nodes at once, enabling substantial bandwidth savings. IPFS makes it possible to efficiently distribute high volumes of data without duplication. If Storj needs more than 80 different widely-distributed servers because of system loads, the many file segments are automatically copied to additional servers.
Another major advantage is anti-censorship: No one controls IPFS, not a government and not any individual. For instance, during the block of Wikipedia in Turkey a few years ago, IPFS was used to create a mirror of Wikipedia, which allowed access to archived static Wikipedia content despite the ban. Even if Wikipedia’s primary (non-IPFS) file servers were censored by the Turkish government and even knocked off-line, the mirrors on IPFS continued to serve the (non-censored) web pages. The mirror has now been expanded to more languages, such as English, Ukrainian, Russian, Arabic, and Chinese.
IPFS also automatically powers the creation of diversely resilient networks that enable persistent availability — with or without internet backbone connectivity. This means better connectivity for the developing world, during natural disasters, or just when you’re on flaky coffee shop wi-fi. You are never dependent on one backbone network.
Perhaps the biggest advantage of all is pricing: Storj provides up to 25 gigabytes free of charge to anyone who establishes a (free) account on the service. That’s a lot, more space than almost all other cloud-based file storage services (with 1 or 2 exceptions). When your stored files exceed 25 gigabytes, Storj charges $4.00 (US) per terabyte for all stored data exceeding 25 gigabytes. In addition, Storj charges $7.00 (US) per terabyte of downloaded bandwidth costs. Again, not the absolute cheapest but certainly one of the cheapest fee structures in the business. Storj claims that its customers save 80% or more on your cloud storage costs compared to the big cloud providers.
You can read more about Storj’s fees at: https://www.storj.io/pricing.
Files are always available and can be downloaded using multi-layered parallelism (pulling from multiple storage nodes around the globe) with low latency, and higher throughput, ensuring faster recovery.
There is a lot more information about Storj on the company’s web servers (distributed across more than 80 different servers around the world, of course) at: https://www.storj.io.
I have been using Storj for more than a year now and am very pleased with the service. It is easy-to-use and works well. I don’t have any testing equipment to measure download speeds, but my “eyeball measurement system” seems to indicate that it retrieves files at least as fast as any of the bigger cloud-based file storage and retrieval services I have used in the past and possibly a bit faster.
You can learn a lot more about Storj by starting at: https://www.storj.io/.
NOTE: By the way, I am not compensated in any way for writing this article. I am simply a very satisfied user of Storj and plan to continue using it until something even better comes along. The folks at Storj do not know that I am planning to publish this article. In fact, I doubt if the folks at Storj even know who I am.