Idea: P2P Secure Distributed File System

Disclaimer: I'm posting this on digg so that this idea might reach the masses quickly. I am not aiming for self-promotion, credit, free cash/hardware or anything. I'd just like to get this idea out there.

I was reading this paper on Google's file system and was inspired at the thought. As I was reading I thought to myself, "I should make an implementation of something like that." Then the sobering reality set in that I lack the necessary hardware to even begin to test rolling out something like this. With only 256 MB of memory on my main development box, even using virtualization to simulate more than one PC was out of the question. I could go out and buy new hardware, but I barely have enough money to pay rent, and I greatly dislike credit cards. I quickly decided against online begging, because that's lame.

Then I thought that I may be able to get together with some people and we could pool our resources to test out something like this. Through a bit of brainstorming, I was thinking that we could even make something like this available just between us. A little more brainstorming later, and I had basically come up with the idea for a peer-to-peer network, and thus had come up with nothing. Yet there was still something to it.

Meshing Google's GFS with the concept of online storage with the concept of peer-to-peer networks, I arrived at this: A Peer-to-Peer Secure Distributed File System. One could refer to it as P2PSDFS for long and P2PFS for short. Think of it as SETI@Home in the business of storing your data, instead of searching for aliens.

Limited technical description

To the user this would look like a program serving as a gateway to a personal data store of infinite size. On the technical side of it though, this is what is going on: people run a program on their computer that sends/accepts chunks of random files from users of the network. Idling computers with significant resources would become automatic masters and would register with a central server, or cluster of central servers. I imagine a DNS-esque protocol would manage the metadata for who is who. Anyway, these chunks would serve as parts of files in a large file tree. Though instead of having one large globally accessible file tree as Google does, one could have one a branch set up for each user, where any data they commit to it would be symmetric-key encrypted (client side) with their chosen password before being committed to the network. Replication of file chunks would have to be far greater than Google's system (3 copies per chunk) as people would be going on/off the network far more often than hardware fails. And no one likes their files unavailable.

Dealing with the downsides

With the privacy of files guaranteed through their encryption, this network would *hopefully* (but not entirely realistically) be free from the legal issues of piracy and such. One issue that I can already foresee is the finite nature of the network, as it can only be as fast/large as the CPU/RAM/HD of the computers connected to it, and the connections that serve them. To overcome this barrier, users could be allocated a certain amount of space dependent upon how much HD space/bandwidth they're willing to commit to the network. Users would earn space credits based upon a ratio of space/time that they provide.

Looking at the great benefit

The great benefit of this though is the dream of limitless, universally accessible storage finally realized. Sure, the Gmail service can provide this to an extent, though not with the geek factor this idea holds. Imagine the convenience of a P2PFS server with a web front end designed to fetch people's files for them (assuming they have their password). One could store/retrieve their files in a universally accessible place without dealing with an FTP server (for example).

Wait, how do I get it?

Unfortunately, this exists only in concept as presented here. In theory all the technologies necessary to make this happen are a reality, or are relatively easy¹ to create. I would certainly jump on this myself, but right now any programming I do for roughly the next half-year could become the intellectual property of one of three different companies, if I understand properly that one document I signed.

That said, I'm presenting the idea for public approval and asking some other skilled devs out there to jump on this. I'll even donate my encryption engine², if it will help, so that there is one less component to worry about.

1 Knock on wood.
2 If you're reading through the source code, ignore the hex conversion code as it was definitely written to get more hours out of the project in school, and then ported mindlessly to all other languages.