You may have noticed I've been scattering around some posts with links to IPFS mirrors of files. The point of this is to hopefully offer a link that will last longer than random file sharing sites. I am irritated when those things disappear and the links are no longer good. The hope is that with a content-addressable system that can grab the files from any mirror, links will stay good for the foreseeable future.
A brief IPFS primer:
IPFS is focused on content-addressibility, so the address of a file is just a hash. An IPFS address like
is mostly the hash of the file (see multihash for the specifics). When an address is requested an IPFS client goes out onto the network to find someone who has the file with that hash.
(Technically the hash is not for the file but for the root of a Merkle DAG ultimately ending in blocks of the file.)
An IPFS object can be a directory (a list of other hashes with names), and an address can include the names of files referenced from a directory, such as
IPFS works something like Bittorrent, Gnutella, or Freenet in that there is a peer-to-peer network distributing the files, with a distributed hash table to figure out what file is where. Unlike Freenet there is no automatic replication, though I understand there is a related project Filecoin that intends to provide monetary incentives to replicate data. Currently the official client (go-ipfs) saves everything that it downloads, and you can choose to "pin" a file within a server so that it will not be cleared out automatically (via a "repo gc" command). Any copy in the network is automatically available as a redundant source.
There is a project called Maelstrom that is directly based on Bittorrent which aims at some of the same ideas. Honestly I hadn't looked into it until just now, so I don't know what the benefits might be of using that instead. It seems like it relies a lot on a special browser. A biased comparison of the systems is here.
Freenet is much more mature, but when I last used it years ago it was very slow and I was confused. It has different goals, and I don't think it is going to be useful for distributing large files.
The reason I got onto IPFS is that it was really easy to set up, and they have a nice HTTP gateway that you can use to directly link to files from the regular web. If you have an IPFS address you can prepend https://ipfs.io and they'll retrieve the file from the network for you (and cache it briefly), for instance Neocities' announcement that they will support IPFS.
I can't be reliant on the ipfs.io gateway always being there (and unrestricted), so those links might die while someone is still hosting the files. At that point it would be possible to put up another gateway, all that would need to change is first part of the URL. However it would be somewhat involved to prevent it from being abused. Arguably my use of ipfs.io is abuse already.
I'm going to make another post in this thread explaining how you can get involved with hosting if you're interested. I also welcome discussion about whether this seems like a good idea, and maybe some better ways to go about it.
Each of those hashes points to a "wrapper" object, which is a directory with a single file in it. This allows me to refer to the file by name in a link. You can create these with "ipfs add -w <file>".
A major problem here is space (and another is bandwidth); this doesn't provide any magic solution to storing something like the huge archives at joshw.info. I'm out of work so I can't afford to fund anything that ambitious (arguably I can't afford to run this web site as is...), so I'm being selective about what stuff I save.
There are discussions about archiving via IPFS here which I should read up on.
I'm keeping everything in the above linked list pinned in at least two places, but I'm only leaving one of them online at the moment. You can pin them yourself with:
ipfs pin add -r <hash>
This command will get the list and pin all the files:
(the -r is recursive, so it goes from the wrapper down through the whole tree for the file)
Caveats: - You need to have already set up (with "ipfs init") and started the daemon (with "ipfs daemon") to actually be in touch with the network. - The "pin add" command doesn't provide any status, so it will look like nothing is happening for a long time. - Everything is very experimental and poorly supported. - Please don't pull down all the files with this unless you intend to try to keep a server running.
For installation information for the official client see ipfs.io
Mostly because there is no HTTP gateway. But Bittorrent has some major benefits: - a well-established ecosystem (seedboxes, software) - client seeding is expected
(An irrelevancy:
Magnet torrent links are tricky, as they are actually hashes of the .torrent seed file, not the original data. Thus how exactly the .torrent was generated matters. IPFS hashes also have a few levels of indirection, though. And this doesn't even really matter, as we are assuming someone is going to publish a hash which will be canonical for the file.)
It feels like it would be a misuse of the network, and I think the speed is going to be unacceptable for large files. It would have several advantages, though: - secrecy - automated mirroring - mature ecosystem
---
Question: Why not just throw files up on an HTTP server?
The intent here is for it to be easy to mirror the files, move them from server to server, without the links rotting. Having a content-addressable system lets me (or anyone else) just dump in a list of hashes to set up a mirror.
---
Question: What other systems might be considered?
Tahoe-LAFS GNUnet but I think they have much worse prospects than the others mentioned here.
---
Question: How long do you think you will be able to get away with using the ipfs.io gateway?
I don't know. I'm sure there are major abuses of their infrastructure going on already. They can't be expected to stay up and completely open forever. IPFS isn't very popular yet, but once it gets more exposure they're going to have to get good at banning abuses, and I don't know what is going to look like an abuse to them. They could simply refuse to connect to my IPFS nodes if they think I'm serving up too much large stuff.
---
Question: What happens when the gateway goes away?
The hope is that there will be support in browsers for IPFS, to directly access the IPFS network, but I think that is a bit of a long shot. It is easy to open up access to my own gateway and change the links, but I will have to whitelist just the files I intend to share if things are not to get out of control.
---
Question: How long are you going to keep this up?
Don't know. It may seem like a really bad idea really soon. My next best hope is Bittorrent magnet links. Yes, I know I took down the tracker, but I really wasn't comfortable with hosting it. I'd like to do some research into easily generating trackerless torrents.
Been looking into BitTorrent again, trackerless DHT stuff actually is seeming like a better idea now that I've tried Deluge instead of Transmission; it seems to be able to pick up a torrent from a btih-only magnet link pretty instantly, and it works across to uTorrent as well. Transmission (at least on the machines I'm trying) just doesn't seem to pick up on DHT stuff as well. I don't know if this is just because Deluge and uTorrent are using a better (and common) DHT bootstrap server.
Also it is much more efficient bandwidth-wise than IPFS.
You do have to have a client, but relying on the public IPFS gateway is a bad idea, and it is pretty unstable most of the time.
As an experiment, compare magnet:?xt=urn:btih:151498147d1a0a18db07a1fc1a76b7aacf33e70a&dn=Crypt%20of%20the%20NecroDancer%20v1.03.zip
(or the base32 version which I think is needed for Vuze/Azureus
Yeah, it's the relatively slow start to eventually find stuff on the DHT that is a bit of a pain. I could go with what this magnet link generator does and add a bunch of public trackers to kick things off faster. The idea was to just have the hash, though.
Just ran across this recently, instant.io uses WebTorrent (BitTorrent over WebRTC), seems to work very well. There is also βTorrent with a somewhat nicer interface.
Some issues:
It's really a separate network from normal BitTorrent, also can't use the common DHT, I think. Actually I'm not at all sure how it resolves Magnet BTIH links, this is probably not purely peer to peer, and it is connected to a number of wss:// trackers.
Vuze is supposed to support this, but I wasn't able to get it seeding on WebTorrent so far. Leaving a browser open is a nonstarter, and I'd really like to have one client supporting both WebTorrent and normal BitTorrent. There are other clients as well, but it seems like all of them use an instance of Chrome to run WebRTC.
Everything is in memory so it's probably going to freak out with large files. The very impressive WebTorrent demo loads a 120 MB video with seemingly no issues on my Chromebook, though.