Decentralized Archive Transport
The Dat Protocol
Dat is a new p2p hypermedia protocol. It provides public-key- & sha256-addressed file archives which can be synced securely and browsed on-demand. Dat supports streaming updates and partial on-demand replication, and has plans for versioned URLs and efficient compaction.
| Fast | Archives sync from multiple sources at once. |
|---|---|
| Secure | Updates are signed, SHAed, and forced to be distributed unformly. |
| Resilient | Archives can change hosts without changing their URLs. |
| Versioned | Changes are written to an append-only log. |
| Decentralized | Any device can host any archive. |
Dat is funded by grants by the Knight and Sloan Foundations. Learn more at the Dat Project.
» Join us in #dat on freenode. » File issues for this website here.
Documentation
Dat Specification
Draft, in progress. A complete specification of the Dat protocol.
Guide: Using Dat from the command-line
How to create and download Dats using the shell.
Guide: Hosting Archives from a Server
Use the Dat CLI and a process-manager to rehost archives from a VPS.
Applications
Dat Command-line
Sync and share folders from the command-line.
Dat Desktop
A simple, GUI-based client for dat archives.
Beaker Browser
An experimental P2P browser, which uses Dat as an alternative to HTTP.
About
Dat uses rabin file-chunking and append-only feeds verified by merkle trees to securely distribute mutable files with "live" update-pushing through a peer-to-peer network. Peers are discovered using multiple overlapping networks (DHT, DNS, mUDP) and contacted via TCP or UTP.
Archives are identified by ed25519 public-keys or sha256 hashes encoded as 64-character hex strings, making the URLs hostless and globally unique.
The site you are currently viewing is served at dat://ad6cae123c812257eea2c20c92fbb06f6e774f73af671a9b306971a3d77efc5f/.
Dat generalizes concepts from certificate transparency to provide its security guarantees. All updates are written to a flattened merkle-tree (the feed) which is signed by the identifying public-key. Data-feeds are distributed uniformly to all peers, and authors are unable to split the change history or serve alternative versions of an archive without detection.
Modules
Dat is authored in Javascript, and is split into reusable, self-contained modules. The main modules are:
- Dat-node: A easy-to-use, comprehensive library for nodejs which wraps all modules.
- Hypercore: The internal protocol for distributing and replicating feeds of binary data.
- Hyperdrive: The internal protocol for reading and writing files using Hypercore.
- Hyperdiscovery: Join the p2p swarm for Dats.
There are many submodules which comprise Dat:
- Cryptography
- Rabin: Node native addon module (C/C++) for Rabin fingerprinting data streams.
- Merkle-tree-stream: A stream that generates a merkle tree based on the incoming data.
- Sodium-signatures: Sodium keypair sign/verify operations.
- Encoding
- Hyperdrive-encoding: The message encoding and schemas used by hyperdrive.
- Bitfield-rle: A run-length-encoder that compresses bitfields.
- Flat-tree: A series of functions to map a binary tree to a list.
- Protocol-buffers: Protocol Buffers for Node.js
- Uint64be: Encode / decode big endian unsigned 64 bit integers
- Networking
- Discovery-Swarm: Uses the Mainline DHT, DNS Servers, and Multicast UDP to discover peers, and then establishes connections using TCP and UTP.
- Hypercore-protocol: A stream that implements the hypercore network protocol.
- Utp-native: Native bindings for libutp.
- Discovery-channel: Search for a key across multiple discovery networks and find peers who answer.
- Dns-discovery: Discover peers in a distributed system using regular dns and multicast dns.
- Bittorrent-dht: Discover peers with the BitTorrent DHT protocol.
There are also some handy community or high-level modules for handling common Dat tasks:
- Hyperdrive-http: Serve a Dat archive over HTTP.
- Hyperdrive-stats: Live & persistent stats module for Dat.
- Hyperdrive-network-speed: Track upload and download speed on a Dat archive.
- Hyperdrive-import-files: Import the contents of a folder into a Dat, and optionally keep watching for changes.
- Zip-to-hyperdrive: Import a .Zip into a Dat archive.
- Tar-dat: Import a .Tar into a Dat archive.
- Hyperdrive-to-zip-stream: Export a Dat archive into a Zip file.
Why javascript? Portability and a rich ecosystem. As new implementations are written, they will be included on this page. (Some community members have started writing a Rust port.)