Data Structures in and on IPFS

ipfs.io
@juanbenet
2015-11-17
DATASTRUCTURES
IN + ON

InfoQ.com: News & Community Site
• 750,000 unique visitors/month
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• News 15-20 / week
• Articles 3-4 / week
• Presentations (videos) 12-15 / week
• Interviews 2-3 / week
• Books 1 / month
Watch the video with slide
synchronization on InfoQ.com!
http://www.infoq.com/presentations
/data-ipfs-ipld

Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide
Presented at QCon San Francisco
www.qconsf.com

ÐΞVCON1
1. IPFS introduction
2. structs in IPFS
3. structs on IPFS

DISTRIBUTED
PERMANENT
MERKLE
THE WEB

Data Structures in and on IPFS

surprise oppression censorship
Human Disasters

NOT AUTHENTICATED
NOT ENCRYPTED
AT REST

PermanentDistributed Safer
SmarterOﬄine
Faster

web
DHT
SFS
routing
network
exchange
merkledag
naming
applications
The Stack

routing
network
exchange
IPLD
naming
applications
IPNS
Bitswap
Kad DHT mDNS
QUIC TCPuTP WebRTC WebSockets
Git ChatVLCEtherpad Bitcoin
CJDNS UDT

routing
network
exchange
IPLD
naming
applications
IPNSDNS Namecoin
Bitswap HTTPBitTorrent
Kad DHTChord mDNSGossip Delegated
FTP
TOR
QUIC TORTCPuTP WebRTC WebSockets
Git BitcoinVLCEtherpad
I2P
I2P
CJDNS UDT
Ethereum
EthNames
Chat

routing
network
exchange
merkldag
naming
applicationsThe Stack
Deﬁning the Data
Moving the Data
Using the Data
IPLD
libp2p
IPNS

routing
network
exchange Bitswap HTTPBitTorrent
FTP
TOR
QUIC TORTCPuTP WebRTC WebSockets I2P
I2P
CJDNS UDT
libp2p - a collection of peer-to-peer protocols
for ﬁnding peers, and connecting to them
for ﬁnding content, and transferring it

libp2p - a collection of peer-to-peer protocols
TCP uTP QUIC SCTP BLE TOR I2PTransports
mDNS
boot
strap
DNS
Kad
DHT
PEX PKIDiscovery
Peer Routing mDNS
Kad
DHT
DNS DVs
mDNS
pub
sub
Kad
DHT
Content Routing
STUN TURN
ICE
Kad
ICE
NAT
Traversal

Transports
Discovery
Peer Routing
NAT Traversal
Content Routing
libp2p

routing
network
exchange
merkledag
naming
applications
IPNSDNS Namecoin
FTP
TOR
Git BitcoinVLCEtherpad Chat
I2P
I2P
CJDNS UDT
INTERNET
OF
DATA

routing
network
exchange
merkledag
naming
applications
IPNSDNS Namecoin
FTP
TOR
Git BitcoinVLCEtherpad Chat
I2P
I2P
CJDNS UDT
INTERNET
OF
DATA STRUCTURES

used by dozens of co’s/org’s

http://10.20.30.40/foo/bar/baz.png
location path
http://example.com/foo/bar/baz.png

you

10.20.30.40
you
HTTP

location path
http://example.com/foo/bar/baz.png
QmW98pJrc6FZ6hash( ) ->

http://
location path
http://
/ipns/example.com/foo/bar/baz.png
/ipfs/QmW98pJrc6FZ6/foo/bar/baz.png
content path

10.20.30.40
you
HTTP

10.20.30.40
you
IPFS

immutable
content addr
MERKLE LINK “URI”

immutable
content addr
MERKLE LINK
fs:/ipfs/QmW98pJrc6FZ6/foo/bar/baz.png
“URI”
ipfs:/ipfs/QmW98pJrc6FZ6/foo/bar/baz.png

/ipns/QmYJPtosPTfoC/foo/bar/baz.png
key name
MAZIERES LINK
mutability
content addr

key name
dns name
/dns/example.com/foo/bar/baz.png
human readability
content addr

has pki based
identity
connects
to others
can be run
as a server
or embedded
in apps
app 
 
can get more
from peers
an IPFS node
can store part
of the dag

ÐΞVCON1
2. structs in IPFS
- Merkle DAGs and Merkle Links
- IPLD: JSON + Merkle-Links
- IPRS: Records
- Mazieres Links for Good
- IPNS: Key Addressing
- DNS and other Human Friendly Names
- Git Versioning with Commits
- unixfs: representing POSIX ﬁles

content path

content path
foo
bar
baz.png
QmW98pJrc6FZ6

content path
foo:HB
HB
HC
HD
bar:HC
baz.png:HDQmW98pJrc6FZ6
foo
bar
baz.png

content path
foo:HB
HB
HC
HD
bar:HC
baz.png:HDQmW98pJrc6FZ6
foo
bar
baz.png
MERKLE LINK

a common hash-chain format
IPLD
for distributed data structures

jbenet @ earth : ~ > ipfs add -r ~/demo/basic
added QmajFHHivh25Qb2cNbnnnEeUe1gDLHX9ta7hs2XKX1vazb basic/cat.jpg
added QmZ7uhHuR3zctKYgiyEntay5xreRyE9NtE8TDQSnxvkE2R basic/foo.jpg
added QmTz3oc4gdpRMKP2sdGUPZTAGRngqjsi99BPoztyP53JMM basic/test/bar
added QmTz3oc4gdpRMKP2sdGUPZTAGRngqjsi99BPoztyP53JMM basic/test/baz/b
added QmYNmQKp6SuaVrpgWRsPTgCQCnpxUYGq76YEKBXuj2N4H6 basic/test/baz/f
added QmX1ebVUtfY11ZCpVmqyE5mDoN62SpLd8eLPpg5GGV1ABt basic/test/baz
added QmYNmQKp6SuaVrpgWRsPTgCQCnpxUYGq76YEKBXuj2N4H6 basic/test/foo
added QmNtpA5TBNqHrKf3cLQ1AiUKXiE4JmUodbG5gXrajg8wdv basic/test
added QmWNj1pTSjbauDHpdyg5HQ26vYcNWnubg1JehmwAE9NnU9 basic/tree-in-cosmos.jpg
added QmbtJUxwLnQ1UF1E8xnjgcZsdGaDdztz7S6rg3rNYcGV9m basic/vm/Makefile
added QmTagEY4NYY8WZkfSWM2xpyH71J7z78Usz2QJRoDaaE8qJ basic/vm/disk.vdi
added QmeLW1ZJPDeLMp7WC4gWTkYvQyAMQECwSVsox3jWdvQpJ5 basic/vm/vbox
added QmbtJUxwLnQ1UF1E8xnjgcZsdGaDdztz7S6rg3rNYcGV9m basic/vm/vm/Makefile
added QmRXgisKJmSSii3UUTn9QqvhKoAAcyt7CqNbFFSpGamB9k basic/vm/vm/disk.vdi
added QmQjgMyaqgouThuGFkZQNNaJpa7QawR4P2yRv7N1Y2ffsE basic/vm/vm
added QmVu5LxhbDo866EmmFJF8zNNfdXvn35FjavhREHrfxWGuA basic/vm
added QmTToTPSAn4y9KRxueN8zHtNye7cupdFstVP3ChJKizpJv basic

jbenet @ earth : ~ > ipfs object get --enc=json QmTToTPSAn4y9KRxueN8zHtNye7cupdFstVP3ChJK
{
"Links": [
{
"Name": "cat.jpg",
"Hash": "QmajFHHivh25Qb2cNbnnnEeUe1gDLHX9ta7hs2XKX1vazb",
"Size": 139795
},
{
"Name": "foo.jpg",
"Hash": "QmZ7uhHuR3zctKYgiyEntay5xreRyE9NtE8TDQSnxvkE2R",
"Size": 1048832
},
{
"Name": "test",
"Hash": "QmNtpA5TBNqHrKf3cLQ1AiUKXiE4JmUodbG5gXrajg8wdv",
"Size": 277
},
{
"Name": "tree-in-cosmos.jpg",

jbenet @ earth : ~ > ipfs object get --enc=protobuf QmTToTPSAn4y9KRxueN8zHtNye7cupdFstVP3
1
" .̧,EfZ;!˄nlr8cat.jpg1
" ,JU>ttX^o)P>LHNfoo.jpg@-
"=D"YЯ
~_[}test<
" weoۡJ
C<H83Ctree-in-cosmos.jpg4-
" pP?ALuzG|Jbvm

IPLD
- merkle-links secure, immutable
- merkle-paths /ipfs/Qmabc…xyz/foo/bar.jpg
- universal nestable URIs
- serialization JSON, PB, XML, RLP
- canonical hashing safe

JSON
{
"username": "@VannevarBush",
"followers": [
"@FredTerman",
"@ClaudeShannon",
"@DougEngelbart",
...
]
}

JSON —> IPLD
{
"username": "@VannevarBush",
"followers": [
"@FredTerman",
"@ClaudeShannon",
"@DougEngelbart",
...
]
}
{
"username": "/ipfs/QmWxCC8BrWSUJR2AJQZu7TE
"followers": [
"/ipfs/QmP8AmEKHHMnaBaQsEJVGs7ZbBSQAxhPn
"/ipfs/QmVDGZWUgo4beprgwcmppoaBXxSainGHd
"/ipfs/QmfVQwj2cyhrKQV1BY1y2nAcQcjiKEkvE
...
]
}

JSON —> IPLD
{
"user": "14321431543254",
"followers": [
"76547653654363",
"85876587647564",
"53476458759856",
...
]
}
{
"user": "/ipfs/QmWxCC8BrWSUJR2AJQZu7TEniAa
"followers": [
"/ipfs/QmP8AmEKHHMnaBaQsEJVGs7ZbBSQAxhPn
"/ipfs/QmVDGZWUgo4beprgwcmppoaBXxSainGHd
"/ipfs/QmfVQwj2cyhrKQV1BY1y2nAcQcjiKEkvE
...
]
}

YML —> IPLD
---
username: @VannevarBush
followers:
- @FredTerman
- @ClaudeShannon
- @DougEngelbart
---
username: /ipfs/QmWxCC8BrWSUJR2AJQZu7TEniAa6
followers:
- /ipfs/QmP8AmEKHHMnaBaQsEJVGs7ZbBSQAxhPn9
- /ipfs/QmVDGZWUgo4beprgwcmppoaBXxSainGHdt
- /ipfs/QmfVQwj2cyhrKQV1BY1y2nAcQcjiKEkvEY

jbenet @ earth : ~ > ipld cat —-fmt json QmWxCC8BrWSUJR2AJQZu7TEniAa6ZinkDLB6zNwBdWTZkL
{
"username": "@VannevarBush"
"name": "Vannevar Bush"
}
Representing Objects

jbenet @ earth : ~ > ipld cat —-fmt json QmWxCC8BrWSUJR2AJQZu7TEniAa6ZinkDLB6zNwBdWTZkL
{
"username": "@VannevarBush"
"name": "Vannevar Bush"
}
jbenet @ earth : ~ > ipld cat —-fmt json QmZqLhjZiFsp7gz1BQ5oBaLY6JegUuKbHyVQY6jFK36KBW
{
"user": {"mlink": "/ipfs/QmWxCC8BrWSUJR2AJQZu7TEniAa6ZinkDLB6zNwBdWTZkL"},
"followers": [
{"mlink": "/ipfs/QmP8AmEKHHMnaBaQsEJVGs7ZbBSQAxhPn9zP2jHQy6Uekj"},
{"mlink": "/ipfs/QmVDGZWUgo4beprgwcmppoaBXxSainGHdtBfhcXXqHUV5Y"},
{"mlink": "/ipfs/QmfVQwj2cyhrKQV1BY1y2nAcQcjiKEkvEYJDGwWFUHcHaQ"},
...
]
}
Representing Objects

jbenet @ earth : ~ > ipld cat —-fmt json QmZqLhjZiFsp7gz1BQ5oBaLY6JegUuKbHyVQY6jFK36KBW
{
"user": {"mlink": "/ipfs/QmWxCC8BrWSUJR2AJQZu7TEniAa6ZinkDLB6zNwBdWTZkL"},
"followers": [
...
]
}
jbenet @ earth : ~ > ipld cat —-fmt json
QmZqLhjZiFsp7gz1BQ5oBaLY6JegUuKbHyVQY6jFK36KBW/followers
[
...
]
Merkle Path Traversals

QmZqLhjZiFsp7gz1BQ5oBaLY6JegUuKbHyVQY6jFK36KBW/followers
[
...
]
QmZqLhjZiFsp7gz1BQ5oBaLY6JegUuKbHyVQY6jFK36KBW/followers/0
{
"username": "@FredTerman"
"name": "Fred Terman"
}
Merkle Path Traversals

jbenet @ earth : ~ > ipld cat —-fmt json $dir
{
"foo": {
"mlink": "/ipfs/QmP8AmEKHHMnaBaQsEJVGs7ZbBSQAxhPn9zP2jHQy6Uekj"
"mode": "0755",
"owner": "jbenet"
},
"cat.jpg": {
"mlink": "/ipfs/QmVDGZWUgo4beprgwcmppoaBXxSainGHdtBfhcXXqHUV5Y"
"mode": "0644",
"owner": "jbenet"
},
"doge.jpg": {
"mlink": "/ipfs/QmfVQwj2cyhrKQV1BY1y2nAcQcjiKEkvEYJDGwWFUHcHaQ",
"mode": "0644",
"owner": "jbenet"
}
}
Link Properties

jbenet @ earth : ~ > ipld cat —-fmt json $dir
{
"foo": {
"mlink": "/ipfs/QmP8AmEKHHMnaBaQsEJVGs7ZbBSQAxhPn9zP2jHQy6Uekj"
"mode": "0755",
"owner": "jbenet"
},
"cat.jpg": {
"mlink": "/ipfs/QmVDGZWUgo4beprgwcmppoaBXxSainGHdtBfhcXXqHUV5Y"
"mode": "0644",
"owner": "jbenet"
},
"doge.jpg": {
"mlink": "/ipfs/QmfVQwj2cyhrKQV1BY1y2nAcQcjiKEkvEYJDGwWFUHcHaQ",
"mode": "0644",
"owner": "jbenet"
}
}
ﬁle systems - dir

jbenet @ earth : ~ > ipld cat —-fmt yml $gitcommit
---
tree: {mlink: e4647147e940e2fab134e7f3d8a40c2022cb36f3}
parents:
- {mlink: /ipfs/QmP8AmEKHHMnaBaQsEJVGs7ZbBSQAxhPn9zP2jHQy6Uekj}
- {mlink: /ipfs/QmfVQwj2cyhrKQV1BY1y2nAcQcjiKEkvEYJDGwWFUHcHaQ}
author:
mlink: /ipfs/QmVoXUXmES4ujTdJ9CQAs8D3jMag2enjtBn2towYHmKxYa
name: Juan Batiz-Benet
email: juan@benet.ai
time: "1435398707 -0700"
committer:
mlink: /ipfs/QmVoXUXmES4ujTdJ9CQAs8D3jMag2enjtBn2towYHmKxYa
name: Juan Batiz-Benet
email: juan@benet.ai
time: "1435398707 -0700"
message: "Merge pull request #7 from ipldnn(WIP) records + merkledag specs"
versioning data structures - git

user: @juanbenet
name: Juan Benet
user: @daviddias
name: David Dias
@juanbenet
follows
@daviddias
user: @timbl
name: Tim Berners-Lee
@juanbenet
follows
@timbl
@juanbenet follows
WEBAPP DATA

AUTHENTICATED
WEBAPP DATA
user: @juanbenet
name: Juan Benet
user: @daviddias
name: David Dias
@juanbenet
follows
@daviddias
user: @timbl
name: Tim Berners-Lee
@juanbenet
follows
@timbl
@juanbenet follows
signed by @juanbenet

Contract
LEGAL RECORDS
Signed Contract
signed by “Bill”
signed by “Joe”
Joe’s Legal Agreements
Secure Timestamp

DNS
is used for all
sorts of things

DNS
is used for all
sorts of things
type: A
sub: subdomain
value: 104.236.176.52
TTL: 180s

IPRS
is used for all
sorts of things
value: QmW98pJrc6FZ6
validity: _____________
key: QmYJPtosPTfoC
sig: Qme8K1jxR7GPG

validity: _____________
key: QmYJPtosPTfoC
sig: Qme8K1jxR7GPG
IPRS
is used for all
sorts of things
Records
are just IPLD objects
that link to other objects
value

validity: _____________
key: QmYJPtosPTfoC
sig: Qme8K1jxR7GPG
IPRS
is used for all
sorts of things
Records
value
key
public key
used to sign

IPRS Validity
- App-speciﬁc value
- Timestamps + TTL
- Cryptographic Freshness
- Ancestry (hash chain)
IPRS
is used for all
sorts of things
validity: _____________
key: QmYJPtosPTfoC
sig: Qme8K1jxR7GPG
Records

QmYJPtosPTfoC
public
key
secret
key
QmW98pJrc6FZ6

public
key
QmW98pJrc6FZ6
secret
key
QmYJPtosPTfoC
QmW98pJrc6FZ6

public
key
QmW98pJrc6FZ6
secret
key
QmYJPtosPTfoC
signed by
QmYJPtosPTfoC
QmW98pJrc6FZ6

/ipfs/QmW98pJrc6FZ6/foo/bar/baz.pngcontent addr
/ipns/jbenet.eth/foo/bar/baz.png
key name
human readability
/eth/jbenet/foo/bar/baz.png
eth name

public
key
QmW98pJrc6FZ6
secret
key
QmYJPtosPTfoC
key: QmYJPtosPTfoC
validity: _____________
sig: Qme8K1jxR7GPG
IPRS Record

jbenet @ earth : ~ > dig TXT ipfs.io
; <<>> DiG 9.8.3-P1 <<>> TXT ipfs.io
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 37227
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;ipfs.io. IN TXT
;; ANSWER SECTION:
ipfs.io. 120 IN TXT "dnslink=/ipfs/QmTgNJEgQaCqRht9KSXNyZsCp2xpHZmBRms28NRMmtcERp"
;; Query time: 169 msec
;; SERVER: 209.244.0.3#53(209.244.0.3)
;; WHEN: Tue Nov 17 16:44:28 2015
;; MSG SIZE rcvd: 98
DNS Links

DNS Links

DNS Links
/ipns/ipfs.io/foo/bar/baz.png
dns name
/dns/ipfs.io/foo/bar/baz.png

DNS Links
dns name
∞
60s-5m

ipfs.io. 120 IN TXT "dnslink=/ipns/QmYJPtosPTfoCht9KSXNyZsCp2xpHZmBRms28NRMmtcERp"
DNS Links
dns name
key name
content addr
60s-5m
∞

ipfs.io. 120 IN TXT "dnslink=/ipns/QmYJPtosPTfoCht9KSXNyZsCp2xpHZmBRms28NRMmtcERp"
DNS Links
dns name
key name
content addr
∞
1ms-1s
60s-5m

2. structs in IPFS
- Merkle DAGs and Merkle Links
- IPLD: JSON + Merkle-Links
- IPRS: Records
- Mazieres Links for Good
- IPNS: Key Addressing
- DNS and other Human Friendly Names
- Git Versioning with Commits
- unixfs: representing POSIX ﬁles

versioning data structures - git

unixfs: representing POSIX ﬁles

which link
to others
directories are
also dag nodes
files are dag nodes
BIG files may be
split into many
unixfs as a dag
unixfs: representing POSIX files

ÐΞVCON1
3. structs on IPFS
- unixfs: Files as you know them
- commit: Git style versioning
- CRDTs: sane mutable data
- keychain: PKI and crypto artifacts
- persona: identity management
- post: sane communication
- html: hey! websites!
- boards: forums, reddit, 4chan
- starship: containers at hyperspeed
- blockchains: interconnected

CRDTs: lattices as IPFS objects

Keychain - cryptographic artifacts as IPFS objects
parent: QmPARENT1
key fmt: _____________
keydata: _____________
parent: QmPARENT2
key fmt: _____________
keydata: _____________

parent: QmPARENT1
key fmt: _____________
keydata: _____________
parent: QmPARENT2
key fmt: _____________
keydata: _____________
parent: 0
key fmt: _____________
keydata: _____________

for CA system

for Web of Trust

3. structs on IPFS
- unixfs: Files as you know them
- commit: Git style versioning
- CRDTs: sane mutable data
- keychain: PKI and crypto artifacts
- persona: identity management
- post: sane communication
- html: hey! websites!
- boards: forums, reddit, 4chan
- starship: containers at hyperspeed
- blockchains: interconnected

Watch the video with slide synchronization on
InfoQ.com!
http://www.infoq.com/presentations/data-
ipfs-ipld

Data Structures in and on IPFS

More Related Content

What's hot (20)

Similar to Data Structures in and on IPFS (20)

More from C4Media (20)

Recently uploaded (20)

Data Structures in and on IPFS