A rough proposal to implement the widely desired pub/sub feature on ipfs without requiring additional capabilities or message-passing behavior from the network or the clients. It could be implemented as an IPFS application right now.
I've just started diving into IPFS so this could be a super naïve approach.
The proposal's core concepts:
- subscription is modeled as querying a non-existent block whose future multihash is known, but has not been added to the DAG. Once this block exists, its payload will use IPNS to provide a forward linked list to one or more future blocks in the publication stream.
- publication is modeled as publishing the block with that desired multihash, and providing a payload that links forward to one or more future blocks in the publication stream.
The proposal's key properties:
- It requires the IPNS router so it's not the fastest ride in town.
- It requires that clients patiently wait for
wantlist
blocks that can't currently found on the network- I believe this is the current behavior of
go-ipfs
- I believe this is the current behavior of
- It is loosely inspired by mutable value chains
- It requires no additional capabilities or message-passing behavior from any actor in the network.
The proposal's minimum necessary flow (or just jump down to the example):
- PUBLISHER creates a PUBLICATION EVENT json object
- The link contains a reference to an
/ipns/
address under PUBLISHER's control. - The reference will contain the future payload for the PUBLICATION EVENT.
- henceforth
<PublicationEvent.PayloadLocation>
- henceforth
- The link contains a sequence ID unique in this stream (e.g. "1")
- In reality we will want to hash this with the private key to avoid spoofing
- The link contains a reference to an
- PUBLISHER calculates the PUBLICATION EVENT multihash, but does not add it to the DAG
- Uses
ipfs add --only-hash
- Henceforth
<PublicationEvent.MultiHash>
- Uses
- PUBLISHER distributes
<PublicationEvent.MultiHash>
widely - SUBSCRIBER adds
<PublicationEvent.MultiHash>
to hiswantlist
. - SUBSCRIBER waits for the object to exist in the DAG.
- PUBLISHER writes content and adds it to the DAG.
- It has ID
<Content.MultiHash>
- Nobody knows about it yet but PUBLISHER.
- It has ID
- PUBLISHER creates a tree/folder at
<PublicationEvent.PayloadLocation>
under his IPNS root.- Henceforth
PayloadTree
- Henceforth
- PUBLISHER writes
<Content.Multihash>
intoPayloadTree/data
- PUBLISHER calculates the hash for a NEW PUBLICATION EVENT using the same process as before
- This link now has sequence ID 2
- It should probably also contain an ipfs (not ipns) backlink to
PayloadTree
- Henceforth
<NewPublicationEvent.MultiHash>
- PUBLISHER writes
<NewPublicationEvent.MultiHash
intoPayloadTree/next
- PUBLISHER publishes the new version of his IPNS root.
- This new version contains the payload we just prepared
- The first publication event's payload is now ready to consume, but nobody knows how to find it.
- PUBLISHER adds and pins the first PUBLICATION EVENT json object to the DAG.
- The publication event (which is really more of a notification) starts percolating through the network as it satisfies subscribers wantlists.
- SUBSCRIBER, who has patiently waited
<PublicationEvent.MultiHash>
in hiswantlist
for ages, receives and re-pins the publication event- Repinning helps other subscribers.
- SUBSCRIBER dereferences
<PublicationEvent.PayloadLocation>
, finding both the content and the next publication event withinPayloadTree/data
andPayloadTree/next
respectively. - SUBSCRIBER acquires the content using
ipfs get
- SUBSCRIBER adds
<NewPublicationEvent.Multihash>
to hiswantlist
, and waits.
Example of basic necessary flow:
Publisher (Alice) |
---|
"Think I'll start a blog. Let me give the world a publication event for when the first post is done." |
// Alice creates a publication event: this JSON content.
// She saves it to ~/first-publication.json
{
"payloadLocation":"/ipns/<Alice.PeerId>/ipfs-pubsub/blog/1",
"publishedBy":["<Alice.PeerId>"],
"sequenceId":1
}
// sequenceId numeric for simplicity. Should probably be Sign(1, PrivateKey) or some other scheme
# Alice discovers the hash of this publication event.
# She does not add it to the DAG.
alice~> ipfs add --only-hash ~/first-publication.json
added <FirstPublication.MultiHash> ~/first-publication.json # Not true! We didn't add it, we just discovered the hash.
Publisher (Alice) | Subscriber (Bob) |
---|---|
"Hey subscribe to my blog at <FirstPublication.MultiHash> !" |
|
"Sure, Alice!" | |
bob~> ipfs object get <FirstPublication.MultiHash> |
|
Prepares to wait the long haul. Distributes this object on his wantlist, as does every other future subscriber to Alice. |
Publisher (Alice) |
---|
"Time for the first post!" |
# Alice writes and adds her first post to the DAG.
# Nobody but her knows about the published content.
alice~> echo "I ate shit for breakfast" > post1.txt && ipfs add post1.txt
added <Post1.MultiHash> post1.txt
# Alice prepares the first publication event's payload.
# She chose her blog's multihash as the payload's data.
alice~> PAYLOAD_DIR=$IPNS_ROOT/<Alice.PeerId>/ipfs-pubsub/blog/1
alice~> mkdir -p $PAYLOAD_DIR
alice~> echo "<Post1.MultiHash>" > $PAYLOAD_DIR/data
// Before publishing the first publication, she needs to prepare the second publication event
// so that her followers will know when she posts again!
// She creates the following json and saves it to ~/second-publication.json
{
"payloadLocation":"/ipns/<Alice.PeerId>/ipfs-pubsub/blog/2",
"publishedBy":["<Alice.PeerId>"],
"sequenceId":2
}
# Alice calculates the new publication event's hash, and posts that hash as the "next" reference
# in the first event's payload. Now we have a forward linked list!
alice~> ipfs add --only-hash ~/second-publication.json
added <SecondPublication.MultiHash> ~/second-publication
alice~> echo "<SecondPublication.MultiHash>" > $PAYLOAD_DIR/next
# The message payload for the first publication is ready to go.
# Alice updates her IPNS in preparation of broadcasting her blog post
alice~> ipfs add --recursive $IPNS_ROOT
added <NewIpnsRoot.MultiHash>
alice~> ipfs name publish <NewIpnsRoot.MultiHash>
# Now she finally publishes to her subscribers, satisfying the wantlist of her many blog followers and notifying
# them that content has updated
alice~> ipfs add --pin ~/first-publication.json
Subscriber (Bob) |
---|
"Oh look, we finally received FirstPublication.MultiHash. I guess Alice finally posted!" |
"Well let's look at the content." |
# Now that Bob has the contents of first-publication.json, he can go get the payload. He does this by dereferencing
# the payloadLocation of publication event he just received, and grabbing the "data" link from that tree.
# You'll recall that the "data" link contained the ID to the actual blog content, and the "next" link
# contained the future ID for the next publication event.
bob~> ipfs cat <FirstPublication.payloadLocation>/data | ipfs cat
I ate shit for breakfast
# "Gross. I wonder what she'll say next. Let's subscribe to the next one."
bob~> ipfs get <FirstPublication.payloadLocation>/next | ipfs get
# This will block until Alice writes her next blog post