Shared stewardship of the "dweb" protocol handler

[lidel]

Summary

Let's make dweb the common URI namespace shared by all p2p protocols, an experiment proving that a path is a viable canonical address and that all kinds of things with different semantics can live in a shared universal namespace

Initially, dweb could be a simple router that redirects to a dedicated handler or a http-based viewer:

Option 1: Path-based URI

URI that supports Unix-like path addressing

dweb:/protocol/path/to/resource

• dweb:/ssb/<id> → `ssb:// (is there a http-gateway?)
• dweb:/dat/<hash>/file → `dat:// (is there a http-gateway?)
• dweb:/ipfs/<cid>/file → ipfs://<cid>/file (or https://<gateway>/ipfs/<cid>/file)
• dweb:/ipns/<cid>/file → ipns://<cid>/file (or https://<gateway>/ipns/<cid>/file)

Main problem: no Origin isolation without writing custom code in browsers.

Option 2: Origin-based URL

A single protocol handler with custom top level domains (TLD suffixes):

dweb://{id}.{ipfs|ipns|dat|ssb|etc}

dweb://bafybeiemxf5abjwjbikoz4mc3a3dla6ual3jsgpdr4cjr3oz3evfyavhwq.ipfs/
dweb://778f8d955175c92e4ced5e4f5563f69bfec0c86cc6f670352c457943666fe639.dat/
dweb://{hash-in-base32}.ssb/

Main win: Origin isolation for free.
More in #28 (comment)

Motivation

As noted in #23 we are working toward improving support for protocol-specific handlers. We proactively define dweb as an URI namespace shared between all DWeb protocols to:

• avoid situation when a single protocol (or a rogue third-party) grabs dweb for own purposes.
• improve access to Dweb resources experience by making it possible to access dweb content via simple http-based viewers

Open Questions

• Is everyone onboard with this idea?
• How should protocol routing be implemented in short term?
  • Option A: create "The DWeb Protocols" web extension under arewedistributedyet org that is a simple browser extension redirecting DWeb protocols to the best viewer currently available.
    • Everyone can PR against it and add a new handler for a specific path, or propose a change for existing ones.
  • Option B: ?

[Gozala]

@lidel From my prior conversations with dat / ssb folks I'm under impression that they don't care about dweb scheme as dat:// encompasses all of the dat and same for ssb, so it appears to me that uniting multiple protocols ipfs + ipns + ipld? is PL specific so I think you should claim it.

[lidel]

Something that come up in recent conversations: making dweb:// a regular URL (with distinct Origins). Putting it here to see what others think about this approach.

Another take on this: what if we reduce the surface of protocol handlers in places like web browsers by using a single protocol handler with custom top level domains (TLD suffixes)?

dweb://{id}.{ipfs|ipns|dat|ssb|etc}

dweb://bafybeiemxf5abjwjbikoz4mc3a3dla6ual3jsgpdr4cjr3oz3evfyavhwq.ipfs/
dweb://778f8d955175c92e4ced5e4f5563f69bfec0c86cc6f670352c457943666fe639.dat/
dweb://{hash-in-base32}.ssb/

Pros:

• single protocol handler enables frictionless cross-dweb-protocol requests (Protocol Handler: Cross-protocol Content mozilla/libdweb#52)
• each dweb protocol gets own TLD, providing Origin-based isolation

Cons:

• demultiplexing becomes an open problem without native support in the client
  • for now, we could make "dweb:// browser extension" that acts as lazy-install prompt for each protocol on the first time IPFS/DAT/SSB address is used
• SSB addressing is case-sensitive, which makes it impossible to use it in Origin label in URLs (potential fix: adopt CID or at least Base32)

Open questions:

• How to handle TLDs existing in DNS?
  • (A) Don't, make DWEB TLDs explicit and required
  • (B) Check for DNSLink. In the beginning we would check DNS, but ENS or other distributed solution could take a priority in the future. If present, websites could get pretty nice URLs like dweb://example.com

[ntninja]

I'll add my 2¢ here (see this historic discussion for my idealistic background):

[The below makes some assumptions about dat & ssb being similar to IPNS in terms of their URL scheme model, definitely correct me on that if there were any misunderstandings there.]

I think your option 2 is both terribly wrong and totally unnecessary. Staying in the realm of traditional URLing we have strings of the form: <scheme>://<authority>/<name> where <scheme> denotes the protocol used to retrieve the given resource, <authority> denotes whom/what we ask/trust to provide us that resource and <name> denotes the resource we actually want from that source.

In that mental model having (sbb|dat|ipns)://<authority>/<name> as separate schemes makes totally sense. The schemes prescribes what protocol to use to fetch the resource and the authority determines whom to ask for and trust on the result. (The main difference being that the authority is likely also cryptographically validated, rather then just an arbitrary name that somebody you asked said to have some address that happens to end up wherever BGP will have it routed to.)

ipfs://<cid>/<path> is slightly off since the “authority” is just the identity of the block addressed, so there is no real trust involved (if we have any valid value we're done), but in terms of mental representation that's the closest thing you can have for content vs authority addressed data. The creators of the ni://-scheme did something very similar for instance.

The dweb:/<protocol>/<authority>/<resource> style scheme OTOH was created for with a different goal in mind: That of unifying the addressing modes of local and remote files. Whether you agree it's worth it or not, there is absolutely no practical benefit of having

dweb://bafybeiemxf5abjwjbikoz4mc3a3dla6ual3jsgpdr4cjr3oz3evfyavhwq.ipfs/
dweb://778f8d955175c92e4ced5e4f5563f69bfec0c86cc6f670352c457943666fe639.dat/
dweb://{hash-in-base32}.ssb/

over

ipfs://bafybeiemxf5abjwjbikoz4mc3a3dla6ual3jsgpdr4cjr3oz3evfyavhwq/
dat://778f8d955175c92e4ced5e4f5563f69bfec0c86cc6f670352c457943666fe639/
ssb://{hash-in-base32}/

Both are easily classified as different origins by browsers and the second/original version is even shorter and does not claim arbitrary TLDs and also has existing scheme/protocol handler support in browsers and many other frameworks.

The way I see this is: Leave dweb:/ the way it is currently designed (option 1) – if the filesystem unification goal proves unworkable or ultimately uninteresting, then just let this scheme die, rather then repurposing it in weird ways.

[fred-wang]

Hi all,

Firefox implemented the WHATWG's whitelist in https://bugzilla.mozilla.org/show_bug.cgi?id=1476035 so I don't think navigator.registerProtocolHandler support dweb protocols anymore.

Regarding whitelisting of dweb protocols, there has been recent progress at blink-dev and WHATWG. It is proposed that new protocols are registered as provisional URL scheme at IANA: whatwg/html#5482 (comment)

Reports on relevant repositories:

• ipfs/ipns: Register URI with IANA ipfs/in-web-browsers#29
• dat: Consider IANA registration of URI Scheme for dat? dat-ecosystem-archive/DEPs#66
• ssb: Consider IANA registration of URI Scheme for ssb? ssbc/ssb-spec-drafts#4

I'm not really clear what the status/plan for dweb: for this one but we can register it too.

I haven't gotten any feedback, so I'll go ahead and register the schemes for ipfs, ipfn, dat, ssb and dweb unless someone opposes. If you have any documentation of the protocol (e.g. spec mentioning the URI schemes), that would be helpful.

Thanks!

[RangerMauve]

I was thinking about this in the context of how different apps based on Dat would do it.

How do you feel about adopting the web+ thing that browsers do, but using dweb+? Like dweb+ipfs:// or dweb+dat://

[Gozala]

I would personally prefer web+ipfs:// and web+dat://, I don't think d prefix communicates anything useful.

[rabble]

@Gozala what does web+ipfs:// or web+ssb:// give us over just ipfs:// and ssb://?

If we're doing this i'd like to consider how we can encode different SSB network keys. You can have the same identity and even message hash on different networks and we'd need a way to link to the right one.

In terms of planetary we'd really like to see ssb uri's adopted so we could support them. Right now we're using apple's extensions to redirect requests for ssb content to planetary.social/p/@identity which works but isn't an open decentralized solution.

[Gozala]

@Gozala what does web+ipfs:// or web+ssb:// give us over just ipfs:// and ssb://?

Former is universally supported across all browsers today (See https://developer.mozilla.org/en-US/docs/Web/API/Navigator/registerProtocolHandler).

I was responding to the @RangerMauve's comment #28 (comment), otherwise I do think ipfs:// and ssb:// make more sense.

If we're doing this i'd like to consider how we can encode different SSB network keys. You can have the same identity and even message hash on different networks and we'd need a way to link to the right one.

I am not sure I fully understand this, could you give an example ?

In terms of planetary we'd really like to see ssb uri's adopted so we could support them. Right now we're using apple's extensions to redirect requests for ssb content to planetary.social/p/@identity which works but isn't an open decentralized solution.

I think that would be wonderful. However I can only speculate how long it would take to convince mainstream browsers to add new protocol schemas to the supported list.

[Gozala]

@fred-wang I know @pfrazee, @mafintosh and rest of the @beakerbrowser are moving from dat:// to hyper:// URL schemas would be great to get them into the loop.

[RangerMauve]

@Gozala We're actually talking about the hyper thing in the dat community right now. So far we're thinking that we're going to be switching all the dat wording to talk about hyper:// as the new protocol stack. So dat:// is probably gonna be deprecated to the old stack.

[serapath]

As part of another more dat specific discussion we had a similar discussion like the one in this thread here and it was about dat:// vs hyper:// vs cabal:// vs. ... because there is not just one dat but rather a diverse ecosystem.

One "solution" is to talk about protocol handlers, but they require browser support or extensions and more than that - some rather centralized authority like browser vendors and/or IANA to even add certain protocols to a certain whitelist - which is not very p2p

This is the reason I personally think a single protocol extension to rule them all is the right way to go.

I also agree with @Alexander255 that the listed options are terrible because they do conflict with how URL's work in general. Now that certain single protocol (e.g. dweb:// to rule them all) can work differently, but in order to translate to existing practices, maybe we can think about more options that heal the issue mentioned by him about the proposed 2 options.

I described a proposal here:

• DDEP-0001:hypercore-header-for-manifestfeed datdotorg/datdot-research#17 (comment)

in short:

• <protocol>://user:pass@subhandler:host/path?params#fragment
  • e.g. dat://cabal:87ed2e3b160f261a032af03921a3bd09227d0a4cde73466c17114816cae43336+5

This way, the host part can still contain a domain and a port and everything that the host usually consists of (which excludes :), so that can be used by the generic protocol handler.

Furthermore - at least in the dat ecosystem a so called topic on a globally available DHT could be used to look up apps or whatever which support certain kinds of protocols if a user doesn't yet know how to handle a specific one :-)

[fred-wang]

Hi dweb community. Thank you again for the feedback. For the record, as I said above the discussion for protocol-specific URL schemes (including dat/hyper) are happening there:

* ipfs/ipns: [ipfs/in-web-browsers#29](https://github.com/ipfs/in-web-browsers/issues/29)
* dat: [datprotocol/DEPs#66](https://github.com/datprotocol/DEPs/issues/66)
* ssb: [ssbc/ssb-spec-drafts#4](https://github.com/ssbc/ssb-spec-drafts/issues/4)

I'm still not sure whether the community reached a consensus regarding "dweb" but I have written a draft in any case (please check):

https://raw.githubusercontent.com/fred-wang/iana-uri-schemes-provisional-registration-requests/master/dweb.txt

I think the decision on dweb will not prevent me from sending the other registrations first. However, I'd like to send patches to the HTML spec and web-platform-tests/chromium/mozilla adding all the new safelisted schemes in one go so I'd still appreciate to know soon whether you are really interested in "dweb".

Regarding registering a generic prefix like "dweb+" (or "hyper+"), I haven't seen anything like this mentioned in IANA's documentation, so I don't known whether they would allow such a generic registration. Similarly, none of the existing safelisted schemes in the HTML specification (or browser implementations) are based on a generic prefix, so that would be something new for WHATWG/browser reviewers. Of course I'm not saying we can't propose/try it, just that it might be harder to convince people that extending current schemes...

The HTML specification provides a generic "web+" prefix but corresponding schemes are not registered at IANA. They are designed to provide generic protocol names for users and there is no guarantee someone else won't use them and so that URI conflicts happen.

[ntninja]

@fred-wang: Just to make sure: Is IPFN vs IPNS in your comments just a typo? I've never about the former except in your comments and it's not mentioned anywhere else either. You even posted a different proposal yourself in another thread.

Regarding the dweb:-scheme: How problematic is it if the scheme is provisionally registered with IANA and later it turns out it won't be used? Does IANA mind if there is no solid spec on it yet? Does WHATWG mind safelisting a “fuzzy” scheme like that?
Probably best to just trying to submit it it, while being upfront about the current state of affairs. Best/worst case they say “no, that's not good enough” – then we could retry later when/if there is a more solid spec. 🙂

[fred-wang]

@fred-wang: Just to make sure: Is IPFN vs IPNS in your comments just a typo? I've never about the former except in your comments and it's not mentioned anywhere else either. You even posted a different proposal yourself in another thread.

Yes, sorry about that. I edited my comments to avoid future confusions.

Regarding the dweb:-scheme: How problematic is it if the scheme is provisionally registered with IANA and later it turns out it won't be used?

Just to be sure: I understand dweb is already used, right? Otherwise it does not make sense to submit it yet. IANA has a "historical" list for "not used anymore" schemes, so I don't think it's a problem to stop using it in the future: https://tools.ietf.org/html/rfc7595#section-5

Does IANA mind if there is no solid spec on it yet?

From https://tools.ietf.org/html/rfc7595#section-7.4 : "A scheme specification is only required for 'permanent' registration." ; so I think it's not a problem.

(In https://tools.ietf.org/html/rfc7595#section-4 they say "If no permanent, citable specification for the scheme definition is included, credible reasons for not providing it SHOULD be given." but this section does not seem up-to-date with respect to 7.4 e.g. they mention "security considerations" while 7.4 say that now "the answers instead belong in the scheme specification". )

Does WHATWG mind safelisting a “fuzzy” scheme like that?

Good question. From whatwg/html#5482 (comment) it seems having a doc somewhere is the only requirement ; in other threads some people were even arguing about switching from a safelist to a blacklist (this means allowing dweb and all other schemes not reserved by browsers).

Nobody complained on decentralized web protocols so far, but for other names there were concerns whatwg/html#1829 whatwg/html#2546 so we can't be sure that won't happen.

Probably best to just trying to submit it it, while being upfront about the current state of affairs. Best/worst case they say “no, that's not good enough” – then we could retry later when/if there is a more solid spec. slightly_smiling_face

I can either submit it or postpone that. I guess if "dweb" is already used it makes sense to try, even if the spec is not solid yet.

---

BTW, this is from the IANA doc, it seems rejection is very unlikely:

Upon receipt of a scheme registration request, the following steps
MUST be followed:

1. IANA checks the submission for completeness; if required sections
   of the scheme registration request are missing or any citations
   are not correct, IANA will reject the registration request. A
   registrant can resubmit a corrected request if desired.

2. If the request is for 'provisional' registration and no entry
   already exists in the current registry for the same name, IANA
   adds the registration to the registry under the First Come First
   Served policy.

3. Otherwise, IANA enters the registration request in the IANA
   registry with the status marked as "Pending Review", and the
   remainder of this section applies.

[lidel]

Just to be sure: I understand dweb is already used, right?

Yes, we are not very vocal about it, but support it in production:
dweb: URIs following "Option 1" from #28 (comment) are supported by IPFS Companion browser extension and a system-wide handler is also registered when IPFS Desktop app is installed.

I think it is enough to argue provisional registration.
We can solidify details later.

[fred-wang]

I sent 7 registrations to IANA yesterday and they have been accepted without any further requests on their side.

IIUC, there isn't a clear consensus regarding dweb as a shared protocol for distributed web, but it's already used and the community doesn't oppose to try to register it. So I believe I should just go ahead and do it?

I updated a bit https://github.com/fred-wang/iana-uri-schemes-provisional-registration-requests/blob/master/dweb.txt ; it references https://docs-beta.ipfs.io/how-to/address-ipfs-on-web/#shared-dweb-namespace which says this is still an experiment and redirect to this github issue. Not sure it is a very solid reference, but I don't think there is anything better for now...

[lidel]

@fred-wang sgtm, go ahead, perhaps it will be enough.
(If not, not a big deal, we can revisit in the future, if more solidified vision for this URI exists)

[fred-wang]

Registration request sent.

[fred-wang]

dweb is registered: https://www.iana.org/assignments/uri-schemes/uri-schemes.xhtml