Unified standard for https links to files

[master255]

The need for this standard is clear to everyone - we need links that we can send to a friend and he will open this link in the application and will be able to view the contents of the catalog or download a file.

There is information that is needed to do this:

1. User ID of the user who has the file stored
2. Path to the file/directory
3. Password or no password

This is the format I use now:
https://p2p/12D3KooWC6iB6hthfr5A8Yd13BHEBJsYRpGdkWLwwNwZtiZtJB3B:321/path/to/folder/or/file
This part - "https://p2p/" means that the file is transferred p2p.
Path means the path
321 means password
If there is no password, the link looks like this:
https://p2p/12D3KooWC6iB6hthfr5A8Yd13BHEBJsYRpGdkWLwwNwZtiZtJB3B:/path/to/folder/or/file
The ID can be any type.

Any suggestions\criticism on this format for p2p data transfer? Which can be retrieved using Libp2p.

A single API for file transfer is there, too. We will discuss later in another question.

[aschmahmann]

Given this is about specs/standards I'd recommend discussing this in https://github.com/libp2p/specs. Going to transfer it over there.

[master255]

But there are more people here. Let's discuss here, and transfer at the end of the discussion. And I don't have the transfer mechanism working.

[MarcoPolo]

I think this specific request is an application level detail, and does not (yet) need a standard.

That said, the more general form of this is about HTTP resources. For which there is this in-progress PR about how to reference them: multiformats/multiaddr#164. I link it since it's vaguely releated, but I don't mean for that PR to incorporate things mentioned in this discussion (namely user id and password as those are application specific).

[master255]

Nice try at not working.
But this link is needed for all Libp2p based applications because Libp2p allows you to transfer information, and in our world information is mostly a file. And if you made standards for information transfer, then you need standards for files and api as well.
Besides, this is a link not only for Libp2p, but for a unified api for future applications. In an environment where there are no DNS servers and the browser is an application with Libp2p or another library.

I know you're a smart programmer. You'd better write your critique of this link. And don't forget that it should be clickable in any browser/chat/application. And not too long, so as not to scare users.

[aschmahmann]

@master255 this comment isn't conducive to meaningful discussion and it strangely seems like you're telling Marco how to spend his time. If you're rude to maintainers it seems unlikely they'll want to engage with you at all. I'd also recommend taking a look at the code of conduct.

If you read the linked issue you can see Marco's put some thought into this already, albeit his usecase is not 100% the same as yours. There's a lot to do to make p2p networking easier and limited people and time to do it. Better to contribute if you want to see something happen then trying to badger maintainers into doing the work.

---

From what I can tell you have a proposal and are not open to some of the issues and suggestions that have been brought up in the discussion so far. If you're not willing productively engage with others it seems unlikely that others will continue to put in the effort to do so. You might also consider prototyping out how this would work end-to-end since I suspect you'll start to see some of the issues that have been pointed out.

FWIW I think it'd be pretty nice to have some standards for referencing resources over libp2p, a multiaddr URI is probably part of it but may not be the only piece here. I'll likely do another round of comments that are hopefully helpful, but I'll give up on engagement here too if the response is more badgering of maintainers.

[master255]

@aschmahmann I completely understand your defense of programmers. I also love programmers and defend them. We're on the same side.
But I am probably too hot and manly after the gym. You can see my pictures on instagram. I've added a link in my profile description.
But at the same time I praised Marco and I know for a fact that he is a good and smart programmer.

[aschmahmann]

A few thoughts on the proposal:

1. Given that the protocol you're relying on is HTTP (or similarly could be FTP, or anything else) it seems like it'd be generally more convenient to follow standard URI (and in this case particularly HTTP URI) semantics https://datatracker.ietf.org/doc/html/rfc3986).
   • The "most correct" thing is probably changing the protocol scheme to something like http+libp2p. If you still want it to be a valid URI then it seems to mean no / in the "authority" so we'd need an encoding scheme there anyway.
   • An alternative might be defining a standard host to use for peerIDs such as p2p, p2p.localhost, etc.
     • With a new protocol scheme this could be http+libp2p://bafzaajaiaejcaiphjgwe4c6hhlyzmjp7ryra4suea5u6uzwjkw6x4ep63xv6mjt6.p2p/path/to/data as meaning to do HTTP-over-libp2p to the given peerID (finding the machine is left to the application's routing system) and then fetching the data from the given path
     • For interoperability something like http://bafzaajaiaejcaiphjgwe4c6hhlyzmjp7ryra4suea5u6uzwjkw6x4ep63xv6mjt6.p2p.localhost/path/to/data could work as a bridge that fits with current tooling that you can intercept if the tooling can't handle it, or the tooling can natively intercept and handle.
   • Related to this, there doesn't seem like a need to come up with new password syntax you could use username:password@host as can be done in other environments for HTTP basic auth
2. You may end up in a position where you start wanting to shove more data into the URI (e.g. IP address, relay addresses, etc.) in which case you're starting to look at some of the benefits of multiaddrs and multiaddrs in URIs Multiaddr URI scheme multiformats/multiaddr#165.
   • Note: the above proposal covers a multiaddr URI on its own, but having a spec for describing multiaddrs as authorities might be useful as well since sometimes a peerID alone won't be enough

[master255]

@aschmahmann You forget that the link must be clickable in any browser/chat/application. And the link should not be too long, so that users are not afraid of it.

And I thought we were doing our best to make Libp2p find a user's computer only by one ID without relays. So there are no plans to add this information for links in the future.

And you can suggest to add more information now.
You suggested:

1. add relays - they are not needed.
2. IP address - no. It is definitely not needed.
3. Add http+libp2p - no. Then the link will not be clickable.
4. Add .p2p.localhost - no. This is useless information that will increase the link size. And using localhost is not correct, because it is not a local link, but a link to a computer on the internet.
5. Use http instead of https - some browsers and applications may give a warning when opening your links, which will not be correct. Since the data transfer is encrypted.

And don't forget, this link is not for browsers, but for applications based on Libp2p and maybe different libraries. It is essentially a link to a file (or even site) on the Web3.0 Internet.

That's good for a start. What other suggestions do you have?

[master255]

in which case you're starting to look at some of the benefits of multiaddrs and multiaddrs in URIs

Here if there are multiaddrs, there will be different links. So doing multiaddrs in the link is not necessary.

[aschmahmann]

To make sure we're on the same page here:

HTTP-over-libp2p is a specific protocol being run over a libp2p connection, which is different from say ping, gossipsub, bitswap, kademlia, ... You could just as easily do say FTP-over-libp2p or SSH-over-libp2p. I assume you're most interested in HTTP-over-libp2p given the initial post, however I suspect it'd be helpful in pursuing this to consider how you'd want say FTP-over-libp2p links to look.

You forget that the link must be clickable in any browser/chat/application
Add http+libp2p - no. Then the link will not be clickable.

You seem to be implying that it has to look like https://<rest-of-url>, but that can't really be accurate "natively" since while say HTTPS is defined over TCP + TLS or QUIC, there's no where Noise, Mplex, Yamux, or the libp2p ALPN in QUIC fit in. This means applications (browsers, chat apps, curl, ...) cannot succeed at using those URLs if they're going to stick with the specs.

AFAICT this leaves us with two options:

1. Use a new protocol scheme
2. Use a "bridge" through standard HTTP(S)

Which is how I arrived at the above options.

Add http+libp2p - no. Then the link will not be clickable.

Depending on the application other URI schemes are clickable, how it works depends on the application. What you'd likely want from a new protocol handler is a way to then handle those requests (e.g. the OS, or applications like web browsers make this doable with things like mailto, tel, onion, ipfs, slack, etc.).

And don't forget, this link is not for browsers, but for applications based on Libp2p and maybe different libraries. It is essentially a link to a file (or even site) on the Web3.0 Internet.

To me this gives an even stronger indication that you're really looking for a new protocol scheme, and that http+libp2p or something should be fine (just as web+<your-thing-here> is a valid new type of protocol scheme).

This is useless information that will increase the link size. And using localhost is not correct, because it is not a local link, but a link to a computer on the internet.

In this case (bridging) it is correct in that you could say that the way you support making http-over-libp2p requests on your machine is by having a small http server that proxies out over libp2p and it handles all requests to *.p2p.localhost.

some browsers and applications may give a warning when opening your links, which will not be correct. Since the data transfer is encrypted.

Note: .localhost shouldn't be a problem here since it can (and frequently is) treated specially. Not sure if there's a more up to date resource on this but you can check out https://datatracker.ietf.org/doc/html/draft-west-let-localhost-be-localhost and do some searching to learn more.

While in theory you can make up and claim some TLD and hope it won't be a problem that you haven't claimed it through IANA, https://<something>.p2p doesn't seem quite valid since the TLD is modifying the protocol itself if you're doing http-over-libp2p. An area where that might be plausible (although the short-term UX would be sad) is to use https://<something>.p2p if you wanted the peerIDs for the purpose of discovering a public IP address you could do a regular HTTPS request to where the certificate was self-signed with the public key behind the peerID. To deal with the short-term UX sadness of everything relying on CAs you'd need a CA to sign all these peerID-based certificates and then try and convince applications that self-signed for .p2p is ok. It'd be a step up, but this also wouldn't help you if you in the more p2p-ish use cases like wanting to fetch from a peer behind a NAT.

And I thought we were doing our best to make Libp2p find a user's computer only by one ID without relays. So there are no plans to add this information for links in the future.

I think you're confused about the point of relays. They're not for discovery of peers, they're for helping establish connections or move traffic for peers in harder to reach environments.

However, you should note that not all applications have some peer routing component to them (e.g. a DHT) that maps peerID -> machine address. Some move around the addresses in other ways. These may not be the use cases you're interested in, but they certainly exist.

[master255]

HTTP-over-libp2p is a specific protocol being run over a libp2p connection, which is different from say ping, gossipsub, bitswap, kademlia, ... You could just as easily do say FTP-over-libp2p or SSH-over-libp2p. I assume you're most interested in HTTP-over-libp2p given the initial post, however I suspect it'd be helpful in pursuing this to consider how you'd want say FTP-over-libp2p links to look.

No. This isn't talking about http over libp2p, it's talking about a more efficient file and data transfer protocol.
Http is for compatibility with something old. I'm talking about a new protocol that doesn't need compatibility.

You seem to be implying that it has to look like https://, but that can't really be accurate "natively" since while say HTTPS is defined over TCP + TLS or QUIC, there's no where Noise, Mplex, Yamux, or the libp2p ALPN in QUIC fit in. This means applications (browsers, chat apps, curl, ...) cannot succeed at using those URLs if they're going to stick with the specs.

I've talked about this before. These links are for p2p apps, not for browsers, chat apps and curl. The main thing required of these links is to be clickable in these applications so that users can click and go to the p2p applications.

Depending on the application other URI schemes are clickable, how it works depends on the application. What you'd likely want from a new protocol handler is a way to then handle those requests (e.g. the OS, or applications like web browsers make this doable with things like mailto, tel, onion, ipfs, slack, etc.).

We're talking about now and the apps that we have now and everyone is using them. Such as Whatsapp, Telegram, Opera, Chrome, Firefox and the similar.... and now the ipfs://qweqwe.com links are not clickable. So it's not possible to use that.

In this case (bridging) it is correct in that you could say that the way you support making http-over-libp2p requests on your machine is by having a small http server that proxies out over libp2p and it handles all requests to *.p2p.localhost.

No. We're not talking about http over libp2p. I don't know where you got that from. And if you think of it that way, all browsers are local servers too.

I think you're confused about the point of relays. They're not for discovery of peers, they're for helping establish connections or move traffic for peers in harder to reach environments.

Relays should be provided by the application, not the link.

[master255]

So we have something like Windows Samba shared directories. There user rights are used, but here password is used.
For many years Windows Samba has not invented anything more than host, share, file path and access control. That's why it's exactly the same in this link:
https://p2p/12D3KooWC6iB6hthfr5A8Yd13BHEBJsYRpGdkWLwwNwZtiZtJB3B:321/path/to/folder/or/file

The ID of the computer where the file is stored is the host
321 - that's the access control
path - is the name of the share (so that no one has the full path to the file. As it is not meaningful and not secure)
to/folder/or/file - the path to the directory/site/file.

[hsanjuan]

You say you want something "clickable", but clicking on your links would open a browser that has no idea how to deal with the proposed URL. So either you make a working http url (assuming files are provided by a gateway-type of server) (as exemplified above), or you come up with a schema that explains what the operating system or eventually the browser needs to do with the URL (as exemplified above too).

There is already ipfs:// (although it routes by content and not by peer)... I used libp2p:// before for things like accesing a file (https://github.com/libp2p/go-libp2p-http/blob/master/p2phttp.go#L96), but I controlled both client and server so I could have chosen anything.

Regarding access keys, there are many ways and having a standard would just add another one. To me it is cleaner when they go as query-param (which can be expanded with other query params when needed without breaking the original url format).

this link is not for browsers, but for applications based on Libp2p

Then don't make a browser link, certainly not an HTTPS link since HTTPS will not be involved. You can't standarize a browser trick.

[master255]

You say you want something "clickable", but clicking on your links would open a browser that has no idea how to deal with the proposed URL.

No. You're wrong. In Windows and Android, you can already register apps to open certain links. Here's a video of how this has worked in Android for a long time https://t.me/media_library/13267

Any app can open https links. Not just the browser.
Even many economies of countries are based on this principle.

libp2p:// and ipfs:// - you can check. These links are not clickable in Whatsapp, Telegram or even in a browser. And nobody cares what's in the beginning. I understand that you want everything to be perfect and work, but in practice it doesn't work.

And if we write https:// - it won't be cheating. Libp2p uses the same encryption mechanisms (TLS, SSL) as https. And if we don't write http, our link will not be clickable.

Our current technology is not advanced enough to make libp2p://qwe.com/ and ipfs://qwe.com/ links clickable. Sorry, but this doesn't work in practice.
And we write https://p2p/ That means we want p2p, not just https.

Basically https://p2p/ - is created to make the link clickable and the user realizes that it is a p2p link.
If you change any element here, the link will stop being clickable and become larger in size, without making any sense.

You can add https://p2p.com/ , but such a site may exist and it may open in some cases and it will be a security issue.
So our goal is to create a site that will never open. Any ideas? We need to add a .com to make the link even more clickable. But we need to add an ending so that the site never exists.

Then don't make a browser link,

This is not possible, as then the link would not be clickable. And no one will be able to use it.

Regarding access keys, there are many ways and having a standard would just add another one. To me it is cleaner when they go as query-param (which can be expanded with other query params when needed without breaking the original url format).

You may be right, but our experience with Samba suggests that we have no other parameters for files and directories. And a colon in the address is allowed by the standard because it is seen as a port. That's why it's clickable.

You suggest:
https://p2p/12D3KooWC6iB6hthfr5A8Yd13BHEBJsYRpGdkWLwwNwZtiZtJB3B/path/to/folder/or/file/or/site?password=321

It's not bad. I need to think about it. I'll answer it later. It's harder to parse.

[master255]

Even here you can see that your links are not clickable

[master255]

You suggest:
https://p2p/12D3KooWC6iB6hthfr5A8Yd13BHEBJsYRpGdkWLwwNwZtiZtJB3B/path/to/folder/or/file/or/site?password=qweqwe

I suggest:
https://p2p/12D3KooWC6iB6hthfr5A8Yd13BHEBJsYRpGdkWLwwNwZtiZtJB3B:qweqwe/path/to/folder/or/file

Disadvantages:

1. This parameter is harder to parse than the password after the colon.
2. it is much longer in the number of characters.
3. There will be problems with unification. Some will use pas and some will use password. Because of this there will be problems with program compatibility.
4. My link can be changed without a password:
   from like this:
   https://p2p/12D3KooWC6iB6hthfr5A8Yd13BHEBJsYRpGdkWLwwNwZtiZtJB3B:/path/to/folder/or/file/or/site
   to this:
   https://p2p/12D3KooWC6iB6hthfr5A8Yd13BHEBJsYRpGdkWLwwNwZtiZtJB3B/path/to/folder/or/file/or/site
   and then your proposal loses the advantage of compactness without password.

And it turns out that my proposal wins. Because it is easier to parse and it is more compact in the number of characters.
Perhaps I missed something?

[aschmahmann]

Here's a video of how this has worked in Android for a long time https://t.me/media_library/13267

This does not work by registering https://t.me to do other things, it works by registering tg://resolve?domain=media_library&post=13267. You'll notice that GitHub doesn't make this highlight either, and yet it's still a valid URI and it's how your browser knows how to forward control to another application. If you open up a browser inspector for the t.me URL you'll see it attempts to forward to the tg: URI.

This is why both Hector and I have explained that you can register a protocol handler for handling this new protocol and why trying to have https:// mean anything other than the standard HTTPS protocol is doomed to failure. You can however you various types of bridge-like strategies in environments where HTTPS is what is supported, for example https://t.me/media_library/13267 both pulls the data and will help forward you to a local application that will handle it more directly. IPFS also has the ipfs:// protocol with handlers in some browsers like Brave and tools like IPFS Desktop and IPFS Companion that can handle them as well (with the same kind of OS support as other protocols like tel, mailto, tg, slack, etc.). It also has public gateways like dweb.link or locally run ones (e.g. .ipfs.localhost) that can allow standard HTTP tooling to interact with the IPFS protocol.

So if you want a clickable link you can use any of these bridging techniques. For example, the telegram style one would mean you'd run a website at https://myhttp-libp2p-bridge.tld that takes all inputs like https://myhttp-libp2p-bridge.tld#your-http+libp2p-URI and then redirects it to http+libp2p://rest-of-URI.

[master255]

@aschmahmann It's good that you wrote this. Maybe others will find it interesting, but we have a different task.
We need a clickable link in any application, any operating system. The brave browser of course is very little. I don't install it myself and can't require users to do so. Especially since it will not help in any way.

Creating a domain to redirect links means breaking p2p. These links in the future will allow you to leave the browser completely, and you propose to return to it and pay money. It doesn't work that way.
So we don't need to go down that road. Today you are there and pay money for the domain, and tomorrow you are not there and the domain was bought by criminals.
P2p - we need to create self-organizing applications. Which do not need servers and domains. We're going down that road. There's no other way.

If windows browsers do not make p2p://qwe.com links clickable, and do not allow us to open https://p2p/.... in other apps like Android does, then that means Windows doesn't support p2p technologies (and all others) and is not letting us develop.

We are discussing here pure p2p links for p2p applications. And ways to integrate them into OS and browsers.

Changing the initial prefix of a link won't make them clickable in Windows, so it's useless.

To use such a link p2p://qwe.com user will need to copy into p2p application the same way as https://p2p/.....
And in case of https://p2p/ the user can at least install a local relay program (dns server + small web server) to open these links. Then p2p://qwe.com links will not be clickable at all. That's why we can't use it.

[hsanjuan]

For encrypted file sharing in p2p ways that works in the browser and has local-only decryption (anchor-based keys that don't get sent around), see: https://github.com/ipfs-shipyard/ipfs-senc

[master255]

Sorry, but the domain https://ipfs.io already means not p2p. And browsers don't support p2p today. So it's client-server interaction.