BEP: Magnet URI extension - Select specific file indices for download

[dessalines]

Here's my BEP in .rst format:

https://gist.github.com/dessalines/9f8c5409f9905502567ef70facfa16fb

[ssiloti]

I'm not strongly opposed to this, but I don't agree with the rational. The proposal is to use this as a workaround for clients lacking a way to download just the metadata then prompt the user to set file priorities. It would require the site serving the magnet link to provide the user with an interface to select the files to download, then generate a personalized magnet link. Otherwise users wouldn't know which file indices correspond to which files. That's a lot of complexity to solve to what is IMO a pretty minor UX issue. I think it would be more appropriate to work with client authors to improve the UX when adding magnet links.

What I could see something like this being useful for is creating a "deep link" to a specific file in a torrent. This could come in handy if you are, for example, serving content from a torrent in a browser.

[dessalines]

What I could see something like this being useful for is creating a "deep link" to a specific file in a torrent.

That is exactly my use case. I'm creating a service that indexes all the files within torrents, basically a file search, not a torrent search, but clients have no way to select only that file. If your back-end service has the torrent bencode, you can easily get the file indices and create magnet links for each specific file, or potentially each folder.

It would require the site serving the magnet link to provide the user with an interface to select the files to download, then generate a personalized magnet link.

True, which wouldn't be difficult from a server point of view. The user wouldn't have to know anything about the specific file indices. All they have to know is that they select a bunch of files and then click a magnet link. I can't see a case where users would ever have to memorize file indices; that work of building those links could be done either on the front or back end.

EDIT: also, waiting for the exact moment when the metadata finishes downloading, then having to pause the torrent, then select the specific files I wanted, is a daily occurence for me.

[arvidn]

another variant of the deep-link use case is to have a "copy magnet link" context menu option on files (not just on torrents themselves as most clients do today)

[the8472]

But what is the usecase for only downloading one file in the first place?

That is exactly my use case. I'm creating a service that indexes all the files within torrents, basically a file search, not a torrent search

And how common is the case that you find that file as one among many similarly-sized ones in a multi-file torrent and you can reasonably infer that the user only wants that one file and not the rest of the content?

[dessalines]

And how common is the case that you find that file as one among many similarly-sized ones in a multi-file torrent and you can reasonably infer that the user only wants that one file and not the rest of the content?

Extremely common. I go through this process at least once a day. They may only want one book out of an author's library, or may only want to download 1 song out of an artists discography torrent. Especially with mobile devices with limited space, you often don't want to download the entire torrent.

I may want to seed my entire book library(could be 10,000 books), and let users download only the specific book they want. Its not easy to scan through 10,000 files, and click the exact one you want. But with a web service, they could click a magnet link specifying that correct file index, and download only the specific book they want.

and you can reasonably infer that the user only wants that one file and not the rest of the content?

A user can decide that for themselves, as long as the link specifies that they are downloading that subset/specific file.

[the8472]

If the user's predominant use-case is downloading a single work then seeding a batch torrent seems like the wrong approach. You're looking for a band-aid where the actual fix is to not aggregate content when aggregation makes no sense.

[jeremyBanks]

It's a mis-use of the design, but it is not a rare user behaviour. Whether it's downloading a specific work from a large Internet Archive collection, or a single album from an artists' entire discography, there are a lot of cases where well-seeded batch torrents are the only source that exists, but a large fraction of users only want some pieces of it.

I can understand hesitation to further encourage that non-optimal usage pattern, but this extension would be useful.

[dessalines]

You're looking for a band-aid where the actual fix is to not aggregate content when aggregation makes no sense.

Many users such as myself have libraries of torrents that would not be practical to make into individual torrents. I know, because I've hit the limit of what rtorrent and deluge could handle when I tried adding millions of torrents. They max out at around a few thousand torrents seeding. Library torrents are even how services like libgen host their files. They create library torrent buckets, I think about 1 GB in size, and when a new book gets added, it goes into a bucket, not an individual torrent.

You're looking for a band-aid where the actual fix is to not aggregate content when aggregation makes no sense.

Aggregation absolutely does make sense, if it didn't then torrent clients wouldn't have the ability to select specific files you want to download at all. They would force you to download the entire collection.

This is a very common use case. I'm sure all of us have selected specific files we want to download from a torrent.

[the8472]

@jeremyBanks

The problem is if user-choices are non-random and clients actually seed then it may become increasingly difficult for clients to download the complete torrent because the majority of peers only have the popular content. For example with discographies that might be That One Hit that the artist is known for.

The protocol offers no way to deal with this other than trying all peers in the swarm until it finds one that doesn't do selective downloading.

Let's go further and assume that torrenting is usable from browsers, for example via addons. Then a lot of people streaming file A (and only file A) from a multi-file torrent would be happy, but at the same time they would make it more difficult for others to stream any other file B, C or D. That's clearly undesirable behavior if users can negatively affect each other due to conflicting goals.

This may all be rather theoretical, but the use-cases mentioned here sounds a lot like like large-scale content distribution of things that are in principle streamable.

TL;DR: It might work to some extent, but it may blow up if you try to scale it up.

@dessalines

Aggregation absolutely does make sense, if it didn't then torrent clients wouldn't have the ability to select specific files you want to download at all. They would force you to download the entire collection.

You assume that aggregation makes sense because selective downloading exists. But that is backwards. The bittorrent protocol was and is only designed to transfer the whole content. Selective downloading is something clients hacked on later because publishers over-aggregated content in a manner where users only want specific files. The protocol does not actively prevent this, but the absence of preventive measures does not imply it is designed for this case or copes well with it.

I know, because I've hit the limit of what rtorrent and deluge could handle when I tried adding millions of torrents. They max out at around a few thousand torrents seeding.

Here you're letting your decisions be dictated by implementation behavior, not what is optimal at the protocol level.

In principle it is possible to seed millions of torrents, arvid has tried (or speculated?) according to a blog post literally titled seeding a million torrents.

Of course you need a client optimized for this case, but providing random access to millions of files is a rather specialized use-case.

If the goal is to distribute individual files out of millions to many different people then I am doubtful whether aggregation is the right solution.

---

In general I am not strongly opposed to this proposal, it just seems more of a bandaid instead of a real solution for the underlying problems that the proponents are facing. It might be good enough™ in practice

[the8472]

You might want to add range coding to the argument. e.g. 0,3-7,10. inclusive ranges, since exclusives ones are not needed with single-element notation.

[dessalines]

Good idea, I updated the gist to show a range example.

[dessalines]

What should I do as a next step? I can issue a PR against this, but I'm not sure how the numbering system works.

[ssiloti]

The BEP should be submitted as a PR with XXX in place of the BEP number. Once a number is assigned the PR is updated before it gets merged.

[gubatron]

👍 👍
We could use this on FrostWire's individual file search results.

FrostWire presents search results both as .torrents and when the metadata is available we're able to show matching search results by looking at the file list, we present these results as individual audio, video, apps, docs, etc. search results.

All .torrent search results now have a "Copy Magnet URI" action available, this would improve sharing a single file from a torrent from FrostWire to FrostWire or other clients.

So, that'd be another use-case out of the realm of websites.

[dessalines]

Closed by #64

[RangerMauve]

Hi, sorry to bring this up, but I wasn't sure if I should post a new issue.

I want to enable loading torrents as websites in a browser with BEP 46. One issue that I'm running into is how to handle relative paths on a web page.

i.e., if my torrent has a structure that looks like:

/index.html
/scripts/main.js
/styles/style.css

I'm planning to focus on magnet URIs which will be navigate to in the browser like you would a website. So if my magnet URI looks like magnet:?xs=urn:btpk:EXAMPLE, what should I use to reference a specific file, for example /script/main.js, within it?

I was thinking of proposing a new enhancement so that one could use the path component of the URI to have something like magnet:/script/main.js?xs=urn:btpk:EXAMPLE. The main reasoning for using this over file indices in another search parameter is to enable people to reuse existing logic that browsers have for relative URLs. But that would break since the xs parameter would be lost in in the new URL.

Would magnet links even make sense for such a use-case, or would this be better served by a new protocol type?

[ssiloti]

@RangerMauve Magnet links could be used for that. A separate issue should be created to propose how such links should be formatted.

One thing to be aware of is that browsers have restrictions on what schemes can be used to serve web content. Mozilla's documentation shows they have magnet whitelisted, but Chrome may not. Most other browsers don't support web-based scheme handlers at all.

[RangerMauve]

Thanks, I'll play around with some ideas and see if there's any that would be worthwhile.

Regarding protocol whitelists, I'm planning on creating a new browser to experiment with things and then working on extensions once APIs like mozilla/libdweb get more flushed out. Also, there's been more of a push lately to whitelist more protocols in order to support p2p web use-cases.

[dessalines]

@RangerMauve if you know the file index, you can use magnet:?xt=urn:btih:HASH&dn=NAME&tr=TRACKER&so=0,2,4,6-8 .

AFAIK only libretorrent has this BEP implemented.

[RangerMauve]

@dessalines: it'd be possible, but when in the context of authoring a website, you don't really know what your torrent file is going to look like until you try to publish it. You know that you have a file called example.png in the root of your folder, but you don't necessarily know what the file index will be until you're done. Ideally, it should be possible to create a website that assumes it was loaded from the filesystem, then publish it on BitTorrent and have it work the same way with no changes to the source code.