raise auto piece size selection limit to 16 MB in create_torrent() #2669

Chocobo1 · 2018-01-05T06:39:04Z

Related issue in qbt: qbittorrent/qBittorrent#8205

I've read various guides on the net, however they doesn't put out suggestions for large torrents (size in tens/hundreds of GB).
I'm only partially aware of its consequences, I definitely need some advice and comments welcome!

ps. This one is targeting RC_1_1 branch, I'll open another one for master branch after merged

Chocobo1 · 2018-01-05T08:26:10Z

I just realized that BEP 30 (Merkle hash torrent extension) is supposed to remedy this kind of issue...
However is the support widespread enough so we can recommend it to users?
Does major torrent clients support it yet?

UPDATE: answering my own question:

No one seems using BEP 30 and it might be broken: Bittorrent v2 #2197 (comment)
There is BEP 52 (The BitTorrent Protocol Specification v2), which probably will mitigate this issue better than BEP 30.

Given the above, does this PR still make sense?

arvidn · 2018-01-05T13:07:57Z

I agree that 2 MiB piece sizes are perhaps a bit small, but 16 MiB feels quite large as well.
Here's my model for reasoning about what a reasonable piece size is:

The larger piece size, the longer it will take a peer to complete downloading a piece. The whole piece need to be downloaded in order to check its hash and checking its hash is required before offering it up to other peers. Larger piece sizes mean longer delays for payload to propagate from one peer to the next, slowing down the dissemination in general.

So, how much worse does it get as the piece size increases? It's hard to say without proper measurements, but it probably depends on the general download speed of the peers in the swarm. i.e. the most relevant property is probably not the piece size directly, but the piece download time.

Do you have any references to what people on the internet recommend?

Another aspect to take into account is whether torrents are expected to primarily be distributed via magnet links or not. If they are, a large torrent file (i.e. small pieces) may not be that big of a deal.

Another aspect is to keep bittorrent v2 in mind, although that's probably some way out, I would expect the piece size to be a lot smaller with a merkle tree. But perhaps these concerns are disconnected.

ssiloti · 2018-01-06T04:00:09Z

Bittorrent v2 doesn't change the calculus for piece size selection that much. The only concerns v2 eliminates are the up-front overhead of downloading the piece hashes when using magnet links and the amount of data wasted on hash fails.

It seems to me that the current algorithm is too aggressive about increasing the piece size as torrents grow into the gigabytes and beyond. I wonder if we should scale the target piece list size with something like the square root of the total size. Maybe put a floor on it so that small torrents don't sacrifice piece size for an insignificant savings on metadata size. For example, if we used target_size = sqrt(total_size) * 2:

4GB torrent = 1MB piece size = 80KB piece list
16GB = 2MB = 160KB
128GB = 4MB = 640KB
1TB = 16MB = 1.25MB

I think it's a safe assumption that anyone on on 1TB torrent has a connection fast enough for 16MB pieces to be reasonable.

Chocobo1 · 2018-01-06T08:35:34Z

Do you have any references to what people on the internet recommend?

Did a quick search on google and found these:
https://wiki.vuze.com/w/Torrent_Piece_Size
http://torrentinvites.org/f29/piece-size-guide-167985/

8mb and 16mb are useful for large file compilations to get around torrent file size limits on sites

Yeah, in the second guide says 1 MB torrent files is a common limitation, maybe it should be taken into consideration in this PR?
Yet I also think 1MB limit is a bit silly for a mega size content (1 TB), however this is tracker operators problem.

I wonder if we should scale the target piece list size with something like the square root of the total size. Maybe put a floor on it so that small torrents don't sacrifice piece size for an insignificant savings on metadata size.

Agree, I was thinking about it too.

For example, if we used target_size = sqrt(total_size) * 2:

Seems you used 1.25 instead of 2 in the examples?
I think the result is not bad.

arvidn · 2018-01-06T14:35:35Z

I also like the square root formula. @Chocobo1 what do you think about updating this PR to something like what @ssiloti suggested?

One thing to keep in mind, I think it's best to keep piece sizes to even powers of 2. Technically the specification doesn't mandate this and libtorrent supports arbitrary piece sizes, but it's a bit funky.

The question then is what should the coefficient be? is 2 right?

ssiloti · 2018-01-06T17:29:08Z

@Chocobo1 It looks like the coefficient is smaller than 2 because we round the piece size up to the nearest power of 2, so the actual piece list size comes out smaller than the target. Bittorrent v2 requires the piece size to be a power of 2.

Chocobo1 · 2018-01-06T18:40:54Z

@Chocobo1 what do you think about updating this PR to something like what @ssiloti suggested?

Okay, I've updated the PR.

Some results for small size torrents:
torrent content, determined piece size, hash list size, piece count
1MB, 16KB, 1KB, 64
10MB, 32KB, 6KB, 320
100MB, 128KB, 15KB, 800
200MB, 256KB, 15KB, 800
700MB, 512KB, 27KB, 1400

UPDATED: fixed wrong values.

maybe the coefficient can be raised for small torrents?

arvidn · 2018-01-06T19:48:34Z

https://ci.appveyor.com/project/arvidn/libtorrent/build/1.0.8084/job/n2hjudil8ojof93d#L376

arvidn · 2018-01-06T19:48:59Z

src/create_torrent.cpp

-			piece_size = int(fs.total_size() / (target_size / 20));
+			const int hash_size = 20;
+
+			double target_list_size = sqrt(fs.total_size()) * 2;


this should be std::sqrt()

std::sqrt is not a template. (see documentation). The integral square root function wasn't added until C++11, and since this patch is against RC_1_1 (which builds with C++98) it will need to convert to a double. I don't think such conversion is that great. In C++11 there are integer overloads for sqrt() though, so that's a possibility for master.

I'm thinking that the value space here is quite small. We want to round up the result to a power of 2 anyway, with only 11 different answers (16kiB, 32kiB, ..., 8 MiB, 16 MiB).

Would it make sense to just have a table instead?

https://ci.appveyor.com/project/arvidn/libtorrent/build/1.0.8089/job/3r5ac86p4a47qhmw#L238

Would it make sense to just have a table instead?

Indeed, PR updated.
I try keeping sqrt() for master branch.

arvidn · 2018-01-07T18:19:28Z

src/create_torrent.cpp

 			}
-			piece_size = i;
+			piece_size = 16*1024 * pow(2.0, i);


I would really prefer to not have any floating point operations, especially if they're unnecessary.
In this case for instance, you could just do: piece_size = 0x4000 << i, if I'm not mistaken.

arvidn · 2018-01-07T18:24:05Z

I think this table-approach is better and simpler than using std::sqrt() in master too.

arvidn

I would prefer this be done without any floating point operations. I don't think it's necessary and I think it adds complexity

@ssiloti

16 MB is chosen to have a bit more future proof Also rewrite the auto piece size selection algorithm, so that it will scale with torrent content size, suggested by @ssiloti.

Chocobo1 mentioned this pull request Jan 5, 2018

Torrent Creator creates 10x the file size of previous torrents? qbittorrent/qBittorrent#8205

Closed

arvidn reviewed Jan 6, 2018

View reviewed changes

arvidn reviewed Jan 7, 2018

View reviewed changes

arvidn requested changes Jan 7, 2018

View reviewed changes

raise auto piece size selection limit to 16 MB in create_torrent()

c074e9b

16 MB is chosen to have a bit more future proof Also rewrite the auto piece size selection algorithm, so that it will scale with torrent content size, suggested by @ssiloti.

Chocobo1 mentioned this pull request Jan 8, 2018

raise auto piece size selection limit to 16 MB in create_torrent() (master branch) #2678

Closed

arvidn approved these changes Jan 8, 2018

View reviewed changes

arvidn merged commit 1877724 into arvidn:RC_1_1 Jan 8, 2018

Chocobo1 deleted the size_1.1 branch January 9, 2018 03:34

Chocobo1 mentioned this pull request Jan 9, 2018

Torrent creator: raise maximum piece size to 32 MiB qbittorrent/qBittorrent#8240

Merged

Chocobo1 mentioned this pull request Mar 23, 2019

Torrent creator automatic piece size defaults qbittorrent/qBittorrent#10402

Closed

arvidn mentioned this pull request Mar 25, 2019

improve piece picker support for small pieces #3746

Closed

FranciscoPombal mentioned this pull request Jun 16, 2020

Suspect Tracker Issue qbittorrent/qBittorrent#13029

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

raise auto piece size selection limit to 16 MB in create_torrent() #2669

raise auto piece size selection limit to 16 MB in create_torrent() #2669

Chocobo1 commented Jan 5, 2018

Chocobo1 commented Jan 5, 2018 •

edited

Loading

arvidn commented Jan 5, 2018 •

edited

Loading

ssiloti commented Jan 6, 2018

Chocobo1 commented Jan 6, 2018 •

edited

Loading

arvidn commented Jan 6, 2018

ssiloti commented Jan 6, 2018 •

edited

Loading

Chocobo1 commented Jan 6, 2018 •

edited

Loading

arvidn commented Jan 6, 2018

arvidn Jan 6, 2018

Chocobo1 Jan 6, 2018

arvidn Jan 6, 2018

arvidn Jan 6, 2018

Chocobo1 Jan 7, 2018

arvidn Jan 7, 2018

Chocobo1 Jan 8, 2018

arvidn commented Jan 7, 2018

arvidn left a comment •

edited

Loading

raise auto piece size selection limit to 16 MB in create_torrent() #2669

raise auto piece size selection limit to 16 MB in create_torrent() #2669

Conversation

Chocobo1 commented Jan 5, 2018

Chocobo1 commented Jan 5, 2018 • edited Loading

arvidn commented Jan 5, 2018 • edited Loading

ssiloti commented Jan 6, 2018

Chocobo1 commented Jan 6, 2018 • edited Loading

arvidn commented Jan 6, 2018

ssiloti commented Jan 6, 2018 • edited Loading

Chocobo1 commented Jan 6, 2018 • edited Loading

arvidn commented Jan 6, 2018

arvidn Jan 6, 2018

Choose a reason for hiding this comment

Chocobo1 Jan 6, 2018

Choose a reason for hiding this comment

arvidn Jan 6, 2018

Choose a reason for hiding this comment

arvidn Jan 6, 2018

Choose a reason for hiding this comment

Chocobo1 Jan 7, 2018

Choose a reason for hiding this comment

arvidn Jan 7, 2018

Choose a reason for hiding this comment

Chocobo1 Jan 8, 2018

Choose a reason for hiding this comment

arvidn commented Jan 7, 2018

arvidn left a comment • edited Loading

Choose a reason for hiding this comment

Chocobo1 commented Jan 5, 2018 •

edited

Loading

arvidn commented Jan 5, 2018 •

edited

Loading

Chocobo1 commented Jan 6, 2018 •

edited

Loading

ssiloti commented Jan 6, 2018 •

edited

Loading

Chocobo1 commented Jan 6, 2018 •

edited

Loading

arvidn left a comment •

edited

Loading