-
Notifications
You must be signed in to change notification settings - Fork 1
/
Copy pathbep_0003.html
385 lines (378 loc) · 21.7 KB
/
bep_0003.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
<?xml version="1.0" encoding="utf-8" ?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="Docutils 0.5: http://docutils.sourceforge.net/" />
<title></title>
<link rel="stylesheet" href="../css/bep.css" type="text/css" />
</head>
<body>
<div class="document">
<div id="upper" class="clear">
<div id="wrap">
<div id="header">
<h1><a href="../index.html">BitTorrent<span>.org</span></a></h1>
</div>
<div id="nav">
<ul>
<li><a href="../index.html">Home</a></li>
<li><a href="../introduction.html">For Users</a></li>
<li><a href="bep_0000.html"><span>For Developers</span></a></li>
<!-- <li><a href="./blog">Blog</a></li> -->
<li><a href="http://forum.bittorrent.org"> Forums </li>
<li><a href="../donate.html">Donate!</a></li>
</ul>
</div> <!-- nav -->
<!-- ### Begin Content ### -->
<div id="second">
<table class="rfc2822 docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field"><th class="field-name">BEP:</th><td class="field-body">3</td>
</tr>
<tr class="field"><th class="field-name">Title:</th><td class="field-body">The BitTorrent Protocol Specification</td>
</tr>
<tr class="field"><th class="field-name">Version:</th><td class="field-body">11031</td>
</tr>
<tr class="field"><th class="field-name">Last-Modified:</th><td class="field-body"><a class="reference" href="http://bittorrent.org/trac/browser/dotorg/trunk/html/beps/bep_0003.rst">2008-02-28 16:43:58 -0800 (Thu, 28 Feb 2008)</a></td>
</tr>
<tr class="field"><th class="field-name">Author:</th><td class="field-body">Bram Cohen <bram at bittorrent.com></td>
</tr>
<tr class="field"><th class="field-name">Status:</th><td class="field-body">Final</td>
</tr>
<tr class="field"><th class="field-name">Type:</th><td class="field-body">Standard</td>
</tr>
<tr class="field"><th class="field-name">Created:</th><td class="field-body">10-Jan-2008</td>
</tr>
<tr class="field"><th class="field-name">Post-History:</th><td class="field-body">24-Jun-2009, clarified the encoding of strings in torrent files</td>
</tr>
</tbody>
</table>
<hr />
<div class="contents topic" id="contents">
<p class="topic-title first">Contents</p>
<ul class="simple">
<li><a class="reference" href="#a-bittorrent-file-distribution-consists-of-these-entities" id="id2">A BitTorrent file distribution consists of these entities:</a></li>
<li><a class="reference" href="#to-start-serving-a-host-goes-through-the-following-steps" id="id3">To start serving, a host goes through the following steps:</a></li>
<li><a class="reference" href="#to-start-downloading-a-user-does-the-following" id="id4">To start downloading, a user does the following:</a></li>
<li><a class="reference" href="#the-connectivity-is-as-follows" id="id5">The connectivity is as follows:</a></li>
<li><a class="reference" href="#metainfo-files-are-bencoded-dictionaries-with-the-following-keys" id="id6">Metainfo files are bencoded dictionaries with the following keys:</a></li>
<li><a class="reference" href="#tracker-get-requests-have-the-following-keys" id="id7">Tracker GET requests have the following keys:</a></li>
<li><a class="reference" href="#all-non-keepalive-messages-start-with-a-single-byte-which-gives-their-type" id="id8">All non-keepalive messages start with a single byte which gives their type.</a></li>
<li><a class="reference" href="#the-possible-values-are" id="id9">The possible values are:</a></li>
<li><a class="reference" href="#copyright" id="id10">Copyright</a></li>
</ul>
</div>
<p>BitTorrent is a protocol for distributing files. It identifies content
by URL and is designed to integrate seamlessly with the web. Its
advantage over plain HTTP is that when multiple downloads of the same
file happen concurrently, the downloaders upload to each other, making
it possible for the file source to support very large numbers of
downloaders with only a modest increase in its load.</p>
<div class="section" id="a-bittorrent-file-distribution-consists-of-these-entities">
<h1>A BitTorrent file distribution consists of these entities:</h1>
<ul class="simple">
<li>An ordinary web server</li>
<li>A static 'metainfo' file</li>
<li>A BitTorrent tracker</li>
<li>An 'original' downloader</li>
<li>The end user web browsers</li>
<li>The end user downloaders</li>
</ul>
<p>There are ideally many end users for a single file.</p>
</div>
<div class="section" id="to-start-serving-a-host-goes-through-the-following-steps">
<h1>To start serving, a host goes through the following steps:</h1>
<ol class="arabic simple">
<li>Start running a tracker (or, more likely, have one running already).</li>
<li>Start running an ordinary web server, such as apache, or have one already.</li>
<li>Associate the extension .torrent with mimetype application/x-bittorrent on their web server (or have done so already).</li>
<li>Generate a metainfo (.torrent) file using the complete file to be served and the URL of the tracker.</li>
<li>Put the metainfo file on the web server.</li>
<li>Link to the metainfo (.torrent) file from some other web page.</li>
<li>Start a downloader which already has the complete file (the 'origin').</li>
</ol>
</div>
<div class="section" id="to-start-downloading-a-user-does-the-following">
<h1>To start downloading, a user does the following:</h1>
<ol class="arabic simple">
<li>Install BitTorrent (or have done so already).</li>
<li>Surf the web.</li>
<li>Click on a link to a .torrent file.</li>
<li>Select where to save the file locally, or select a partial download to resume.</li>
<li>Wait for download to complete.</li>
<li>Tell downloader to exit (it keeps uploading until this happens).</li>
</ol>
</div>
<div class="section" id="the-connectivity-is-as-follows">
<h1>The connectivity is as follows:</h1>
<ul class="simple">
<li>Strings are length-prefixed base ten followed by a colon and the string. For example <tt class="docutils literal"><span class="pre">4:spam</span></tt> corresponds to 'spam'.</li>
<li>Integers are represented by an 'i' followed by the number in base 10
followed by an 'e'. For example <tt class="docutils literal"><span class="pre">i3e</span></tt> corresponds to 3 and
<tt class="docutils literal"><span class="pre">i-3e</span></tt> corresponds to -3. Integers have no size
limitation. <tt class="docutils literal"><span class="pre">i-0e</span></tt> is invalid. All encodings with a leading
zero, such as <tt class="docutils literal"><span class="pre">i03e</span></tt>, are invalid, other than
<tt class="docutils literal"><span class="pre">i0e</span></tt>, which of course corresponds to 0.</li>
<li>Lists are encoded as an 'l' followed by their elements (also
bencoded) followed by an 'e'. For example <tt class="docutils literal"><span class="pre">l4:spam4:eggse</span></tt>
corresponds to ['spam', 'eggs'].</li>
<li>Dictionaries are encoded as a 'd' followed by a list of alternating
keys and their corresponding values followed by an 'e'. For example,
<tt class="docutils literal"><span class="pre">d3:cow3:moo4:spam4:eggse</span></tt> corresponds to {'cow': 'moo',
'spam': 'eggs'} and <tt class="docutils literal"><span class="pre">d4:spaml1:a1:bee</span></tt> corresponds to
{'spam': ['a', 'b']}. Keys must be strings and appear in sorted order
(sorted as raw strings, not alphanumerics).</li>
</ul>
</div>
<div class="section" id="metainfo-files-are-bencoded-dictionaries-with-the-following-keys">
<h1>Metainfo files are bencoded dictionaries with the following keys:</h1>
<dl class="docutils">
<dt>announce</dt>
<dd>The URL of the tracker.</dd>
<dt>info</dt>
<dd><p class="first">This maps to a dictionary, with keys described below.</p>
<p>The <tt class="docutils literal"><span class="pre">name</span></tt> key maps to a UTF-8 encoded string which is the
suggested name to save the file (or directory) as. It is purely advisory.</p>
<p><tt class="docutils literal"><span class="pre">piece</span> <span class="pre">length</span></tt> maps to the number of bytes in each piece
the file is split into. For the purposes of transfer, files are
split into fixed-size pieces which are all the same length except for
possibly the last one which may be truncated. <tt class="docutils literal"><span class="pre">piece</span>
<span class="pre">length</span></tt> is almost always a power of two, most commonly 2 18 =
256 K (BitTorrent prior to version 3.2 uses 2 20 = 1 M as
default).</p>
<p><tt class="docutils literal"><span class="pre">pieces</span></tt> maps to a string whose length is a multiple of
20. It is to be subdivided into strings of length 20, each of which is
the SHA1 hash of the piece at the corresponding index.</p>
<p>There is also a key <tt class="docutils literal"><span class="pre">length</span></tt> or a key <tt class="docutils literal"><span class="pre">files</span></tt>,
but not both or neither. If <tt class="docutils literal"><span class="pre">length</span></tt> is present then the
download represents a single file, otherwise it represents a set of
files which go in a directory structure.</p>
<p>In the single file case, <tt class="docutils literal"><span class="pre">length</span></tt> maps to the length of
the file in bytes.</p>
<p>For the purposes of the other keys, the multi-file case is treated as
only having a single file by concatenating the files in the order they
appear in the files list. The files list is the value
<tt class="docutils literal"><span class="pre">files</span></tt> maps to, and is a list of dictionaries containing
the following keys:</p>
<p><tt class="docutils literal"><span class="pre">length</span></tt> - The length of the file, in bytes.</p>
<p><tt class="docutils literal"><span class="pre">path</span></tt> - A list of UTF-8 encoded strings corresponding to subdirectory
names, the last of which is the actual file name (a zero length list
is an error case).</p>
<p>In the single file case, the name key is the name of a file, in the
muliple file case, it's the name of a directory.</p>
<p class="last">All strings in a .torrent file that contains text must be UTF-8
encoded.</p>
</dd>
</dl>
</div>
<div class="section" id="tracker-get-requests-have-the-following-keys">
<h1>Tracker GET requests have the following keys:</h1>
<dl class="docutils">
<dt>info_hash</dt>
<dd>The 20 byte sha1 hash of the bencoded form of the info value from the
metainfo file. Note that this is a substring of the metainfo
file. This value will almost certainly have to be escaped.</dd>
<dt>peer_id</dt>
<dd>A string of length 20 which this downloader uses as its id. Each
downloader generates its own id at random at the start of a new
download. This value will also almost certainly have to be escaped.</dd>
<dt>ip</dt>
<dd>An optional parameter giving the IP (or dns name) which this peer is
at. Generally used for the origin if it's on the same machine as the
tracker.</dd>
<dt>port</dt>
<dd>The port number this peer is listening on. Common behavior is for a
downloader to try to listen on port 6881 and if that port is taken try
6882, then 6883, etc. and give up after 6889.</dd>
<dt>uploaded</dt>
<dd>The total amount uploaded so far, encoded in base ten ascii.</dd>
<dt>downloaded</dt>
<dd>The total amount downloaded so far, encoded in base ten ascii.</dd>
<dt>left</dt>
<dd>The number of bytes this peer still has to download, encoded in
base ten ascii. Note that this can't be computed from downloaded and
the file length since it might be a resume, and there's a chance that
some of the downloaded data failed an integrity check and had to be
re-downloaded.</dd>
<dt>event</dt>
<dd>This is an optional key which maps to <tt class="docutils literal"><span class="pre">started</span></tt>,
<tt class="docutils literal"><span class="pre">completed</span></tt>, or <tt class="docutils literal"><span class="pre">stopped</span></tt> (or
<tt class="docutils literal"><span class="pre">empty</span></tt>, which is the same as not being present). If not
present, this is one of the announcements done at regular
intervals. An announcement using <tt class="docutils literal"><span class="pre">started</span></tt> is sent when a
download first begins, and one using <tt class="docutils literal"><span class="pre">completed</span></tt> is sent
when the download is complete. No <tt class="docutils literal"><span class="pre">completed</span></tt> is sent if
the file was complete when started. Downloaders send an announcement
using <tt class="docutils literal"><span class="pre">stopped</span></tt> when they cease downloading.</dd>
</dl>
<p>Tracker responses are bencoded dictionaries. If a tracker response
has a key <tt class="docutils literal"><span class="pre">failure</span> <span class="pre">reason</span></tt>, then that maps to a human
readable string which explains why the query failed, and no other keys
are required. Otherwise, it must have two keys: <tt class="docutils literal"><span class="pre">interval</span></tt>,
which maps to the number of seconds the downloader should wait between
regular rerequests, and <tt class="docutils literal"><span class="pre">peers</span></tt>. <tt class="docutils literal"><span class="pre">peers</span></tt> maps to
a list of dictionaries corresponding to <tt class="docutils literal"><span class="pre">peers</span></tt>, each of
which contains the keys <tt class="docutils literal"><span class="pre">peer</span> <span class="pre">id</span></tt>, <tt class="docutils literal"><span class="pre">ip</span></tt>, and
<tt class="docutils literal"><span class="pre">port</span></tt>, which map to the peer's self-selected ID, IP
address or dns name as a string, and port number, respectively. Note
that downloaders may rerequest on nonscheduled times if an event
happens or they need more peers.</p>
<p>If you want to make any extensions to metainfo files or tracker
queries, please coordinate with Bram Cohen to make sure that all
extensions are done compatibly.</p>
<p>BitTorrent's peer protocol operates over TCP. It performs efficiently
without setting any socket options.</p>
<p>Peer connections are symmetrical. Messages sent in both directions
look the same, and data can flow in either direction.</p>
<p>The peer protocol refers to pieces of the file by index as
described in the metainfo file, starting at zero. When a peer finishes
downloading a piece and checks that the hash matches, it announces
that it has that piece to all of its peers.</p>
<p>Connections contain two bits of state on either end: choked or not,
and interested or not. Choking is a notification that no data will be
sent until unchoking happens. The reasoning and common techniques
behind choking are explained later in this document.</p>
<p>Data transfer takes place whenever one side is interested and the
other side is not choking. Interest state must be kept up to date at
all times - whenever a downloader doesn't have something they
currently would ask a peer for in unchoked, they must express lack of
interest, despite being choked. Implementing this properly is tricky,
but makes it possible for downloaders to know which peers will start
downloading immediately if unchoked.</p>
<p>Connections start out choked and not interested.</p>
<p>When data is being transferred, downloaders should keep several
piece requests queued up at once in order to get good TCP performance
(this is called 'pipelining'.) On the other side, requests which can't
be written out to the TCP buffer immediately should be queued up in
memory rather than kept in an application-level network buffer, so
they can all be thrown out when a choke happens.</p>
<p>The peer wire protocol consists of a handshake followed by a
never-ending stream of length-prefixed messages. The handshake starts
with character ninteen (decimal) followed by the string 'BitTorrent
protocol'. The leading character is a length prefix, put there in the
hope that other new protocols may do the same and thus be trivially
distinguishable from each other.</p>
<p>All later integers sent in the protocol are encoded as four bytes
big-endian.</p>
<p>After the fixed headers come eight reserved bytes, which are all
zero in all current implementations. If you wish to extend the
protocol using these bytes, please coordinate with Bram Cohen to make
sure all extensions are done compatibly.</p>
<p>Next comes the 20 byte sha1 hash of the bencoded form of the info
value from the metainfo file. (This is the same value which is
announced as <tt class="docutils literal"><span class="pre">info_hash</span></tt> to the tracker, only here it's raw
instead of quoted here). If both sides don't send the same value, they
sever the connection. The one possible exception is if a downloader
wants to do multiple downloads over a single port, they may wait for
incoming connections to give a download hash first, and respond with
the same one if it's in their list.</p>
<p>After the download hash comes the 20-byte peer id which is reported
in tracker requests and contained in peer lists in tracker
responses. If the receiving side's peer id doesn't match the one the
initiating side expects, it severs the connection.</p>
<p>That's it for handshaking, next comes an alternating stream of
length prefixes and messages. Messages of length zero are keepalives,
and ignored. Keepalives are generally sent once every two minutes, but
note that timeouts can be done much more quickly when data is
expected.</p>
</div>
<div class="section" id="all-non-keepalive-messages-start-with-a-single-byte-which-gives-their-type">
<h1>All non-keepalive messages start with a single byte which gives their type.</h1>
</div>
<div class="section" id="the-possible-values-are">
<h1>The possible values are:</h1>
<ul class="simple">
<li>0 - choke</li>
<li>1 - unchoke</li>
<li>2 - interested</li>
<li>3 - not interested</li>
<li>4 - have</li>
<li>5 - bitfield</li>
<li>6 - request</li>
<li>7 - piece</li>
<li>8 - cancel</li>
</ul>
<p>'choke', 'unchoke', 'interested', and 'not interested' have no payload.</p>
<p>'bitfield' is only ever sent as the first message. Its payload is a
bitfield with each index that downloader has sent set to one and the
rest set to zero. Downloaders which don't have anything yet may skip
the 'bitfield' message. The first byte of the bitfield corresponds to
indices 0 - 7 from high bit to low bit, respectively. The next one
8-15, etc. Spare bits at the end are set to zero.</p>
<p>The 'have' message's payload is a single number, the index which
that downloader just completed and checked the hash of.</p>
<p>'request' messages contain an index, begin, and length. The last
two are byte offsets. Length is generally a power of two unless it
gets truncated by the end of the file. All current implementations use
2 15 , and close connections which request an amount greater than 2
17.</p>
<p>'cancel' messages have the same payload as request messages. They
are generally only sent towards the end of a download, during what's
called 'endgame mode'. When a download is almost complete, there's a
tendency for the last few pieces to all be downloaded off a single
hosed modem line, taking a very long time. To make sure the last few
pieces come in quickly, once requests for all pieces a given
downloader doesn't have yet are currently pending, it sends requests
for everything to everyone it's downloading from. To keep this from
becoming horribly inefficient, it sends cancels to everyone else every
time a piece arrives.</p>
<p>'piece' messages contain an index, begin, and piece. Note that they
are correlated with request messages implicitly. It's possible for an
unexpected piece to arrive if choke and unchoke messages are sent in
quick succession and/or transfer is going very slowly.</p>
<p>Downloaders generally download pieces in random order, which does a
reasonably good job of keeping them from having a strict subset or
superset of the pieces of any of their peers.</p>
<p>Choking is done for several reasons. TCP congestion control behaves
very poorly when sending over many connections at once. Also, choking
lets each peer use a tit-for-tat-ish algorithm to ensure that they get
a consistent download rate.</p>
<p>The choking algorithm described below is the currently deployed
one. It is very important that all new algorithms work well both in a
network consisting entirely of themselves and in a network consisting
mostly of this one.</p>
<p>There are several criteria a good choking algorithm should meet. It
should cap the number of simultaneous uploads for good TCP
performance. It should avoid choking and unchoking quickly, known as
'fibrillation'. It should reciprocate to peers who let it
download. Finally, it should try out unused connections once in a
while to find out if they might be better than the currently used
ones, known as optimistic unchoking.</p>
<p>The currently deployed choking algorithm avoids fibrillation by
only changing who's choked once every ten seconds. It does
reciprocation and number of uploads capping by unchoking the four
peers which it has the best download rates from and are
interested. Peers which have a better upload rate but aren't
interested get unchoked and if they become interested the worst
uploader gets choked. If a downloader has a complete file, it uses its
upload rate rather than its download rate to decide who to
unchoke.</p>
<p>For optimistic unchoking, at any one time there is a single peer
which is unchoked regardless of it's upload rate (if interested, it
counts as one of the four allowed downloaders.) Which peer is
optimistically unchoked rotates every 30 seconds. To give them a
decent chance of getting a complete piece to upload, new connections
are three times as likely to start as the current optimistic unchoke
as anywhere else in the rotation.</p>
</div>
<div class="section" id="copyright">
<h1>Copyright</h1>
<p>This document has been placed in the public domain.</p>
<!-- Local Variables:
mode: indented-text
indent-tabs-mode: nil
sentence-end-double-space: t
fill-column: 70
coding: utf-8
End: -->
</div>
</div>
<div id="footer">
<hr/>
</div>
</div>
</body>
</html>