-
Notifications
You must be signed in to change notification settings - Fork 6
/
rfc3640.txt
executable file
·2411 lines (1672 loc) · 100 KB
/
rfc3640.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Network Working Group J. van der Meer
Request for Comments: 3640 Philips Electronics
Category: Standards Track D. Mackie
Apple Computer
V. Swaminathan
Sun Microsystems Inc.
D. Singer
Apple Computer
P. Gentric
Philips Electronics
November 2003
RTP Payload Format for Transport of MPEG-4 Elementary Streams
Status of this Memo
This document specifies an Internet standards track protocol for the
Internet community, and requests discussion and suggestions for
improvements. Please refer to the current edition of the "Internet
Official Protocol Standards" (STD 1) for the standardization state
and status of this protocol. Distribution of this memo is unlimited.
Copyright Notice
Copyright (C) The Internet Society (2003). All Rights Reserved.
Abstract
The Motion Picture Experts Group (MPEG) Committee (ISO/IEC JTC1/SC29
WG11) is a working group in ISO that produced the MPEG-4 standard.
MPEG defines tools to compress content such as audio-visual
information into elementary streams. This specification defines a
simple, but generic RTP payload format for transport of any non-
multiplexed MPEG-4 elementary stream.
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3
2. Carriage of MPEG-4 Elementary Streams Over RTP . . . . . . . . 4
2.1. Signaling by MIME Format Parameters . . . . . . . . . . 4
2.2. MPEG Access Units . . . . . . . . . . . . . . . . . . . 5
2.3. Concatenation of Access Units . . . . . . . . . . . . . 5
2.4. Fragmentation of Access Units . . . . . . . . . . . . . 6
2.5. Interleaving . . . . . . . . . . . . . . . . . . . . . . 6
2.6. Time Stamp Information . . . . . . . . . . . . . . . . . 7
2.7. State Indication of MPEG-4 System Streams . . . . . . . 8
2.8. Random Access Indication . . . . . . . . . . . . . . . . 8
van der Meer, et al. Standards Track [Page 1]
RFC 3640 Transport of MPEG-4 Elementary Streams November 2003
2.9. Carriage of Auxiliary Information . . . . . . . . . . . 8
2.10. MIME Format Parameters and Configuring Conditional Field 8
2.11. Global Structure of Payload Format . . . . . . . . . . . 9
2.12. Modes to Transport MPEG-4 Streams . . . . . . . . . . . 9
2.13. Alignment with RFC 3016 . . . . . . . . . . . . . . . . 10
3. Payload Format . . . . . . . . . . . . . . . . . . . . . . . . 10
3.1. Usage of RTP Header Fields and RTCP . . . . . . . . . . 10
3.2. RTP Payload Structure . . . . . . . . . . . . . . . . . 11
3.2.1. The AU Header Section . . . . . . . . . . . . . 11
3.2.1.1. The AU-header . . . . . . . . . . . . 12
3.2.2. The Auxiliary Section . . . . . . . . . . . . . 14
3.2.3. The Access Unit Data Section . . . . . . . . . . 15
3.2.3.1. Fragmentation. . . . . . . . . . . . . 16
3.2.3.2. Interleaving . . . . . . . . . . . . . 16
3.2.3.3. Constraints for Interleaving . . . . . 17
3.2.3.4. Crucial and Non-Crucial AUs with
MPEG-4 System Data . . . . . . . . . . 20
3.3. Usage of this Specification. . . . . . . . . . . . . . . 21
3.3.1. General. . . . . . . . . . . . . . . . . . . . . 21
3.3.2. The Generic Mode . . . . . . . . . . . . . . . . 22
3.3.3. Constant Bit Rate CELP . . . . . . . . . . . . . 22
3.3.4. Variable Bit Rate CELP . . . . . . . . . . . . . 23
3.3.5. Low Bit Rate AAC . . . . . . . . . . . . . . . . 24
3.3.6. High Bit Rate AAC. . . . . . . . . . . . . . . . 25
3.3.7. Additional Modes . . . . . . . . . . . . . . . . 26
4. IANA Considerations. . . . . . . . . . . . . . . . . . . . . . 27
4.1. MIME Type Registration . . . . . . . . . . . . . . . . . 27
4.2. Registration of Mode Definitions with IANA . . . . . . . 33
4.3. Concatenation of Parameters. . . . . . . . . . . . . . . 33
4.4. Usage of SDP . . . . . . . . . . . . . . . . . . . . . . 34
4.4.1. The a=fmtp Keyword . . . . . . . . . . . . . . . 34
5. Security Considerations. . . . . . . . . . . . . . . . . . . . 34
6. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 35
APPENDIX: Usage of this Payload Format. . . . . . . . . . . . . . 36
Appendix A. Interleave Analysis . . . . . . . . . . . . . . . . . 36
A. Examples of Delay Analysis with Interleave. . . . . . . . . . 36
A.1. Introduction . . . . . . . . . . . . . . . . . . . . . . 36
A.2. De-interleaving and Error Concealment . . . . . . . . . 36
A.3. Simple Group Interleave . . . . . . . . . . . . . . . . 36
A.3.1. Introduction . . . . . . . . . . . . . . . . . . 36
A.3.2. Determining the De-interleave Buffer Size . . . 37
A.3.3. Determining the Maximum Displacement . . . . . . 37
A.4. More Subtle Group Interleave . . . . . . . . . . . . . . 38
A.4.1. Introduction . . . . . . . . . . . . . . . . . . 38
A.4.2. Determining the De-interleave Buffer Size. . . . 38
A.4.3. Determining the Maximum Displacement . . . . . . 39
A.5. Continuous Interleave . . . . . . . . . . . . . . . . . 39
A.5.1. Introduction . . . . . . . . . . . . . . . . . . 39
van der Meer, et al. Standards Track [Page 2]
RFC 3640 Transport of MPEG-4 Elementary Streams November 2003
A.5.2. Determining the De-interleave Buffer Size . . . 40
A.5.3. Determining the Maximum Displacement . . . . . . 40
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
Normative References . . . . . . . . . . . . . . . . . . . . . . . 41
Informative References . . . . . . . . . . . . . . . . . . . . . . 41
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 42
Full Copyright Statement . . . . . . . . . . . . . . . . . . . . . 43
1. Introduction
The MPEG Committee is Working Group 11 (WG11) in ISO/IEC JTC1 SC29
that specified the MPEG-1, MPEG-2 and, more recently, the MPEG-4
standards [1]. The MPEG-4 standard specifies compression of audio-
visual data into, for example an audio or video elementary stream.
In the MPEG-4 standard, these streams take the form of audio-visual
objects that may be arranged into an audio-visual scene by means of a
scene description. Each MPEG-4 elementary stream consists of a
sequence of Access Units; examples of an Access Unit (AU) are an
audio frame and a video picture.
This specification defines a general and configurable payload
structure to transport MPEG-4 elementary streams, in particular
MPEG-4 audio (including speech) streams, MPEG-4 video streams and
also MPEG-4 systems streams, such as BIFS (BInary Format for Scenes),
OCI (Object Content Information), OD (Object Descriptor) and IPMP
(Intellectual Property Management and Protection) streams. The RTP
payload defined in this document is simple to implement and
reasonably efficient. It allows for optional interleaving of Access
Units (such as audio frames) to increase error resiliency in packet
loss.
Some types of MPEG-4 elementary streams include "crucial" information
whose loss cannot be tolerated. However, RTP does not provide
reliable transmission, so receipt of that crucial information is not
assured. Section 3.2.3.4 specifies how stream state is conveyed so
that the receiver can detect the loss of crucial information and
cease decoding until the next random access point has been received.
Applications transmitting streams that include crucial information,
such as OD commands, BIFS commands, or programmatic content such as
MPEG-J (Java) and ECMAScript, should include random access points, at
a suitable periodicity depending upon the probability of loss, in
order to reduce stream corruption to an acceptable level. An example
is the carousel mechanism as defined by MPEG in ISO/IEC 14496-1 [1].
Such applications may also employ additional protocols or services to
reduce the probability of loss. At the RTP layer, these measures
include payload formats and profiles for retransmission or forward
error correction (such as in RFC 2733 [10]), that must be employed
van der Meer, et al. Standards Track [Page 3]
RFC 3640 Transport of MPEG-4 Elementary Streams November 2003
with due consideration to congestion control. Another solution that
may be appropriate for some applications is to carry RTP over TCP
(such as in RFC 2326 [8], section 10.12). At the network layer,
resource allocation or preferential service may be available to
reduce the probability of loss. For a general description of methods
to repair streaming media, see RFC 2354 [9].
Though the RTP payload format defined in this document is capable of
transporting any MPEG-4 stream, other, more specific, formats may
exist, such as RFC 3016 [12] for transport of MPEG-4 video (ISO/IEC
14496 [1] part 2).
Configuration of the payload is provided to accommodate the
transportation of any MPEG-4 stream at any possible bit rate.
However, for a specific MPEG-4 elementary stream typically only very
few configurations are needed. So as to allow for the design of
simplified, but dedicated receivers, this specification requires that
specific modes be defined for transport of MPEG-4 streams. This
document defines modes for MPEG-4 CELP and AAC streams, as well as a
generic mode that can be used to transport any MPEG-4 stream. In the
future, new RFCs are expected to specify additional modes for the
transportation of MPEG-4 streams.
The RTP payload format defined in this document specifies carriage of
system-related information that is often equivalent to the
information that may be contained in the MPEG-4 Sync Layer (SL) as
defined in MPEG-4 Systems [1]. This document does not prescribe how
to transcode or map information from the SL to fields defined in the
RTP payload format. Such processing, if any, is left to the
discretion of the application. However, to anticipate the need for
the transportation of any additional system-related information in
the future, an auxiliary field can be configured that may carry any
such data.
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in BCP 14, RFC 2119 [4].
2. Carriage of MPEG-4 Elementary Streams over RTP
2.1. Signaling by MIME Format Parameters
With this payload format, a single MPEG-4 elementary stream can be
transported. Information on the type of MPEG-4 stream carried in the
payload is conveyed by MIME format parameters, as in an SDP [5]
message or by other means (see section 4). These MIME format
parameters specify the configuration of the payload. To allow for
simplified and dedicated receivers, a MIME format parameter is
van der Meer, et al. Standards Track [Page 4]
RFC 3640 Transport of MPEG-4 Elementary Streams November 2003
available to signal a specific mode of using this payload. A mode
definition MAY include the type of MPEG-4 elementary stream, as well
as the applied configuration, so as to avoid the need for receivers
to parse all MIME format parameters. The applied mode MUST be
signaled.
2.2. MPEG Access Units
For carriage of compressed audio-visual data, MPEG defines Access
Units. An MPEG Access Unit (AU) is the smallest data entity to which
timing information is attributed. In the case of audio, an Access
Unit may represent an audio frame and in the case of video, a
picture. MPEG Access Units are octet-aligned by definition. If, for
example, an audio frame is not octet-aligned, up to 7 zero-padding
bits MUST be inserted at the end of the frame to achieve the octet-
aligned Access Units, as required by the MPEG-4 specification.
MPEG-4 decoders MUST be able to decode AUs in which such padding is
applied.
Consistent with the MPEG-4 specification, this document requires that
each MPEG-4 part 2 video Access Unit include all the coded data of a
picture, any video stream headers that may precede the coded picture
data, and any video stream stuffing that may follow it, up to but not
including the startcode indicating the start of a new video stream or
the next Access Unit.
2.3. Concatenation of Access Units
Frequently it is possible to carry multiple Access Units in one RTP
packet. This is particularly useful for audio; for example, when AAC
is used for encoding a stereo signal at 64 kbits/sec, AAC frames
contain on average, approximately 200 octets. On a LAN with a 1500
octet MTU, this would allow an average of 7 complete AAC frames to be
carried per RTP packet.
Access Units may have a fixed size in octets, but a variable size is
also possible. To facilitate parsing in the case of multiple
concatenated AUs in one RTP packet, the size of each AU is made known
to the receiver. When concatenating in the case of a constant AU
size, this size is communicated "out of band" through a MIME format
parameter. When concatenating in case of variable size AUs, the RTP
payload carries "in band" an AU size field for each contained AU.
In combination with the RTP payload length, the size information
allows the RTP payload to be split by the receiver back into the
individual AUs.
van der Meer, et al. Standards Track [Page 5]
RFC 3640 Transport of MPEG-4 Elementary Streams November 2003
To simplify the implementation of RTP receivers, it is required that
when multiple AUs are carried in an RTP packet, each AU MUST be
complete, i.e., the number of AUs in an RTP packet MUST be integral.
In addition, an AU MUST NOT be repeated in other RTP packets; hence
repetition of an AU is only possible when using a duplicate RTP
packet.
2.4. Fragmentation of Access Units
MPEG allows for very large Access Units. Since most IP networks have
significantly smaller MTU sizes, this payload format allows for the
fragmentation of an Access Unit over multiple RTP packets. Hence,
when an IP packet is lost after IP-level fragmentation, only an AU
fragment may get lost instead of the entire AU. To simplify the
implementation of RTP receivers, an RTP packet SHALL either carry one
or more complete Access Units or a single fragment of one AU, i.e.,
packets MUST NOT contain fragments of multiple Access Units.
2.5. Interleaving
When an RTP packet carries a contiguous sequence of Access Units, the
loss of such a packet can result in a "decoding gap" for the user.
One method of alleviating this problem is to allow for the Access
Units to be interleaved in the RTP packets. For a modest cost in
latency and implementation complexity, significant error resiliency
to packet loss can be achieved.
To support optional interleaving of Access Units, this payload format
allows for index information to be sent for each Access Unit. After
informing receivers about buffer resources to allocate for de-
interleaving, the RTP sender is free to choose the interleaving
pattern without propagating this information a priori to the
receiver(s). Indeed, the sender could dynamically adjust the
interleaving pattern based on the Access Unit size, error rates, etc.
The RTP receiver does not need to know the interleaving pattern used;
it only needs to extract the index information of the Access Unit and
insert the Access Unit into the appropriate sequence in the decoding
or rendering queue. An example of interleaving is given below.
van der Meer, et al. Standards Track [Page 6]
RFC 3640 Transport of MPEG-4 Elementary Streams November 2003
For example, if we assume that an RTP packet contains 3 AUs, and that
the AUs are numbered 0, 1, 2, 3, 4, and so forth, and if an
interleaving group length of 9 is chosen, then RTP packet(i) contains
the following AU(n):
RTP packet(0): AU(0), AU(3), AU(6)
RTP packet(1): AU(1), AU(4), AU(7)
RTP packet(2): AU(2), AU(5), AU(8)
RTP packet(3): AU(9), AU(12), AU(15)
RTP packet(4): AU(10), AU(13), AU(16) Etc.
2.6. Time Stamp Information
The RTP time stamp MUST carry the sampling instant of the first AU
(fragment) in the RTP packet. When multiple AUs are carried within
an RTP packet, the time stamps of subsequent AUs can be calculated if
the frame period of each AU is known. For audio and video, this is
possible if the frame rate is constant. However, in some cases it is
not possible to make such a calculation (for example, for variable
frame rate video, or for MPEG-4 BIFS streams carrying composition
information). To support such cases, this payload format can be
configured to carry a time stamp in the RTP payload for each
contained Access Unit. A time stamp MAY be conveyed in the RTP
payload only for non-first AUs in the RTP packet, and SHALL NOT be
conveyed for the first AU (fragment), as the time stamp for the first
AU in the RTP packet is carried by the RTP time stamp.
MPEG-4 defines two types of time stamps: the composition time stamp
(CTS) and the decoding time stamp (DTS). The CTS represents the
sampling instant of an AU, and hence the CTS is equivalent to the RTP
time stamp. The DTS may be used in MPEG-4 video streams that use
bi-directional coding, i.e., when pictures are predicted in both
forward and backward direction by using either a reference picture in
the past, or a reference picture in the future. The DTS cannot be
carried in the RTP header. In some cases, the DTS can be derived
from the RTP time stamp using frame rate information; this requires
deep parsing in the video stream, which may be considered
objectionable. If the video frame rate is variable, the required
information may not even be present in the video stream. For both
reasons, the capability has been defined to optionally carry the DTS
in the RTP payload for each contained Access Unit.
To keep the coding of time stamps efficient, each time stamp
contained in the RTP payload is coded as a difference. For the CTS,
the offset from the RTP time stamps is provided, and for the DTS, the
offset from the CTS.
van der Meer, et al. Standards Track [Page 7]
RFC 3640 Transport of MPEG-4 Elementary Streams November 2003
2.7. State Indication of MPEG-4 System Streams
ISO/IEC 14496-1 defines states for MPEG-4 system streams. So as to
convey state information when transporting MPEG-4 system streams,
this payload format allows for the optional carriage in the RTP
payload of the stream state for each contained Access Unit. Stream
states are used to signal "crucial" AUs that carry information whose
loss cannot be tolerated and are also useful when repeating AUs
according to the carousel mechanism defined in ISO/IEC 14496-1.
2.8. Random Access Indication
Random access to the content of MPEG-4 elementary streams may be
possible at some but not all Access Units. To signal Access Units
where random access is possible, a random access point flag can
optionally be carried in the RTP payload for each contained Access
Unit. Carriage of random access points is particularly useful for
MPEG-4 system streams in combination with the stream state.
2.9. Carriage of Auxiliary Information
This payload format defines a specific field to carry auxiliary data.
The auxiliary data field is preceded by a field that specifies the
length of the auxiliary data, so as to facilitate the skipping of
data without parsing it. The coding of the auxiliary data is not
defined in this document; instead, the format, meaning and signaling
of auxiliary information is expected to be specified in one or more
future RFCs. Auxiliary information MUST NOT be transmitted until its
format, meaning and signaling have been specified and its use has
been signaled. Receivers that have knowledge of the auxiliary data
MAY decode the auxiliary data, but receivers without knowledge of
such data MUST skip the auxiliary data field.
2.10. MIME Format Parameters and Configuring Conditional Fields
To support the features described in the previous sections, several
fields are defined for carriage in the RTP payload. However, their
use strongly depends on the type of MPEG-4 elementary stream that is
carried. Sometimes a specific field is needed with a certain length,
while in other cases such a field is not needed. To be efficient in
either case, the fields to support these features are configurable by
means of MIME format parameters. In general, a MIME format parameter
defines the presence and length of the associated field. A length of
zero indicates absence of the field. As a consequence, parsing of
the payload requires knowledge of MIME format parameters. The MIME
format parameters are conveyed to the receiver via SDP [5] messages,
as specified in section 4.4.1, or through other means.
van der Meer, et al. Standards Track [Page 8]
RFC 3640 Transport of MPEG-4 Elementary Streams November 2003
2.11. Global Structure of Payload Format
The RTP payload following the RTP header, contains three octet-
aligned data sections, of which the first two MAY be empty, see
Figure 1.
+---------+-----------+-----------+---------------+
| RTP | AU Header | Auxiliary | Access Unit |
| Header | Section | Section | Data Section |
+---------+-----------+-----------+---------------+
<----------RTP Packet Payload----------->
Figure 1: Data sections within an RTP packet
The first data section is the AU (Access Unit) Header Section, that
contains one or more AU-headers; however, each AU-header MAY be
empty, in which case the entire AU Header Section is empty. The
second section is the Auxiliary Section, containing auxiliary data;
this section MAY also be configured empty. The third section is the
Access Unit Data Section, containing either a single fragment of one
Access Unit or one or more complete Access Units. The Access Unit
Data Section MUST NOT be empty.
2.12. Modes to Transport MPEG-4 Streams
While it is possible to build fully configurable receivers capable of
receiving any MPEG-4 stream, this specification also allows for the
design of simplified, but dedicated receivers, that are for example,
capable of receiving only one type of MPEG-4 stream. This is
achieved by requiring that specific modes be defined in order to use
this specification. Each mode may define constraints for transport
of one or more types of MPEG-4 streams, for instance on the payload
configuration.
The applied mode MUST be signaled. Signaling the mode is
particularly important for receivers that are only capable of
decoding one or more specific modes. Such receivers need to
determine whether the applied mode is supported, so as to avoid
problems with processing of payloads that are beyond the capabilities
of the receiver.
In this document several modes are defined for the transportation of
MPEG-4 CELP and AAC streams, as well as a generic mode that can be
used for any MPEG-4 stream. In the future, new RFCs may specify
other modes of using this specification. However, each mode MUST be
in full compliance with this specification (see section 3.3.7).
van der Meer, et al. Standards Track [Page 9]
RFC 3640 Transport of MPEG-4 Elementary Streams November 2003
2.13. Alignment with RFC 3016
This payload can be configured as nearly identical to the payload
format defined in RFC 3016 [12] for the MPEG-4 video configurations
recommended in RFC 3016. Hence, receivers that comply with RFC 3016
can decode such RTP payload, provided that additional packets
containing video decoder configuration (VO, VOL, VOSH) are inserted
in the stream, as required by RFC 3016 [12]. Conversely, receivers
that comply with the specification in this document SHOULD be able to
decode payloads, names and parameters defined for MPEG-4 video in RFC
3016 [12]. In this respect, it is strongly RECOMMENDED that the
implementation provide the ability to ignore "in band" video decoder
configuration packets that may be found in streams conforming to the
RFC 3016 video payload.
Note the "out of band" availability of the video decoder
configuration is optional in RFC 3016 [12]. To achieve maximum
interoperability with the RTP payload format defined in this
document, applications that use RFC 3016 to transport MPEG-4 video
(part 2) are recommended to make the video decoder configuration
available as a MIME parameter.
3. Payload Format
3.1. Usage of RTP Header Fields and RTCP
Payload Type (PT): The assignment of an RTP payload type for this
packet format is outside the scope of this document; it is
specified by the RTP profile under which this payload format is
used, or signaled dynamically out-of-band (e.g., using SDP).
Marker (M) bit: The M bit is set to 1 to indicate that the RTP packet
payload contains either the final fragment of a fragmented Access
Unit or one or more complete Access Units.
Extension (X) bit: Defined by the RTP profile used.
Sequence Number: The RTP sequence number SHOULD be generated by the
sender in the usual manner with a constant random offset.
Timestamp: Indicates the sampling instant of the first AU contained
in the RTP payload. This sampling instant is equivalent to the
CTS in the MPEG-4 time domain. When using SDP, the clock rate of
the RTP time stamp MUST be expressed using the "rtpmap" attribute.
If an MPEG-4 audio stream is transported, the rate SHOULD be set
to the same value as the sampling rate of the audio stream. If an
MPEG-4 video stream is transported, it is RECOMMENDED that the
rate be set to 90 kHz.
van der Meer, et al. Standards Track [Page 10]
RFC 3640 Transport of MPEG-4 Elementary Streams November 2003
In all cases, the sender SHALL make sure that RTP time stamps are
identical only if the RTP time stamp refers to fragments of the same
Access Unit.
According to RFC 3550 [2] (section 5.1), it is RECOMMENDED that RTP
time stamps start at a random value for security reasons. This is
not an issue for synchronization of multiple RTP streams. However,
when streams from multiple sources are to be synchronized (for
example one stream from local storage, another from an RTP streaming
server), synchronization may become impossible if the receiver only
knows the original time stamp relationships. In such cases the time
stamp relationship required for obtaining synchronization may be
provided by out of band means. The format of such information, as
well as methods to convey such information, are beyond the scope of
this specification.
SSRC: set as described in RFC 3550 [2].
CC and CSRC fields are used as described in RFC 3550 [2].
RTCP SHOULD be used as defined in RFC 3550 [2]. Note that time
stamps in RTCP Sender Reports may be used to synchronize multiple
MPEG-4 elementary streams and also to synchronize MPEG-4 streams with
non-MPEG-4 streams, in case the delivery of these streams uses RTP.
3.2. RTP Payload Structure
3.2.1. The AU Header Section
When present, the AU Header Section consists of the AU-headers-length
field, followed by a number of AU-headers, see Figure 2.
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+- .. -+-+-+-+-+-+-+-+-+-+
|AU-headers-length|AU-header|AU-header| |AU-header|padding|
| | (1) | (2) | | (n) | bits |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+- .. -+-+-+-+-+-+-+-+-+-+
Figure 2: The AU Header Section
The AU-headers are configured using MIME format parameters and MAY be
empty. If the AU-header is configured empty, the AU-headers-length
field SHALL NOT be present and consequently the AU Header Section is
empty. If the AU-header is not configured empty, then the AU-
headers-length is a two octet field that specifies the length in bits
of the immediately following AU-headers, excluding the padding bits.
Each AU-header is associated with a single Access Unit (fragment)
contained in the Access Unit Data Section in the same RTP packet.
van der Meer, et al. Standards Track [Page 11]
RFC 3640 Transport of MPEG-4 Elementary Streams November 2003
For each contained Access Unit (fragment), there is exactly one AU-
header. Within the AU Header Section, the AU-headers are bit-wise
concatenated in the order in which the Access Units are contained in
the Access Unit Data Section. Hence, the n-th AU-header refers to
the n-th AU (fragment). If the concatenated AU-headers consume a
non-integer number of octets, up to 7 zero-padding bits MUST be
inserted at the end in order to achieve octet-alignment of the AU
Header Section.
3.2.1.1. The AU-header
Each AU-header may contain the fields given in Figure 3. The length
in bits of the fields, with the exception of the CTS-flag, the
DTS-flag and the RAP-flag fields, is defined by MIME format
parameters; see section 4.1. If a MIME format parameter has the
default value of zero, then the associated field is not present. The
number of bits for fields that are present and that represent the
value of a parameter MUST be chosen large enough to correctly encode
the largest value of that parameter during the session.
If present, the fields MUST occur in the mutual order given in Figure
3. In the general case, a receiver can only discover the size of an
AU-header by parsing it since the presence of the CTS-delta and DTS-
delta fields is signaled by the value of the CTS-flag and DTS-flag,
respectively.
+---------------------------------------+
| AU-size |
+---------------------------------------+
| AU-Index / AU-Index-delta |
+---------------------------------------+
| CTS-flag |
+---------------------------------------+
| CTS-delta |
+---------------------------------------+
| DTS-flag |
+---------------------------------------+
| DTS-delta |
+---------------------------------------+
| RAP-flag |
+---------------------------------------+
| Stream-state |
+---------------------------------------+
Figure 3: The fields in the AU-header. If used, the AU-Index field
only occurs in the first AU-header within an AU Header
Section; in any other AU-header, the AU-Index-delta field
occurs instead.
van der Meer, et al. Standards Track [Page 12]
RFC 3640 Transport of MPEG-4 Elementary Streams November 2003
AU-size: Indicates the size in octets of the associated Access Unit
in the Access Unit Data Section in the same RTP packet. When the
AU-size is associated with an AU fragment, the AU size indicates
the size of the entire AU and not the size of the fragment. In
this case, the size of the fragment is known from the size of the
AU data section. This can be exploited to determine whether a
packet contains an entire AU or a fragment, which is particularly
useful after losing a packet carrying the last fragment of an AU.
AU-Index: Indicates the serial number of the associated Access Unit
(fragment). For each (in decoding order) consecutive AU or AU
fragment, the serial number is incremented by 1. When present,
the AU-Index field occurs in the first AU-header in the AU Header
Section, but MUST NOT occur in any subsequent (non-first) AU-
header in that Section. To encode the serial number in any such
non-first AU-header, the AU-Index-delta field is used.
AU-Index-delta: The AU-Index-delta field is an unsigned integer that
specifies the serial number of the associated AU as the difference
with respect to the serial number of the previous Access Unit.
Hence, for the n-th (n>1) AU, the serial number is found from:
AU-Index(n) = AU-Index(n-1) + AU-Index-delta(n) + 1
If the AU-Index field is present in the first AU-header in the AU
Header Section, then the AU-Index-delta field MUST be present in
any subsequent (non-first) AU-header. When the AU-Index-delta is
coded with the value 0, it indicates that the Access Units are
consecutive in decoding order. An AU-Index-delta value larger
than 0 signals that interleaving is applied.
CTS-flag: Indicates whether the CTS-delta field is present. A value
of 1 indicates that the field is present, a value of 0 indicates
that it is not present.
The CTS-flag field MUST be present in each AU-header if the length
of the CTS-delta field is signaled to be larger than zero. In
that case, the CTS-flag field MUST have the value 0 in the first
AU-header and MAY have the value 1 in all non-first AU-headers.
The CTS-flag field SHOULD be 0 for any non-first fragment of an
Access Unit.
van der Meer, et al. Standards Track [Page 13]
RFC 3640 Transport of MPEG-4 Elementary Streams November 2003
CTS-delta: Encodes the CTS by specifying the value of CTS as a 2's
complement offset (delta) from the time stamp in the RTP header of
this RTP packet. The CTS MUST use the same clock rate as the time
stamp in the RTP header.
DTS-flag: Indicates whether the DTS-delta field is present. A value
of 1 indicates that DTS-delta is present, a value of 0 indicates
that it is not present.
The DTS-flag field MUST be present in each AU-header if the length
of the DTS-delta field is signaled to be larger than zero. The
DTS-flag field MUST have the same value for all fragments of an
Access Unit.
DTS-delta: Specifies the value of the DTS as a 2's complement offset
(delta) from the CTS. The DTS MUST use the same clock rate as the
time stamp in the RTP header. The DTS-delta field MUST have the
same value for all fragments of an Access Unit.
RAP-flag: When set to 1, indicates that the associated Access Unit
provides a random access point to the content of the stream. If
an Access Unit is fragmented, the RAP flag, if present, MUST be
set to 0 for each non-first fragment of the AU.
Stream-state: Specifies the state of the stream for an AU of an
MPEG-4 system stream; each state is identified by a value of a
modulo counter. In ISO/IEC 14496-1, MPEG-4 system streams use the
AU_SequenceNumber to signal stream states. When the stream state
changes, the value of the stream-state MUST be incremented by one.
Note: no relation is required between stream-states of different
streams.
3.2.2. The Auxiliary Section
The Auxiliary Section consists of the auxiliary-data-size field
followed by the auxiliary-data field. Receivers MAY (but are not
required to) parse the auxiliary-data field; to facilitate skipping
of the auxiliary-data field by receivers, the auxiliary-data-size
field indicates the length in bits of the auxiliary-data. If the
concatenation of the auxiliary-data-size and the auxiliary-data
fields consume a non-integer number of octets, up to 7 zero padding
bits MUST be inserted immediately after the auxiliary data in order
to achieve octet-alignment. See Figure 4.
van der Meer, et al. Standards Track [Page 14]
RFC 3640 Transport of MPEG-4 Elementary Streams November 2003
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+- .. -+-+-+-+-+-+-+-+-+
| auxiliary-data-size | auxiliary-data |padding bits |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+- .. -+-+-+-+-+-+-+-+-+
Figure 4: The fields in the Auxiliary Section
The length in bits of the auxiliary-data-size field is configurable
by a MIME format parameter; see section 4.1. The default length of
zero indicates that the entire Auxiliary Section is absent.
auxiliary-data-size: specifies the length in bits of the immediately
following auxiliary-data field;
auxiliary-data: the auxiliary-data field contains data of a format
not defined by this specification.
3.2.3. The Access Unit Data Section
The Access Unit Data Section contains an integer number of complete
Access Units or a single fragment of one AU. The Access Unit Data
Section is never empty. If data of more than one Access Unit is
present, then the AUs are concatenated into a contiguous string of
octets. See Figure 5. The AUs inside the Access Unit Data Section
MUST be in decoding order, though not necessarily contiguous in the
case of interleaving.
The size and number of Access Units SHOULD be adjusted such that the
resulting RTP packet is not larger than the path MTU. To handle
larger packets, this payload format relies on lower layers for
fragmentation, which may result in reduced performance.
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|AU(1) |
+ |
| |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |AU(2) |
+-+-+-+-+-+-+-+-+ |
| |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| | AU(n) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|AU(n) continued|
|-+-+-+-+-+-+-+-+
Figure 5: Access Unit Data Section; each AU is octet-aligned.
van der Meer, et al. Standards Track [Page 15]
RFC 3640 Transport of MPEG-4 Elementary Streams November 2003
When multiple Access Units are carried, the size of each AU MUST be
made available to the receiver. If the AU size is variable, then the
size of each AU MUST be indicated in the AU-size field of the
corresponding AU-header. However, if the AU size is constant for a
stream, this mechanism SHOULD NOT be used; instead, the fixed size
SHOULD be signaled by the MIME format parameter "constantSize"; see
section 4.1.
The absence of both AU-size in the AU-header and the constantSize
MIME format parameter indicates the carriage of a single AU
(fragment), i.e., that a single Access Unit (fragment) is transported
in each RTP packet for that stream.
3.2.3.1. Fragmentation
A packet SHALL carry either one or more complete Access Units, or a
single fragment of an Access Unit. Fragments of the same Access Unit
have the same time stamp but different RTP sequence numbers. The
marker bit in the RTP header is 1 on the last fragment of an Access
Unit, and 0 on all other fragments.
3.2.3.2. Interleaving
Unless prohibited by the signaled mode, a sender MAY interleave
Access Units. Receivers that are capable of receiving modes that
support interleaving MUST be able to decode interleaved Access Units.
When a sender interleaves Access Units, it needs to provide
sufficient information to enable a receiver to unambiguously
reconstruct the original order, even in the case of out-of-order
packets, packet loss or duplication. The information that senders
need to provide depends on whether or not the Access Units have a
constant time duration. Access Units have a constant time duration,
if:
TS(i+1) - TS(i) = constant
for any i, where:
i indicates the index of the AU in the original order, and
TS(i) denotes the time stamp of AU(i)
The MIME parameter "constantDuration" SHOULD be used to signal that
Access Units have a constant time duration; see section 4.1.
If the "constantDuration" parameter is present, the receiver can
reconstruct the original Access Unit timing based solely on the RTP
timestamp and AU-Index-delta. Accordingly, when transmitting Access
Units of constant duration, the AU-Index, if present, MUST be set to
van der Meer, et al. Standards Track [Page 16]
RFC 3640 Transport of MPEG-4 Elementary Streams November 2003
the value 0. Receivers of constant duration Access Units MUST use
the RTP timestamp to determine the index of the first AU in the RTP
packet. The AU-Index-delta header and the signaled
"constantDuration" are used to reconstruct AU timing.
If the "constantDuration" parameter is not present, then senders MAY
signal AUs of constant duration by coding the AU-Index with zero in
each RTP packet. In the absence of the constantDuration parameter
receivers MUST conclude that the AUs have constant duration if the
AU-index is zero in two consecutive RTP packets.
When transmitting Access Units of variable duration, then the
"constantDuration" parameter MUST NOT be present, and the transmitter
MUST use the AU-Index to encode the index information required for
re-ordering, and the receiver MUST use that value to determine the
index of each AU in the RTP packet. The number of bits of the AU-
Index field MUST be chosen so that valid index information is
provided at the applied interleaving scheme, without causing problems
due to roll-over of the AU-Index field. In addition, the CTS-delta
MUST be coded in the AU header for each non-first AU in the RTP
packet, so that receivers can place the AUs correctly in time.
When interleaving is applied, a de-interleave buffer is needed in
receivers to put the Access Units in their correct logical
consecutive decoding order. This requires the computation of the
time stamp for each Access Unit. In case of a constant time duration
per Access Unit, the time stamp of the i-th access unit in an RTP
packet with RTP time stamp T is calculated as follows:
Timestamp[0] = T
Timestamp[i, i > 0] = T +(Sum(for k=1 to i of (AU-Index-delta[k]
+ 1))) * access-unit-duration
When AU-Index-delta is always 0, this reduces to T + i * (access-
unit-duration). This is the non-interleaved case, where the frames
are consecutive in decoding order. Note that the AU-Index field
(present for the first Access Unit) is indeed not needed in this
calculation.
3.2.3.3. Constraints for Interleaving
The size of the packets should be suitably chosen to be appropriate
to both the path MTU and the capacity of the receiver's de-interleave
buffer. The maximum packet size for a session SHOULD be chosen to
not exceed the path MTU.
van der Meer, et al. Standards Track [Page 17]
RFC 3640 Transport of MPEG-4 Elementary Streams November 2003
To allow receivers to allocate sufficient resources for de-
interleaving, senders MUST provide the information to receivers as
specified in this section.
AUs enter the decoder in decoding order. The de-interleave buffer is
used to re-order a stream of interleaved AUs back into decoding
order. When interleaving is applied, the decoding of "early" AUs has
to be postponed until all AUs that precede it in decoding order are
present. Therefore, these "early" AUs are stored in the de-
interleave buffer. As an example in Figure 6, the interleaving
pattern from section 2.5 is considered.
+--+--+--+--+--+--+--+--+--+--+--+-
Interleaved AUs | 0| 3| 6| 1| 4| 7| 2| 5| 8| 9|12|..
+--+--+--+--+--+--+--+--+--+--+--+-
Storage of "early" AUs 3 3 3 3 3 3
6 6 6 6 6 6
4 4 4
7 7 7
12 12
Figure 6: Storage of "early" AUs in the de-interleave buffer per
interleaved AU.
AU(3) is to be delivered to the decoder after AU(0), AU(1) and AU(2);
of these AUs, AU(2) arrives from the network last and hence AU(3)
needs to be stored until AU(2) is present in the pattern. Similarly,
AU(6) is to be stored until AU(5) is present, while AU(4) and AU(7)
are to be stored until AU(2) and AU(5) are present, respectively.
Note that the fullness of the de-interleave buffer varies in time.
In Figure 6, the de-interleave buffer contains at most 4, but often
less AUs.
So as to give a rough indication of the resources needed in the
receiver for de-interleaving, the maximum displacement in time of an
AU is defined. For any AU(j) in the pattern, each AU(i) with i<j
that is not yet present can be determined. The maximum displacement
in time of an AU is the maximum difference between the time stamp of
an AU in the pattern and the time stamp of the earliest AU that is
not yet present. In other words, when considering a sequence of
interleaved AUs, then: