Should we revisit and document getUserMedia() track order?

[jan-ivar]

The spec says: "The tracks of a MediaStream are stored in a track set. The track set MUST contain the MediaStreamTrack objects that correspond to the tracks of the stream. The relative order of the tracks in the set is User Agent defined and the API will never put any requirements on the order. The proper way to find a specific MediaStreamTrack object in the set is to look it up by its id."

But #1020 is concerned about deterministic event order between track clones, looking at using track creation order.

If we're chasing determinism at this level, might getUserMedia need to be clearer about whether it creates its audio or video track first?

I suspect audio comes first, given "2. Let requestedMediaTypes be the set of media types in constraints with either a dictionary value or a value of true... 11.3 For each media type kind in requestedMediaTypes ...", and the fact that dictionary members are sorted lexographically.

While use of the word "set" in the sentences above do not link to set, they might be understood to mean the same thing. It reads:

Thankfully, an implementation check reveals the same order in all browsers: audio, video. Is it time to document this?

[bradisbell]

@jan-ivar Is there ever a time when a MediaStream would be returned from getUserMedia() with tracks added later? For example, an audio device is usually available seconds before a video device initializes and starts returning video data. I suppose the opposite could happen. Just thinking ahead, wondering if specifying track order might hurt optimizations in the future that return available tracks immediately and add the other requested track(s) later.

[jan-ivar]

getUserMedia({video: true, audio: true}) is guaranteed to return two tracks. The algorithm ensures that.

[fippo]

See webrtcHacks/adapter#969 and the related discussion.
From what I recall this has continued to confuse "web developers".

I am not sure if lexographically order of tracks sorted by the id is meaningful to anyone, sorting by kind and then by id seems more natural.

I'd suggest checking how getTracks + addTrack behaves and look forward to the WPT!

edit: done, see this fiddle:

• Chrome returns audio, video
• Firefox returns audio, video
• Safari returns in order of id
  If this gets you confused enough think about the average web developer.

[fippo]

Note: chromium returns audiotracks-then-videotracks from getTracks(), see here, changed in 2014

Which makes sense for getUserMedia but not so much when you consider manually added tracks (empty stream + video + audio => [audio, video] or clones.

[fippo]

[audio,video] + video clone + audio clone: https://jsfiddle.net/fippo/w7z91b8k/1/ show link
Current state:

Browser	order
"make everyone unhappy"	audio, video, video, audio
Chrome	audio, audio, video, video
Firefox	audio, video, video, audio
Safari	track-id-order as we learned

I do not think this this is an improvement.

We also need to be careful about remote tracks as shown by this which puts video first, then audio in the SDP:
https://jsfiddle.net/fippo/ec1puh6w/
The risk of causing breakage in how this translates to SDP is too big.
Multiple tracks per kind and then using that with a peer connection does not seem like a big use-case either.

I'm happy to send a PR updating https://w3c.github.io/webrtc-pc/#legacy-interface-extensions but the biggest compat-win would be Safari stopping to randomize the order. (and note that adapter does not interfere with MediaStream.getTracks(), only with legacy addStream order)

[jan-ivar]

Working like Firefox is not "breakage". There's no risk here as far as I'm concerned. Only upside.

We also need to be careful about remote tracks as shown by this which puts video first, then audio in the SDP:
https://jsfiddle.net/fippo/ec1puh6w/
The risk of causing breakage in how this translates to SDP is too big.

SDP is garbage in garbage out. E.g. put the m=audio line first instead and audio track is surfaced first. Same on input.

It seems clear we would have heard about it by now if this were problematic.

Knowing there's no breakage, it's disappointing we cannot align around the spec which has the most natural and logical behavior, which happens to be how Firefox already works in this case.

[fippo]

As an exercise let me walk you through https://www.chromium.org/blink/launching-features/#behavior-changes:

• Can the change cause user-visible or functional breakage? YES, changes the order of SDP m-lines
  • Is it possible that existing code relies on the current behavior? YES, see Safari/Freeswitch or Asterisk interop woes
  • What would that coding pattern look like? How likely it is that this coding pattern is used in the wild? see specification-shown example of stream.getTracks().forEach(track => pc.addTrack(track, stream)
• What are other browser engines doing? VARIES
  • Would this change align Chromium's behaviour with other vendors? YES, matches Firefox. Safari orders by UUID
  • Or would Chromium be the first to roll-out this behavior change? NO
• What is the usage of the feature being changed (typically measured with a UseCounter? No use counter
• How likely is breakage in sites that only support Chromium browsers? some webrtc services do not support Firefox so might not have encountered the new behavior and are not expecting it

Since we do not have a use-counter I'll make a guess:

• 99.8% of usage is going straight from GUM to addTrack
• 0.0063% of usage is stream.addTrack with a screensharing track and a microphone one. For extra fun 40% of those adds the mic to the screensharing stream, 60% the other way round. I bet this is where the new behavior would break.

Should we ask some Chromium API owners if they would sign this off as a "web-facing change"?
Even ignoring the risk (I am all for it, moar content!) is "most natural and logical behavior" sufficient motivation to justify the effort?

And keep in mind that this affects SDP... you know why Firefox continues to put version 99 into its SDP? Does not seem logical to me...

[jan-ivar]

I see no evidence that changing Chrome’s track order would cause breakage. Other browsers (Firefox and Safari) have always had different track orders with no ill effect.

Any site depending on a particular browser’s order is already non-interoperable, since the spec explicitly allows any order.

Unifying behavior reduces confusion and helps developers. If there’s usage data showing a real dependency, that would be valuable, but absent that, fears of “breakage” seem unfounded.

• 99.8% of usage is going straight from GUM to addTrack

IOW no issue for Chromium whose gUM order would remain unchanged: [audio, video].

• 0.0063% of usage is stream.addTrack with a screensharing track and a microphone one. For extra fun 40% of those adds the mic to the screensharing stream, 60% the other way round. I bet this is where the new behavior would break.

And keep in mind that this affects SDP...

There's no evidence websites rely on this order or that anything would break. There's evidence they don't (interoperable websites).

If usage data for ordering assumptions exists, it would be good to see it. Absent that, we should not assume a hidden dependency that has somehow gone unreported despite existing browser differences.

[guidou]

I see no evidence that changing Chrome’s track order would cause breakage. Other browsers (Firefox and Safari) have always had different track orders with no ill effect.

Absence of evidence is very different from evidence of absence.

Any site depending on a particular browser’s order is already non-interoperable, since the spec explicitly allows any order.

We don't want to break those sites either.

Unifying behavior reduces confusion and helps developers. If there’s usage data showing a real dependency, that would be valuable, but absent that, fears of “breakage” seem unfounded.

Let's unify then around Chrome's behavior. If we want to guarantee order for getTracks() we should also do it for getAudioTracks() and getVideoTracks().

We can specify it as internally having two ordered sets, one for getAudioTracks() and one for getVideoTracks(). The order would be the order in which tracks are added to the corresponding set.
getTracks() would return the concatenation of the audio set and the video set.

[jan-ivar]

It seems incumbent on those making unsubstantiated claims of "breakage" to provide evidence, and explain how breakage would manifest. I've provided plenty of evidence of absence: years of different browser orders without bug reports.

Any site depending on a particular browser’s order is already non-interoperable, since the spec explicitly allows any order.

We don't want to break those sites either.

Which sites are those?

Let's unify then around Chrome's behavior.

Unifying on Chrome’s historic ordering seems arbitrary. If we truly suspect real-world reliance, a usage counter could confirm that. Otherwise, there is no evidence of harm in converging on a single, simpler order that aligns track set with https://infra.spec.whatwg.org/#sets

If we want to guarantee order for getTracks() we should also do it for getAudioTracks() and getVideoTracks().

Agreed. Those already appear expressed as the track set filtered on kind, so we'd just need to remove "The conversion from the track set to the sequence is User Agent defined and the order does not have to be stable between calls."

[guidou]

It seems incumbent on those making unsubstantiated claims of "breakage" to provide evidence, and explain how breakage would manifest.

What's unsubstantiated is claiming that this change is risk-free.

I've provided plenty of evidence of absence: years of different browser orders without bug reports.

That's only evidence that the status quo does not cause problems to anyone, which is strong evidence in favor of maintaining the status quo. It is not evidence of absence of harm after we deploy the change.

Any site depending on a particular browser’s order is already non-interoperable, since the spec explicitly allows any order.

We don't want to break those sites either.

Which sites are those?

I don't want to find out.

Let's unify then around Chrome's behavior.

Unifying on Chrome’s historic ordering seems arbitrary.

No more arbitrary than any other ordering.

If we truly suspect real-world reliance, a usage counter could confirm that. Otherwise, there is no evidence of harm in converging on a single, simpler order that aligns track set with https://infra.spec.whatwg.org/#sets

In the past, behavior changes to established APIs like this one have caused breakage for Chrome users.
The benefit of making this change (aligning the spec with a new infra guideline) does not outweigh this risk.
Given this, the only change I find acceptable is aligning the spec with Chrome's ordering. It has the following benefits:

• Aligns the spec with the infra guideline, which is the motivation for the proposal.
• Trivial to implement for any UA that implements getAudioTracks() and getVideoTracks() using addition order (for which we seem to have agreement).
• Zero risk for users of Chromium-based browsers. Any other ordering fails to address this concern.
• Zero risk for Firefox users, as per Mozilla's analysis.

But we still need to wait for WebKit's assessment.

If we want to guarantee order for getTracks() we should also do it for getAudioTracks() and getVideoTracks().

Agreed. Those already appear expressed as the track set filtered on kind, so we'd just need to remove "The conversion from the track set to the sequence is User Agent defined and the order does not have to be stable between calls."

I support proceeding with this one if no one objects.

[jan-ivar]

It’s contradictory to cite “no bug reports” as evidence supporting the status quo while also dismissing the same “no bug reports” as no evidence supporting change. If lack of bug reports proves anything, it’s that different orders have coexisted for years without incident—i.e. the absence of bug reports is actually evidence that changes like these pose little risk.

[guidou]

It’s contradictory to cite “no bug reports” as evidence supporting the status quo while also dismissing the same “no bug reports” as no evidence supporting change. If lack of bug reports proves anything, it’s that different orders have coexisted for years without incident—i.e. the absence of bug reports is actually evidence that changes like these pose little risk.

An outcome completely consistent with the evidence presented so far is that the browser that changes its behavior starts to receive bug reports, and the browsers that keep their behavior continue to receive no bug reports.
Keeping the status quo where no one changes its behavior is likely to continue to result in no bugs.

But I have already presented our concerns and a compromise solution that seems to address all concerns presented so far.

I think we should now wait for WebKit's assessment and continue from there.

[fippo]

There are four separate technical issues here.

1/ getUserMedia track order (what the title says) and how it affects peerconnection

• is the same for Chromium and Firefox
• sorted by random id in Safari. As pointed out above this caused some issues in the past. I do not think a webkit bug was filed, web developers moved on.

2/ how does getTracks behave if a video track and a audio track are added to a GUM track and then used in a peerconnection

• There are difference in this case.
• How often developers do this is unknown
  Since "more than a single audio and video track" is not well supported in other APIs like the HTMLMediaElement or MediaRecorder (which surprisingly does not throw on multiple video tracks) I do not think this is actually relevant.

3/ how does getTracks behave if an audio track from getUserMedia is added to a video track from getDisplayMedia and then used in a peerconnection

• There are difference in this case.
• How often developers do this is unknown but since audio-only+screensharing is common
  I believe this use-case is common enough that we would have heard if the differences cause any issue.
  My guess is that audio-screen is the more common order here since JS has already acquired your microphone when you start sharing.

4/ SDP

• https://jsfiddle.net/fippo/6t7pa0d3/1/ shows the ontrack order on the remote end matches the addition order on the receiving end.
• I could not find a WPT.

Of those four problems I have only seen evidence of breakage for (1, link above) and of course (4) is a headache.
I have not heard of any "web developer" asking for changes in (2) or (3) and my rolodex is quite extensive.

Speaking as a web developer please @youennf consider (1)

[alvestrand]

I explained on the call the history of why track order is implemented as "group by type": The original specification had only "getAudioTracks" and "getVideoTracks"; "getTracks" was added later. Thus, a natural implementation was to return the concatenation of getAudioTracks and getVideoTracks, and since we had explicit language stating that the order of tracks was not guaranteed, this was a safe implementation choice vs the spec at the time.

[jan-ivar]

Is there any evidence the Firefox and Safari implementations are unsafe?

[jan-ivar]

An outcome completely consistent with the evidence presented so far is that the browser that changes its behavior starts to receive bug reports, and the browsers that keep their behavior continue to receive no bug reports.
Keeping the status quo where no one changes its behavior is likely to continue to result in no bugs.

That argument ignores the actual evidence that browsers have already changed this behavior (or never matched Chrome’s) with no bug reports. If track ordering were a real point of failure, we’d see it in Safari or Firefox, which have behaved differently for years. Suggesting a new change might produce bug reports assumes a risk we’ve effectively tested in the real world already—and we’ve seen no problems.

The status quo proves we can safely change behavior here to align with Web Platform Infra. Why do so? Because future uses of WebRTC might be more plentiful and involve more tracks than today's use. Just because it's no problem today doesn't mean it won't be tomorrow.

[fippo]

WRT "effectively tested" we had no tests even for basic assertions in WPT (added above), no tests which would have shown Safaris behavior for single-track-of-kind and no tests for multi-track behavior.

Your "works in Firefox therefore no risk" theory was very recently proven when libWebRTC changed how it assigned rtx payload types (not intentionally). It turned out that folks relied on "rtx pt == pt + 1" even though that is non-spec and Firefox does not follow that pattern. That was far from "no cost".

[jan-ivar]

Lack of tests is a red herring. Thanks for contributing them, but tests have nothing to do with the risk of breakage we're discussing. WPT only tests browser conformance to behaviors we specify. A decision here would inform tests, not the other way around.

What I said was that track order dependency has been "effectively tested in the real world" by Safari and Firefox behaving differently for years (in the case of Safari having no reliable order from one track to the next).

Therefore, we know interoperable websites—by definition—do not rely on track order.

Kudos to Safari for proving this point and taking one for the team here!

Your "works in Firefox therefore no risk" theory was very recently proven when libWebRTC changed how it assigned rtx payload types (not intentionally). It turned out that folks relied on "rtx pt == pt + 1" even though that is non-spec and Firefox does not follow that pattern. That was far from "no cost".

In case it's unclear to readers, by "folks" we're down to maintainers of non-interoperable chromium-only websites.

I would venture these websites represent a much larger "cost" to us than to you.

[fippo]

What I said was that track order dependency has been "effectively tested in the real world" by Safari and Firefox behaving differently for years (in the case of Safari having no reliable order from one track to the next).

Please look at the very beginning of the issue where your testing claimed

Thankfully, an implementation check reveals the same order in all browsers: audio, video

The meeting slide states this which is one of the very few points where we agree:

Safari sorts by random (lexicographical order of track.id, adapter.js tries to correct it) Needlessly confusing for web developers

Therefore, we know interoperable websites—by definition—do not rely on track order.
Kudos to Safari for proving this point and taking one for the team here!

"Needlessly confusing web developers" is not a good thing.

websites may
1/ not care about the order or do not rely on getTracks() order. There is no problem in having different orders so no action needed.
2/ do what adapter.js does in the legacy addStream method: add all audio tracks, then add all video tracks. Possibly the first of each kind because the use-case for more is quite a niche.
3/ be unmaintained. I do not want to know how many versions of rtcmulticonnection are floating around.

The public numbers we have show that addStream are still hovering at ~5% of Chrome getUserMedia calls and pretty stable for the last years:
https://webrtchacks.github.io/chromestatus/?buckets=1401,1402,1641
This shows two things:

• the lack of addStream in Safari has not affected this number nor has any decision in this WG.
• there is a stable, non-declining usage of legacy navigator.getUserMedia (and this might well be webkitGetUserMedia). This is not interoperable with Firefox or Safari. Yet developers insist on using those APIs.

Same for https://webrtchacks.github.io/chromestatus/?buckets=1656 which shows RTCIceServer.url. Safari throws here, Firefox shows a deprecation on:

const pc = new RTCPeerConnection({iceServers: [{url: 'stun:stun.l.google.com:19302'}]})

Unshipping this is a great way to test your theory! And we can see if this moves the needle in the public Chrome metrics.

This demonstrates that the working group has a problem with the way it ignores WPT:
https://wpt.fyi/results/webrtc/historical.html?label=experimental&label=master&aligned
This fails in all browsers but Safari.
Chromium issue here with some comments.
Firefox issue filed by you here.
Since this throws in Safari (thank you for taking one for the team but not much impact) this should be higher priority than the merely confusing track order?
Out of curiosity, why is this not part of Interop 2025?

In case it's unclear to readers, by "folks" we're down to maintainers of non-interoperable chromium-only websites.

"Maintainers" seems the right word, from what I saw the affected product had reached "end of sale".
I'll ping you the name of the person you want to discuss this with over 🐑👁

[alvestrand]

Just for fun, I found the git commit that added getTracks().

commit 1149581
Author: Adam Bergkvist adam.bergkvist@ericsson.com
Date: Wed Jun 4 10:15:19 2014 +0200

Bug 25860: make sure we have a bug to have a getTracks that gives you all the tracks

The very first instantiation of this function had the "order is user agent defined and doesn't have to be stable" language.
I can't find any mailing list discussion on this topic, but this is the kind of language we ended up with when there was no consensus on a rule to impose. Someone with the patience to search the 2014 meeting minutes may find more.

The Chrome implementation (https://source.chromium.org/chromium/chromium/src/+/main:third_party/blink/renderer/modules/mediastream/media_stream.cc;drc=5b1ceafef667af5fa64c3002e9427ddf01e105a1;l=291) is literally "first take all the audio tracks, and then all the video tracks" - due to a Blink reformat, I can't see its history beyond 2016, but it's been unchanged since that time.

"User needs come before the needs of web page authors, which come before the needs of user agent implementors, which come before the needs of specification writers, which come before theoretical purity."

For Chrome-as-a-product, changing this behavior risks hurting users (those using software that was working fine on earlier versions, and may no longer be maintained, but suddenly stops working). In addition, changing the spec harms web page authors that actually believe the spec to reflect realilty until the time such a behavior change can be agreed to and experimented with to the degree that one can have evidence of absence of harm rather than absence of evidence of harm.

For Chrome, no benefit has been demonstrated. A "theoretical purity" argument carries very little weight.

[dontcallmedom-bot]

This issue had an associated resolution in WebRTC February 2025 meeting – (Issue #1028 Revisit and document getUserMedia() track order):

RESOLUTION: No resolution. Try to get feedback from Safari.

[jan-ivar]

Let's not broaden discussion to include unrelated issues. Evidence of B is not evidence of A.

websites may
1/ not care about the order or do not rely on getTracks() order. There is no problem in having different orders so no action needed.

That's a false choice. The status quo relies on interop-hygiene among web developers, which can be effective AND not great at the same time.

It's our job to ensure long-term interop. So it seems a totally reasonable position for me to suggest there's no breakage today given the evidence, AND saying this may not hold in the future as more advanced multi-track uses appear, or say interest in interop among WebRTC-related web developers dips.

"Needlessly confusing web developers" is not a good thing.

The current spec relies on interop hygiene among web developers. We should thank them for staying interoperable so far, and fix the spec to relieve them of carrying this interop burden, which they should never have had (we didn't know better back then).

Hopefully, we can all agree that standardizing one behavior would be beneficial, and that we're down to picking what that behavior should be.

Any evidence of track order dependency would be helpful to this WG in making the decision of what behavior to pick.

A "theoretical purity" argument carries very little weight.

Track order is directly observable, and hence not "theoretical". I think you mean logically consistent observable behavior.

[youennf]

A few thoughts:

• This seems to be a valid interop issue. It apparently caused some issues for Safari in the past AIUI so we should fix it. The status quo is not good, web implementors should do their best to converge and help web developers.
• Saying we don't know whether that could break, so let's do nothing is really disappointing and not healthy. I hear this argument much more often here than in other web areas.
• I don't have a strong preference on which behavior to pick. I am ok trying to update WebKit behavior to what the WG deems appropriate. Let's first agree on trying to solve this issue or not.

web-platform-tests/interop#855 is a good example of trying to keep the web a sane place.
Chrome and Safari will try to align with the spec. If it proves too difficult to align with the spec, we will likely discuss updating the spec (and Firefox).