Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support raw AAC in HLS #1083

Closed
joeyparrish opened this issue Oct 24, 2017 · 16 comments
Closed

Support raw AAC in HLS #1083

joeyparrish opened this issue Oct 24, 2017 · 16 comments
Assignees
Labels
component: HLS The issue involves Apple's HLS manifest format status: archived Archived and locked; will not be updated type: enhancement New feature or request
Milestone

Comments

@joeyparrish
Copy link
Member

Originally raised in #887 (comment)

Our HLS parser must guess container and codecs from the limited information provided in an HLS playlist. Currently, we don't understand raw AAC.

MediaSource seems to support raw AAC with the MIME type "audio/aac" (no codecs parameter), so we should be able to handle this.

Sample streams:
https://content.jwplatform.com/manifests/vM7nH0Kl.m3u8

@joeyparrish joeyparrish added type: enhancement New feature or request component: HLS The issue involves Apple's HLS manifest format labels Oct 24, 2017
@joeyparrish joeyparrish added this to the v2.3.0 milestone Oct 24, 2017
@joeyparrish joeyparrish self-assigned this Oct 24, 2017
@joeyparrish joeyparrish modified the milestones: v2.3.0, v2.4.0 Oct 24, 2017
@joeyparrish
Copy link
Member Author

Recognizing raw AAC is easy, but given the new timestamp parsing code we are using to fix #1011, we still won't be able to fully support raw AAC until we have a parser for it.

@joeyparrish joeyparrish removed their assignment Oct 26, 2017
@joeyparrish joeyparrish modified the milestones: v2.4.0, Backlog Dec 4, 2017
@avelad
Copy link
Member

avelad commented Jan 29, 2018

Hi @joeyparrish,

With this patch should be enough?

diff --git a/lib/hls/hls_parser.js b/lib/hls/hls_parser.js
index b684d280..602afa05 100644
--- a/lib/hls/hls_parser.js
+++ b/lib/hls/hls_parser.js
@@ -1372,6 +1372,8 @@ shaka.hls.HlsParser.prototype.getStartTime_ =
       // There is no standard way to embed a timestamp in an mp3 file, so the
       // start time is presumably 0.
       return 0;
+    } else if (mimeType == 'audio/aac') {
+      return 0
     } else if (mimeType == 'video/mp2t') {
       return this.getStartTimeFromTsSegment_(responses[0].data);
     } else if (mimeType == 'application/mp4' ||
@@ -1867,6 +1869,7 @@ shaka.hls.HlsParser.AUDIO_EXTENSIONS_TO_MIME_TYPES_ = {
   'm4s': 'audio/mp4',
   'm4i': 'audio/mp4',
   'm4a': 'audio/mp4',
+  'aac': 'audio/aac',
   // mpeg2 ts aslo uses video/ for audio: http://goo.gl/tYHXiS
   'ts': 'video/mp2t'
 };

@media-square
Copy link

Is there an update concerning this bug, since we are facing the same issue?

@joeyparrish
Copy link
Member Author

@media-square, no update. We have not worked on this at all yet.

Re: the patch suggested by @avelad, no, that would not work for live streams as far as I can tell.

@bafolts
Copy link

bafolts commented Jul 3, 2018

Will the work involved here be a function to getStartTimeFromAacSegment_ implemented to parse the aac segment and pull it's start time?

@joeyparrish
Copy link
Member Author

Yes, that's the idea.

@media-square
Copy link

So basically, there is a solution which should work for live streams but...
When can we expect a patch or will this issue not being fixed?

@bes
Copy link

bes commented Oct 2, 2018

We are interested in this patch as well

@peet-j
Copy link

peet-j commented Nov 6, 2018

+1

@ismena
Copy link
Contributor

ismena commented Nov 6, 2018

@bes @peet-j Thanks for expressing interest!

@joeyparrish Could you, please, advise if this is on our road map?

@joeyparrish
Copy link
Member Author

As far as I can tell, containerless formats like MP3 and raw AAC do not have their own timestamps. So long as the video and audio content are properly aligned and the content is VOD, we could just use the patch from @avelad. But I don't think there are any timestamps to parse out, so we're left with guessing at "0".

I'm not certain what will happen with that patch and any non-zero-start-time content or any live content, though. What happens for raw AAC in live depends on how MediaSource behaves with timestamp-less raw content. I'm reaching out to the Chrome team for clarification, but I suspect this will not work yet for live.

@joeyparrish
Copy link
Member Author

I heard from Chrome team. The way AAC and MP3 work in MediaSource is that timestampOffset determines the starting time of each new append in the SourceBuffer, and after each append, the browser updates timestampOffset to reflect the amount of data that was appended. This is 'sequence' mode, even if you didn't set it explicitly on the SourceBuffer.

If we had the right metadata, we could make that work, including for live. But it will require refactoring in our HLS implementation. That refactor would overlap significantly with #1558.

@kevinscroggins-youi
Copy link

Hi @joeyparrish, are there any updates on when this feature might be implemented? Thanks!

@JPeMu
Copy link

JPeMu commented May 10, 2019

I happened to see this issue in passing. Just to correct the previous comment about containerless formats not having timestamps - this isn't true. The HLS spec requires that all packed audio formats (such as raw AAC/MP3 etc) have an ID3 PRIV tag inserted at the very start with the timestamp information.

Please see: https://tools.ietf.org/html/draft-pantos-http-live-streaming-23#section-3.4

This is a "MUST" item in the spec, and it goes so far as to add that clients "SHOULD NOT" play back audio data like this unless it contains this timestamp!

BR
Tony

@joeyparrish
Copy link
Member Author

It turns out that the timestamps in the HLS parser are only a small part of the problem of supporting raw AAC. The bigger issue is support for containerless formats generally at the MediaSource level. Every time we clear the buffer or seek, the timestampOffset needs to be set before appending containerless segments.

I filed issue #2337 to track full containerless format support. I will close this issue as soon as the HLS parser has been taught to recognize raw AAC. Until we can play it correctly, though (#2337), we will reject raw AAC HLS content.

joeyparrish added a commit that referenced this issue Jan 15, 2020
This makes the HLS parser recognize raw AAC, but refuse to play it.
It will not play correctly until we have full support for
containerless formats at the MediaSource level.

Closes #1083 (raw AAC, can't do anything more for this right now)
Issue #2337 (full support for containerless formats in general)

Backported to v2.5.x

Change-Id: I50c7f06a1aa08d515f4d9f74603954a6d015dc29
@joeyparrish
Copy link
Member Author

The change to explicitly reject raw AAC has been cherry-picked for v2.5.8.

@joeyparrish joeyparrish modified the milestones: Backlog, v2.6 Jan 23, 2020
@shaka-project shaka-project locked and limited conversation to collaborators Mar 13, 2020
@shaka-bot shaka-bot added the status: archived Archived and locked; will not be updated label Apr 15, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
component: HLS The issue involves Apple's HLS manifest format status: archived Archived and locked; will not be updated type: enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

10 participants