Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Toggling Live caption via media-playback button dynamically downloads Google SODA components #15864

Open
stephendonner opened this issue May 14, 2021 · 7 comments
Open

Toggling Live caption via media-playback button dynamically downloads Google SODA components #15864

stephendonner opened this issue May 14, 2021 · 7 comments
Labels
accessibility needs-discussion Although the issue is clear, we haven't yet reached a decision about the right solution. OS/Desktop priority/P3 The next thing for us to work on. It'll ride the trains. privacy/connect This requires making a network connection to a third-party service. privacy security

Comments

@stephendonner
Copy link

Description

Toggling Live caption via media-playback button dynamically downloads Google SODA components; this is new in the Chromium 91 major bump - we need to figure out how we want to handle this

Steps to Reproduce

  1. new profile
  2. play https://www.youtube.com/watch?v=kwAGY1PwMkE
  3. click on the media-control menu item on the right-hand side of the toolbar
  4. toggle Live Caption to On
  5. look at your terminal/console window

Actual result:

live-caption

Looks like this downloads Google's SODA (Speech On-Device API) components, dynamically:

[6492:1116:0514/082151.546:VERBOSE1:network_delegate.cc(32)] NetworkDelegate::NotifyBeforeURLRequest: https://www.youtube.com/api/stats/watchtime?ns=yt&el=detailpage&cpn=PaQe6mmwRVE79tyy&docid=kwAGY1PwMkE&ver=2&referrer=https%3A%2F%2Fwww.youtube.com%2Fresults%3Fsearch_query%3Djonathan%2Bsampson&cmt=136.601&ei=DZWeYIvwNs2Dkwavnq3IDw&fmt=134&fs=0&rt=137&of=wbAUJoLFDaeeOSJrHXH1sg&euri&lact=3412&cl=372995990&state=playing&vm=CAEQABgEKiAwdkUxUVZoUUY5M0NfYURuSzlRbzUwX053b3Y2WFg2OToyQU9HdF9PSzQwYkVsaTNZNjNjRXNGTERzdWdnZmhwamJzTnI2ZWRkMHh6dVVOei1vN2c&volume=100&cbr=Chrome&cbrver=91.0.4472.38&c=WEB&cver=2.20210512.06.00&cplayer=UNIPLAYER&cos=Windows&cosver=10.0&cplatform=DESKTOP&hl=en_US&cr=US&len=1648.281&rtn=177&feature=search&afmt=251&idpj=-1&ldpj=-14&rti=137&st=96.601&et=136.601&muted=0
[6492:1116:0514/082151.546:VERBOSE1:tls_handshaker.cc(96)] TlsHandshaker: continuing handshake
[6492:1116:0514/082151.573:VERBOSE1:tls_handshaker.cc(96)] TlsHandshaker: continuing handshake
[6492:1116:0514/082151.573:VERBOSE1:tls_handshaker.cc(96)] TlsHandshaker: continuing handshake
[6492:1116:0514/082151.574:VERBOSE1:tls_client_handshaker.cc(470)] Client: handshake finished
[6492:1116:0514/082155.001:VERBOSE1:network_delegate.cc(32)] NetworkDelegate::NotifyBeforeURLRequest: https://r4---sn-n4v7knl6.googlevideo.com/videoplayback?expire=1621027181&ei=DZWeYIvwNs2Dkwavnq3IDw&ip=99.152.113.108&id=o-ADGDADG68oPksEnAu0IeM4FdrNUwVM6g-2QXr2VNX4ry&itag=251&source=youtube&requiressl=yes&mh=Rf&mm=31%2C26&mn=sn-n4v7knl6%2Csn-a5mlrn7s&ms=au%2Conr&mv=m&mvi=4&pl=20&initcwndbps=1120000&vprv=1&mime=audio%2Fwebm&ns=MGv1UU8Ye9SARsiTk4nIcegF&gir=yes&clen=26249892&otfp=1&dur=1648.281&lmt=1575389686247419&mt=1621004696&fvip=4&keepalive=yes&fexp=24001373%2C24007246&c=WEB&txp=2211222&n=zO1ysO_CI9s7fA&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cvprv%2Cmime%2Cns%2Cgir%2Cclen%2Cotfp%2Cdur%2Clmt&sig=AOq0QJ8wRQIhAPkD_9VAx-HIOLqSMdfrsTVcyFAgZEoKpb2QtFHd8_jKAiAeIL_O0azsOiehp2iN6FQgI6ERQBI6OnNIdYfdHUKsyA%3D%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRAIgNOmVEXzF982XNOoYiS2QSqdp_JYjAmHB5PSdLhokbs8CIFOj6HB-azpcSiC0P2JCsYCAx_61_HESy-GwJa4pHPD2&alr=yes&cpn=PaQe6mmwRVE79tyy&cver=2.20210512.06.00&range=4112753-4597340&rn=23&rbuf=119945
[6988:14308:0514/082155.254:VERBOSE1:component_installer.cc(267)] StartRegistration for SODA Library
[6988:14780:0514/082155.254:VERBOSE1:component_installer.cc(267)] StartRegistration for SODA en-US Models
[6988:9976:0514/082155.255:VERBOSE1:component_installer.cc(411)] FinishRegistration for SODA Library
[6988:9976:0514/082155.256:VERBOSE1:component_installer.cc(411)] FinishRegistration for SODA en-US Models
[6492:1116:0514/082155.257:VERBOSE1:network_delegate.cc(32)] NetworkDelegate::NotifyBeforeURLRequest: https://go-updater.brave.com/extensions
[6492:1116:0514/082155.257:VERBOSE1:network_delegate.cc(32)] NetworkDelegate::NotifyBeforeURLRequest: https://go-updater.brave.com/extensions
[6492:1116:0514/082155.294:VERBOSE1:network_delegate.cc(32)] NetworkDelegate::NotifyBeforeURLRequest: https://componentupdater.brave.com/service/update2/json
[6492:1116:0514/082155.296:VERBOSE1:network_delegate.cc(32)] NetworkDelegate::NotifyBeforeURLRequest: https://componentupdater.brave.com/service/update2/json
[6988:9976:0514/082155.346:VERBOSE1:request_sender.cc(181)] Request completed from url: https://update.googleapis.com/service/update2/json
[6492:1116:0514/082155.347:VERBOSE1:network_delegate.cc(32)] NetworkDelegate::NotifyBeforeURLRequest: https://crlsets.brave.com/edgedl/release2/chrome_component/AqV3dEfu7BD5b9UQsUmlNA_1.0.7/AN4h7FThv35v2FE8zZO1BHs
[6988:9976:0514/082155.348:VERBOSE1:request_sender.cc(181)] Request completed from url: https://update.googleapis.com/service/update2/json
[6492:1116:0514/082155.349:VERBOSE1:network_delegate.cc(32)] NetworkDelegate::NotifyBeforeURLRequest: https://crlsets.brave.com/edgedl/release2/chrome_component/KER6p0Gr9-8E8ZmiXE7F4A_1.0.0/ALoRDWbCNFaNcA17KLAsXzQ
[6988:9976:0514/082155.358:VERBOSE1:url_fetcher_downloader.cc(136)] url fetcher response started for: http://edgedl.me.gvt1.com/edgedl/release2/chrome_component/AqV3dEfu7BD5b9UQsUmlNA_1.0.7/AN4h7FThv35v2FE8zZO1BHs
[6988:9976:0514/082155.407:VERBOSE1:url_fetcher_downloader.cc(136)] url fetcher response started for: http://edgedl.me.gvt1.com/edgedl/release2/chrome_component/KER6p0Gr9-8E8ZmiXE7F4A_1.0.0/ALoRDWbCNFaNcA17KLAsXzQ
[6492:1116:0514/082156.009:VERBOSE1:network_delegate.cc(32)] NetworkDelegate::NotifyBeforeURLRequest: https://i.ytimg.com/an_webp/eGoSdLNceEk/mqdefault_6s.webp?du=3000&sqp=CLqZ-oQG&rs=AOn4CLACq8V-ALZRLmtxwi2X8e3NMSHACA
[6988:9976:0514/082156.099:VERBOSE1:url_fetcher_downloader.cc(113)] Downloaded 20266623 bytes in 752ms from http://edgedl.me.gvt1.com/edgedl/release2/chrome_component/AqV3dEfu7BD5b9UQsUmlNA_1.0.7/AN4h7FThv35v2FE8zZO1BHs to C:\Users\Stephen\AppData\Local\Temp\chrome_url_fetcher_6988_1606468182\AN4h7FThv35v2FE8zZO1BHs
[6988:16568:0514/082156.158:VERBOSE1:component_unpacker.cc(59)] Verifying component: C:\Users\Stephen\AppData\Local\Temp\chrome_url_fetcher_6988_1606468182\AN4h7FThv35v2FE8zZO1BHs
[6988:16568:0514/082156.359:VERBOSE1:component_unpacker.cc(77)] Verification successful: C:\Users\Stephen\AppData\Local\Temp\chrome_url_fetcher_6988_1606468182\AN4h7FThv35v2FE8zZO1BHs
[6988:16568:0514/082156.360:VERBOSE1:component_unpacker.cc(90)] Unpacking in: C:\Users\Stephen\AppData\Local\Temp\6988_512607737
[6988:9976:0514/082156.654:VERBOSE1:url_fetcher_downloader.cc(113)] Downloaded 41780976 bytes in 1305ms from http://edgedl.me.gvt1.com/edgedl/release2/chrome_component/KER6p0Gr9-8E8ZmiXE7F4A_1.0.0/ALoRDWbCNFaNcA17KLAsXzQ to C:\Users\Stephen\AppData\Local\Temp\chrome_url_fetcher_6988_1603615579\ALoRDWbCNFaNcA17KLAsXzQ
[6988:8408:0514/082156.688:VERBOSE1:component_unpacker.cc(103)] Unpacked successfully
[6988:8408:0514/082156.691:VERBOSE1:component_installer.cc(119)] Install: version=1.0.7 current version=0.0.0.0
[6988:8408:0514/082156.691:VERBOSE1:component_installer.cc(137)] unpack_path=C:\Users\Stephen\AppData\Local\Temp\6988_512607737 install_path=C:\Users\Stephen\AppData\Local\BraveSoftware\Brave-Browser-Nightly\User Data\SODA\1.0.7
[6988:9976:0514/082156.694:VERBOSE1:component_installer.cc(454)] Component ready, version 1.0.7 in C:\Users\Stephen\AppData\Local\BraveSoftware\Brave-Browser-Nightly\User Data\SODA\1.0.7
[6988:9976:0514/082156.694:VERBOSE1:soda_component_installer.cc(143)] Component ready, version 1.0.7 in C:\Users\Stephen\AppData\Local\BraveSoftware\Brave-Browser-Nightly\User Data\SODA\1.0.7
[6988:9976:0514/082156.695:VERBOSE1:component_updater_service.cc(387)] Update completed with error 0
[6492:1116:0514/082156.695:VERBOSE1:network_delegate.cc(32)] NetworkDelegate::NotifyBeforeURLRequest: https://go-updater.brave.com/extensions
[6492:1116:0514/082156.754:VERBOSE1:network_delegate.cc(32)] NetworkDelegate::NotifyBeforeURLRequest: https://componentupdater.brave.com/service/update2/json
[6988:16568:0514/082156.762:VERBOSE1:component_unpacker.cc(59)] Verifying component: C:\Users\Stephen\AppData\Local\Temp\chrome_url_fetcher_6988_1603615579\ALoRDWbCNFaNcA17KLAsXzQ
[6988:9976:0514/082156.787:VERBOSE1:request_sender.cc(181)] Request completed from url: https://update.googleapis.com/service/update2/json
[6988:16568:0514/082157.136:VERBOSE1:component_unpacker.cc(77)] Verification successful: C:\Users\Stephen\AppData\Local\Temp\chrome_url_fetcher_6988_1603615579\ALoRDWbCNFaNcA17KLAsXzQ
[6988:16568:0514/082157.137:VERBOSE1:component_unpacker.cc(90)] Unpacking in: C:\Users\Stephen\AppData\Local\Temp\6988_2124775342
[6988:16568:0514/082157.629:VERBOSE1:component_unpacker.cc(103)] Unpacked successfully
[6988:16568:0514/082157.635:VERBOSE1:component_installer.cc(119)] Install: version=1.0.0 current version=0.0.0.0
[6988:16568:0514/082157.635:VERBOSE1:component_installer.cc(137)] unpack_path=C:\Users\Stephen\AppData\Local\Temp\6988_2124775342 install_path=C:\Users\Stephen\AppData\Local\BraveSoftware\Brave-Browser-Nightly\User Data\SODALanguagePacks\en-US\1.0.0
[6988:9976:0514/082157.645:VERBOSE1:component_installer.cc(454)] Component ready, version 1.0.0 in C:\Users\Stephen\AppData\Local\BraveSoftware\Brave-Browser-Nightly\User Data\SODALanguagePacks\en-US\1.0.0
[6988:9976:0514/082157.645:VERBOSE1:soda_language_pack_component_installer.cc(126)] Component ready, version 1.0.0 in C:\Users\Stephen\AppData\Local\BraveSoftware\Brave-Browser-Nightly\User Data\SODALanguagePacks\en-US\1.0.0
[6492:1116:0514/082157.652:VERBOSE1:network_delegate.cc(32)] NetworkDelegate::NotifyBeforeURLRequest: https://go-updater.brave.com/extensions
[6988:9976:0514/082157.652:VERBOSE1:component_updater_service.cc(387)] Update completed with error 0
[6492:1116:0514/082157.690:VERBOSE1:network_delegate.cc(32)] NetworkDelegate::NotifyBeforeURLRequest: https://componentupdater.brave.com/service/update2/json
[6988:9976:0514/082157.724:VERBOSE1:request_sender.cc(181)] Request completed from url: https://update.googleapis.com/service/update2/json

These show up in brave://components:

  • SODA en-US Models - Version: 1.0.0
  • SODA Library - Version: 1.0.7

soda-components

Expected result:

Unsure; but we should figure out if we want this enabled at all, and/or wrapped via our proxy calls, as we do with other component downloads.

Reproduces how often:

100%

Brave version (brave://version info)

Brave 1.26.19 Chromium: 91.0.4472.38 (Official Build) nightly (64-bit)
Revision 8155e7afab5d695cf0e028f4d77203287523cda9-refs/branch-heads/4472_35@{#6}
OS Windows 10 OS Version 2009 (Build 21376.1)

Version/Channel Information:

  • Can you reproduce this issue with the current release? no
  • Can you reproduce this issue with the beta channel? no
  • Can you reproduce this issue with the nightly channel? yes

@rebron @bsclifton @karenkliu @brave/legacy_qa
@stephendonner stephendonner added needs-discussion Although the issue is clear, we haven't yet reached a decision about the right solution. OS/Desktop labels May 14, 2021
@stephendonner
Copy link
Author

Should also mention:

  1. the UI options for these are missing in Brave, for which I filed Consider pulling in some options from Chromium's Accessibility settings page #15844
  2. there are two feature flags in brave://flags:
    Live Caption
    SODA for Live Caption

Both are set to default even with the component downloaded and presumably active.

soda-feature-flags

@stephendonner stephendonner mentioned this issue May 14, 2021
Closed
@stephendonner
Copy link
Author

I'll test and report back on whether disabling either/both of those flags disables the UI and network requests (as I'd expect them to).

@wchen342 wchen342 mentioned this issue May 15, 2021
Closed
@mkarolin mkarolin mentioned this issue May 17, 2021
Closed
@bsclifton
Copy link
Member

We can proxy this; besides proxying, we'll want to create a security/privacy review before re-enabling

@diracdeltas diracdeltas added privacy/connect This requires making a network connection to a third-party service. security labels May 18, 2021
@diracdeltas
Copy link
Member

cc @pes10k @jumde

@jumde
Copy link
Contributor

jumde commented May 18, 2021

This will be disabled for now since it requires an API key. Following up with the chromium team to check if this will be available for chromium derivatives.

@rebron rebron added the priority/P3 The next thing for us to work on. It'll ride the trains. label May 18, 2021
@jumde jumde added the privacy label Sep 1, 2021
@genesiscz
Copy link

Any update?

1 similar comment
@Crefix
Copy link

Crefix commented Apr 29, 2023

Any update?

@rebron rebron added this to General May 28, 2024
@rebron rebron moved this to P3 Backlog in General May 28, 2024
@win32ss win32ss mentioned this issue Oct 29, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
accessibility needs-discussion Although the issue is clear, we haven't yet reached a decision about the right solution. OS/Desktop priority/P3 The next thing for us to work on. It'll ride the trains. privacy/connect This requires making a network connection to a third-party service. privacy security
Projects
General
Status: P3 Backlog
Milestone
No milestone
Development

No branches or pull requests
7 participants
@stephendonner @diracdeltas @genesiscz @jumde @bsclifton @rebron @Crefix