Upgrade webrtc-audio-processing lib to v1.3 #23

skywhale · 2021-08-30T01:47:24Z

Upgrade the underlying webrtc-audio-processing library to v1.3. This is a major update, which include significant changes to the AEC behaviors.

Tasks

Give feedback

Expose some pameters for AEC3 (https://webrtc.googlesource.com/src/+/refs/heads/main/api/audio/echo_canceller3_config.h)
Options

strohel

Whoa, this must have been a ton of work! I've reviewed roughly the first half of the expanded diff, added some comments. I haven't worked with the library yet, so please take them with a grain of salt.

Thanks for taking care to update and extend tests, that's admirable you found energy to do that despite overall size of the change.

src/config.rs

strohel · 2021-08-31T16:29:35Z

src/config.rs

+    /// Analyze the output of the linear AEC instead of the capture frame. Has no effect if
+    /// echo_canceller.export_linear_aec_output is false.
+    pub analyze_linear_aec_output_when_available: bool,


Optional brainstorm: we can try to combine this with echo_canceller.export_linear_aec_output into 3-state enum, so that it is not possible to represent state that has no effect. Caveat: would have to be in some higher-level structure and do more harm than good.

Thank you for the brainstorm. I think I will wait to introduce a 3-state enum for now. I want to keep the API flat and less opinionated until I have deeper understanding of the internals and what are the recommended configurations. To improve a bit, I removed EchoCanceller::export_linear_aec_output, and changed it to configure automatically based on NoiseSuppressor:: analyze_linear_aec_output, after confirming the webrtc implementation that linear AEC output is only used by noise cancellation.

src/config.rs

webrtc-audio-processing-sys/build.rs

webrtc-audio-processing-sys/src/lib.rs

skywhale · 2021-09-07T14:45:53Z

I gave up on this TODO. Bindgen cannot bind virtual functions (rust-lang/rust-bindgen#27).

bschwind · 2021-09-13T02:44:54Z

@skywhale are you able to build this on MacOS? I'm getting the following error:

<bunch of CMake output>
...
-- Installing: /Users/brian/projects/tonari/webrtc-audio-processing/target/release/build/webrtc-audio-processing-sys-0357fd24f02783fb/out/lib/libabsl_time_zone.a
-- Installing: /Users/brian/projects/tonari/webrtc-audio-processing/target/release/build/webrtc-audio-processing-sys-0357fd24f02783fb/out/lib/libabsl_bad_any_cast_impl.a
-- Installing: /Users/brian/projects/tonari/webrtc-audio-processing/target/release/build/webrtc-audio-processing-sys-0357fd24f02783fb/out/lib/libabsl_bad_optional_access.a
-- Installing: /Users/brian/projects/tonari/webrtc-audio-processing/target/release/build/webrtc-audio-processing-sys-0357fd24f02783fb/out/lib/libabsl_bad_variant_access.a
cargo:root=/Users/brian/projects/tonari/webrtc-audio-processing/target/release/build/webrtc-audio-processing-sys-0357fd24f02783fb/out/./abseil-cpp

--- stderr
CMake Warning:
  Manually-specified variables were not used by the project:

    CMAKE_ASM_COMPILER
    CMAKE_ASM_FLAGS
    CMAKE_C_COMPILER
    CMAKE_C_FLAGS


/Users/brian/projects/tonari/webrtc-audio-processing/webrtc-audio-processing-sys/abseil-cpp/absl/time/internal/cctz/src/time_zone_info.cc:721:35: warning: declaration shadows a local variable [-Wshadow-uncaptured-local]
      name, [](const std::string& name) -> std::unique_ptr<ZoneInfoSource> {
                                  ^
/Users/brian/projects/tonari/webrtc-audio-processing/webrtc-audio-processing-sys/abseil-cpp/absl/time/internal/cctz/src/time_zone_info.cc:709:44: note: previous declaration is here
bool TimeZoneInfo::Load(const std::string& name) {
                                           ^
/Users/brian/projects/tonari/webrtc-audio-processing/webrtc-audio-processing-sys/abseil-cpp/absl/time/internal/cctz/src/time_zone_info.cc:722:18: warning: declaration shadows a local variable [-Wshadow-uncaptured-local]
        if (auto zip = FileZoneInfoSource::Open(name)) return zip;
                 ^
/Users/brian/projects/tonari/webrtc-audio-processing/webrtc-audio-processing-sys/abseil-cpp/absl/time/internal/cctz/src/time_zone_info.cc:720:8: note: previous declaration is here
  auto zip = cctz_extension::zone_info_source_factory(
       ^
/Users/brian/projects/tonari/webrtc-audio-processing/webrtc-audio-processing-sys/abseil-cpp/absl/time/internal/cctz/src/time_zone_info.cc:723:18: warning: declaration shadows a local variable [-Wshadow-uncaptured-local]
        if (auto zip = AndroidZoneInfoSource::Open(name)) return zip;
                 ^
/Users/brian/projects/tonari/webrtc-audio-processing/webrtc-audio-processing-sys/abseil-cpp/absl/time/internal/cctz/src/time_zone_info.cc:720:8: note: previous declaration is here
  auto zip = cctz_extension::zone_info_source_factory(
       ^
/Users/brian/projects/tonari/webrtc-audio-processing/webrtc-audio-processing-sys/abseil-cpp/absl/time/internal/cctz/src/time_zone_libc.cc:166:22: warning: declaration shadows a local variable [-Wshadow]
        if (std::tm* tmp = local_time(&lo, &tm)) {
                     ^
/Users/brian/projects/tonari/webrtc-audio-processing/webrtc-audio-processing-sys/abseil-cpp/absl/time/internal/cctz/src/time_zone_libc.cc:156:18: note: previous declaration is here
    if (std::tm* tmp = local_time(&mid, &tm)) {
                 ^
/Users/brian/projects/tonari/webrtc-audio-processing/webrtc-audio-processing-sys/abseil-cpp/absl/time/internal/cctz/src/time_zone_libc.cc:191:9: warning: result of comparison 'const std::int_fast64_t' (aka 'const long long') < -9223372036854775808 is always false [-Wtautological-type-limit-compare]
  if (s < std::numeric_limits<std::time_t>::min()) {
      ~ ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/Users/brian/projects/tonari/webrtc-audio-processing/webrtc-audio-processing-sys/abseil-cpp/absl/time/internal/cctz/src/time_zone_libc.cc:195:9: warning: result of comparison 'const std::int_fast64_t' (aka 'const long long') > 9223372036854775807 is always false [-Wtautological-type-limit-compare]
  if (s > std::numeric_limits<std::time_t>::max()) {
      ~ ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/Users/brian/projects/tonari/webrtc-audio-processing/webrtc-audio-processing-sys/abseil-cpp/absl/time/internal/cctz/src/time_zone_libc.cc:84:6: warning: unused function template 'tm_gmtoff' [-Wunused-template]
auto tm_gmtoff(const T& tm) -> decltype(tm.__tm_gmtoff) {
     ^
/Users/brian/projects/tonari/webrtc-audio-processing/webrtc-audio-processing-sys/abseil-cpp/absl/time/internal/cctz/src/time_zone_libc.cc:100:6: warning: unused function template 'tm_zone' [-Wunused-template]
auto tm_zone(const T& tm) -> decltype(tm.__tm_zone) {
     ^
5 warnings generated.
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: libabsl_debugging_internal.a(elf_mem_image.cc.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: libabsl_debugging_internal.a(vdso_support.cc.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: libabsl_debugging_internal.a(elf_mem_image.cc.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: libabsl_debugging_internal.a(vdso_support.cc.o) has no symbols
3 warnings generated.
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: libabsl_flags.a(flag.cc.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: libabsl_flags.a(flag.cc.o) has no symbols
warning: /Library/Developer/CommandLineTools/usr/bin/ranlib: warning for library: libabsl_flags.a the table of contents is empty (no object file members in the library define global symbols)
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: /Users/brian/projects/tonari/webrtc-audio-processing/target/release/build/webrtc-audio-processing-sys-0357fd24f02783fb/out/lib/libabsl_debugging_internal.a(elf_mem_image.cc.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: /Users/brian/projects/tonari/webrtc-audio-processing/target/release/build/webrtc-audio-processing-sys-0357fd24f02783fb/out/lib/libabsl_debugging_internal.a(vdso_support.cc.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: /Users/brian/projects/tonari/webrtc-audio-processing/target/release/build/webrtc-audio-processing-sys-0357fd24f02783fb/out/lib/libabsl_flags.a(flag.cc.o) has no symbols
warning: /Library/Developer/CommandLineTools/usr/bin/ranlib: warning for library: /Users/brian/projects/tonari/webrtc-audio-processing/target/release/build/webrtc-audio-processing-sys-0357fd24f02783fb/out/lib/libabsl_flags.a the table of contents is empty (no object file members in the library define global symbols)
Error: Os { code: 2, kind: NotFound, message: "No such file or directory" }

bschwind · 2021-09-13T02:50:44Z

Oh, this is caused by me not having meson installed. I swear Rust's file handling error message is the cause of way too much misery: Error: Os { code: 2, kind: NotFound, message: "No such file or directory" }

skywhale · 2021-09-13T02:53:34Z

Ah yes, meson and ninja are two build tools you need. I've only tested on my macbook so far; there may be some more work necessary on Linux.

bschwind · 2021-09-13T03:36:22Z

I'm getting more build errors on linux:

  --- stderr
  /home/tonari/webrtc-audio-processing/webrtc-audio-processing-sys/abseil-cpp/absl/synchronization/internal/graphcycles.cc: In member function ‘void absl::lts_2020_02_25::synchronization_internal::GraphCycles::RemoveNode(void*)’:
  /home/tonari/webrtc-audio-processing/webrtc-audio-processing-sys/abseil-cpp/absl/synchronization/internal/graphcycles.cc:451:26: error: ‘numeric_limits’ is not a member of ‘std’
    451 |   if (x->version == std::numeric_limits<uint32_t>::max()) {
        |                          ^~~~~~~~~~~~~~
  /home/tonari/webrtc-audio-processing/webrtc-audio-processing-sys/abseil-cpp/absl/synchronization/internal/graphcycles.cc:451:49: error: expected primary-expression before ‘>’ token
    451 |   if (x->version == std::numeric_limits<uint32_t>::max()) {
        |                                                 ^
  /home/tonari/webrtc-audio-processing/webrtc-audio-processing-sys/abseil-cpp/absl/synchronization/internal/graphcycles.cc:451:52: error: ‘::max’ has not been declared; did you mean ‘std::max’?
    451 |   if (x->version == std::numeric_limits<uint32_t>::max()) {
        |                                                    ^~~
        |                                                    std::max
  In file included from /usr/include/c++/11.1.0/algorithm:62,
                   from /home/tonari/webrtc-audio-processing/webrtc-audio-processing-sys/abseil-cpp/absl/synchronization/internal/graphcycles.cc:38:
  /usr/include/c++/11.1.0/bits/stl_algo.h:3467:5: note: ‘std::max’ declared here
   3467 |     max(initializer_list<_Tp> __l, _Compare __comp)
        |     ^~~
  make[2]: *** [absl/synchronization/CMakeFiles/graphcycles_internal.dir/build.make:76: absl/synchronization/CMakeFiles/graphcycles_internal.dir/internal/graphcycles.cc.o] Error 1
  make[1]: *** [CMakeFiles/Makefile2:2150: absl/synchronization/CMakeFiles/graphcycles_internal.dir/all] Error 2
  make[1]: *** Waiting for unfinished jobs....
  make: *** [Makefile:136: all] Error 2
  thread 'main' panicked at '
  command did not execute successfully, got: exit status: 2

  build script failed, must exit now', /home/tonari/.cargo/registry/src/github.com-1ecc6299db9ec823/cmake-0.1.45/src/lib.rs:894:5
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Possibly related? abseil/abseil-cpp#832

skywhale · 2023-11-24T02:11:43Z

This PR has been rebased on webrtc-audio-processing v1.3.

morajabi · 2024-03-04T13:35:13Z

@skywhale I'm getting an error message trying to use this branch with bundled flag on macOS:


  cargo:warning=In file included from src/wrapper.hpp:8:

  cargo:warning=/Users/mo/dev/noor/target/debug/build/webrtc-audio-processing-sys-4ab82baded4bbe6f/out/include/webrtc-audio-processing-1/modules/audio_processing/include/audio_processing.h:26:10: fatal error: 'absl/types/optional.h' file not found

  cargo:warning=#include "absl/types/optional.h"

  cargo:warning=         ^~~~~~~~~~~~~~~~~~~~~~~

  cargo:warning=1 error generated.

  exit status: 1

  --- stderr

Any idea how to fix / debug this?

Update: Fixed by adding abseil back

jacksongoode · 2025-01-06T20:34:17Z

Just curious why the change was made to remove the abseil-cpp builder? I've found, that at least on my machine (macOS 15) with all the build requirements, that I get:

The system library `webrtc-audio-processing-1` required by crate `webrtc-audio-processing-sys` was not found.
The file `webrtc-audio-processing-1.pc` needs to be installed and the PKG_CONFIG_PATH environment variable must contain its parent directory.
The PKG_CONFIG_PATH environment variable is not set.

HINT: if you have installed the library, try setting PKG_CONFIG_PATH to the directory containing `webrtc-audio-processing-1.pc`.

I've made a branch to link the abseil library back and it built successfully. I can open a PR to merge into this branch.

skywhale · 2025-01-07T11:50:44Z

@jacksongoode Thank you for investigating this and opening up a PR :)

Just curious why the change was made to remove the abseil-cpp builder?

I removed it because the new build system using meson pulls the dependency automatically and installs it under the build directory, which we can reference. I believe in your and @morajabi's case, meson found a system-installed abseil-cpp and skipped the step to pull the dependency. I fixed build.rs so that we correctly refer to the same abseil-cpp used by meson.

I also rebased this branch on top of main. I had to squash it as I do that 🙏

skywhale · 2025-01-07T11:57:56Z

@jacksongoode Could you pull the latest branch and see if the build succeeds?

The following commands should be sufficient for running the example karaoke program.

git clone https://github.com/tonarino/webrtc-audio-processing && cd webrtc-audio-processing
git switch upgrade-v1
git submodule update --init --recursive
cargo run --example karaoke --features bundled --features derive_serde

jacksongoode · 2025-01-07T18:54:21Z

I removed it because the new build system using meson pulls the dependency automatically and installs it under the build directory, which we can reference.

This makes sense and also the build error as well. I cleaned and it works now! I'll close my PR.

skywhale force-pushed the upgrade-v1 branch from 797e5a8 to 330f6fe Compare August 30, 2021 03:16

strohel reviewed Aug 31, 2021

View reviewed changes

skywhale marked this pull request as ready for review September 7, 2021 15:41

skywhale force-pushed the upgrade-v1 branch from 6a1a304 to 1ea411a Compare September 9, 2021 11:41

skywhale force-pushed the upgrade-v1 branch 2 times, most recently from c40fcb4 to 3b9aa5a Compare September 25, 2021 12:01

morajabi mentioned this pull request Apr 7, 2023

What's blocking using latest freedesktop/pulseaudio-webrtc-audio-processing? #33

Closed

skywhale changed the title ~~Upgrade webrtc-audio-processing lib to v1.0~~ Upgrade webrtc-audio-processing lib to v1.3 Nov 23, 2023

morajabi mentioned this pull request Apr 2, 2024

Transient noise suppression seems to not work #36

Closed

Upgrade webrtc-audio-processing lib to v1.3 (not tested to build)

35e5e21

skywhale force-pushed the upgrade-v1 branch from 6e23717 to 35e5e21 Compare January 7, 2025 11:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade webrtc-audio-processing lib to v1.3 #23

Upgrade webrtc-audio-processing lib to v1.3 #23

skywhale commented Aug 30, 2021 •

edited

Loading

Tasks

strohel left a comment

strohel Aug 31, 2021

skywhale Sep 7, 2021

skywhale commented Sep 7, 2021

bschwind commented Sep 13, 2021

bschwind commented Sep 13, 2021

skywhale commented Sep 13, 2021

bschwind commented Sep 13, 2021

skywhale commented Nov 24, 2023

morajabi commented Mar 4, 2024 •

edited

Loading

jacksongoode commented Jan 6, 2025

skywhale commented Jan 7, 2025 •

edited

Loading

skywhale commented Jan 7, 2025 •

edited

Loading

jacksongoode commented Jan 7, 2025

Upgrade webrtc-audio-processing lib to v1.3 #23

Are you sure you want to change the base?

Upgrade webrtc-audio-processing lib to v1.3 #23

Conversation

skywhale commented Aug 30, 2021 • edited Loading

Tasks

strohel left a comment

Choose a reason for hiding this comment

strohel Aug 31, 2021

Choose a reason for hiding this comment

skywhale Sep 7, 2021

Choose a reason for hiding this comment

skywhale commented Sep 7, 2021

bschwind commented Sep 13, 2021

bschwind commented Sep 13, 2021

skywhale commented Sep 13, 2021

bschwind commented Sep 13, 2021

skywhale commented Nov 24, 2023

morajabi commented Mar 4, 2024 • edited Loading

jacksongoode commented Jan 6, 2025

skywhale commented Jan 7, 2025 • edited Loading

skywhale commented Jan 7, 2025 • edited Loading

jacksongoode commented Jan 7, 2025

skywhale commented Aug 30, 2021 •

edited

Loading

morajabi commented Mar 4, 2024 •

edited

Loading

skywhale commented Jan 7, 2025 •

edited

Loading

skywhale commented Jan 7, 2025 •

edited

Loading