WASM: sdk-common #1170

dangeross · 2025-02-18T20:12:20Z

This PR updates the sdk-common crate to be compilable to the wasm32-unknown-unknown target.

Exchange the use of the tonic transport feature in WASM for tonic_web_wasm_client
Update async traits to be WASM compatible
Update tests to be WASM compatible including adding a mockito alternative for WASM server mocking
Add CI tests for WASM and sdk-common as a WASM dependency check
Add sdk-macros crate to simplify WASM/non-WASM use
Add DNS-over-HTTPS lookup for WASM

roeierez · 2025-02-19T06:29:54Z

BIP353: Would a Breez API to do DNS TXT lookups be acceptable?

There is DNS over http protocol and services. https://developers.google.com/speed/public-dns/docs/doh
I think google has a public endpoint also.

roeierez

Thanks Ross for this PR! It really allows us to see what are the main issues and requires us to discuss an approach before we continue with the higher levels.
One thing we should think and discuss is how to minimize the wasm specific code changes and how to isolate it as much as possible (IMO even if we have to add more "duplicate" code).
I still don't have a silver bullet here but one think worth to consider is if we treat the SDK common as internal (not exposing structures to the browser) we might avoid wasmify all the structures.

roeierez · 2025-02-19T08:44:03Z

libs/sdk-common/src/breez_server.rs

            api_key,
        })
    }

+    #[cfg(not(target_arch = "wasm32"))]


Since both grpc_client & grpc_channel implements the same required interfaces perhaps we can just abstract their creation instead of duplicate all methods?
If it is not possible to have one struct field then perhaps we can just create a generic method that returns the right interface implementations and chooses internally which structure to return?

libs/sdk-common/src/fiat.rs

roeierez · 2025-02-19T08:58:53Z

libs/sdk-common/src/input_parser.rs

@@ -285,6 +301,7 @@ fn extract_bip353_record(records: Vec<String>) -> Option<String> {
    bip353_record.first().cloned()
 }

+#[cfg(not(target_arch = "wasm32"))]
 async fn bip353_parse(
    input: &str,
    dns_resolver: &AsyncResolver<GenericConnector<TokioRuntimeProvider>>,


Seems like we can implement a DNS resolver over https and pass it as a trait to the bip353_parse function.

dangeross · 2025-02-19T09:52:26Z

Can test by cloning my WASM playground

JssDWt

Great milestone this one!

In general I think we should try keeping the wasm feature flags as compact as possible. Like moving things that need it to their own file. Or creating our own macros for testing or model derivations for example.

.github/workflows/main.yml

libs/sdk-common/Cargo.toml

JssDWt · 2025-02-19T09:54:29Z

libs/sdk-common/Cargo.toml

+] } 
+
+[target.'cfg(not(target_arch = "wasm32"))'.build-dependencies]
+tonic-build = "^0.8"


Noting tonic-build needs an upgrade next.

Makefile

libs/sdk-common/Cargo.toml

libs/sdk-common/src/buy/mod.rs

libs/sdk-common/src/utils/rest_client.rs

libs/sdk-common/justfile

danielgranhao

Awesome work! Found a couple of issues with tests, otherwise is looking good

libs/sdk-common/Makefile

danielgranhao · 2025-02-20T12:48:54Z

libs/sdk-common/Makefile

+	$(CLANG_PREFIX) cargo clippy --target=wasm32-unknown-unknown -- -D warnings -A clippy::uninlined-format-args
+
+wasm-test:
+	$(CLANG_PREFIX) wasm-pack test --headless --firefox


We should add firefox to the readme prerequisites.

Also, I tried running the tests on chrome and there a couple of tests failed:

failures: ---- input_parser::tests::test_external_parsing_bitcoin_address_and_bolt11 output ---- JS exception that was thrown: Error: Unrecognized input type at imports.wbg.__wbindgen_error_new (http://127.0.0.1:8000/wasm-bindgen-test:1070:21) at sdk_common-0e2972322936c71e.wasm.__wbindgen_error_new externref shim (http://127.0.0.1:8000/wasm-bindgen-test_bg.wasm:wasm-function[20745]:0x720e87) at sdk_common-0e2972322936c71e.wasm.wasm_bindgen::JsError::new::hd2e35b77ed3041d8 (http://127.0.0.1:8000/wasm-bindgen-test_bg.wasm:wasm-function[12491]:0x6a22bf) at sdk_common-0e2972322936c71e.wasm.<core::result::Result<(),E> as wasm_bindgen_test::__rt::Termination>::into_js_result::{{closure}}::h31537578390b5f52 (http://127.0.0.1:8000/wasm-bindgen-test_bg.wasm:wasm-function[3019]:0x4cf863) at sdk_common-0e2972322936c71e.wasm.core::result::Result<T,E>::map_err::h259c253d9c7c5cbd (http://127.0.0.1:8000/wasm-bindgen-test_bg.wasm:wasm-function[7480]:0x5fef42) at sdk_common-0e2972322936c71e.wasm.<core::result::Result<(),E> as wasm_bindgen_test::__rt::Termination>::into_js_result::h42b88e6f5f052ddd (http://127.0.0.1:8000/wasm-bindgen-test_bg.wasm:wasm-function[12792]:0x6a8f03) at sdk_common-0e2972322936c71e.wasm.wasm_bindgen_test::__rt::Context::execute_async::{{closure}}::hcd1cc5f1b4de1bf2 (http://127.0.0.1:8000/wasm-bindgen-test_bg.wasm:wasm-function[2721]:0x4abf78) at sdk_common-0e2972322936c71e.wasm.<wasm_bindgen_test::__rt::TestFuture<F> as core::future::future::Future>::poll::{{closure}}::{{closure}}::h923e23a775fadbaf (http://127.0.0.1:8000/wasm-bindgen-test_bg.wasm:wasm-function[7577]:0x60341f) at sdk_common-0e2972322936c71e.wasm.wasm_bindgen::convert::closures::invoke0_mut::h5c7b6a0834f2c50d (http://127.0.0.1:8000/wasm-bindgen-test_bg.wasm:wasm-function[9230]:0x644f6f) at __wbg_adapter_64 (http://127.0.0.1:8000/wasm-bindgen-test:309:10) ---- input_parser::tests::test_external_parsing_lnurlp_first_response output ---- JS exception that was thrown: Error: Unrecognized input type at imports.wbg.__wbindgen_error_new (http://127.0.0.1:8000/wasm-bindgen-test:1070:21) at sdk_common-0e2972322936c71e.wasm.__wbindgen_error_new externref shim (http://127.0.0.1:8000/wasm-bindgen-test_bg.wasm:wasm-function[20745]:0x720e87) at sdk_common-0e2972322936c71e.wasm.wasm_bindgen::JsError::new::hd2e35b77ed3041d8 (http://127.0.0.1:8000/wasm-bindgen-test_bg.wasm:wasm-function[12491]:0x6a22bf) at sdk_common-0e2972322936c71e.wasm.<core::result::Result<(),E> as wasm_bindgen_test::__rt::Termination>::into_js_result::{{closure}}::h31537578390b5f52 (http://127.0.0.1:8000/wasm-bindgen-test_bg.wasm:wasm-function[3019]:0x4cf863) at sdk_common-0e2972322936c71e.wasm.core::result::Result<T,E>::map_err::h259c253d9c7c5cbd (http://127.0.0.1:8000/wasm-bindgen-test_bg.wasm:wasm-function[7480]:0x5fef42) at sdk_common-0e2972322936c71e.wasm.<core::result::Result<(),E> as wasm_bindgen_test::__rt::Termination>::into_js_result::h42b88e6f5f052ddd (http://127.0.0.1:8000/wasm-bindgen-test_bg.wasm:wasm-function[12792]:0x6a8f03) at sdk_common-0e2972322936c71e.wasm.wasm_bindgen_test::__rt::Context::execute_async::{{closure}}::ha848c2cec0289637 (http://127.0.0.1:8000/wasm-bindgen-test_bg.wasm:wasm-function[2720]:0x4abd62) at sdk_common-0e2972322936c71e.wasm.<wasm_bindgen_test::__rt::TestFuture<F> as core::future::future::Future>::poll::{{closure}}::{{closure}}::h9c7ec505bdf6b1f7 (http://127.0.0.1:8000/wasm-bindgen-test_bg.wasm:wasm-function[7579]:0x603595) at sdk_common-0e2972322936c71e.wasm.wasm_bindgen::convert::closures::invoke0_mut::h5c7b6a0834f2c50d (http://127.0.0.1:8000/wasm-bindgen-test_bg.wasm:wasm-function[9230]:0x644f6f) at __wbg_adapter_64 (http://127.0.0.1:8000/wasm-bindgen-test:309:10) failures: input_parser::tests::test_external_parsing_bitcoin_address_and_bolt11 input_parser::tests::test_external_parsing_lnurlp_first_response test result: FAILED. 43 passed; 2 failed; 0 ignored; 0 filtered out; finished in 0.45s

Added docs and fixed tests 17c1765

Can you pull and test safari for me? I get errors starting the tests. I added make wasm-test-safari

Running headless tests in Safari on `http://127.0.0.1:54724/` Try find `webdriver.json` for configure browser's capabilities: Ok driver status: signal: 9 (SIGKILL) Error: http://127.0.0.1:54724/session: status code 500

I get the same error when trying to run headless. Running on the browser works fine.

libs/sdk-common/mock_service_worker.js

libs/sdk-common/Makefile

danielgranhao · 2025-02-20T17:48:05Z

libs/sdk-common/Makefile

+	$(CLANG_PREFIX) cargo clippy --target=wasm32-unknown-unknown -- -D warnings -A clippy::uninlined-format-args
+
+wasm-test:
+	$(CLANG_PREFIX) wasm-pack test --headless --firefox


I get the same error when trying to run headless. Running on the browser works fine.

Timeout fn is not available on ClientBuilder in the WASM client

roeierez · 2025-02-26T10:16:51Z

libs/sdk-common/src/test_utils/mock_server_wasm.rs

+    pub port: u16,
+}
+
+impl MockServer {


Something to think of regarding the MockServer. Currently we are using a Mock server for lnurl testing.
We are also dynamically replacingthe urls with a maybe_replace_host_with_mock_test_host function tha is only available for testing and registering the expected result on the mock server.
It also forces us to implement this service worker for wasm.
Since all lnurl requests are going through the function get_parse_and_log_response perhaps we should just create a testing version of it that returns the expected result and get rid of the mock server?
If we are able to do it it will remove a lot of specific WASM code and also the tests will use a lot of shared code.

danielgranhao

Looking good! Left some comments. Also, as @roeierez proposed, I agree it would be nice to get rid of the mock server.

libs/Cargo.toml

libs/sdk-common/src/breez_server.rs

libs/sdk-common/src/utils/rest_client.rs

libs/sdk-common/webdriver.json

danielgranhao · 2025-02-26T12:19:00Z

libs/sdk-macros/Cargo.toml

+[package]
+name = "sdk-macros"
+edition = "2021"
+version.workspace = true


Maybe we should publish this to crates.io? It will be pretty useful for the boltz client crate as well :)

dangeross requested review from roeierez, JssDWt, danielgranhao and hydra-yse February 18, 2025 20:12

roeierez reviewed Feb 19, 2025

View reviewed changes

dangeross force-pushed the savage-wasm-sdk-common branch from 9ed23ec to 72c6a65 Compare February 19, 2025 09:52

JssDWt reviewed Feb 19, 2025

View reviewed changes

dangeross force-pushed the savage-wasm-sdk-common branch 2 times, most recently from 88e7f0e to 19ed955 Compare February 19, 2025 13:19

danielgranhao reviewed Feb 20, 2025

View reviewed changes

dangeross force-pushed the savage-wasm-sdk-common branch 9 times, most recently from 80ea651 to 9922a8c Compare February 25, 2025 16:57

dangeross added 8 commits February 25, 2025 18:45

Compile sdk-common to WASM

d9f95f2

Simplify breez_server

53177f1

Replace justfile with Makefile

dd296c4

Isolate WASM implementations for tonic_wrap and grpc client

cbd33fa

Bump reqwest and change timeout to RequestBuilder

5b7d517

Timeout fn is not available on ClientBuilder in the WASM client

Fix brew directory selection, fix chrome test errors

f8aa77a

Update WASM target

00a5d19

Update README

b8731ac

dangeross added 4 commits February 25, 2025 18:46

Add minimal tsify decorators

5beba10

Add sdk_macros crate

3762ad6

Add extern_wasm_bindgen macro

dcd7f5e

Add DNS over HTTPS lookup for WASM

33bd7af

dangeross force-pushed the savage-wasm-sdk-common branch from 9922a8c to 33bd7af Compare February 25, 2025 17:46

CI Clippy

249c838

dangeross requested review from roeierez, danielgranhao and JssDWt February 25, 2025 18:35

Validate DNS label size

17d16db

roeierez reviewed Feb 26, 2025

View reviewed changes

Reimplement WASM TXT lookup to use DNS wireformat

3d37a6a

dangeross force-pushed the savage-wasm-sdk-common branch from fc3138a to 3d37a6a Compare February 26, 2025 11:38

danielgranhao reviewed Feb 26, 2025

View reviewed changes

Apply review feedback

a9a1c18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WASM: sdk-common #1170

WASM: sdk-common #1170

dangeross commented Feb 18, 2025 •

edited

Loading

roeierez commented Feb 19, 2025

roeierez left a comment

roeierez Feb 19, 2025

roeierez Feb 19, 2025

dangeross commented Feb 19, 2025

JssDWt left a comment

JssDWt Feb 19, 2025

danielgranhao left a comment

danielgranhao Feb 20, 2025 •

edited

Loading

dangeross Feb 20, 2025

danielgranhao Feb 20, 2025

danielgranhao Feb 20, 2025

roeierez Feb 26, 2025

danielgranhao left a comment

danielgranhao Feb 26, 2025

WASM: sdk-common #1170

Are you sure you want to change the base?

WASM: sdk-common #1170

Conversation

dangeross commented Feb 18, 2025 • edited Loading

roeierez commented Feb 19, 2025

roeierez left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dangeross commented Feb 19, 2025

JssDWt left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

danielgranhao left a comment

Choose a reason for hiding this comment

danielgranhao Feb 20, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

danielgranhao left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dangeross commented Feb 18, 2025 •

edited

Loading

danielgranhao Feb 20, 2025 •

edited

Loading