Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Inline pthread worker.js file into the main output file #21701

Merged
merged 1 commit into from
Apr 16, 2024

Conversation

sbc100
Copy link
Collaborator

@sbc100 sbc100 commented Apr 5, 2024

This method has many advantages over the previous method of generating a separate file:

  • Avoids separate output file simplifying deployment.
  • Avoids confusing laying of global scopes
  • Avoids exporting symbols on the Module simply for visibility within the worker file.
  • Avoids code duplication
  • Avoids the needs to importScripts call, and the node polyfill for this.
  • Allows optimizers such as closure and JSDCE to operate on the combined code.
  • -sSINGLE_FILE now works with pthreads
  • Fewer network requests
  • No need for locateFile logic to run on the worker to find the worker.js

The test_pthread_custom_pthread_main_url test was completely removed
since it was testing how worker.js was located and worker.js no longer
exists.

Fixes: #9796

@msqr1
Copy link

msqr1 commented Apr 6, 2024

Would you consider doing this for wasm workers and audio worklets? Also, if this has many advantages, I would suggest making this the default, while keeping option to NOT embed the wasm file, for reduced size, module caching and stuff.

@sbc100
Copy link
Collaborator Author

sbc100 commented Apr 6, 2024

Would you consider doing this for wasm workers and audio worklets? Also, if this has many advantages, I would suggest making this the default, while keeping option to NOT embed the wasm file, for reduced size, module caching and stuff.

Absolutely! One step a time though. Getting pthreads done first because it probably most used (and likely the hardest to get working) so a good proof of concept.

@msqr1
Copy link

msqr1 commented Apr 6, 2024

What about this being default and wasm file embedding? What do you think? We can make another setting called EMBED_WASM

@sbc100
Copy link
Collaborator Author

sbc100 commented Apr 6, 2024

What about this being default and wasm file embedding? What do you think? We can make another setting called EMBED_WASM

My plan is to make this not only the default, but the only option. I don't see any reason at this point to keep the old method around.

Don't we already have -sSINGLE_FILE to controle the Wasm embedding? I don't see that ever being the default BTW because it comes with several downsides, the most major of which it prevent streaming compilation and caching the wasm module.

@msqr1
Copy link

msqr1 commented Apr 6, 2024

SINGLE_FILE seems to mean embed everything into one file, not just the wasm, I think we should have another name for embedding or separate the wasm since this can be quite misleading

@sbc100
Copy link
Collaborator Author

sbc100 commented Apr 6, 2024

SINGLE_FILE seems to mean embed everything into one file, not just the wasm, I think we should have another name for embedding or separate the wasm since this can be quite misleading

But if we remove the separate worker files, what other file are there? Another way of putting it: What files would you not want to embed when you use -sSINGLE_FILE?

@msqr1
Copy link

msqr1 commented Apr 6, 2024

You're right, my mistake. But should we just rename the settings anyway? I'm really excited for this to land ngl

@sbc100
Copy link
Collaborator Author

sbc100 commented Apr 6, 2024

You're right, my mistake. But should we just rename the settings anyway? I'm really excited for this to land ngl

Doesn't SINGLE_FILE make sense? IIUC it is normally by for folks who don't want to worry about hosting multiple files, and want do distribute just one.. so that name seem to match the purpose. What about it do you think is confusing?

@msqr1
Copy link

msqr1 commented Apr 7, 2024

We should use a global switch statement for this? For simplicity I suggest just doing globalThis.constructor.name == "xxxGlobalScope", not using the ENVIRONMENT_IS_XXX

@curiousdannii
Copy link
Contributor

curiousdannii commented Apr 7, 2024

Rather than using a query string (which may complicate caching), can you use the name option?

Only potential issue I can see is that it's not supported in Android Webview, but I'm not sure if that's a supported browser.

@sbc100
Copy link
Collaborator Author

sbc100 commented Apr 7, 2024

Rather than using a query string (which may complicate caching), can you use the name option?

Sure we can try that. Can you explain what you mean by "may complicate caching"?

Only potential issue I can see is that it's not supported in Android Webview, but I'm not sure if that's a supported browser.

@curiousdannii
Copy link
Contributor

curiousdannii commented Apr 7, 2024

Just that many caches may distinguish query strings. It may not be added to service worker caches either. I was thinking of a case where you add all the files in a service worker for offline use, but then it fails because of the query string.

If someone was using a service worker they could work around that though. But it might be more predictable to avoid it possible. I don't know that the name property is a viable alternative however.

@sbc100
Copy link
Collaborator Author

sbc100 commented Apr 7, 2024

Just that many caches may distinguish query strings. It may not be added to service worker caches either. I was thinking of a case where you add all the files in a service worker for offline use, but then it fails because of the query string.

If someone was using a service worker they could work around that though. But it might be more predictable to avoid it possible. I don't know that the name property is a viable alternative however.

Are you saying that adding the query string would mean that the file would not cached at all, or that maybe we would end up with two seperate cache entries (one for the main.js and another for the main.js?phthread=1)? I like the idea of using name since we do control that.

src/library_pthread.js Outdated Show resolved Hide resolved
src/runtime_init_memory.js Outdated Show resolved Hide resolved
src/shell.js Outdated Show resolved Hide resolved
sbc100 added a commit that referenced this pull request Apr 25, 2024
The issue here was the special case for the setting of `scriptDirectory` under
nodejs when `ENVIRONMENT_IS_WORKER`.

Previously this path was not being hit when the main thread was running in
a worker because we were not setting `ENVIRONMENT_IS_WORKER` in
this case.   This was fixed in #21701 and `ENVIRONMENT_IS_WORKER`
was correctly set, but that meant that `scriptDirectory` then started being
set incorrectly.

Fixes: #21827
sbc100 added a commit to sbc100/emscripten that referenced this pull request Apr 28, 2024
sbc100 added a commit that referenced this pull request Apr 29, 2024
sbc100 added a commit to sbc100/emscripten that referenced this pull request May 29, 2024
In emscripten-core#21701 the running of extern-pre/post-js in pthreads was disabled.
The reason was that we had a couple of tests that seemed to be assuming
this.  However, it turns out that there are important use cases that
depend on this code running in threads.

Instead, fix the two test cases by adding an extra condition.

Fixes: emscripten-core#22012
sbc100 added a commit to sbc100/emscripten that referenced this pull request May 29, 2024
In emscripten-core#21701 the running of extern-pre/post-js in pthreads was disabled.
The reason was that we had a couple of tests that seemed to be assuming
this.  However, it turns out that there are important use cases that
depend on this code running in threads.

Instead, fix the two test cases by adding an extra condition.

Fixes: emscripten-core#22012
sbc100 added a commit that referenced this pull request May 30, 2024
In #21701 the running of extern-pre/post-js in pthreads was disabled.
The reason was that we had a couple of tests that seemed to be assuming
this.  However, it turns out that there are important use cases that
depend on this code running in threads.

Instead, fix the two test cases by adding an extra condition.

Fixes: #22012
msorvig pushed a commit to msorvig/emscripten that referenced this pull request Jun 4, 2024
…-core#22013)

In emscripten-core#21701 the running of extern-pre/post-js in pthreads was disabled.
The reason was that we had a couple of tests that seemed to be assuming
this.  However, it turns out that there are important use cases that
depend on this code running in threads.

Instead, fix the two test cases by adding an extra condition.

Fixes: emscripten-core#22012
@sbc100 sbc100 mentioned this pull request Jun 19, 2024
sbc100 added a commit to sbc100/emscripten that referenced this pull request Sep 10, 2024
In the part using unquoted key here meant that closure would minify
these key and this was a problem because the separate worker.js files
would not then be able to interpret them.

However since emscripten-core#21701 landed there is no longer separate worker.js file
so closure can minify these keys and it will effect both sending and
receiving code.
sbc100 added a commit to sbc100/emscripten that referenced this pull request Sep 10, 2024
In the part using unquoted key here meant that closure would minify
these key and this was a problem because the separate worker.js files
would not then be able to interpret them.

However since emscripten-core#21701 landed there is no longer separate worker.js file
so closure can minify these keys and it will effect both sending and
receiving code.
sbc100 added a commit that referenced this pull request Sep 11, 2024
In the part using unquoted key here meant that closure would minify
these key and this was a problem because the separate worker.js files
would not then be able to interpret them.

However since #21701 landed there is no longer separate worker.js file
so closure can minify these keys and it will effect both sending and
receiving code.
sbc100 added a commit to sbc100/emscripten that referenced this pull request Sep 20, 2024
We stopped using a separate worker file back in emscripten-core#21701.  That was
released in 3.1.58 back in April.

This change ends the transition period by no longer generating a
dummy/useless worker.js file alongside the output.
sbc100 added a commit to sbc100/emscripten that referenced this pull request Sep 20, 2024
We stopped using a separate worker file back in emscripten-core#21701.  That was
released in 3.1.58 back in April.

This change ends the transition period by no longer generating a
dummy/useless worker.js file alongside the output.
sbc100 added a commit to sbc100/emscripten that referenced this pull request Sep 20, 2024
We stopped using a separate worker file back in emscripten-core#21701.  That was
released in 3.1.58 back in April.

This change ends the transition period by no longer generating a
dummy/useless worker.js file alongside the output.
sbc100 added a commit to sbc100/emscripten that referenced this pull request Sep 20, 2024
We stopped using a separate worker file back in emscripten-core#21701.  That was
released in 3.1.58 back in April.

This change ends the transition period by no longer generating a
dummy/useless worker.js file alongside the output.
sbc100 added a commit that referenced this pull request Sep 20, 2024
We stopped using a separate worker file back in #21701. That was
released in 3.1.58 back in April.

This change ends the transition period by no longer generating a
dummy/useless worker.js file alongside the output.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add option to inline webworker
7 participants