Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

initial implementation of an executable server #168

Closed
wants to merge 1 commit into from

Conversation

paul-marechal
Copy link
Member

@paul-marechal paul-marechal commented Oct 12, 2021

The goal of this PR is to setup a browser version of Blueprint packaged
as a "drag and drop" solution that's easy to deploy.

The setup is a bit tricky and might not apply well to every kind of
product made using the Theia framework, but it should work for most
usages.

Add theia-blueprint-browser Theia extension that contributes the
required customizations to run in the packaged environment (mostly to
accommodate to the new file layout).

Use a node-specific Webpack configuration to bundle the backend along
with the browser-specific customizations.

Use a forked pkg package to support forking child processes with
execArgv options. The fork is setup as a git submodule.

Upstream patches to pkg

How to test

Run:

# setup git submodules
git submodule update --init

# build and package
yarn && yarn browser bundle && yarn browser package

This should result in the browser application being packaged into applications/browser/packaged.

Review checklist

Reminder for reviewers

@paul-marechal
Copy link
Member Author

@vince-fugnitto @marcdumais-work I am seeing errors coming from a few builtins, do you guys know if this is expected?

@paul-marechal
Copy link
Member Author

I was wondering what would be good to do in this PR before merging it in master:

  • Do we want a CI that publishes this new application?
  • Is there any particular utilization case that is currently missing and that should be contributed via Theia extension?
  • Is there any weird bugs with the packaged version that need fixing?

I'd like help to try on different platforms as well. I only tested from Windows 10.

@paul-marechal paul-marechal force-pushed the mp/browser-package branch 2 times, most recently from 78b1294 to d5db3df Compare October 13, 2021 14:45
@marcdumais-work marcdumais-work mentioned this pull request Oct 19, 2021
.gitmodules Outdated Show resolved Hide resolved
@marcdumais-work
Copy link
Contributor

One thing I noticed: the help -> about dialog seems broken.

@paul-marechal paul-marechal force-pushed the mp/browser-package branch 3 times, most recently from faec2b2 to 2b0a931 Compare November 2, 2021 18:21
@marcdumais-work
Copy link
Contributor

Here are the vscode extensions that seem to fail for me at startup (more might when e.g. .a new LS is triggered):

  • vscode.merge-conflict
  • vscode.emmet
  • vscode.npm

More from backend startup log:

root INFO [hosted-plugin: 14507] PLUGIN_HOST(14507): PluginManagerExtImpl/loadPlugin(/opt/ericsson-dev/theia-blueprint/applications/browser/packaged/builtins/vscode.merge-conflict/extension/dist/mergeConflictMain)
root ERROR [hosted-plugin: 14507] Activating extension 'Merge Conflict (built-in)' failed: TypeError: i.id.endsWith is not a function
at Object.loadPlugin (/snapshot/theia-blueprint/applications/browser/bundled/plugin-host.js:1:429)
at /snapshot/theia-blueprint/applications/browser/bundled/921.js:1:166617
at processTicksAndRejections (internal/process/task_queues.js:97:5)
at async l.$activatePlugin (/snapshot/theia-blueprint/applications/browser/bundled/921.js:1:167690)
at async Promise.all (index 0)
at async l.activateBySingleEvent (/snapshot/theia-blueprint/applications/browser/bundled/921.js:1:167592)
at async l.$activateByEvent (/snapshot/theia-blueprint/applications/browser/bundled/921.js:1:167258)
root INFO [hosted-plugin: 14507] PLUGIN_HOST(14507): PluginManagerExtImpl/loadPlugin(/opt/ericsson-dev/theia-blueprint/applications/browser/packaged/builtins/vscode.emmet/extension/dist/node/emmetNodeMain)
root ERROR [hosted-plugin: 14507] Activating extension 'Emmet (built-in)' failed: TypeError: i.id.endsWith is not a function
at Object.loadPlugin (/snapshot/theia-blueprint/applications/browser/bundled/plugin-host.js:1:429)
at /snapshot/theia-blueprint/applications/browser/bundled/921.js:1:166617
at processTicksAndRejections (internal/process/task_queues.js:97:5)
at async l.$activatePlugin (/snapshot/theia-blueprint/applications/browser/bundled/921.js:1:167690)
at async Promise.all (index 1)
at async l.activateBySingleEvent (/snapshot/theia-blueprint/applications/browser/bundled/921.js:1:167592)
at async l.$activateByEvent (/snapshot/theia-blueprint/applications/browser/bundled/921.js:1:167258)
root INFO [hosted-plugin: 14507] PLUGIN_HOST(14507): PluginManagerExtImpl/loadPlugin(/opt/ericsson-dev/theia-blueprint/applications/browser/packaged/builtins/vscode.npm/extension/dist/npmMain)
root ERROR [hosted-plugin: 14507] Activating extension 'NPM support for VS Code (built-in)' failed: TypeError: i.id.endsWith is not a function
at Object.loadPlugin (/snapshot/theia-blueprint/applications/browser/bundled/plugin-host.js:1:429)
at /snapshot/theia-blueprint/applications/browser/bundled/921.js:1:166617
at processTicksAndRejections (internal/process/task_queues.js:97:5)
at async l.$activatePlugin (/snapshot/theia-blueprint/applications/browser/bundled/921.js:1:167690)
at async Promise.all (index 0)
at async l.activateBySingleEvent (/snapshot/theia-blueprint/applications/browser/bundled/921.js:1:167592)
at async l.$activateByEvent (/snapshot/theia-blueprint/applications/browser/bundled/921.js:1:167258)
root INFO [5b44c0a8-6407-43bf-ad2b-8228f1da8e98] Start of 85 plugins took: 731.6 ms [Finished 4.769 s after frontend start]

@marcdumais-work
Copy link
Contributor

When I try to search for a file in the repo (Ctrl-p), it's not able to spawn ripgrep:

root ERROR Failed to search: file:///opt/ericsson-dev/theia-blueprint Error: spawn /opt/ericsson-dev/theia-blueprint/applications/browser/packaged/bin/rg EACCES
at Process.ChildProcess._handle.onexit (internal/child_process.js:268:19)
at onErrorNT (internal/child_process.js:470:16)
at processTicksAndRejections (internal/process/task_queues.js:84:21)

I notice the ripgrep executable does not have execute permission. If I manually add it (chmod +x <rg executable>, now search works.

@marcdumais-work
Copy link
Contributor

I missed a few non-working extensions (or at least extensions that throw exceptions at startup):

  • vscode.git
  • vscode.debug-auto-launch
  • vscode.typescript-language-features
  • vscode.json-language-features/
  • vscode.configuration-editing

@marcdumais-work
Copy link
Contributor

Many seem to have this same error: TypeError: i.id.endsWith is not a function

Couple of example:

root INFO [hosted-plugin: 14507] PLUGIN_HOST(14507): PluginManagerExtImpl/loadPlugin(/opt/ericsson-dev/theia-blueprint/applications/browser/packaged/builtins/vscode.json-language-features/extension/client/dist/node/jsonClientMain)
root ERROR [hosted-plugin: 14507] Activating extension 'JSON Language Features (built-in)' failed: TypeError: i.id.endsWith is not a function
    at Object.loadPlugin (/snapshot/theia-blueprint/applications/browser/bundled/plugin-host.js:1:429)
    at /snapshot/theia-blueprint/applications/browser/bundled/921.js:1:166617
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
    at async l.$activatePlugin (/snapshot/theia-blueprint/applications/browser/bundled/921.js:1:167690)
    at async Promise.all (index 1)
    at async l.activateBySingleEvent (/snapshot/theia-blueprint/applications/browser/bundled/921.js:1:167592)
    at async l.$activateByEvent (/snapshot/theia-blueprint/applications/browser/bundled/921.js:1:167258)
    at async l.$start (/snapshot/theia-blueprint/applications/browser/bundled/921.js:1:165034)
root ERROR [hosted-plugin: 14507] Activating extension 'Node Debug Auto-attach (built-in)' failed: TypeError: i.id.endsWith is not a function
    at Object.loadPlugin (/snapshot/theia-blueprint/applications/browser/bundled/plugin-host.js:1:429)
    at /snapshot/theia-blueprint/applications/browser/bundled/921.js:1:166617
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
    at async l.$activatePlugin (/snapshot/theia-blueprint/applications/browser/bundled/921.js:1:167690)
    at async Promise.all (index 2)
    at async l.activateBySingleEvent (/snapshot/theia-blueprint/applications/browser/bundled/921.js:1:167592)
    at async l.$activateByEvent (/snapshot/theia-blueprint/applications/browser/bundled/921.js:1:167258)
    at async l.$start (/snapshot/theia-blueprint/applications/browser/bundled/921.js:1:164967)

@marcdumais-work
Copy link
Contributor

Many seem to have this same error: TypeError: i.id.endsWith is not a function

seems to come from new code: https://github.com/eclipse-theia/theia-blueprint/pull/168/files#diff-f909d9e80fd47722da8e8ea2c689895ecfe35d5e60adfc9040af4fad1397b9b7R32

@marcdumais-work
Copy link
Contributor

marcdumais-work commented Nov 15, 2021

@paul-marechal
Copy link
Member Author

Many seem to have this same error: TypeError: i.id.endsWith is not a function

seems to come from new code: https://github.com/eclipse-theia/theia-blueprint/pull/168/files#diff-f909d9e80fd47722da8e8ea2c689895ecfe35d5e60adfc9040af4fad1397b9b7R32

I might have a fix for this locally that I did not push yet. This code is a copy/paste of the original implementation with the intent of fixing this i.id.endsWith is not a function error. When bundled using Webpack, module reflection like this doesn't seem to work well anymore. I think it used to, but now doesn't for some reason... I'll push my changes once I get back to it.

@paul-marechal
Copy link
Member Author

paul-marechal commented Nov 15, 2021

OK I figured out this specific error: I had set the node webpack config to development mode locally, for debugging purposes. But when pushing I reverted back to production mode. Turns out in production mode Webpack gives numerical ids to modules, and the code expects strings.

I just fixed this specific issue and rebased.

Now I think the biggest issue is getting TypeScript to run.

@marcdumais-work
Copy link
Contributor

marcdumais-work commented Nov 16, 2021

Thanks for the update Paul. Now it fails for me during yarn browser bundle:

18 warnings have detailed information that is not shown.
Use 'stats.errorDetails: true' resp. '--stats-error-details' to show it.

ERROR in 684.js
684.js from Terser plugin
Unexpected token: punc ()) [684.js:2317,38]
at js_error (/opt/ericsson-dev/theia-blueprint/node_modules/terser/dist/bundle.min.js:550:11)
at croak (/opt/ericsson-dev/theia-blueprint/node_modules/terser/dist/bundle.min.js:1274:9)
at token_error (/opt/ericsson-dev/theia-blueprint/node_modules/terser/dist/bundle.min.js:1282:9)
at unexpected (/opt/ericsson-dev/theia-blueprint/node_modules/terser/dist/bundle.min.js:1288:9)
at semicolon (/opt/ericsson-dev/theia-blueprint/node_modules/terser/dist/bundle.min.js:1326:56)
at simple_statement (/opt/ericsson-dev/theia-blueprint/node_modules/terser/dist/bundle.min.js:1583:73)
at statement (/opt/ericsson-dev/theia-blueprint/node_modules/terser/dist/bundle.min.js:1396:19)
at embed_tokens_wrapper (/opt/ericsson-dev/theia-blueprint/node_modules/terser/dist/bundle.min.js:1339:26)
at block
(/opt/ericsson-dev/theia-blueprint/node_modules/terser/dist/bundle.min.js:2172:20)
at _function_body (/opt/ericsson-dev/theia-blueprint/node_modules/terser/dist/bundle.min.js:2084:21)

@marcdumais-work
Copy link
Contributor

Maybe relevant, I have interesting warnings just before the failure, that I do not remember seeing before. Several similar to this:

WARNING in ../../node_modules/@theia/node-pty/lib/windowsPtyAgent.js 35:35-74
Module not found: Error: Can't resolve '../build/Release/conpty.node' in '/opt/ericsson-dev/theia-blueprint/node_modules/@theia/node-pty/lib'
@ ../../node_modules/@theia/node-pty/lib/windowsTerminal.js 22:24-52
@ ../../node_modules/@theia/node-pty/lib/index.js 10:19-63
@ ../../node_modules/@theia/process/lib/node/terminal-process.js 36:19-45
@ ../../node_modules/@theia/process/lib/node/process-backend-module.js 20:27-56
@ ./src-gen/backend/server.js 48:47-104
@ ./src-gen/backend/main.js 10:21-40

WARNING in ../../node_modules/@theia/node-pty/lib/windowsPtyAgent.js 39:39-76
Module not found: Error: Can't resolve '../build/Debug/conpty.node' in '/opt/ericsson-dev/theia-blueprint/node_modules/@theia/node-pty/lib'
@ ../../node_modules/@theia/node-pty/lib/windowsTerminal.js 22:24-52
@ ../../node_modules/@theia/node-pty/lib/index.js 10:19-63
@ ../../node_modules/@theia/process/lib/node/terminal-process.js 36:19-45
@ ../../node_modules/@theia/process/lib/node/process-backend-module.js 20:27-56
@ ./src-gen/backend/server.js 48:47-104
@ ./src-gen/backend/main.js 10:21-40

@paul-marechal
Copy link
Member Author

paul-marechal commented Nov 18, 2021

Yikes, since the rebase a new version of a source parser dependency used for minification might be bogus :S

edit: They patched it and we now depend on the bug-free library!

@marcdumais-work
Copy link
Contributor

CI is failing and I have no clue why

Locally I get this test failure: "Element with selector "#vsx-extensions:builtin .theia-TreeContainer" not found"

2021-12-13T16:27:38.723Z INFO devtools: COMMAND findElement("css selector", "#vsx-extensions:builtin .theia-TreeContainer")
2021-12-13T16:27:38.726Z INFO devtools: RESULT Error: Element with selector "#vsx-extensions:builtin .theia-TreeContainer" not found
at DevToolsDriver.findElement (/opt/ericsson-dev/theia-blueprint/node_modules/devtools/build/utils.js:78:16)
at processTicksAndRejections (internal/process/task_queues.js:97:5)
at async Browser.wrappedCommand (/opt/ericsson-dev/theia-blueprint/node_modules/devtools/build/devtoolsdriver.js:62:26)
at async Browser.runCommandWithHooks (/opt/ericsson-dev/theia-blueprint/node_modules/@wdio/sync/build/wrapCommand.js:70:25)
2021-12-13T16:27:38.758Z INFO devtools: COMMAND findElements("css selector", "#vsx-extensions:builtin .theia-TreeContainer")

@marcdumais-work
Copy link
Contributor

@paul-marechal the latest yarn.lock is not checked in. Maybe related, on Linux the package does not start for me.

$ ./packaged/blueprint.exe 
[...]
Error: node-loader:
Error: /tmp/pkg/694f1da7fa5cb433d1e9645b71e964d078001515fe7d30a7e0105232de015663/drivelist.node: cannot open shared object file: No such file or directory
    at Object.20840 (/snapshot/theia-blueprint/applications/browser/bundled/249.js:2:2068497)
    at o (/snapshot/theia-blueprint/applications/browser/bundled/blueprint.js:1:10120)
    at e.exports (/snapshot/theia-blueprint/applications/browser/bundled/blueprint.js:1:82)
    at Object.60841 (/snapshot/theia-blueprint/applications/browser/bundled/249.js:2:1363319)
    at o (/snapshot/theia-blueprint/applications/browser/bundled/blueprint.js:1:10120)
    at Object.74102 (/snapshot/theia-blueprint/applications/browser/bundled/249.js:2:105092)
    at o (/snapshot/theia-blueprint/applications/browser/bundled/blueprint.js:1:10120)
    at Object.54406 (/snapshot/theia-blueprint/applications/browser/bundled/249.js:2:106592)
    at o (/snapshot/theia-blueprint/applications/browser/bundled/blueprint.js:1:10120)
    at Object.27846 (/snapshot/theia-blueprint/applications/browser/bundled/249.js:2:92421)

@paul-marechal
Copy link
Member Author

@marcdumais-work the latest issue you are reporting is fixed in a newer version of the pkg fork. That and the outdated yarn.lock makes me think that you have an out of date checkout of the branch, do you mind trying to fetch and force reset?

@marcdumais-work
Copy link
Contributor

Thanks @paul-marechal - I will be sure to -ff it too, when I retry :)

@@ -0,0 +1,120 @@
{
"private": true,
"name": "theia-blueprint-node",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

theia-blueprint-browser? Though a private package, we still see the name when e.g. running yarn why pkg

@marcdumais-work
Copy link
Contributor

Even starting with a fresh clone, I was not able to get the PR to work.

@vince-fugnitto confirmed offline that he has the startup crash too, like above.

@paul-marechal paul-marechal force-pushed the mp/browser-package branch 2 times, most recently from e6a2fb8 to 3f15db3 Compare January 10, 2022 22:42
Copy link
Member

@vince-fugnitto vince-fugnitto left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I confirmed that the build was now successful, and the web application started on Linux 👍


I verified the following:

Successful:

  • scm works well, I can see changes under version control
  • searching works well from file search to search-in-workspace
  • syntax highlighting works
  • renaming works (refactoring)
  • find all references works
  • terminals work, can be spawned and used
  • executing tasks works
  • searching and installing extensions work

Unsuccessful:

root ERROR Error starting the debug session Error: It is not possible to provide debug adapter executable.
    at t.DebugExtImpl.createDebugAdapter (/snapshot/theia-blueprint/applications/browser/bundled/921.js:1:133563)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
    at async t.DebugExtImpl.$createDebugSession (/snapshot/theia-blueprint/applications/browser/bundled/921.js:1:131411)

@paul-marechal
Copy link
Member Author

ETA: Everything but JS debugging seems to be working. We are struggling to find why we can't debug JS. Once this issue is solved it should be good to go.

@paul-marechal paul-marechal force-pushed the mp/browser-package branch 7 times, most recently from 7480e48 to eea2f98 Compare March 1, 2022 01:40
@paul-marechal paul-marechal marked this pull request as ready for review March 1, 2022 02:36
@paul-marechal paul-marechal force-pushed the mp/browser-package branch 2 times, most recently from 81d1188 to 0ddbc47 Compare March 2, 2022 20:56
The goal of this PR is to setup a browser version of Blueprint packaged
as a "drag and drop" that's easy to deploy.

The setup is a bit tricky and might not apply well to every kind of
product made using the Theia framework, but it should work for most
usages.

Add `theia-blueprint-browser` Theia extension that contributes the
required customizations to run in the packaged environment (mostly to
accommodate to the new file layout).

Use a `node`-specific Webpack configuration to bundle the backend along
with the browser-specific customizations.

Use a forked `pkg` package to support forking child processes with
`execArgv` options. The fork is setup as a git submodule.
@paul-marechal
Copy link
Member Author

There was a bug on Windows that froze the backend completely when trying to open a terminal. This is now fixed.

I also got rid of the Git submodule in favor of patch-package.

Node.js 16 is now required to build the browser package.

@paul-marechal
Copy link
Member Author

@jfaltermeier I have been referred to you to get help with building the browser binaries on Jenkins? I have no idea where to start with this, or how to test that the configuration works?

Here are the build steps for the browser package:

yarn
yarn browser bundle
yarn browser package

The output will be the whole contents of applications/browser/packaged/*: it includes the node binary as well as folders for the frontend bundles, builtin plugins and extra binaries (e.g. ripgrep).

@jfaltermeier
Copy link
Contributor

Hi @paul-marechal ,
the starting point would be the Jenkinsfile in the root of the git repository, especially the def buildInstaller() part. There you can add the additional build steps.
When you edit the title of the PR to include jenkins, it should get built on the Eclipse CI as well (https://ci.eclipse.org/theia/job/Theia%20PRs/view/change-requests/).

You could temporarily add an archiveArtifacts step (https://www.jenkins.io/doc/pipeline/steps/core/#archiveartifacts-archive-the-artifacts) to the pipeline to store the results on Jenkins for testing.
Probabily similar to this: archiveArtifacts artifacts: 'applications/browser/packaged/**', fingerprint: false
(from a former testing PR: 7c40012#diff-e6ffa5dc854b843b3ee3c3c28f8eae2f436c2df2b1ca299cca1fa5982e390cf8)

In the long run there are also groovy functions defined for signing and notarizing (on Mac), and finally to upload the results to download.eclipse.org.
Also there is the next/Jenkinsfile, which is a nightly build that consumes the next version of Theia to help us identify any breaking changes early.
Those have to be adjusted as well, in the long run.

But I guess a good first step would be to have the browser package available for testing from Jenkins.

Please let me know if there are any problems

@paul-marechal
Copy link
Member Author

Closing because backend bundling will be taken care of by eclipse-theia/theia#12412

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants