temp test setup: try to migrate the LSP job

[mbien]

this is just a test setup which tries to run the LSP job on github without running anything else.

not intended to be merged, only for debugging.

[mbien]

not sure if it matters for this test. But maybe inline the $OPTS here. Since I forgot a '-' where it is declared which causes yaml to add a newline at the end of the list.

  OPTS: >-
    -Dmetabuild.jsonurl=https://raw.githubusercontent.com/apache/netbeans-jenkins-lib/master/meta/netbeansrelease.json
    -Dtest-unit-sys-prop.ignore.random.failures=true

would be correct (note the - after >)

it is fixed in the original PR, forgot to add it here.

[sdedic]

This is really weird. I think I've found the culprit, but WHY the thing happens is not (yet) obvious. The $JSCallback$ is generated during ant build, but **is deleted during ant testbefore thejunit` task launches:

2022-11-14T19:26:54.2848252Z 
2022-11-14T19:26:54.2848365Z -do-compile:
2022-11-14T19:26:54.2848719Z     [mkdir] Skipping /home/runner/work/netbeans/netbeans/java/java.lsp.server/build/classes because it already exists.
2022-11-14T19:26:54.8304004Z    [depend] Deleting file /home/runner/work/netbeans/netbeans/java/java.lsp.server/build/classes/org/netbeans/modules/java/lsp/server/refactoring/MoveRefactoring$JavaDoc.class since org.netbeans.modules.java.lsp.server.refactoring.MoveRefactoring out of date
2022-11-14T19:26:54.8308302Z    [depend] Top level class = org.netbeans.modules.java.lsp.server.refactoring.MoveRefactoring
2022-11-14T19:26:54.8311636Z    [depend] Deleting file /home/runner/work/netbeans/netbeans/java/java.lsp.server/build/classes/org/netbeans/modules/java/lsp/server/refactoring/MoveRefactoring.class since one of its inner classes was removed
2022-11-14T19:26:54.8315332Z    [depend] Deleting file /home/runner/work/netbeans/netbeans/java/java.lsp.server/build/classes/org/netbeans/modules/java/lsp/server/refactoring/MoveRefactoring$MoveElementControl.class since org.netbeans.modules.java.lsp.server.refactoring.MoveRefactoring out of date
2022-11-14T19:26:54.8318743Z    [depend] Top level class = org.netbeans.modules.java.lsp.server.refactoring.MoveRefactoring
2022-11-14T19:26:54.8321940Z    [depend] Deleting file /home/runner/work/netbeans/netbeans/java/java.lsp.server/build/classes/org/netbeans/modules/java/lsp/server/refactoring/MoveRefactoring$ElementControl.class since org.netbeans.modules.java.lsp.server.refactoring.MoveRefactoring out of date
2022-11-14T19:26:54.8325250Z    [depend] Top level class = org.netbeans.modules.java.lsp.server.refactoring.MoveRefactoring
2022-11-14T19:26:54.8328737Z    [depend] Deleting file /home/runner/work/netbeans/netbeans/java/java.lsp.server/build/classes/org/netbeans/modules/java/lsp/server/refactoring/MoveRefactoring$MoveElementControl$1.class since org.netbeans.modules.java.lsp.server.refactoring.MoveRefactoring out of date
2022-11-14T19:26:54.8332064Z    [depend] Top level class = org.netbeans.modules.java.lsp.server.refactoring.MoveRefactoring

[...]
2022-11-14T19:26:54.8739460Z  [nb-javac] org/netbeans/modules/java/lsp/server/db/DBAddConnection$1.class will be deleted since DBAddConnection.class is missing
2022-11-14T19:26:54.8743656Z  [nb-javac] org/netbeans/modules/java/lsp/server/debugging/launch/NbLaunchDelegate$1.class will be deleted since NbLaunchDelegate.class is missing
2022-11-14T19:26:54.8747259Z  [nb-javac] org/netbeans/modules/java/lsp/server/debugging/launch/NbLaunchDelegate$1CheckBuildProgress.class will be deleted since NbLaunchDelegate.class is missing
2022-11-14T19:26:54.8750713Z  [nb-javac] org/netbeans/modules/java/lsp/server/debugging/launch/NbLaunchDelegate$1W.class will be deleted since NbLaunchDelegate.class is missing
2022-11-14T19:26:54.8754045Z  [nb-javac] org/netbeans/modules/java/lsp/server/debugging/launch/NbLaunchDelegate$2.class will be deleted since NbLaunchDelegate.class is missing
2022-11-14T19:26:54.8757368Z  [nb-javac] org/netbeans/modules/java/lsp/server/debugging/launch/NbLaunchDelegate$3$1.class will be deleted since NbLaunchDelegate.class is missing
2022-11-14T19:26:54.8760671Z  [nb-javac] org/netbeans/modules/java/lsp/server/debugging/launch/NbLaunchDelegate$3.class will be deleted since NbLaunchDelegate.class is missing
2022-11-14T19:26:54.8763959Z  [nb-javac] org/netbeans/modules/java/lsp/server/debugging/launch/NbLaunchDelegate$4.class will be deleted since NbLaunchDelegate.class is missing
2022-11-14T19:26:54.8767812Z  [nb-javac] org/netbeans/modules/java/lsp/server/explorer/TreeItem$CollapsibleState.class will be deleted since TreeItem.class is missing
2022-11-14T19:26:54.8771151Z  [nb-javac] org/netbeans/modules/java/lsp/server/explorer/TreeItem$IconDescriptor.class will be deleted since TreeItem.class is missing
2022-11-14T19:26:54.8774811Z  [nb-javac] org/netbeans/modules/java/lsp/server/htmlui/$JsCallbacks$$1.class will be deleted since .class is missing
2022-11-14T19:26:54.8778046Z  [nb-javac] org/netbeans/modules/java/lsp/server/htmlui/$JsCallbacks$.class will be deleted since .class is missing
2022-11-14T19:26:54.8781266Z  [nb-javac] org/netbeans/modules/java/lsp/server/htmlui/WebView$1.class will be deleted since WebView.class is missing
2022-11-14T19:26:54.8784542Z  [nb-javac] org/netbeans/modules/java/lsp/server/htmlui/WebView$Command$1Wrap.class will be deleted since WebView.class is missing
2022-11-14T19:26:54.8787760Z  [nb-javac] org/netbeans/modules/java/lsp/server/htmlui/WebView$Command.class will be deleted since WebView.class is missing
2022-11-14T19:26:54.8790991Z  [nb-javac] org/netbeans/modules/java/lsp/server/htmlui/WebView$Page.class will be deleted since WebView.class is missing

But WHY has been MoveRefactoring invalidated I do not understand. I've logged timestamps of .class files and the source .java files before ant test is run -- and seems the timestamps are correct (classes newer than sources).

And from the nbjavac messages, I'd say that

2022-11-14T19:26:54.8778046Z  [nb-javac] org/netbeans/modules/java/lsp/server/htmlui/$JsCallbacks$.class will be deleted since .class is missing

is also strange: as if the nbjavac would consider "" as a toplevel class name.

[sdedic]

clearly touching any of the source after ant build on the java.lsp.server module and before ant test triggers the behaviour.

[mbien]

would it help to upload the workspace for inspection and then diff (e.g with meld) the build before and after test? Maybe there is more wrong which could give clues what is going on there.

      - name: Prepare Artifact
        run: tar -I 'zstd -9 -T0' -cf /tmp/ws.tar.zst --exclude ".git" .

      - name: Upload Workspace
        uses: actions/upload-artifact@v3
        with:
          name: ws
          path: /tmp/ws.tar.zst
          retention-days: 5
          if-no-files-found: error

[sdedic]

would it help to upload the workspace for inspection and then diff (e.g with meld) the build before and after test? Maybe there is more wrong which could give clues what is going on there.

Interesting -- how would I fetch that artifact ?

[sdedic]

So - cause found. This is a combination of 3 participants:

1. a bug in github action setup-java@v3. The action seems to unpack the JDK installation with improper timestamps: instead of leaving the file stats as they are stored in the archive, the unpacked contents is timestamped with the date/time of the action execution. If an application links to a jar inside a JDK, such as tools.jar for many NB modules, various up-to-date checks might fail, since the referenced JAR's time will be that of the JDK installation, not the time of the JAR's creation.

2. the CustomJavac task used in nbbuild attempts to clean debris after depend ANT task - the depend task does not handle inner classes well, so CustomJavac cleans up inner classes that depend lefts behind, based on the enclosing class' presence. Because of a bug it mishandles classes whose $ mistakes them for innerclasses. Naturally no toplevel class '' ever existed :)

3. The sequence of steps in the workflow is:
   a. install JDK 11
   b. build
   c. install JDK 8
   d. test
   At the last step, tools.jar from JDK8 appears more recent than compiled classes from step (b), the netbeans build fails up-to-date check and a module (not test, but production code) is rebuilt, partially.

As JDK8 is itypically not reinstalled each time on developer's machine - and if it was, it would be through some conventional unzip or tar that preserves file timestamps, so the up-to-date check always succeeded on local machine(s) and the recompilation step was never done in a local environment.

Then, depend task because of (again) failed up-to-date check deleted some classes that (in)directly used tools.jar types; typically refactoring classes. After that, because of the bug (2), nb-javac deleted $JsCallback$ class - that was never generated again.

The tests were finally run, but the required $JsCallback* classes were gone already.

[sdedic]

Relevant parts from the log:

Cause of the actual test failure:

2022-11-15T11:54:18.9353555Z     [junit] Error during dispatch at server 
2022-11-15T11:54:18.9353858Z     [junit] java.util.concurrent.CompletionException: java.lang.NoClassDefFoundError: org/netbeans/modules/java/lsp/server/htmlui/$JsCallbacks$
2022-11-15T11:54:18.9354114Z     [junit] 	at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)
2022-11-15T11:54:18.9354372Z     [junit] 	at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280)
2022-11-15T11:54:18.9354596Z     [junit] 	at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1606)
2022-11-15T11:54:18.9354796Z     [junit] 	at org.openide.util.RequestProcessor$Task.run(RequestProcessor.java:1418)
2022-11-15T11:54:18.9355031Z     [junit] 	at org.netbeans.modules.openide.util.GlobalLookup.execute(GlobalLookup.java:45)
2022-11-15T11:54:18.9355235Z     [junit] 	at org.openide.util.lookup.Lookups.executeWith(Lookups.java:278)
2022-11-15T11:54:18.9355443Z     [junit] 	at org.openide.util.RequestProcessor$Processor.run(RequestProcessor.java:2033)

• will add more logging to both Server.java and ServerTest.java to display these in the future

Files present at the end of the compilation:

2022-11-15T11:50:59.9605425Z java/java.lsp.server/build/classes/org/netbeans/modules/java/lsp/server/htmlui:
2022-11-15T11:50:59.9605683Z total 64
2022-11-15T11:50:59.9606013Z -rw-r--r-- 1 runner docker 1802 2022-11-15 11:49:39.123093473 $JsCallbacks$$1.class
2022-11-15T11:50:59.9606474Z -rw-r--r-- 1 runner docker 4391 2022-11-15 11:49:39.127093433 $JsCallbacks$.class
2022-11-15T11:50:59.9606882Z -rw-r--r-- 1 runner docker 4497 2022-11-15 11:49:39.163093069 Buttons.class
2022-11-15T11:50:59.9607278Z -rw-r--r-- 1 runner docker  257 2022-11-15 11:49:38.707097682 WebView$1.class
2022-11-15T11:50:59.9607701Z -rw-r--r-- 1 runner docker 1702 2022-11-15 11:49:38.707097682 WebView$Command$1Wrap.class
2022-11-15T11:50:59.9608116Z -rw-r--r-- 1 runner docker 9974 2022-11-15 11:49:38.707097682 WebView$Command.class
2022-11-15T11:50:59.9608525Z -rw-r--r-- 1 runner docker 1592 2022-11-15 11:49:38.707097682 WebView$Page.class
2022-11-15T11:50:59.9608936Z -rw-r--r-- 1 runner docker 4261 2022-11-15 11:49:38.707097682 WebView$Server.class
2022-11-15T11:50:59.9609335Z -rw-r--r-- 1 runner docker 8391 2022-11-15 11:49:38.707097682 WebView.class
2022-11-15T11:50:59.9609511Z 

Files present at the time of test execution:

2022-11-15T11:54:18.9352721Z     [junit] File list in classes: WebView$Command.class
2022-11-15T11:54:18.9352816Z     [junit] WebView$1.class
2022-11-15T11:54:18.9352924Z     [junit] WebView$Command$1Wrap.class
2022-11-15T11:54:18.9353014Z     [junit] WebView$Server.class
2022-11-15T11:54:18.9353106Z     [junit] WebView.class
2022-11-15T11:54:18.9353200Z     [junit] WebView$Page.class
2022-11-15T11:54:18.9353286Z     [junit] Buttons.class
2022-11-15T11:54:18.9353432Z     [junit] File list in generated classes: $JsCallbacks$.java

There's no $JsCallback$.class in the classes directory

Why the uptodate task thinks the module should be recompiled:

2022-11-15T11:51:05.1806399Z    [depend] Class org.netbeans.modules.java.lsp.server.refactoring.PullUpRefactoring$1 is out of date with respect to /opt/hostedtoolcache/Java_Zulu_jdk/8.0.352-8/x64/lib/tools.jar

Because of

2022-11-15T11:50:59.9453309Z JAVA_HOME/lib contents:
2022-11-15T11:50:59.9468383Z total 38272
2022-11-15T11:50:59.9468880Z drwxr-xr-x+ 3 runner docker     4096 Nov 15 11:50 amd64
2022-11-15T11:50:59.9469371Z -rw-r--r--+ 1 runner docker 17898406 Nov 15 11:50 ct.sym
2022-11-15T11:50:59.9469725Z -rw-r--r--+ 1 runner docker   163083 Nov 15 11:50 dt.jar
2022-11-15T11:50:59.9470064Z -rw-r--r--+ 1 runner docker    19429 Nov 15 11:50 ir.idl
2022-11-15T11:50:59.9470418Z -rw-r--r--+ 1 runner docker   407704 Nov 15 11:50 jconsole.jar
2022-11-15T11:50:59.9470780Z -rwxr-xr-x+ 1 runner docker    23224 Nov 15 11:50 jexec
2022-11-15T11:50:59.9471135Z -rw-r--r--+ 1 runner docker     1637 Nov 15 11:50 orb.idl
2022-11-15T11:50:59.9471496Z -rw-r--r--+ 1 runner docker  2312771 Nov 15 11:50 sa-jdi.jar
2022-11-15T11:50:59.9471865Z -rw-r--r--+ 1 runner docker 18348414 Nov 15 11:50 tools.jar
2022-11-15T11:50:59.9472383Z nb module jar:
2022-11-15T11:50:59.9484419Z -rw-r--r-- 1 runner docker 1963066 Nov 15 11:49 nbbuild/netbeans/java/modules/org-netbeans-modules-java-lsp-server.jar

Why the $JsCallback$ is actually deleted during compilation step:

2022-11-15T11:51:05.2022221Z  [nb-javac] org/netbeans/modules/java/lsp/server/htmlui/$JsCallbacks$$1.class will be deleted since .class is missing

I can fix this bug.

[sdedic]

I've force-pushed clean up changes + fixes.

[mbien]

would it help to upload the workspace for inspection and then diff (e.g with meld) the build before and after test? Maybe there is more wrong which could give clues what is going on there.

Interesting -- how would I fetch that artifact ?

scroll down in the summary page of the workflow run (which didn't set ci:no-build). It lists all artifacts. The regular workflow already uploads a shared artifact after the JDK 11 build which is a tar of the whole workspace. The cleanup job however removes that again if the build is green, if it isn't green it is kept. This allows restarts of failed jobs in the middle of the pipeline without starting from scratch. (and inspection of the workspace too)

1. a bug in github action setup-java@v3. The action seems to unpack the JDK installation with improper timestamps: instead of leaving the file stats as they are stored in the archive, the unpacked contents is timestamped with the date/time of the action execution. If an application links to a jar inside a JDK, such as tools.jar for many NB modules, various up-to-date checks might fail, since the referenced JAR's time will be that of the JDK installation, not the time of the JAR's creation.

interesting! We could setup all JDKs first, then use their env vars, since they all should set different vars. However, the regular workflow would build in a primary job which produces the tar artifact as described above, then this test would run in a secondary job on a different JDK. This means each job would only setup one JDK. So this situation of setting up two JDKs in the same job should not occur outside of the job we did setup for this debugging PR here. (but we have other places which do that - so this is good to know)

Also keep in mind that this is the first thing I tried. I simply added the LSP job to the existing pipeline. Interesting that the job intended for debugging purposes added another problem to the mix :)

2. the CustomJavac task used in nbbuild attempts to clean debris after depend ANT task - the depend task does not handle inner classes well, so CustomJavac cleans up inner classes that depend lefts behind, based on the enclosing class' presence.

i was afraid that something like this could happen. All the hooks and wedges added a lot of non standard behavior over the years.

I would suggest the following: lets extract all java changes from this PR into a fresh PR and get this PR in. Once its in, I update #4817 so that it is using a regular secondary job for the LSP testing - just like everything else.

would this be ok?

amazing work @sdedic, thanks a lot!