Generate identical atn simulation output across targets to compare parsing functionality

[parrt]

Per #3814 trying to debug ATN sim. For ref, previous thoughts: #3817

[parrt]

@kaby76 @ericvergnaud @KvanTTT getting somewhere on this. Added some basic infrastructure and a command line tool.

[parrt]

Run this so we get right jars before trying this script:

cd ANTLR-ROOT-DIR
mvn install -DskipTests=true
cd runtime-tests 
mvn install -DskipTests=true # yes do it again

Run the script from runtime-tests dir with

../scripts/traceatn.sh /tmp/JSON.g4 json -target Go /tmp/foo.json

or whatever your test grammar, start rule, target, test input are.

Here is traceatn.sh:

export ANTLRJAR=/Users/parrt/.m2/repository/org/antlr/antlr4/4.11.2-SNAPSHOT/antlr4-4.11.2-SNAPSHOT-complete.jar
export TESTJAR=/Users/parrt/.m2/repository/org/antlr/antlr4-runtime-testsuite/4.11.2-SNAPSHOT/antlr4-runtime-testsuite-4.11.2-SNAPSHOT-tests.jar
export JUPITER=/Users/parrt/.m2/repository/org/junit/jupiter/junit-jupiter-api/5.9.0/junit-jupiter-api-5.9.0.jar
export OPENTEST=/Users/parrt/.m2/repository/org/opentest4j/opentest4j/1.2.0/opentest4j-1.2.0.jar
java -classpath $ANTLRJAR:$TESTJAR:$JUPITER:$OPENTEST org.antlr.v4.test.runtime.TraceATN $@

[kaby76]

Besides the hardwired user info in traceatn.sh, ':' path separators, it works fine on Windows for Java, but get no output for anything else (I have all targets installed).

[kaby76]

$ cat ../scripts/traceatn.sh
# Run this so we get right jars before trying this script:
#  cd ANTLR-ROOT-DIR
#  mvn install -DskipTests=true
#  cd runtime-tests
#  mvn install jar:test-jar -DskipTests=true
#
# Run script with
#
#  traceatn.sh /tmp/JSON.g4 json /tmp/foo.json

USERPROFILE=c:/Users/Kenne
export ANTLRJAR=$USERPROFILE/.m2/repository/org/antlr/antlr4/4.11.2-SNAPSHOT/antlr4-4.11.2-SNAPSHOT-complete.jar
export TESTJAR=$USERPROFILE/.m2/repository/org/antlr/antlr4-runtime-testsuite/4.11.2-SNAPSHOT/antlr4-runtime-testsuite-4.11.2-SNAPSHOT-tests.jar
export JUPITER=$USERPROFILE/.m2/repository/org/junit/jupiter/junit-jupiter-api/5.9.0/junit-jupiter-api-5.9.0.jar
export OPENTEST=$USERPROFILE/.m2/repository/org/opentest4j/opentest4j/1.2.0/opentest4j-1.2.0.jar
java -classpath $ANTLRJAR\;$TESTJAR\;$JUPITER\;$OPENTEST org.antlr.v4.test.runtime.TraceATN $@

Kenne@DESKTOP-DL44R7B MINGW64 ~/issue-3814/antlr4/runtime-testsuite

$ bash ../scripts/traceatn.sh ~/big-index/grammars-v4/abnf/Abnf.g4 rulelist -target CSharp ~/big-index/grammars-v4/abnf/examples/postal.abnf


Kenne@DESKTOP-DL44R7B MINGW64 ~/issue-3814/antlr4/runtime-testsuite

Note: dotnet is the "go to" build tool for CSharp. MSBuild is not. In fact, I call "dotnet msbuild" if I want that explicitly. "dotnet" has tool packages that can be easily installed and works without any environment variable changes. That's why "Trash" package the apps as dotnet tools.

All the other target tools are on PATH.

For Cpp:

$ bash ../scripts/traceatn.sh ~/big-index/grammars-v4/abnf/Abnf.g4 rulelist -target Cpp ~/big-index/grammars-v4/abnf/examples/postal.
abnf
java.lang.Exception: command "MSBuild antlr4cpp-vs2022.vcxproj /p:configuration=Release DLL /p:platform=x64"
  in C:\msys64\home\Kenne\issue-3814\antlr4\runtime\Cpp\runtime failed:
  can't build c++ ANTLR runtime using MSBuild
        at org.antlr.v4.test.runtime.RuntimeRunner.runCommand(RuntimeRunner.java:334)
        at org.antlr.v4.test.runtime.cpp.CppRunner.initRuntime(CppRunner.java:107)
        at org.antlr.v4.test.runtime.RuntimeRunner.initAntlrRuntimeIfRequired(RuntimeRunner.java:272)
        at org.antlr.v4.test.runtime.RuntimeRunner.run(RuntimeRunner.java:181)
        at org.antlr.v4.test.runtime.TraceATN.test(TraceATN.java:164)
        at org.antlr.v4.test.runtime.TraceATN.execParse(TraceATN.java:217)
        at org.antlr.v4.test.runtime.TraceATN.main(TraceATN.java:230)
Caused by: java.io.IOException: Cannot run program "MSBuild" (in directory "C:\msys64\home\Kenne\issue-3814\antlr4\runtime\Cpp\runtime"): CreateProcess error=2, The system cannot find the file specified
        at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1143)
        at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1073)
        at org.antlr.v4.test.runtime.Processor.start(Processor.java:52)
        at org.antlr.v4.test.runtime.Processor.run(Processor.java:29)
        at org.antlr.v4.test.runtime.RuntimeRunner.runCommand(RuntimeRunner.java:327)
        ... 6 more
Caused by: java.io.IOException: CreateProcess error=2, The system cannot find the file specified
        at java.base/java.lang.ProcessImpl.create(Native Method)
        at java.base/java.lang.ProcessImpl.<init>(ProcessImpl.java:494)
        at java.base/java.lang.ProcessImpl.start(ProcessImpl.java:159)
        at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1110)
        ... 10 more
State: Compile; java.lang.Exception: C++ ANTLR runtime is not initialized

[parrt]

Note: dotnet is the "go to" build tool for CSharp. MSBuild is not. In fact, I call "dotnet msbuild" if I want that explicitly. "dotnet" has tool packages that can be easily installed and works without any environment variable changes.

Yeah, that must be something that's baked into the test rig. If you turn on WATCH_COMMANDS_EXEC in RuntimeRunner, you'll see the sequence of command during testing. Happy to replace as you suggest.

[parrt]

Ok, and now C# looks like Java! @ericvergnaud 19344b8

[parrt]

@marcospassos Could you update PHP target to reflect one of this commits so PHP ATN simulation trace output is same so we can compare parsing behavior? Not required but could be useful to you. Found issues with Go and JS for example :)

Here is C# commit: 19344b8

[parrt]

I can't get Swift to build so I can't update that target. I'm out of gas on this so I'll merge if tests pass, leaving Swift/PHP to update if somebody wants to help.

[marcospassos]

@marcospassos Could you update PHP target to reflect one of this commits so PHP ATN simulation trace output is same so we can compare parsing behavior? Not required but could be useful to you. Found issues with Go and JS for example :)

Here is C# commit: 19344b8

Sure!

[kaby76]

Trying to use this in comparing CSharp vs PHP.

In my CSharp driver, I set parser.Trace = true;. Works great. Output goes to stdout, one line at a time per entry.

$ ./bin/Debug/net6.0/Test.exe ../examples/access.aql
enter   arangodb_query, LT(1)=RETURN
enter   data_query, LT(1)=RETURN
enter   data_access_query, LT(1)=RETURN
enter   return_expr, LT(1)=RETURN
consume [@0,0:5='RETURN',<9>,1:0] rule return_expr
enter   expr, LT(1)=1
enter   literal, LT(1)=1
enter   numeric_literal, LT(1)=1
consume [@2,7:7='1',<86>,1:7] rule numeric_literal
exit    numeric_literal, LT(1)=<EOF>
exit    literal, LT(1)=<EOF>
exit    expr, LT(1)=<EOF>
exit    return_expr, LT(1)=<EOF>
exit    data_access_query, LT(1)=<EOF>
exit    data_query, LT(1)=<EOF>
consume [@4,9:8='<EOF>',<-1>,2:0] rule arangodb_query
exit    arangodb_query, LT(1)=<EOF>
CSharp 0 ../examples/access.aql success 0.0505942
Total Time: 0.0921106

In my PHP driver, I set $parser.setTrace(true);. The trace output for PHP has zero new lines separating each trace entry. Not sure where it goes to, but that's moot.

$ php -d memory_limit=2G Test.php -file ../examples/access.aql
enter   arangodb_query, LT(1)=RETURNenter   data_query, LT(1)=RETURNenter   data_access_query, LT(1)=RETURNenter   return_expr, LT(1)=RETURNconsume [@0,0:5='RETURN',<9>,1:0] rule return_exprenter   expr, LT(1)=1enter   literal, LT(1)=1enter   numeric_literal, LT(1)=1consume [@2,7:7='1',<86>,1:7] rule numeric_literalexit    numeric_literal, LT(1)=<EOF>exit    literal, LT(1)=<EOF>exit    expr, LT(1)=<EOF>exit    return_expr, LT(1)=<EOF>exit    data_access_query, LT(1)=<EOF>exit    data_query, LT(1)=<EOF>consume [@4,9:8='<EOF>',<-1>,2:0] rule arangodb_queryexit    arangodb_query, LT(1)=<EOF>

I would expect the trace functionality to have the same semantics. Basically the PHP trace is unusable at this moment in dev branch of https://github.com/antlr/antlr-php-runtime. I can sed -E the output, but doing so isn't right.

[kaby76]

Well, it seems that the "trace" that is activated by parser.Trace = true; and $parser.setTrace(true); settings is actually for a parser visitor tracer. CSharp:

antlr4/runtime/CSharp/src/Parser.cs

Line 30 in 76fa05c

public virtual void EnterEveryRule(ParserRuleContext ctx)

Go: https://github.com/antlr/antlr-php-runtime/blob/ac71eb377c816c886b674a7ef805b54ca881d96c/src/ParserTraceListener.php#L20.

This code is pretty old, and has nothing to do with this PR. But it looks like nobody tested the PHP target. echo in PHP is a write without a newline character. Will submit an Issue in the PHP runtime repo.