Fleet-compatible generation for Grammar-Kit (#359)

* [fleet] Fixing test compilation

* [fleet] Adjusting Parser generation for Fleet

* [fleet]Passing generate for Fleet as an option + modifying parse() function

* [fleet]Making tests for Fleet

* [fleet]Moving Fleet-related generation logic outside of general generator

* [fleet]Fix import generation for Fleet

* [fleet]Adding generation action for Fleet

* [fleet]Refactor ParserGenerator

* [fleet]Separate tests for fleet generator

* [fleet]Lexer generator action for Fleet

* [fleet]Improving import directives in output files

* [fleet]Polishing imports in generated files

* [fleet]Polishing tests for fleet

* [fleet]Adding import setting support for lexer generation

* [fleet]forcing main to generate Fleet files

* [fleet]fix accidental typo

* [fleet]Moving generated files to proper folder

* [fleet]fix lexer generation location

* [fleet] adjust type holder generation to not generate factory methods

* [fleet] Generate additional fleet-specific files

* [fleet] Adjust testing data

* [fleet] Polishing outputs

* [fleet] Polishing for the review

* [fleet] Clarification on Main class usage

* [fleet] adjusting GenOptions + including adjustPackagesForFleet

* [fleet] Renamed fleet-related actions

* [fleet] Cleaning-up test

* [fleet] Removed IFileType generation from the BNF grammar

* [fleet] Created "Run Fleet-Compatible JFlex Generator"

* [fleet] adding filetype generation to the main function parameters + adjusting tests for the new API

* [fleet] idea API version update

* [fleet] fix lexer imports not adjusting to fleet

* Revert "[fleet] idea API version update"

This reverts commit 129503f.

* Revert "[fleet] Fixing test compilation"

This reverts commit f511e14.

* [fleet] configuration files rollback + removal of unused resource

* [fleet] removing unintentional whitespaces

* [fleet] hiding fleet-related items for fleet-unrelated projects + changing string prompts

* [fleet] reverting "hiding fleet-related items for fleet-unrelated projects"

* [fleet] fixing BnfRunFleetJFlexAction adding "fleet." to "fleet."

* [fleet] updating CHANGELOG.md

* [fleet] Extract basic generator functionality to make a separate IFileType generator for fleet

* [fleet] remove unused getAllPossibleAttributeValues

* [fleet] remove unused GenOptions parameter from FleetFileTypeGenerator constructor

* [fleet] implement FleetBnfFileImpl that adjusts class names for Fleet

* [fleet] extract adjustPackagesForFleet into a separate attribute for easier access from outside of GenOptions

* [fleet] fix tests and test data

* [fleet] add TOKEN_TYPE_FACTORY default value override

* [fleet] improve naming and streamline logic in FleetBnfFileWrapper

* [fleet] restructure FleetConstants

* [fleet] refactor ParserGenerator so that ParserGenerator extends GeneratorBase

* [fleet] fix hasAttributeValue logic

* [fleet] make ExpressionGeneratorHelper reference constant set, not BnfConstants

* [fleet] fix parser generation action

* [fleet] adjust test data

* [fleet] fix Main after refactoring

* [fleet] refactor contxt creation for generate lexer action

* [fleet] move adjustPackages attribute inside generate attribute

* [fleet] adjust test data for testExprParser

* [fleet] adjust code style

* [fleet] Move RuleInfo + NameShortener to ParserGenerator

* [fleet] Move RuleInfo + NameShortener to ParserGenerator

* [fleet] rename ParserConstantSet to IntelliJPlatformConstants

* [fleet] cleanup GenerateAction

* [fleet] move fleet-related classes to flat "fleet" package

* [fleet] move fleet-related classes to flat "fleet" package

* [fleet] adding spaces

* [fleet] simplifying BnfRunJFlexAction

* [fleet] remove json.bnf from test data

* [fleet] reformat test + inline myFile

* [fleet] remove adjustPackagesForFleet GenOption

* [fleet] rename ExprParser test and add FleetExternalRules test

* [fleet] replace FleetBnfFileWrapper as PsiFile for BNF for Fleet-related generation

* [fleet] refactor FleetBnfFileWrapper creation

* [fleet] rename file to minimize diff with master

* [fleet] add explanation for forceCachedPsi(wrapped) call in FleetBnfFileWrapper.wrapBnfFile

---------

Co-authored-by: Boris Krylov <boris.krylov@jetbrains.com>

CHANGELOG.md

@@ -3,6 +3,9 @@
 ## [Unreleased]
 * JFlex: update library from jflex-1.9.1 to jflex-1.9.2 #349
 * Compatibility: IntelliJ IDEA 2024.2
+* Context menu: Fleet: Generate Parser Code
+* Context menu: Fleet: Generate JFlex Lexer
+* Context menu: Fleet: Run JFlex Generator
 
 ## [2022.3.2]
 * JFlex: complete classes in java references

resources/META-INF/MANIFEST.MF

@@ -1,8 +1,10 @@
 Manifest-Version: 1.0
 Class-Path: light-psi-all.jar
-  lib/3rd-party-rt.jar
-  lib/platform-api.jar
-  lib/platform-impl.jar
   lib/util.jar
+  lib/util-8.jar
+  lib/util_rt.jar
+  lib/app-client.jar
+  lib/lib-client.jar
+  lib/opentelemetry.jar
 Main-Class: org.intellij.grammar.Main
 

resources/META-INF/plugin-java.xml

@@ -22,6 +22,10 @@
       </action>
       <action id="grammar.Generate.ParserUtil" class="org.intellij.grammar.actions.BnfGenerateParserUtilAction"/>
       <action id="grammar.Generate.JFlexLexer" class="org.intellij.grammar.actions.BnfGenerateLexerAction"/>
+      <separator/>
+      <action id="grammar.fleet.Generate" class="org.intellij.grammar.fleet.GenerateFleetAction"/>
+      <action id="grammar.fleet.Generate.JFlexLexer" class="org.intellij.grammar.fleet.BnfGenerateFleetLexerAction"/>
+      <action id="grammar.fleet.Run.JFlex" class="org.intellij.grammar.fleet.BnfRunFleetJFlexAction"/>
       <add-to-group group-id="grammar.file.group" anchor="first"/>
     </group>
   </actions>

resources/messages/GrammarKitBundle.properties

@@ -14,6 +14,9 @@ action.grammar.Generate.JFlexLexer.text=Generate JFlex Lexer
 action.grammar.Generate.ParserUtil.text=Generate Parser Util
 action.grammar.Run.JFlex.text=Run JFlex Generator
 action.grammar.Generate.text=Generate Parser Code
+action.grammar.fleet.Generate.text=Fleet: Generate Parser Code
+action.grammar.fleet.Generate.JFlexLexer.text=Fleet: Generate JFlex Lexer
+action.grammar.fleet.Run.JFlex.text=Fleet: Run JFlex Generator
 
 intention.category.bnf=Grammar-Kit BNF
 intention.flip.arguments.text=Flip arguments

resources/templates/fleet.lexer.flex.template

@@ -0,0 +1,47 @@
+package $packageName;
+
+import fleet.com.intellij.lexer.FlexLexer;
+import fleet.com.intellij.psi.tree.IElementType;
+
+import static fleet.com.intellij.psi.TokenType.BAD_CHARACTER;
+import static fleet.com.intellij.psi.TokenType.WHITE_SPACE;
+import static $typesClass.*;
+
+%%
+
+%{
+  public $lexerClass() {
+    this((java.io.Reader)null);
+  }
+%}
+
+%public
+%class $lexerClass
+%implements FlexLexer
+%function advance
+%type IElementType
+%unicode
+
+EOL=\R
+WHITE_SPACE=\s+
+
+#foreach( $token in $regexpTokens.keySet() )
+$token=$regexpTokens.get($token)
+#end
+
+#macro(spaces $len) #set( $count = $maxTokenLength - $len ) $StringUtil.repeat(" ", $count) #end
+%%
+<YYINITIAL> {
+  {WHITE_SPACE} #spaces(11) { return WHITE_SPACE; }
+
+#foreach( $token in $simpleTokens.keySet() )
+  "$simpleTokens.get($token)" #spaces($simpleTokens.get($token).length()) { return ${tokenPrefix}$token; }
+#end
+
+#foreach( $token in $regexpTokens.keySet() )
+  {$token} #spaces($token.length()) { return ${tokenPrefix}$token; }
+#end
+
+}
+
+[^] { return BAD_CHARACTER; }

src/org/intellij/grammar/LightPsi.java

@@ -128,6 +128,11 @@ private static boolean shouldAddEntry(String path) {
            path.contains("platform-api.jar") ||
            path.contains("platform-impl.jar") ||
            path.contains("util.jar") ||
+           path.contains("util-8.jar") ||
+           path.contains("util_rt.jar") ||
+           path.contains("app-client.jar") ||
+           path.contains("lib-client.jar") ||
+           path.contains("opentelemetry.jar") ||
            path.contains("extensions.jar");
   }
 

src/org/intellij/grammar/Main.java

@@ -10,6 +10,8 @@
 import com.intellij.psi.PsiFile;
 import com.intellij.psi.impl.DebugUtil;
 import org.intellij.grammar.generator.ParserGenerator;
+import org.intellij.grammar.fleet.FleetBnfFileWrapper;
+import org.intellij.grammar.fleet.FleetFileTypeGenerator;
 import org.intellij.grammar.psi.BnfFile;
 
 import java.io.File;
@@ -18,16 +20,16 @@
 /**
  * Command-line interface to parser generator.
  * Required community jars on classpath:
- * jdom.jar, trove4j.jar, extensions.jar, picocontainer.jar, junit.jar, idea.jar, openapi.jar, util.jar.
+ * app-client.jar, lib-client.jar, opentelemetry.jar, util.jar, util-8.jar, util_rt.jar
  *
  * @author gregsh
- *
  * @noinspection UseOfSystemOutOrSystemErr
  */
 public class Main {
   public static void main(String[] args) {
     if (args.length < 2) {
-      System.out.println("Usage: Main <output-dir> <grammars or patterns>");
+      System.out.println(
+        "Usage: Main <output-dir> <grammar-or-pattern 1> [--fleet] [--generateFileTypeElement --className=<fqn> --debugName=<debugName> --languageClass=<fqn>] [ ... <grammar-or-pattern n> [--fleet] [--generateFileTypeElement...]]");
       return;
     }
     File output = new File(args[0]);
@@ -43,6 +45,12 @@ public static void main(String[] args) {
       BnfParserDefinition parserDefinition = new BnfParserDefinition();
 
       for (int i = 1; i < args.length; i++) {
+        boolean generateForFleet = false;
+        boolean generateFileTypeElement = false;
+        String className = "";
+        String languageClass = "";
+        String debugName = "FILE";
+
         String grammar = args[i];
         int idx = grammar.lastIndexOf(File.separator);
         File grammarDir = new File(idx >= 0 ? grammar.substring(0, idx) : ".");
@@ -53,30 +61,99 @@ public static void main(String[] args) {
           return;
         }
 
+        while (i + 1 < args.length && (args[i + 1].startsWith("--fleet") || args[i + 1].startsWith("--generateFileTypeElement"))) {
+          i++;
+          var arg = args[i];
+          if (arg.equals("--fleet")) {
+            generateForFleet = true;
+          }
+          if (arg.startsWith("--generateFileTypeElement")) {
+            var hasClassName = false;
+            var hasLanguageClass = false;
+            while (i + 1 < args.length &&
+                   (args[i + 1].startsWith("--className") ||
+                    args[i + 1].startsWith("--debugName") ||
+                    args[i + 1].startsWith("--languageClass"))) {
+              i++;
+              var argInner = args[i];
+              if (argInner.startsWith("--className")) {
+                String[] keyValuePair = argInner.split("=");
+                if (keyValuePair.length == 2) {
+                  className = keyValuePair[1];
+                  hasClassName = true;
+                }
+                else {
+                  System.out.println("Error parsing parameters: " + argInner);
+                  return;
+                }
+              }
+              if (argInner.startsWith("--languageClass")) {
+                String[] keyValuePair = argInner.split("=");
+                if (keyValuePair.length == 2) {
+                  languageClass = keyValuePair[1];
+                  hasLanguageClass = true;
+                }
+                else {
+                  System.out.println("Error parsing parameters: " + argInner);
+                  return;
+                }
+              }
+              if (argInner.startsWith("--debugName")) {
+                String[] keyValuePair = argInner.split("=");
+                if (keyValuePair.length == 2) {
+                  debugName = keyValuePair[1];
+                }
+                else {
+                  System.out.println("Error parsing parameters: " + argInner);
+                  return;
+                }
+              }
+            }
+
+            if (!hasClassName) {
+              System.out.println("Error parsing parameters: --className missing");
+              return;
+            }
+            if (!hasLanguageClass) {
+              System.out.println("Error parsing parameters: --languageClass missing");
+              return;
+            }
+            generateFileTypeElement = true;
+          }
+        }
         File[] files = grammarDir.listFiles();
         int count = 0;
         if (files != null) {
           for (File file : files) {
             if (file.isDirectory() || !grammarPattern.matcher(file.getName()).matches()) continue;
-            PsiFile bnfFile = LightPsi.parseFile(file, parserDefinition);
-            if (!(bnfFile instanceof BnfFile)) continue;
+            PsiFile psiFile = LightPsi.parseFile(file, parserDefinition);
+            if (!(psiFile instanceof BnfFile)) continue;
 
             // for light-psi-all building:
             if (args[0].contains("lightpsi")) {
               Class.forName("org.jetbrains.annotations.NotNull");
               Class.forName("org.jetbrains.annotations.Nullable");
               Class.forName("org.intellij.lang.annotations.Pattern");
               Class.forName("org.intellij.lang.annotations.RegExp");
-              DebugUtil.psiToString(bnfFile, false);
+              DebugUtil.psiToString(psiFile, false);
+            }
+            count++;
+
+            BnfFile bnfFile = (generateForFleet) ? FleetBnfFileWrapper.wrapBnfFile((BnfFile)psiFile) : (BnfFile)psiFile;
+            new ParserGenerator(bnfFile, grammarDir.getAbsolutePath(), output.getAbsolutePath(), "").generate();
+            if (generateFileTypeElement) {
+              new FleetFileTypeGenerator((BnfFile)psiFile,
+                                         grammarDir.getAbsolutePath(),
+                                         output.getAbsolutePath(),
+                                         "",
+                                         className, debugName, languageClass).generate();
             }
 
-            count ++;
-            new ParserGenerator((BnfFile) bnfFile, grammarDir.getAbsolutePath(), output.getAbsolutePath(), "").generate();
             System.out.println(file.getName() + " parser generated to " + output.getCanonicalPath());
           }
         }
         if (count == 0) {
-          System.out.println("No grammars matching '"+wildCard+"' found in: "+ grammarDir);
+          System.out.println("No grammars matching '" + wildCard + "' found in: " + grammarDir);
         }
       }
     }

src/org/intellij/grammar/actions/BnfGenerateLexerAction.java

@@ -61,6 +61,13 @@
  * @author greg
  */
 public class BnfGenerateLexerAction extends AnAction {
+
+  private static final String LEXER_FLEX_TEMPLATE = "/templates/lexer.flex.template";
+
+  protected String getLexerFlexTemplate() {
+    return LEXER_FLEX_TEMPLATE;
+  }
+
   @Override
   public @NotNull ActionUpdateThread getActionUpdateThread() {
     return ActionUpdateThread.BGT;
@@ -172,22 +179,31 @@ public void visitElement(@NotNull PsiElement element) {
     }
     catch (Throwable ignore) {}
     ve.init();
-
-    VelocityContext context = new VelocityContext();
+
+    VelocityContext context = makeContext(bnfFile, packageName, simpleTokens, regexpTokens, maxLen);
+
+    StringWriter out = new StringWriter();
+    InputStream stream = getClass().getResourceAsStream(getLexerFlexTemplate());
+    ve.evaluate(context, out, "lexer.flex.template", new InputStreamReader(stream));
+    return StringUtil.convertLineSeparators(out.toString());
+  }
+
+  protected VelocityContext makeContext(BnfFile bnfFile,
+                                             @Nullable String packageName,
+                                             Map<String, String> simpleTokens,
+                                             Map<String, String> regexpTokens,
+                                             int[] maxLen) {
+    var context = new VelocityContext();
     context.put("lexerClass", getLexerName(bnfFile));
-    context.put("packageName", StringUtil.notNullize(packageName, StringUtil.getPackageName(getRootAttribute(bnfFile, KnownAttribute.PARSER_CLASS))));
+    context.put("packageName",
+                StringUtil.notNullize(packageName, StringUtil.getPackageName(getRootAttribute(bnfFile, KnownAttribute.PARSER_CLASS))));
     context.put("tokenPrefix", getRootAttribute(bnfFile, KnownAttribute.ELEMENT_TYPE_PREFIX));
     context.put("typesClass", getRootAttribute(bnfFile, KnownAttribute.ELEMENT_TYPE_HOLDER_CLASS));
-    context.put("tokenPrefix", getRootAttribute(bnfFile, KnownAttribute.ELEMENT_TYPE_PREFIX));
     context.put("simpleTokens", simpleTokens);
     context.put("regexpTokens", regexpTokens);
     context.put("StringUtil", StringUtil.class);
     context.put("maxTokenLength", maxLen[0]);
-
-    StringWriter out = new StringWriter();
-    InputStream stream = getClass().getResourceAsStream("/templates/lexer.flex.template");
-    ve.evaluate(context, out, "lexer.flex.template", new InputStreamReader(stream));
-    return StringUtil.convertLineSeparators(out.toString());
+    return context;
   }
 
   public static @NotNull String token2JFlex(@NotNull String tokenText) {
@@ -243,7 +259,7 @@ static String getFlexFileName(BnfFile bnfFile) {
     return getLexerName(bnfFile) + ".flex";
   }
 
-  private static String getLexerName(BnfFile bnfFile) {
+  protected static String getLexerName(BnfFile bnfFile) {
     return "_" + BnfGenerateParserUtilAction.getGrammarName(bnfFile) + "Lexer";
   }
 

src/org/intellij/grammar/actions/BnfRunJFlexAction.java

@@ -58,7 +58,7 @@
 import java.io.File;
 import java.io.IOException;
 import java.util.ArrayList;
-import java.util.Iterator;
+import java.util.Collection;
 import java.util.List;
 import java.util.regex.Matcher;
 import java.util.regex.Pattern;
@@ -91,7 +91,7 @@ public void update(@NotNull AnActionEvent e) {
     e.getPresentation().setEnabledAndVisible(project != null && !files.isEmpty());
   }
 
-  private static List<VirtualFile> getFiles(@NotNull AnActionEvent e) {
+  protected static List<VirtualFile> getFiles(@NotNull AnActionEvent e) {
     return JBIterable.of(e.getData(CommonDataKeys.VIRTUAL_FILE_ARRAY)).filter(file -> {
       FileType fileType = file.getFileType();
       return fileType == JFlexFileType.INSTANCE ||
@@ -115,21 +115,20 @@ public void actionPerformed(@NotNull AnActionEvent e) {
       return;
     }
     String batchId = "jflex@" + System.nanoTime();
-    new Runnable() {
-      final Iterator<VirtualFile> it = files.iterator();
-      @Override
-      public void run() {
-        if (it.hasNext()) {
-          doGenerate(project, it.next(), flexFiles, batchId).doWhenProcessed(this);
-        }
-      }
-    }.run();
+    doGenerate(project, files, flexFiles, batchId);
+  }
+
+  protected void doGenerate(@NotNull Project project,
+                                      Collection<VirtualFile> flexFiles,
+                                      @NotNull Couple<File> jflex,
+                                      @NotNull String batchId){
+    flexFiles.forEach(file -> doGenerateInner(project, file, jflex, batchId));
   }
 
-  public static ActionCallback doGenerate(@NotNull Project project,
-                                          @NotNull VirtualFile flexFile,
-                                          @NotNull Couple<File> jflex,
-                                          @NotNull String batchId) {
+  protected static ActionCallback doGenerateInner(@NotNull Project project,
+                                      @NotNull VirtualFile flexFile,
+                                      @NotNull Couple<File> jflex,
+                                      @NotNull String batchId) {
     FileDocumentManager fileDocumentManager = FileDocumentManager.getInstance();
     Document document = fileDocumentManager.getDocument(flexFile);
     if (document == null) return ActionCallback.REJECTED;
@@ -252,7 +251,7 @@ private static void attachAndActivate(@NotNull Project project,
     }
   }
 
-  private static @Nullable Couple<File> getOrDownload(@NotNull Project project) {
+  protected static @Nullable Couple<File> getOrDownload(@NotNull Project project) {
     LibraryTable libraryTable = LibraryTablesRegistrar.getInstance().getLibraryTable();
     for (Library library : libraryTable.getLibraries()) {
       Couple<File> result = findInUrls(library.getUrls(OrderRootType.CLASSES));