[Bug] new support for Python f-strings breaks with multi-line f-strings

[dylanscott]

Reproducible in vscode.dev or in VS Code Desktop?

• Not reproducible in vscode.dev or VS Code Desktop

Reproducible in the monaco editor playground?

• Not reproducible in the monaco editor playground

Monaco Editor Playground Link

Link

Monaco Editor Playground Code

single_line_f_string f'''nothing {'to see'} here'''

multi_line_f_string = f"""first line looks fine
{'uh oh'}
now the highlighting is broken down here :(
"""

also = "it's broken highlighting for everything after"
four = 2 + 2

Reproduction Steps

1. Configure an editor using the built-in python language
2. Enter a multi-line f-string which has an expression after the 1st line

Actual (Problematic) Behavior

The portion of the f-string after the expression is highlighted as python code instead of a string, and then the closing f-string delimiter causes code after that to be highlighted as if it were a string

Expected Behavior

Self evident

Additional Context

I believe this was added with #4401, which notably does not have multi-line f-strings in its test coverage.

[ivanksh]

It is reproducible for me too

[rubenfiszel]

Reproducible for us too at windmill.dev, it seems a pretty severe bug for the editing experience

[PaGrom]

Affects me too.

[charliemeyer]

This affects us as well, and is a pretty bad bug for UX on our code editor.

[dylanscott]

FWIW I was able to disable the broken syntax highlighting (at the cost of not getting any highlighting within template segments) with a hack like:

import { languages as Languages } from "monaco-editor";
import { language as pythonMonarchLanguage } from "monaco-editor/esm/vs/basic-languages/python/python.js";

// here or on an initialization path:
fixFStringBug(pythonMonarchLanguage);

function fixFStringBug({ tokenizer }: Languages.IMonarchLanguage): void {
  // HACK: unbreak highlighting in the presence of multiline f-strings until
  // https://github.com/microsoft/monaco-editor/issues/4601 is fixed
  const badRules = ["fStringBody", "fDblStringBody", "fStringDetail"];
  for (const rule of badRules) {
    delete tokenizer[rule];
  }

  const badReferences = new Set(badRules.map((rule) => `@${rule}`));
  const isBadRule = (rule: Languages.IMonarchLanguageRule): boolean =>
    Array.isArray(rule) && rule.length === 3 && badReferences.has(rule[2]);
  for (const name of Object.keys(tokenizer)) {
    tokenizer[name] = tokenizer[name].filter((rule) => !isBadRule(rule));
  }
}

You may need to ignore a type error from the 2nd import, or you can add a type stub like

declare module "monaco-editor/esm/vs/basic-languages/python/python.js" {
  export const language: IMonarchLanguage;
}

[samstrohkorbatt]

Proposed solution in PR #4827
Additionally, other f-string usages were added, like lambda function, walrus operator, and more. Details in the PR.