API to iteratively apply themes

[Gerrit0]

This is admittedly a bit of a stretch, and I might be getting to the point where I should just be using vscode-textmate directly... but... it is awfully convenient to have Shiki's handling of theme and grammar loading.

I would like a way to get unthemed tokens, then ask Shiki to apply a theme to the token to give me styles. Being able to do this would mean that I can tokenize text once when doing highlighting with multiple themes,thereby avoiding the overhead of tokenization for the 2nd-N'th theme. TypeDoc currently has code that looks like this:

function highlight(code: string, lang: string) {
  const lightTokens = this.highlighter.codeToThemedTokens(code, lang, lightTheme, { includeExplanation: false });
  const darkTokens = this.highlighter.codeToThemedTokens(code, lang, darkTheme, { includeExplanation: false });

  // merge tokens together to handle light/dark switches

Which means that I am highlighting all code twice. This isn't a big deal if it's only a couple hundred lines, but with a single 1000 line codeblock, I'm already spending an extra 400ms doing highlighting...

There are a few different possibilities ways the API could look. I'm not very attached to any of them. A way to get a token that has getColor(theme) instead of color would work, or something that just included the metadata so that I could pass it back to the highlighter to get the color.

[naiyerasif]

I am currently writing a custom renderer for Shiki and I stumbled across this usecase. I believe it'd be nice to have some sort of annotation API that can provide rendering information for the tokens instead of coupling them together. Having token annotations separated from actual tokens can also help fleshing out the Decoration API being discussed here #5

[Gerrit0]

I looked at doing this last weekend and unfortunately I'm not sure how possible this is, without essentially re-implementing tokenizeLine2 in vscode-textmate...

[octref]

Planning to add dark mode support, so there's an API for using multiple themes to convert code to HTML. Will that fully solve your use-case?

[antfu]

This new usage should get you covered: https://shiki.style/guide/dual-themes