feat: Added the ability to add custom themes/languages to Shiki #2518

JuanM04 · 2022-02-01T19:05:35Z

Changes

Added wrap to Astro Markdown shiki config
Added a way to import custom themes and languages to Shiki (both for <Code /> and Markdown plugin)

Follow up to #2497

Testing

yarn test:match "Shiki|Code"

Docs

changeset-bot · 2022-02-01T19:05:37Z

🦋 Changeset detected

Latest commit: 321a468

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 2 packages

Name	Type
astro	Patch
@astrojs/markdown-remark	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

netlify · 2022-02-01T19:10:09Z

✔️ Deploy Preview for astro-docs-2 ready!

🔨 Explore the source changes: ffbe290

🔍 Inspect the deploy log: https://app.netlify.com/sites/astro-docs-2/deploys/61fc0fd6fa3f020008a82611

😎 Browse the preview: https://deploy-preview-2518--astro-docs-2.netlify.app/en/guides/markdown-content

JuanM04 · 2022-02-01T21:36:53Z

It seems like there are some timeout errors that aren't related to this PR. The difference between the latest two commits is in the indentation, and in the first commit windows fails and in the second macos fails

matthewp · 2022-02-02T15:40:19Z

@JuanM04 the OSX test can be a little flakey, so I've rerun it.

retronav

The changesets have Shiki probably misspelled as "Shika"

JuanM04 · 2022-02-02T16:57:58Z

@obnoxiousnerd thanks!

@matthewp In the latest re-run that you've made, Windows threw this error:

 1) Astro Markdown Shiki
       Themes
         Custom theme
           "before all" hook for "Markdown file":
     Error: EPERM: operation not permitted, mkdir 'D:\a\astro\astro\packages\astro\test\fixtures\astro-markdown-shiki\themes\dist\astro'
      at Object.mkdirSync (node:fs:1334:3)
      at mkdirpath (D:\a\astro\astro\node_modules\rollup\dist\shared\rollup.js:21723:8)
      at D:\a\astro\astro\node_modules\rollup\dist\shared\rollup.js:21727:9
      at new Promise (<anonymous>)
      at writeFile (D:\a\astro\astro\node_modules\rollup\dist\shared\rollup.js:21726:12)
      at writeOutputFile (D:\a\astro\astro\node_modules\rollup\dist\shared\rollup.js:23708:25)
      at D:\a\astro\astro\node_modules\rollup\dist\shared\rollup.js:23632:65
      at Array.map (<anonymous>)
      at handleGenerateWrite (D:\a\astro\astro\node_modules\rollup\dist\shared\rollup.js:23632:52)
      at runMicrotasks (<anonymous>)

Edit: ~~I found the error. Windows fails if you put fixtures in other place that isn't fixtures/. For example, fixtures/markdown/themes needs to be fixtures/markdown-themes~~

Edit 2: no, it was a race condition.

matthewp · 2022-02-02T22:30:38Z

Nice! Thanks @JuanM04

FredKSchott · 2022-02-03T05:31:42Z

examples/with-markdown-shiki/astro.config.mjs

@@ -6,6 +6,9 @@
 // helpful tooltips, and warnings if your exported object is invalid.
 // You can disable this by removing "@ts-check" and `@type` comments below.
 import astroRemark from '@astrojs/markdown-remark';
+import fs from 'fs';
+
+const riGrammar = JSON.parse(fs.readFileSync('./src/shiki/ri.tmLanguage.json'));


this feels like a really advanced case, I feel like most users just want to change the theme, and not add a custom lang. I'd recommend removing all changes from this example.

If you wanted to mention something about this, you could maybe instead link to the docs with something like:

{ syntaxHighlight: 'shiki', shikiTheme: 'dracula', + // full documentation: ASTRO_DOCS_URL

Yeah, that's a good appreciation

I've made the changes. What do you think about them?

FredKSchott · 2022-02-03T05:33:35Z

packages/astro/components/Code.astro

@@ -43,7 +50,10 @@ function repairShikiTheme(html: string): string {
 }

 const highlighter = await shiki.getHighlighter({ theme });
-const _html = highlighter.codeToHtml(code, { lang });
+if(typeof lang !== 'string') await highlighter.loadLanguage(lang)


is it okay if this runs on every codeblock render? If not, can you do a check that the language hasn't already been loaded?

The language is loaded to the highlighter instance. Because the instance is being generated on every render, so does the loadLanguage function. I'll see if I can optimize it a bit more.

Edit: I've found a faster way to load only one language

packages/markdown/remark/src/types.ts

FredKSchott · 2022-02-03T20:00:27Z

packages/astro/components/Code.astro

+const highlighter = await shiki.getHighlighter({ 
+	theme,
+	// Load custom lang if passed an object, otherwise load the default
+	langs: typeof lang !== 'string' ? [lang] : undefined


FredKSchott · 2022-02-03T20:03:45Z

packages/markdown/remark/src/remark-shiki.ts

@@ -12,6 +44,13 @@ const remarkShiki = async (theme: shiki.Theme) => {
 			html = html.replace('<pre class="shiki"', '<pre class="astro-code"');
 			// Replace "shiki" css variable naming with "astro".
 			html = html.replace(/style="(background-)?color: var\(--shiki-/g, 'style="$1color: var(--astro-code-');
+			// Handle code wrapping
+			// if wrap=null, do nothing.
+			if (wrap === false) {


I would really like to not mess with the default style by default, unless we consider them broken. I would expect shiki's default output without changes to be fine / not overflow. Is that not the case / do we consider the default shiki output broken?

Honestly, I don't know. I've just mimicked <Code /> behavior

Right, but we don't currently add overflow-x in <Code> today, right? That's what I'm wondering about, since I hadn't heard this complaint / that a fix for this was needed and I generally like us to match default behaviors when we can.

Right, but we don't currently add overflow-x in <Code> today, right?

Actually, yes. I double checked with a brand new app.

That's what I'm wondering about, since I hadn't heard this complaint

I'm as confused as you. So, I did a little research (thanks Linus for making git blame) and I found this comment: #1208 (comment)

lol I did this??? Welp, nevermind then 🤦 I'm suprised that I didn't file this as a bug of Shiki then, maybe its trapped in one of my other issues on that repo.

Either way, thanks for digging! this LGTM then

FredKSchott

LGTM

FredKSchott · 2022-02-04T20:02:11Z

uh oh, tests failing!

JuanM04 · 2022-02-04T20:04:42Z

@FredKSchott I've been fighting Windows tests for so long, and I don't know why some times they fail and some times they pass

FredKSchott · 2022-02-04T22:19:03Z

hmm, race condition? Did a recent commit introduce or has this always been happening?

I'm about to spin up my own PR, will verify if its an issue on main

JuanM04 · 2022-02-04T23:25:46Z

@FredKSchott I think it's because I do multiple builds with the same folder. But this isn't the only test that does that. Is there a way to force mocha/chai to run a suit synchronously?

Also, it never happened on Linux nor Mac (although Mac sometimes throw an error in "before all" hook in "{root}", but I've seen it in other PRs)

JuanM04 · 2022-02-05T03:03:52Z

In fact, was a race problem. @FredKSchott, could you re-run the test another time just to be sure?

Btw, I've read somewhere that Vitest lets you control concurrency in a test-by-test basis

jonathantneal · 2022-02-07T15:21:43Z

Hey @JuanM04. Like I’m mentioning in your other PR, we recently migrated our docs site over to https://github.com/withastro/docs

I have another PR to remove our old version of the docs from this monorepo, #2517.

I can make sure your work gets added into that repository. Would you be interested in making that PR? You don’t have to, but I would like you to get the commit credit. I can also attribute the improvement to you, but it would not be the same as a signed commit from your machine.

FredKSchott · 2022-02-07T16:30:43Z

LGTM!

…astro#2518) * Replaced `shikiTheme` with `shikiConfig` * Code.astro now accepts custom themes/langs * Updated docs * Updated tests * Fixed language loading * Added customization examples * Updated documentation * Added more tests * Changelogs * Changed some spaces to tabs * Fixed typo in changesets * Moved tests fixtures * Rolled back changes to with-markdown-shiki * Removed lang example in docs * Optimized Code component * Try to fix windows errors * Try to see if this new tests work

JuanM04 added 8 commits February 1, 2022 11:02

Replaced shikiTheme with shikiConfig

d295815

Code.astro now accepts custom themes/langs

f703715

Updated docs

5874e38

Updated tests

273fbeb

Fixed language loading

948de06

Added customization examples

415e403

Updated documentation

27be244

Added more tests

030cb64

github-actions bot added pkg: astro Related to the core `astro` package (scope) docs pkg: example Related to an example package (scope) feat: markdown Related to Markdown (scope) labels Feb 1, 2022

JuanM04 added 2 commits February 1, 2022 16:13

Changelogs

d2677f0

Changed some spaces to tabs

5ab34b8

retronav suggested changes Feb 2, 2022

View reviewed changes

JuanM04 added 2 commits February 2, 2022 13:54

Merge branch 'main' into shiki-config

0ded936

Fixed typo in changesets

d318359

Moved tests fixtures

63972e4

FredKSchott reviewed Feb 3, 2022

View reviewed changes

packages/markdown/remark/src/types.ts Show resolved Hide resolved

Rolled back changes to with-markdown-shiki

993e988

JuanM04 added 4 commits February 3, 2022 12:48

Removed lang example in docs

ef1873e

Merge branch 'main' into shiki-config

ffbe290

Optimized Code component

034f974

Try to fix windows errors

bb1fa19

FredKSchott reviewed Feb 3, 2022

View reviewed changes

FredKSchott approved these changes Feb 3, 2022

View reviewed changes

Merge branch 'main' into shiki-config

af8e6c8

Try to see if this new tests work

321a468

FredKSchott merged commit 2bc9154 into withastro:main Feb 7, 2022

github-actions bot mentioned this pull request Feb 7, 2022

[ci] release (next) #2533

Merged

JuanM04 deleted the shiki-config branch February 7, 2022 16:34

github-actions bot mentioned this pull request Feb 18, 2022

[ci] release #2619

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Added the ability to add custom themes/languages to Shiki #2518

feat: Added the ability to add custom themes/languages to Shiki #2518

JuanM04 commented Feb 1, 2022 •

edited

Loading

changeset-bot bot commented Feb 1, 2022 •

edited

Loading

netlify bot commented Feb 1, 2022 •

edited

Loading

JuanM04 commented Feb 1, 2022 •

edited

Loading

matthewp commented Feb 2, 2022

retronav left a comment

JuanM04 commented Feb 2, 2022 •

edited

Loading

matthewp commented Feb 2, 2022

FredKSchott Feb 3, 2022 •

edited

Loading

JuanM04 Feb 3, 2022

JuanM04 Feb 3, 2022

FredKSchott Feb 3, 2022

JuanM04 Feb 3, 2022 •

edited

Loading

FredKSchott Feb 3, 2022

FredKSchott Feb 3, 2022

JuanM04 Feb 3, 2022

FredKSchott Feb 3, 2022

JuanM04 Feb 3, 2022 •

edited

Loading

FredKSchott Feb 3, 2022

JuanM04 Feb 3, 2022

FredKSchott left a comment

FredKSchott commented Feb 4, 2022

JuanM04 commented Feb 4, 2022

FredKSchott commented Feb 4, 2022

JuanM04 commented Feb 4, 2022 •

edited

Loading

JuanM04 commented Feb 5, 2022

jonathantneal commented Feb 7, 2022

FredKSchott commented Feb 7, 2022

feat: Added the ability to add custom themes/languages to Shiki #2518

feat: Added the ability to add custom themes/languages to Shiki #2518

Conversation

JuanM04 commented Feb 1, 2022 • edited Loading

Changes

Testing

Docs

changeset-bot bot commented Feb 1, 2022 • edited Loading

🦋 Changeset detected

netlify bot commented Feb 1, 2022 • edited Loading

JuanM04 commented Feb 1, 2022 • edited Loading

matthewp commented Feb 2, 2022

retronav left a comment

Choose a reason for hiding this comment

JuanM04 commented Feb 2, 2022 • edited Loading

matthewp commented Feb 2, 2022

FredKSchott Feb 3, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JuanM04 Feb 3, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JuanM04 Feb 3, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

FredKSchott left a comment

Choose a reason for hiding this comment

FredKSchott commented Feb 4, 2022

JuanM04 commented Feb 4, 2022

FredKSchott commented Feb 4, 2022

JuanM04 commented Feb 4, 2022 • edited Loading

JuanM04 commented Feb 5, 2022

jonathantneal commented Feb 7, 2022

FredKSchott commented Feb 7, 2022

JuanM04 commented Feb 1, 2022 •

edited

Loading

changeset-bot bot commented Feb 1, 2022 •

edited

Loading

netlify bot commented Feb 1, 2022 •

edited

Loading

JuanM04 commented Feb 1, 2022 •

edited

Loading

JuanM04 commented Feb 2, 2022 •

edited

Loading

FredKSchott Feb 3, 2022 •

edited

Loading

JuanM04 Feb 3, 2022 •

edited

Loading

JuanM04 Feb 3, 2022 •

edited

Loading

JuanM04 commented Feb 4, 2022 •

edited

Loading