Thoughts .toString()

Astro update broke my code blocks!

astro shiki typescript

Astro integrates a compatibility build of Shiki, but up until at least version 5.1.1, it was actually compatible with the fully featured Shiki v2.0.0. That’s why I was able to highlight my inline code with getSingletonHighlightCore instead of the “compat” createHighlighter. I upgraded to 5.1.7 this morning and all my code blocks broke. I could’ve rolled it back, but I wanted to see what was up.

If some context is needed, this post explains a bit more in depth what I had to highlight inline code.

The Problem

There were two problems. The first is that Astro insisted on using its own version of Shiki types. So every Shiki import became plagued with errors that looked like this.

shell
1Type 'import("j:/Projects/devblog-astro/node_modules/@shikijs/types/dist/index").ShikiTransformer' is not assignable to type 'import("j:/Projects/devblog-astro/node_modules/@astrojs/markdown-remark/node_modules/@shikijs/types/dist/index").ShikiTransformer'.

In this case, I was doing import { ShikiTransformer } from 'shiki';, but Astro really wanted me to use the identical version under @astrojs/markdown-remark that isn’t even exposed/exported.

The second problem was more serious. I got the following error and neither my inline nor blocked code work.

shell
1ShikiError: Failed to parse Markdown file "J:\Projects\devblog-astro\src\content\blog\dev-blog\aws-deploy-series\14-cache-control\14-cache-control.md":
2Must invoke loadWasm first.
3    at new OnigString (file:///J:/Projects/devblog-astro/node_modules/@shikijs/engine-oniguruma/dist/index.mjs:259:19)
4    at Object.createString (file:///J:/Projects/devblog-astro/node_modules/@shikijs/engine-oniguruma/dist/index.mjs:494:20)
5    at Object.createOnigString (file:///J:/Projects/devblog-astro/node_modules/@shikijs/core/dist/index.mjs:1640:39)
6    at Grammar.createOnigString (file:///J:/Projects/devblog-astro/node_modules/@shikijs/vscode-textmate/dist/index.mjs:2272:26)
7    at Grammar._tokenize (file:///J:/Projects/devblog-astro/node_modules/@shikijs/vscode-textmate/dist/index.mjs:2428:31)
8    at Grammar.tokenizeLine2 (file:///J:/Projects/devblog-astro/node_modules/@shikijs/vscode-textmate/dist/index.mjs:2366:20)
9    at _tokenizeWithTheme (file:///J:/Projects/devblog-astro/node_modules/@shikijs/core/dist/index.mjs:863:28)
10    at tokenizeWithTheme (file:///J:/Projects/devblog-astro/node_modules/@shikijs/core/dist/index.mjs:809:18)
11    at codeToTokensBase (file:///J:/Projects/devblog-astro/node_modules/@shikijs/core/dist/index.mjs:785:10)
12    at file:///J:/Projects/devblog-astro/node_modules/@shikijs/core/dist/index.mjs:982:21
1316:25:22 [ERROR] [glob-loader] Error rendering dev-blog/aws-deploy-series/14-cache-control/14-cache-control.md: Failed to parse Markdown file "J:\Projects\devblog-astro\src\content\blog\dev-blog\aws-deploy-series\14-cache-control\14-cache-control.md":
14Must invoke loadWasm first.

I tried filing a bug report with Astro, but their test environment on StackBlitz won’t show the same error message. It just failed silently. After some digging, I figured out that I could get block code to work, but not inline code, by adding this line.

/astro.config.tstypescript
3import { createOniguramaEngine } from 'shiki/engine/onigurama';

skip to line 10

10// https://astro.build/config
11export default defineConfig({
12    site: SITE_URL,
13    integrations: [sitemap(), svelte()],
14    markdown: {
15        shikiConfig: {
languages, themes, transformers, etc.

skip to line 21

21            engine: createOniguramaEngine(import('shiki/wasm'))
22        }
23    }
24});

But it turned out this wasn’t the main issue.

The real problem

I eventually found that if I commented out specifically the lines that called the highlighter outside of Astro integration and did e.g. codeToHast with it, the code would run (without highlighting my inline code of course). To fix it, it was something like this.

/src/markdown/inline-code.tstypescript
1import {

skip to line 8

     getSingletonHighlighterCore 
8} from 'shiki';
9import { createShikiHighlighter } from '@astrojs/markdown-remark';

skip to line 104

104const highlightCode = () => {
     const cachedHighlighter = getSingletonHighlighterCore({
         langs: [resolveLanguage('plaintext')],
         themes: Object.values(shikiThemes).map(resolveTheme)
     });
105    const cachedHighlighter = createShikiHighlighter({
Type 'string' is not assignable to type 'LanguageRegistration'.ts(2322)
106        langs: ['plaintext'],
107        themes: shikiThemes 
108    });
109    return async (tree: Root) => {

skip to line 151

151        const highlighter = await cachedHighlighter;
152        for (const instance in inlineInstances) {
153            const resolvedLanguage = resolveLanguage(instance.language);
154            if (resolvedLanguage) {
                 await highlighter.loadLanguage(
                     resolveLanguage(resolvedLanguage));
155                await h

Uh, what? Astro’s “convenience” function wraps around Shiki’s createHighlighter and doesn’t actually return a highlighter, making it impossible to add languages later. Not to mention it wrapping the parameters in unnecessary types.

astro/packages/markdown/remark/src/shiki.tstypescript
72export async function createShikiHighlighter({
73	langs = [],
74	theme = 'github-dark',
75	themes = {},
76	langAlias = {},
77}: CreateShikiHighlighterOptions = {}): Promise<ShikiHighlighter> {
78	theme = theme === 'css-variables' ? cssVariablesTheme() : theme;
79
80	const highlighter = await createHighlighter({
81		langs: ['plaintext', ...langs],
82		langAlias,
83		themes: Object.values(themes).length ? Object.values(themes) : [theme],
84	});
85
86	async function highlight(

skip to line 188

188	}
189
190	return {
191		codeToHast(code, lang, options = {}) {
192			return highlight(code, lang, options, 'hast') as Promise<Root>;
193		},
194		codeToHtml(code, lang, options = {}) {
195			return highlight(code, lang, options, 'html') as Promise<string>;
196		},
197	};
198}

Astro’s createShikiHighlighter from its Github repo

If that’s the case, I’d rather import { createHighlighter } from 'shiki'; and set up options.meta.__raw myself.

The solution

So we cut out any mention of “astro” from the script and end up with this.

/src/markdown/inline-code.tstypescript
 import { createShikiHighlighter } from '@astrojs/markdown-remark';
9import { createHighlighter } from 'shiki';

skip to line 104

104const highlightCode = () => {
     const cachedHighlighter = createShikiHighlighter({
         langs: ['plaintext'],
         themes: shikiThemes 
     });
105    const cachedHighlighter = createHighlighter({
106        langs: ['plaintext'], 
107        themes: Object.values(shikiThemes).map(
108            (theme: ThemeTypes) => resolveTheme(theme))
109    });
110    return async (tree: Root) => {

skip to line 151

151        const highlighter = await cachedHighlighter;
152        for (const instance in inlineInstances) {
153            const resolvedLanguage = resolveLanguage(instance.language);
154            if (resolvedLanguage) {
Shiki's highlighters have the same methods, so we didn't need to change this after all.
155                await highlighter.loadLanguage(
156                    resolveLanguage(resolvedLanguage));

Meanwhile, Astro is still complaining about the types, so I decided to just swap out its Shiki integration altogether, for both inline and code blocks.

/astro.config.tstypescript
1import type { ShikiConfig, AstroUserConfig } from 'astro';

skip to line 5

5type MarkdownConfig = NonNullable<AstroUserConfig['markdown']>;

skip to line 10

10export const config: Partial<MarkdownConfig> = {
     shikiConfig: {
         themes: shikiThemes,
         defaultColor: false,
         transformers: [transformer]
     },
11    syntaxHighlight: false,
12    rehypePlugins: [highlightCode]
13};

I already had the code to find inline code from HAST code (a tree representation of a document). All I need to do then is to add a branch to detect code blocks.

/src/markdown/inline-code.tstypescript
1import { Element, Text, Root } from 'hast';
2import { visit } from 'unist-util-visit';

skip to line 24

24const isInlineCode = (
25    element: Element,
26    parent: Element | Root | undefined
27): boolean => {
The default render of inline code is <code>Text</code>.
28    return (
29        element.tagName === 'code' &&
30        parent.type === 'element' &&
31        parent.tagName !== 'pre' &&
32        element.children.length === 1 &&
33        element.children[0].type === 'text'
34    );
35};
36const isBlockCode = (
37    element: Element,
38    parent: Element | Root | undefined
39): boolean => {
The default render of a code block is <pre><code class=language-[language]>Text</code></pre>. The <pre> block distinguishes between inline and block code.
40    return (
41        element.tagName === 'pre' &&
42        element.children.length === 1 &&
43        element.children[0].type === 'element' &&
44        element.children[0].tagName === 'code' &&
45        element.children[0].children[0].type === 'text'
46    );
47};

skip to line 100

100interface VisitCallback {
101    node: Element;
102    index?: number;
103    parent?: Element | Root;
104}
105interface CodeInstance {
106    node: Element;
107    code: string;
108    language: string;
109    meta?: string
110}
Inline code are expected to be in the format of Text{:Language}.
111const regexp = /^(.+){:(\w+)}$/;
112const highlightCode = () => {
Create cached highlighter.

skip to line 117

117    return async(tree: Root) => {
118        const inlineInstances: CodeInstance[] = [];
119        const blockInstances: CodeInstance[] = [];
Visit each element in the document in order.
120        visit(tree, 'element', ({node, index, parent}: VisitCallback) => {
121            if (isInlineCode(node, parent)) {
122                const textNode = node.children[0] as Text;
123                const match = textNode.value.match(regexp);
124                if (match) {
125                    const [_, code, language] = match;
126                    inlineInstances.push({ node, code, language });
127                }
128                else {
129                    inlineInstances.push({ node, code, 'plaintext' });
130                }
131            }
132            else if (isBlockCode(node, parent)) {
133                const codeElement = node.children[0] as Element;
134                let meta = codeElement.data?.meta;
135                if (!meta) meta = undefined;
136                const textNode = codeElement.children[0] as Text;
Expect <code class=language-[language]>.
137                const language = (
138                    codeElement.properties['className'] as string
139                    )[0].split('-')[1];
140                blockInstances.push({ node, code, language, meta });
141            }
142        });

skip to line 151

151        const highlighter = await cachedHighlighter;
152        for (const instance of inlineInstances) {
Load languages if valid.

skip to line 163

163            const root = highlighter.codeToHast(instance.code, {
164                lang: instance.language,
165                themes: shikiThemes,
166                defaultColor: false,
167                transformers: [{
168                    // Do some styling
169                }]
170            });
171            const pre = root.children[0] as Element;
172            const code = pre.children[0] as Element;
173            instance.node.properties = {...pre.properties};
174            instance.node.children = code.children;
175        }
176        for (const instance of blockInstances) {
Load languages if valid.

skip to line 185

185            const root = highlighter.codeToHast(instance.code.trimEnd(), {
186                lang: instance.language,
187                themes: shikiThemes,
188                defaultColor: false,
189                transformers: [
190                    {
191                        pre(pre) {
192                            pre.properties['data-language'] = instance.language;
193                        }
194                    },
Import all the code block transformers that used to run on shikiConfig.
195                    transformer(instance.meta)
196                ]
197            });
198            const figure = root.children[0] as Element;
199            instance.node = {...figure, properties: {...figure.properties}};
200        }
201    }
202};
203export default highlightCode;

Finally, somewhere in the beginning of the transformers function, we need to initialize the meta data as it worked in Shiki.

typescript
1const transformer = (metadata: string): ShikiTransformer => {
2    return {
3        preprocess(code, options) {
4            options.meta = options.meta || {};
5            options.meta.__raw = metadata;
... the rest of the transform

skip to line 128

128        }
129    }
130};
131export default transformer;

What’s next?

I think there’s no disadvantage to reducing dependencies. Before with the Astro integrations, I’m at the mercy of both Astro and Shiki. If either of these tools update, I may have to spend a day figuring it out. Now besides issues with Vite, I’m at the mercy of Shiki only. I can breathe easy updating Astro knowing that it likely won’t mess with my blog posts, since the syntax highlighting is the most complicated piece of machinery in there. That trepidation shouldn’t hold me back from potential bug fixes and new features.

One question remains: Is it still possible to not use the shiki/compat createHighlighter? I’ll be on the lookout for that, although this problem might be too involved for me right now. It seems to have something to do with Vite’s interaction with web assembly, and that’s not something I can diagnose easily. I’ll most likely set this aside at some point and move on to the next project.