Asciidoc Integration

[louiss0]

Body

Summary

I want to ask the team to create an official loader for Asciidoc files.
Then integrate other tools for creating Asciidoc files.

Background & Motivation

I recently discovered Asciidoc after watching a YouTube video on it. The video made compelling arguments about why Asciidoc was better than Markdown. I tried it out and I like it. It supports things that most people want without having to download too many things. The problem with Asciidoc is that it isn't supported by any Static Site Generator that I know of.

There are things that the team can provide that I probably won't be able to provide. Like Shiki Support or good configuration.

Goals

• Create a loader for Asciidoc files.
• Inject Astro's Components into Asciidoc files by creating Marcos then making the Asciidoc Processor use them.
• Generate Syntax Highlighting by using Antfu's Shiki.
• Using a config file to extend Asciidoc.

Example

The integration of Asciidoc is one that takes on many different APIs, so I decided to split them off into section's. These examples aren't going to be fully concrete. The code written here should be able to help with general idea.

Loader API

The loader API's job is to first use a function to gather a list of paths to an Asciidoc page. Then the paths are then passed on to the asciidoc processor's load function. The loadFile function will execute by using each path. Each file's information will be extracted; then sent into the store.

The code below is a template. The asciidoc code written is the proper API!

import asciidoc from "asciidoc"

const processor = asciidoc()

function useGlobPatternToGetFilePaths (glob:string) {

  // Gathers file names using the glob.  

	return [
		"posts/post.adoc"
	]

 }


 function asciidocLoader(glob:string) {

   const filePathsFromGlob = useGlobPatternToGetFilePaths(glob)


	return {
		name: "asciidoc-loader",
		load: async ({store, parseData }) => {

	   const paths = filePathsFromGlob.map(
		   (path)=> processor.loadFile(path)
	   ) 

	   for (const path of paths) {

		const document = processor.loadFile(path)

		const attributes = document.getAttributes()

		  parseData(attributes)

		  store.set({
			id: document.getTitle().toLowerCase().replaceAll(/\s+/g, "-"),
		    data:attrbutes,
		    rendered: {
			    metadata:{
			      headings: document.getSections()
				      .map(section => section.getTitle()),
				  
			    },
				html: document.getContent()
		    }
		  })
	   
	   }
		},

	}
   
 }

Asciidoc Marcos

Asciidoc can be extended by using the Extensions API. It's an API that allows the developer to tell Asciidoc how to process certain code. A macro is a syntax that is written like this. element:[] This is Asciidoc's way or describing elements. There are two kinds of macros. A block and inline macro.

The image macro would work like this.

image::img.jpg[alt="Image"]`

The picture macro won't work in Asciidoc. Block macros don't accept blocks as content. But a picture block can be created.

[picture]
----
source::img2.jpg[alt="Image"]`
image::img.jpg[alt="Image"]`
----

I found out that Asciidoc processor function's can't be used asynchronously at all.

Highlighter

A custom highlighter can be created in Asciidoc by using Asciidoc's Syntax Highlighter class. It's supposed to be done like this.

import asciidoctor, {

type Block, 
type Document, 
type SyntaxHighlighterFormatOptions,
type SyntaxHighlighterHighlightOptions,

} from 'asciidoctor'

const processor = asciidoctor()

class ShikiHighlighter extends processor.SyntaxHighlighter {


// This method is responsible for changing the string.
override format(node: Block, lang: string, opts?:SyntaxHighlighterFormatOptions): string {

return "I'm shiki"

}

// This method is for changing the highlight.
highlight(node: Block, source: string, lang: string, opts:SyntaxHighlighterHighlightOptions) {

	return "I'm Shiki"


}

}

Using a Config File to Extend Asciidoc

I don't know how to read file system things, so I decided to represent just an object instead. But A config object can be used to register custom blocks and macros. The config object allows its user to
pass in attributes blocks and macros. The attributes: property takes an object of curated props that are meant to be added to every page. The blocks: prop takes an object that accepts a word as it's key and an object with two properties. The context: and the processor. The context is the type of block that will be created. The processor is the function that will pass content that needs to be rendered.

This is the type for the Asciidoc config.

type AsciidocGlobalVariables = {

'source-highlighter': string

author: string

backend: string

filetype: boolean

localdir: string

localdate: string

localdatetime: string

localtime: string

localyear: number

'attribute-missing': 'drop' | 'drop-line' | 'skip' | 'warn'

'attribute-undefined': 'drop' | 'drop-line'

experimental: boolean

'appendix-caption': string

'appendix-number': string

'appendix-refsig': string

'caution-caption': string

'caution-number': string

'caution-refsig': string

'caution-signifier': string

'example-caption': string

'example-number': string

'figure-caption': string

'figure-number': number

'footnote-number': number

'important-caption': string

'last-update-label': string

'listing-caption': string

'listing-number': number

'note-caption': string

'part-refsig': string

'part-signifier': string

'preface-title': string

'table-caption': string

'table-number': string

'tip-caption': string

'toc-title': string

'untitled-label': string

'warning-caption': string

'app-name': string

authors: string

idprefix: string

idseparator: string

leveloffset: 0 | 1 | 2 | 3 | 4 | 5

partnums: boolean

setanchors: boolean

sectids: boolean

sectlinks: boolean

sectnums: boolean

sectnumlevels: 0 | 1 | 2 | 3 | 4 | 5

'title-separator': string

toc: true | 'auto' | 'left' | 'right' | 'macro' | 'preamble'

toclevels: 1 | 2 | 3 | 4 | 5

fragment: boolean

stylesheet: string

}

  
  

type AsciidocConfig = Partial<{

attributes: Partial<AsciidocGlobalVariables>

blocks: Record<string, {

context: 'example' | 'listing' | 'literal' | 'pass' | 'quote' | 'sidebar'

processor: (attributes: Record<string, NonNullable<unknown>>, reader: Reader) => string

}>

macros: Partial<{

inline: Record<

string,

{

context: 'quoted' | 'anchor'

processor: (target: string, attributes: Record<string, NonNullable<unknown>>) => string

}

  

>

block: Record<string,

{

context: 'example' | 'listing' | 'literal' | 'pass' | 'quote' | 'sidebar'

processor: (target: string, attributes: Record<string, NonNullable<unknown>>) => string

}>

}>

}>

The code at the bottom is the function for using the config file. The name is registerBasedOnConfig It's a function that does the following things.

1. Extract attributes, blocks, inline and block macros.
2. Checks If blocks exists. !f so then it registers blocks by using the properties it contains.
3. Checks If macros.inline exists. If so then it registers inline macros by using its properties,
4. Checks If macros.block exists. If so then registers block macros by using its properties.
5. Then it returns a function that takes in a file name as its parameter. That function will be used to load whatever file is passed to it returning an Asciidoc.Document

I don't know what the config file name should be called, but I think ascii.config.ts is a good name.

function registerBasedOnConfig(config: AsciidocConfig) {

  

const processor = asciidoctor()

  

const registry = processor.Extensions.create()

  

const { attributes, blocks, macros: { block, inline } } = config

  

if (blocks) {

  

for (const [blockName, blockContextAndProcessor] of Object.entries(blocks)) {

  
  

registry.block(blockName, function () {

  
  

this.onContext(blockContextAndProcessor.context)

  

this.process(function (parent, reader, attributes) {

  
  

return this.createBlock(

parent,

blockContextAndProcessor.context,

blockContextAndProcessor.processor(

attributes,

reader

),

attributes

)

  

})

  

})

  

}

}

  
  
  
  

if (inline) {

  
  

for (const [inlineMacroName, inlineMacroContextAndProcessor] of Object.entries(inline)) {

  

registry.inlineMacro(inlineMacroName, function () {

  
  

this.process(function (parent, target, attributes) {

  
  
  

return this.createInline(

parent,

inlineMacroContextAndProcessor.context,

inlineMacroContextAndProcessor.processor(target, attributes),

attributes

)

  

})

  

})

}

  

}

  

if (block) {

  
  

for (const [blockMacroName, blockMacroContextAndProcessor] of Object.entries(block)) {

  

registry.blockMacro(blockMacroName, function () {

  
  

this.process(function (parent, target, attributes) {

  
  

return this.createBlock(

parent,

blockMacroContextAndProcessor.context,

blockMacroContextAndProcessor.processor(target, attributes),

attributes

)

  

})

  

})

}

  
  

}

  
   
return (filename: string) => {

  
  

return processor.loadFile(filename, { attributes, extension_registry: registry })

  

}

  
}

[louiss0]

For more resources you can follow these sites.

Documentation on the Asciidoc Language: https://docs.asciidoctor.org

Documentation on the Asciidoc JavaScript library: https://docs.asciidoctor.org/asciidoctor.js/latest/

[louiss0]

I also set up a repo for you guys to use to play with Asciidoc.
https://github.com/louiss0/asciidoc

All you have to do is fork it then do what ever you need to do with it it's a simple Astro project that uses pnpm as it's package manager, Vitest for test running and it has the asciidoctor library installed for you guys to use. I hope this is enough. If there are any problems just respond to this repo.

[florian-lefebvre]

It's unlikely we'll work on this anytime soon. Feel free to create a community integration for it!

[louiss0]

It's unlikely we'll work on this anytime soon. Feel free to create a community integration for it!

Could you at least get Ben Holmes to help me. He knows this stuff best! I don't know how to deal with file issues! What about Astro Image and picture.

If he could give me some advice, I'd be OK with doing things myself.

[louiss0]

It's unlikely we'll work on this anytime soon. Feel free to create a community integration for it!

Could you at least get Ben Holmes to help me. He knows this stuff best! I don't know how to deal with file issues! What about Astro Image and picture.

If he could give me some advice, I'd be OK with doing things myself.

I don't care about this being done now. I think it's best to start the project. After Astro Containers and the loader API is stable.

[florian-lefebvre]

If he could give me some advice, I'd be OK with doing things myself.

Well we have the discord server for that! You can ask anything you want in #integrations

[louiss0]

If he could give me some advice, I'd be OK with doing things myself.

Well, we have the discord server for that! You can ask anything you want in #integrations

I wish I could. I got kicked out for spam.

Thanks for responding!

[ematipico]

There's nothing stopping you from creating and publishing an integration yourself.

It's unlikely we will have this in core

[louiss0]

There's nothing stopping you from creating and publishing an integration yourself.

It's unlikely we will have this in core

I'm ok with that. I was hoping that you guys would start this project after being done with Astro 5. But since you'd be too busy even after that, I guess I might have to do this myself.

I don't know a good library for loading config files. I'm scared because I don't want to deal with cross-platform OS issues.
Another problem is with the way Windows deals with globs. I need a library that will do away with that problem.

I know you created the glob loader. I just need the library that was used to generate path names from globs.

Please tell me what library is good for creating globs and one for loading configs, and then I can finish this project myself!

[florian-lefebvre]

Glob library we use: https://github.com/withastro/astro/blob/main/packages/astro/src/content/loaders/glob.ts#L3. For configuration loading, you may be interested in https://github.com/unjs/c12

[louiss0]

Glob library we use: https://github.com/withastro/astro/blob/main/packages/astro/src/content/loaders/glob.ts#L3. For configuration loading, you may be interested in https://github.com/unjs/c12

Thank you for the help. This discussion is over. I'm going to take notes then close things up.

[louiss0]

Thanks @florian-lefebvre!

[louiss0]

I'm sorry to bring up this discussion again, but I'm banned from the server. @ascorbic . Can you please help me figure out how to watch for file changes. I need to find a way to make sure that asciidoc files and the content collection config is watched for changes so that people don't have to shutdown the server after every change.

[louiss0]

I'm sorry to bring up this discussion again, but I'm banned from the server. @ascorbic . Can you please help me figure out how to watch for file changes. I need to find a way to make sure that asciidoc files and the content collection config is watched for changes so that people don't have to shutdown the server after every change.

I want to watch for changes in the file where the folder for watching content is supposed to be placed. I ask the user what is the folder name you want to use for writing collections.
I want to watch for changes in that repo then reload the server every time changes happen to files in that folder.

[ascorbic]

You can use the existing glob loader if you create an integration to handle the content type. Take a look at how the ContentEntryType is handled in markdown and markdoc for examples. Essentially you register as a handler for a particular file extension.

[louiss0]

You can use the existing glob loader if you create an integration to handle the content type. Take a look at how the ContentEntryType is handled in markdown and markdoc for examples. Essentially you register as a handler for a particular file extension.

This is confusing for me because. Pages are generated only by the loader that's it. The code I see has some thing to do with processing. Which only the loader is supposed to do? All I want to do is tell vite to reload the server every time a .adoc file changes that's it.

[louiss0]

@ascorbic
So I looked at the code for both and this is what I found from the Markdoc implementation. I tried to look up the docs to see addContentEntryType, but I saw nothing. I don't know what this function does at all? What am I supposed to do with it. The code below is the one I think I need to complete.

{
		name: "",
		hooks: {
			'astro:config:setup': async (params) => {
			
				addContentEntryType({
					extensions: [],
					
					getEntryInfo({ fileUrl, contents }) {

						return {
							data: {},
							body: "",
							slug: "", 
							rawData: {} 
						}
					},
					async getRenderModule() {
				
						return { code: "" }

					}
					
				});

			},
			'astro:server:setup': async ({ server }) => {
				server.watcher.on('all', (_event, entry) => {
					if (SUPPORTED_MARKDOC_CONFIG_FILES.some((f) => entry.endsWith(f))) {
					server.restart();
					}
				});
			},
		},
	}

What do getRenderModule and getEntryInfo do? Am I allowed to return any kind of information from either one of them? I'm guessing that the content is supposed to be returned from getRenderModule. But get getEntryinfo is supposed to return metadata about the content.

First question is do I need the functions? If I do then what am I supposed to return from either functions. I don't plan on manipulating the content myself that is supposed to be done by Asciidoc. @bholmesdev can you help here.

[louiss0]

So I finally figured out what I'm supposed to to with addContentEntryType.
The thing is now that I'm not sure what I'm supposed to return from getEntryInfo

What is rawData: supposed to have the parsed content or the contents.
Is the body: supposed to have the just the parsed content?

[louiss0]

Thanks for your help @ascorbic. The answer to my problem all along was the watcher. Who ever came up with that idea good job.
Things like this are why I love Astro soo much! The loader is pretty much complete all I have to do now is see how it works in the real world. I might introduce Asciidoc soon!