Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prevent loading the full Markdown package when converting cached markdown fragments #487

Closed
Witiko opened this issue Aug 22, 2024 · 1 comment · Fixed by #491
Closed
Assignees
Labels
lua Related to the Lua interface and implementation speed Related to speed improvements
Milestone

Comments

@Witiko
Copy link
Owner

Witiko commented Aug 22, 2024

In pdfTeX, long documents with many short markdown fragments present a speed challenge, since every fragment loads the whole Markdown package with its requirements and PEG parsers. The cache implemented by the option eagerCache improves the situation by preventing the repeated parsing of the same fragments on subsequent runs. However, the package is still loaded.

We should separate the file markdown.lua into a new file markdown-parser.lua, which would contain most of the current implementation, and a smaller file markdown.lua, which would only contain a skeleton method M.new(). This method would lazily load the Markdown package only when there is a conversion attempt that cannot be satisfied by the cache.

Adding an extra file would make the manual installation somewhat more error-prone and would slightly increase the chance of version mismatch between files. However, it would provide us with some extra breathing room when it comes to speed.

@Witiko Witiko added lua Related to the Lua interface and implementation speed Related to speed improvements labels Aug 22, 2024
@Witiko Witiko added this to the 3.7.1 milestone Aug 22, 2024
@Witiko Witiko self-assigned this Aug 22, 2024
@Witiko
Copy link
Owner Author

Witiko commented Aug 23, 2024

Interestingly, this change would also allow us to use other parsers instead of markdown-parser.lua. I don't have any clear use cases but using a Gruber-style parser from Markdown 2 instead of the CommonMark parser from Markdown 3 or a parser for different flavor of markdown altogether, using a simpler and faster parser for simple documents with no syntax extensions, or having a separate PEG parser for YAML documents instead of tinyyaml, which is regex-based, kinda slow, and only supports a subset of YAML, all seem feasible and potentially useful.

More generally, this change could be a stepping stone towards a more general architecture that might subsume the Markdown package and support different types of lightweight markup languages with separate parsers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
lua Related to the Lua interface and implementation speed Related to speed improvements
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant