-
Notifications
You must be signed in to change notification settings - Fork 320
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Pandoc filters are not working anymore with quarto #5392
Comments
Thanks for the report. I pinged @cscheid who knows more about our filter execution layer. |
Test can be also made with pandoc-plantuml-filter which I based my > pip list
Package Version
---------------------- -------
argcomplete 3.0.5
click 8.1.3
packaging 23.0
pandoc-plantuml-filter 0.1.2
pandocfilters 1.5.0
panflute 2.3.0
pip 23.0.1
pipx 1.2.0
PyYAML 6.0
setuptools 65.5.0
userpath 1.8.0 This file was working on previous version of quarto. ---
execute:
echo: false
pagination_next: null
filters:
- pandoc-plantuml
editor:
render-on-save: true
---
# Welcome for fun
```plantuml
@startuml
actor "User" as user
@enduml
``` Got the same error with it. But the command following command is working: > pandoc -s docs/index.qmd -o docs/index.html --filter pandoc-plantuml
[WARNING] Could not deduce format from file extension .qmd
Defaulting to markdown
Created directory plantuml-images
Created image plantuml-images/8e377f6639c577b8eec950fe2b906a90812ee344.svg
[WARNING] This document format requires a nonempty <title> element.
Defaulting to 'index' as the title.
To specify a title, use 'title' in metadata or --metadata title="...". |
Note the error message:
Panflute should work (there are a number of caveats wrt custom AST nodes though); the issue here is that we are not finding the filter where we expect to, and we no longer resolve arbitrary executable files (like pandoc does). We can reconsider that decision, but the easiest way for you to solve it right now is to add (eg) a python script to the path that then calls the files on your path. |
This was my missing part. I wasn't ware of that change. Thanks for the clarification |
We have to wrap user filters so they can handle custom AST nodes, and that requires special code. The last time I thought through this I came to the conclusion that it would be hard to do this portably in Lua, but I could be wrong. I also think that allowing a filter from the user's PATH goes against quarto's philosophy of keeping everything important in the project source, so I'm actually happy with this new restriction. (Otherwise, the result of rendering this file would be non-obviously dependent on the surrounding environment.) |
Yes I agree - it is just a big difference with how it works with Pandoc itself. So I understand the user-side questioning on this for a "pandoc wrapper". Our extension system support non-lua filter already right ? It seems to be case as in our doc. Extensions seems to be the good way to user (any) filter with quarto. if it is not possible already, should be adapt our extension to resolve that @FabienSe did you try using the Extension form to use your filter with a Quarto project ? On Lua filter side, some new org is following the _extension organisation to be quarto compatible for example: https://github.com/pandoc-ext/abstract-section |
Thanks @cderv and @cscheid for your answers.
I succeed to do it with the following python script named from custom_filter.main import main
from panflute import Doc
def test(doc: Doc = None) -> Doc:
return main(doc=doc)
if __name__ == "__main__":
test() The documentation metadata seems to be wrong. Indeed the provided document format is I also changed the filter definition to
I did not tested it yet. |
Yes we don't document it with example, but we say
So I am assuming we support already other type of filter. @dragonstyle do we support non lua filter in extension ? Do we have an example somewhere ? |
Extensions won't know the difference, this distinction happens entirely inside of the Lua filter chain now. (quartodoc used to work like that but I helped those folks convert it to a Lua filter exactly because of these limitations.) |
I was thinking that using Extension would be easier to find non-lua filter as it would be relative to the project, and not from the user's PATH. I understand now:
Not sure it is all crystal clear to me yet. Anyway, as always you're precision helped me understand the changes better. Thanks! |
Unfortunately that metadata is not wrong, and the behavior is currently inevitable; this is the output format in quarto (because reasons). That's one of the problems with using non-Lua filters in quarto. We have Lua APIs for checking formats, but we don't yet have language-agnostic APIs in filters. We intend to do that, but not in the near future. |
Thanks again for your answers @cderv and @cscheid. Short term, I will keep the quick solution with an additional python script which call my custom script. I looked at the discussion at quartodoc regarding the limitation of non-lua filter. Is it possible to do stuff like create folder or file, call external command line in Lua ? |
Yes it is possible Look at our doc https://quarto.org/docs/extensions/lua-api.html and the other linked doc
In addition to Quartodoc example, here is another: shinylive calls a custom program (from a python package) from within the LUA filter You can do a lot with Lua ! |
I can confirm that this is a backward incompatible change in Quarto that has not been advertised. We had filters installed via a Python package, and that simply had to be on the path. Those filters used to work, but no longer do. We will also follow the Python wrapper approach for now, but this is perhaps a common enough use case that it warrants a section in the docs? |
Just to say - the ability to use Panflute for filters was a big sell for us, as Python programmers using Quarto. It made it easy and pleasant to switch from e.g. Jupyter-book. Forcing the use of Lua would be a significant barrier to uptake, I suspect - because filters are so important to customising output. |
Let me be clear: you can 100% still use panflute. The only change is that binaries on your path are no longer interpreted as JSON filters. All you need to do is provide a |
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
I opened a new issue for another |
Is there an easy / documented way to wrap an executable pandoc filter? I want to use the filter Would it be reasonable to look in |
@ChrisJefferson Interesting. Did you try just adding
And I think we use the same codepath in extensions. Here, |
It works if I write
Then put a symlink to I'd prefer not to put filters in the same directory as my |
Hm. This looks like a bug on our side. |
Do I understand it correctly, that the following should work currently for Rust filters?: ---
title: Test Rust Filter
filters:
- _extensions/pandoc-katex
---
Foo I ask because it is unclear from the conversation above it that was the exact syntax proposed, and also if it was confirmed working or not. And because when I try that syntax with pandoc-filter-diagram it fails like this:
|
To be clear the following message is not recommend quarto practice, might not work and may break in future, because I'm just some person, getting my filter working, not giving official quarto guidance. Long term, I imagine there may well be better ways of doing this. Having said that, here's what I did in your situation, I'm assuming you've globally installed
You can pop a link to it into the current directory (
Then if you just refer to it, like this:
The
Just so the pandoc filter diagram link was in a subdirectory, rather than mixed in with my |
I really don't think there's a bug here; if there is one, I'd like to ask folks to open a new issue. Here's a specific example of using a binary inside a Quarto filter extension:
In this example, |
Bug description
I was using quarto with a filter for pandoc, it was working great but it is not working anymore.
I am working on a custom filter developed with panflute to render plantuml diagram in image. A library used to develop pandoc filter with python.
A similar filter can be found here. I was trying to improve it.
I got the same error with this filter too.
I was using it with the
filters
parameter like this in aindex.qmd
file:This file is in a quarto project to render it in a docusaurus website.
The code block was rendered into image by my custom filter. I assume the filter was passed to pandoc directly because the filter is working great with
--filter
of pandoc cli.Now I got the following error:
Operating system : WSL on Ubuntu 20.04.6
Quarto check:
Checklist
quarto check
so we know which version of quarto and its dependencies you're running.The text was updated successfully, but these errors were encountered: