Collection error v0.11.0

[pawamoy]

Describe the bug
Collection error due to JSON failing to load the file line read on stdout. I suspect the Python interpreter to pre-execute code that is writing to stdout, or simply writing some blank lines to stdout for some reason.

To Reproduce
Waiting for feedback in rdilweb/docs@cc20660

Expected behavior
No collection error.

Information (please complete the following information):

• mkdocstrings version: 0.11.0

Additional context
Possible fix:

diff --git a/src/mkdocstrings/handlers/python.py b/src/mkdocstrings/handlers/python.py
index 88e8da3..c8127eb 100644
--- a/src/mkdocstrings/handlers/python.py
+++ b/src/mkdocstrings/handlers/python.py
@@ -125,13 +125,19 @@ class PythonCollector(BaseCollector):
         """
         log.debug("mkdocstrings.handlers.python: Opening 'pytkdocs' subprocess")
         env = os.environ.copy()
+        env["PYTHONUNBUFFERED"] = "1"
 
-        python_str = "; ".join(setup_commands) + "; " if setup_commands else ""
-        cmd = python_str + "from pytkdocs.cli import main; main(['--line-by-line'])"
+        if setup_commands:
+            cmd = [
+                "python",
+                "-c",
+                "; ".join(setup_commands) + "; from pytkdocs.cli import main; main(['--line-by-line'])"
+            ]
+        else:
+            cmd = ["pytkdocs", "--line-by-line"]
 
-        env["PYTHONUNBUFFERED"] = "1"
         self.process = Popen(  # noqa: S603,S607 (we trust the input, and we don't want to use the absolute path)
-            ["python", "-c", cmd], universal_newlines=True, stderr=PIPE, stdout=PIPE, stdin=PIPE, bufsize=-1, env=env,
+            cmd, universal_newlines=True, stderr=PIPE, stdout=PIPE, stdin=PIPE, bufsize=-1, env=env,
         )
 
     def collect(self, identifier: str, config: dict) -> Any:

Ping @RDIL @rossmechanic

[rossmechanic]

Hi @pawamoy , just tried this locally, and haven't been able to produce any errors (it has all been working perfectly, with or without setup_commands). However, I have a fix in order to get the print statements back to stdout. We can do the following to catch the stdout and flush it:

        catch_command_stdout ="from io import StringIO; import sys; sys.stdout = StringIO(); "
        reset_stdout = "sys.stdout.flush(); sys.stdout = sys.__stdout__; "
        python_str = "; ".join(setup_commands) + "; " if setup_commands else ""
        cmd = catch_command_stdout + python_str + reset_stdout + "from pytkdocs.cli import main; main(['--line-by-line'])"

Then you shouldn't need that if statement. What do you think? (Please no judgement on the variable names I just threw this together 🙈 )

[RDIL]

Hey, sorry for not responding sooner, I was asleep. So this happens on https://gitpod.io, but not in CI or locally on my machine, which is interesting. Maybe this is because I have gitpod pre-configured to run mkdocs serve as soon as the workspace is opened. Not quite sure why this is happening though.

[pawamoy]

@rossmechanic, thank you very much for looking into this! And thanks for the snippet (no judgement 😄)! I've refactored it in the following snippet. However if Python itself is executing a pre-script, faking sys.stdout might happen too late.

@RDIL no problem! Is it possible to patch an installed Python library on gitpod.io? If yes, could you try to apply this patch to mkdocstrings to see if it fixes the issue?

diff --git a/src/mkdocstrings/handlers/python.py b/src/mkdocstrings/handlers/python.py
index 88e8da3..f50d3f1 100644
--- a/src/mkdocstrings/handlers/python.py
+++ b/src/mkdocstrings/handlers/python.py
@@ -125,13 +125,27 @@ class PythonCollector(BaseCollector):
         """
         log.debug("mkdocstrings.handlers.python: Opening 'pytkdocs' subprocess")
         env = os.environ.copy()
+        env["PYTHONUNBUFFERED"] = "1"
 
-        python_str = "; ".join(setup_commands) + "; " if setup_commands else ""
-        cmd = python_str + "from pytkdocs.cli import main; main(['--line-by-line'])"
+        if setup_commands:
+            # prevent the Python interpreter or the setup commands
+            # from writing to stdout as it would break pytkdocs output
+            commands = [
+                "import sys",
+                "from io import StringIO",
+                "from pytkdocs.cli import main as pytkdocs",
+                "sys.stdout = StringIO()",  # redirect stdout to memory buffer
+                *setup_commands,
+                "sys.stdout.flush()",
+                "sys.stdout = sys.__stdout__",  # restore stdout
+                "pytkdocs(['--line-by-line'])"
+            ]
+            cmd = "; ".join(commands)
+        else:
+            cmd = ["pytkdocs", "--line-by-line"]
 
-        env["PYTHONUNBUFFERED"] = "1"
         self.process = Popen(  # noqa: S603,S607 (we trust the input, and we don't want to use the absolute path)
-            ["python", "-c", cmd], universal_newlines=True, stderr=PIPE, stdout=PIPE, stdin=PIPE, bufsize=-1, env=env,
+            cmd, universal_newlines=True, stderr=PIPE, stdout=PIPE, stdin=PIPE, bufsize=-1, env=env,
         )
 
     def collect(self, identifier: str, config: dict) -> Any:

Otherwise could you tell us if the PYTHONSTARTUP environment variable is defined, or if running python -Sc "import site" outputs anything (even a blank line)?

[RDIL]

I can try it out.

[RDIL]

Hey @pawamoy, I tried the patch but it has just been frozen for 15 minutes. From the output of mkdocs-plugin-progress, I can see it has frozen on INFO - Processing page "libraries/filehandlers/api.md"..., which is one of the pages I use mkdocstrings on.

[pawamoy]

Hmmm forget about it. I'm trying it myselfin gitpod.io on your repo, and the error happens only when trying to collect filehandlers.

[pawamoy]

Really not sure to understand what's happening here 😅

Thank you for trying it out @RDIL

I'll investigate a bit more.

Oh and the patch I gave you was wrong anyway... terribly sorry...

[RDIL]

@pawamoy no worries!

[pawamoy]

Well I can confirm the issue only happens when running the process with python -c, and only through mkdocs... so weird.

mkdocs serve  # fails
python3 -c "from mkdocs.commands.serve import serve; serve()"  # works

[pawamoy]

OK seems to be related to pyenv:

/home/gitpod/.pyenv/shims/python3 -c "from mkdocs.commands.serve import serve; serve()"
# works

/home/gitpod/.pyenv/versions/3.8.2/bin/python3 -c "from mkdocs.commands.serve import serve; serve()"
# fails, the mkdocs script uses this path

[pawamoy]

If think they have some issues on gitpod.io with Python venvs because I just can't install anything in a virtualenv. It works, message says "Successfully installed..." but then pip freezes outputs nothing, and if I install again, it installs again and doesn't say "Requirement already satisfied".

EDIT: needs export PIP_USER=no

[pawamoy]

Conclusion:

1. • the bug is coming from gitpod's workspace setups, particularly how pyenv and pip are setup: calling python directly from pyenv's versions does not work while calling the shim works, and when scripts are installed with pip install their shebang point directly to the version, not the shim.
   • This could surely be debugged further: on my system, installing a Python script creates a shim (shell script running pyenv exec program), while on gitpod.io it does not create a shim but directly the python-shebang script. Maybe coming from one of their plugins: pyenv-doctor pyenv-installer pyenv-update pyenv-virtualenv pyenv-which-ext python-build.
   • As to why running Python like that makes the mkdocs command fail, I have no idea. pytkdocs never writes blank lines on stdout, it either completely fails (cannot read stdin) and prints nothing, prints a JSON dict containing an error, or a JSON dict containing 0 or more objects, but never just blank lines
2. mkdocstrings 0.11.0 is working correctly without setup commands
3. it will break if a setup commands writes to stdout
4. a fixed version of the snippet above fixes this issue
5. it will break if code imported while collecting documentation writes to stdout
6. and this will be fixed in pytkdocs itself, see print statements in the loaded code break the output pytkdocs#24

In the meantime @RDIL you can use python3 -c "from mkdocs.commands.serve import serve; serve()" instead of mkdocs serve in your startup configuration!

I'll close this issue once the fix mentioned in point 4 is implemented!

Thank you @RDIL and @rossmechanic for your help 🙂