sphinx.ext.automodsumm broken with Sphinx 1.3

[embray]

After updating to Sphinx 1.3 the Sphinx build for Astropy and all affiliated packages doesn't work any more!?

$ python setup.py build_sphinx
...
Traceback (most recent call last):
  File "<stdin>", line 28, in <module>
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/sphinx/application.py", line 188, in __init__
    self._init_builder(buildername)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/sphinx/application.py", line 250, in _init_builder
    self.emit('builder-inited')
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/sphinx/application.py", line 497, in emit
    results.append(callback(self, *args))
  File "/Users/deil/code/astropy/astropy_helpers/astropy_helpers/sphinx/ext/automodsumm.py", line 225, in process_automodsumm_generation
    filestosearch = [x + ext for x in env.found_docs
  File "/Users/deil/code/astropy/astropy_helpers/astropy_helpers/sphinx/ext/automodsumm.py", line 226, in <listcomp>
    if os.path.isfile(env.doc2path(x))]\
TypeError: Can't convert 'list' object to str implicitly

cc @eteq @astrofrog

[cdeil]

On Python 2.7 I get this different error:

$ python setup.py build_sphinx
...
/usr/bin/clang -bundle -undefined dynamic_lookup -L/opt/local/lib -Wl,-headerpad_max_install_names -L/opt/local/lib/db48 build/temp.macosx-10.10-x86_64-2.7/astropy/convolution/boundary_wrap.o -o build/lib.macosx-10.10-x86_64-2.7/astropy/convolution/boundary_wrap.so
  File "<stdin>", line 38
    print(darkred('reST markup error:'), file=sys.stderr)
                                             ^
SyntaxError: invalid syntax
Sphinx Documentation subprocess failed with return code 1

Is this a recently introduced bug in Sphinx (https://github.com/sphinx-doc/sphinx/blame/548a6dc22e4a3edff76413b6bdeedcf2b8c3f30c/sphinx/setup_command.py#L171) or some weird installation issue on my machine?

[embray]

I've seen the latter issue, and it's totally just a bug in Sphinx--nothing astropy-helpers can do about. I know it's been fixed more recently. I just have Sphinx pinned to 1.2.3 for Python 2.7, I think.

I'm trying to figure out what could cause the first one though. What does your sphinx conf have for the source_suffix option? It seems like this could happen if it is a list instead of a string (it should just be source_suffix = '.rst').

[cdeil]

I have the git master version of astropy, which has source_suffix = '.rst':

astropy-helpers/astropy_helpers/sphinx/conf.py

Line 58 in 25d6ad8

source_suffix = '.rst'

So you can't reproduce this issue with Python 3 and it's something specific to my setup?

[embray]

I think that, to date, the docs build still hasn't been guaranteed to work fully on Python 3. I'm not 100% sure what the last word on that was though. I always use Python 2 to build the docs.

[embray]

Just looking at the code though, the only way I can see to get the exception you're getting is if somewhere source_suffix is being set to a list :\

[embray]

@cdeil Well in any case it's not just you--I was able to reproduce this on Python 3. It's bizarre though.

[embray]

I see: http://sphinx-doc.org/config.html#confval-source_suffix

Changed in version 1.3: Can now be a list of extensions.

I guess it automatically converts it to a list.

[embray]

However, fixing that issue reveals several more. So I think it's safe to say Sphinx 1.3 isn't support yet.

[astrofrog]

(regarding Python 3 support, I think it should actually work, and if not, it was pretty close last time I tried. I guess the question is whether to add a Travis build for it in astropy core to avoid regressions, but then it's an additional build :-/)

[embray]

Yes, I think Python 3 should mostly work, but maybe has a few little issues. On Sphinx 1.3 I think it will mostly work too, but there are some minor incompatibilities in the automodsumm extension, and possibly others, that need to be chased down... (hopefully by people who actually understand them which is not me, though I was able to fix a few of them...)

[cdeil]

I've been using python3.4 setup.py build_sphinx for the past month or two with the Astropy main repo and affiliated packages ... it worked fine before the update to Sphinx 1.3.

[astrofrog]

I agree, with Sphinx 1.2.3 there are no errors/warnings :)

[embray]

Good to know--then I guess this really is mostly just a Sphinx version issue. Astropy is still using 1.2.3 so best to stick with that until the other issues are sorted out.

[cdeil]

@embray You mentioned that you fixed a few of the issues with automodsumm and Sphinx 1.3. What's the remaining error you get when you try to run python setup.py build_sphinx for Astropy? I think @eteq wrote automodsumm and maybe he can help?

[embray]

I'll try to post the fixes I have so far at some point. I just haven't yet because I was trying to get everything fixed (since none of the fixes I had by themselves were of much help to get the thing running).

[embray]

Well, the following patch at least gets the damn thing to run without crashing:

diff --git a/astropy_helpers/sphinx/ext/automodsumm.py b/astropy_helpers/sphinx/ext/automodsumm.py
index 9a43680..88f2462 100644
--- a/astropy_helpers/sphinx/ext/automodsumm.py
+++ b/astropy_helpers/sphinx/ext/automodsumm.py
@@ -220,10 +220,12 @@ class Automoddiagram(InheritanceDiagram):
 #<---------------------automodsumm generation stuff--------------------------->
 def process_automodsumm_generation(app):
     env = app.builder.env
-    ext = app.config.source_suffix

-    filestosearch = [x + ext for x in env.found_docs
-                     if os.path.isfile(env.doc2path(x))]\
+    filestosearch = []
+    for docname in env.found_docs:
+        filename = env.doc2path(docname)
+        if os.path.isfile(filename):
+            filestosearch.append(docname + os.path.splitext(filename)[1])

     liness = []
     for sfn in filestosearch:
@@ -238,10 +240,11 @@ def process_automodsumm_generation(app):
                         f.write('\n')

     for sfn, lines in zip(filestosearch, liness):
+        suffix = os.path.splitext(sfn)[1]
         if len(lines) > 0:
             generate_automodsumm_docs(lines, sfn, builder=app.builder,
                                       warn=app.warn, info=app.info,
-                                      suffix=app.config.source_suffix,
+                                      suffix=suffix,
                                       base_path=app.srcdir)

 #_automodsummrex = re.compile(r'^(\s*)\.\. automodsumm::\s*([A-Za-z0-9_.]+)\s*'

but then it outputs "WARNING: document isn't included in any toctree" for all the generated API pages. Interestingly, all, or at least most, of the API references seem to work--the links go to the corresponding API pages. But they're rendered in black instead of blue, as though the reference wasn't resolved.

[embray]

I think this might have something to do with automodapi not working correctly, though I haven't figured out where it goes wrong.

[embray]

Digging a little further, with the fixes I made above (and a couple other minor ones) it doesn't seem there's any difference in what is output by automodsumm or automodapi, but there does seem there may be a problem with how Sphinx is interpreting the resulting directives--possibly changes in how the :toctree: directive work, though I haven't foudn anything obvious yet.

[embray]

Now I'm tracking another issue--maybe related?--that automodapi seems to get run twice, though the first time it's run on files with the filename extension in the docname (ex. "wcs/index.rst"), and then later a second time with just "wcs/index" without the extension as the docname...

Update: I see; that's just because automodsumm calls automodapi directly, but automodapi is also connected to the Sphinx "source-read" event, so it gets called twice on each document. I don't know if that matters or not.

[embray]

Aha! This change seems to be at the heart of the toctree generation for API items.

The check_module method that is called there checks whether the function or object really belongs directly to the module it's defined in. So if we're documenting astropy.config.get_config as part of the astropy.config module, it breaks here because the actual fully-qualified name of that function is astropy.config.configuration.get_config. This is an annoying regression though from our perspective. I'll see if there's some way to bypass that.

[embray]

I added a comment about this in sphinx-doc/sphinx#1061.

[embray]

Okay, I think I've found a workaround for this. It's hacky but gets the job done.

[embray]

I've converted this to a PR with some related fixes attached. This should get the docs build pretty much working with Sphinx 1.3 (I'm not sure if everything works perfectly but it at least runs and doesn't seem to have any warnings).

Maybe after the tests for this run I'll attach a new build configuration to the .travis.yml to test against Sphinx 1.3.

[cdeil]

Thanks @embray for fixing this so quickly!

If it works with Sphinx 1.3 on your machine or travis-ci it's probably not necessary, but let me know if you want me to try this branch on my machine.

[embray]

One issue it still has is that links to API references are still showing up in black instead of blue. I wonder if that's just something that needs to be tweaked in the stylesheet for the theme.

[cdeil]

@embray – Any chance to get this in soon?
(I'm not looking forward to having to teach my package manager to go back to an old Sphinx version, but I do want to build the docs for the affiliated packages I work on.)

[embray]

Hm, well Travis-CI lost its damn mind on that last build, so I'll restart it. It looks like this change might have broken things with Sphinx 1.2.x now though. I'll have to go back and re-tinker.

[embray]

@astrofrog Any idea what this is about:

Error: Cowardly refusing to `sudo brew install`

You can use brew with sudo, but only if the brew executable is owned by root.

However, this is both not recommended and completely unsupported so do so at

your own risk.

The command "source continuous-integration/travis/install_graphviz_$TRAVIS_OS_NAME.sh" failed and exited with 1 during .

? I think you set this up. The travis builds are losing their mind over it.

[embray]

@cdeil I think this is nearly fixed, but I need to find out why the CI builds are going belly up.

[embray]

Rebased to force a rebuild, now that #152 is in; hopefully now the build should pass on all platforms.