Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix CI part 2 #1025

Merged
merged 11 commits into from
Nov 3, 2021
Merged

Fix CI part 2 #1025

merged 11 commits into from
Nov 3, 2021

Conversation

ianthomas23
Copy link
Member

Following on from PR #1022, this is an attempt to fix the remaining CI failures. There are 3 changes:

  1. Exception text that needs to precisely match that expected in pandas (like in Fix Ragged tests #1022).
  2. Remove the upper version limit on nbconvert as certain versions were causing problems when running pytest "--nbsmoke-run" "-k" ".ipynb"
  3. Change of np.int to int which is only causes a DeprecationWarning but I fixed it whilst I was there.

Fixes work locally for me, but need to check across the full CI matrix.

@ianthomas23
Copy link
Member Author

The CI errors are different, so there is some form of progress here. This may take a number of iterations to sort out.

@ianthomas23
Copy link
Member Author

So I understand the remaining problem and know how to fix it. But the fix is really in nbsmoke not datashader.

Before this PR, nbconvert was pinned to <6. I've removed this to fix one problem, but an old problem resurfaces in that nbconvert had some breaking changes (from datashader point of view) between versions 5 and 6, in particular how template files are referenced. The template use occurs in nbsmoke here

https://github.com/pyviz-dev/nbsmoke/blob/a9c58a43af3b57c4c76ea0efc8315e0ea3344d87/nbsmoke/verify.py#L68-L72

From a datashader point of view, if you change

    html_exporter.template_file = 'basic'

to

    html_exporter.template_file = 'classic/base.html.j2'

everything works fine. It would also need a check of nbconvert version number to select which code path to follow.

So the best way to fix this is:

  1. PR in nbsmoke with the above fix.
  2. New release of nbsmoke, presumably 0.0.6.
  3. Pin nbsmoke to >=0.0.6 in datashader within this PR, and no further changes are required.

I have three concerns:

  1. Changes to nbsmoke may break other projects that I know nothing about.
  2. nbsmoke hasn't seen any code changes for about a year so I am nervous about getting involved with it.
  3. I would need the support of someone who has sufficient permissions to do a release of nbsmoke.

@jbednar
Copy link
Member

jbednar commented Oct 18, 2021

Thanks! We can handle updating nbsmoke. The breaking changes in nbconvert have caused all sorts of problems in our ecosystem, alas. Thanks for the investigative work and for sharing the details!

@ianthomas23
Copy link
Member Author

PR submitted to nbsmoke holoviz-dev/nbsmoke#57.

Over to you @jbednar!

@jbednar
Copy link
Member

jbednar commented Oct 20, 2021

Thanks! And I've quickly punted it over to @maximlt, who should have much more brain space available to deal with it! :-)

@maximlt
Copy link
Member

maximlt commented Nov 1, 2021

Ok things got better on the nbsmoke/nbconvert front!

But it seems like datashader needs to be updated wrt bokeh. This test is failing:

=================================== FAILURES ===================================
________________________ test_interactive_image_update _________________________

    def test_interactive_image_update():
        # jbednar: This test uses 1x1 images that are not actually supported
        # (as they have infinite resolution), but as InteractiveImage is deprecated
        # anyway the tests have not been updated.
        p = figure(x_range=(0, 1), y_range=(0, 1), plot_width=2, plot_height=2)
        img = InteractiveImage(p, create_image)
    
        # Ensure bokeh Document is instantiated
        img._repr_html_()
        assert isinstance(img.doc, Document)
    
        # Ensure image is updated
        img.update_image({'xmin': 0.5, 'xmax': 1, 'ymin': 0.5, 'ymax': 1, 'w': 1, 'h': 1})
        out = np.array([[4287299584]], dtype=np.uint32)
        assert img.ds.data['x'] == [0.5]
        assert img.ds.data['y'] == [0.5]
        assert img.ds.data['dh'] == [0.5]
        assert img.ds.data['dw'] == [0.5]
        assert np.array_equal(img.ds.data['image'][0], out)
    
        # Ensure patch message is correct
>       msg = img.get_update_event()

datashader/tests/test_bokeh_ext.py:65: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
datashader/bokeh_ext.py:319: in get_update_event
    return patch_event(self)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

image = <datashader.bokeh_ext.InteractiveImage object at 0x7f8fd42b7450>

    def patch_event(image):
        """
        Generates a bokeh patch event message given an InteractiveImage
        instance. Uses the bokeh messaging protocol for bokeh>=0.12.10
        and a custom patch for previous versions.
    
        Parameters
        ----------
        image: InteractiveImage
            InteractiveImage instance with a plot
    
        Returns
        -------
        msg: str
            JSON message containing patch events to update the plot
        """
        if bokeh_version > '0.12.9':
>           events = list(image.doc._held_events)
E           AttributeError: 'Document' object has no attribute '_held_events'

@ianthomas23
Copy link
Member Author

@maximlt Is there anything I can do to help with this?

@maximlt
Copy link
Member

maximlt commented Nov 1, 2021

@ianthomas23 I've pushed a commit that fixes a compatibility issue with bokeh>=2.4.

Now the tests are all failing on Windows for another reason. This is the error report on py38:

datashader/tests/test_mpl_ext.py::test_image_initialize PASSED           [ 56%]

Windows fatal exception: access violation

Current thread 0x0000046c (most recent call first):
  File "C:\Miniconda3\envs\test-environment\lib\site-packages\matplotlib\backends\backend_agg.py", line 240 in get_text_width_height_descent
  File "C:\Miniconda3\envs\test-environment\lib\site-packages\matplotlib\text.py", line 306 in _get_layout
  File "C:\Miniconda3\envs\test-environment\lib\site-packages\matplotlib\text.py", line 903 in get_window_extent
  File "C:\Miniconda3\envs\test-environment\lib\site-packages\matplotlib\axis.py", line 1068 in <listcomp>
  File "C:\Miniconda3\envs\test-environment\lib\site-packages\matplotlib\axis.py", line 1068 in _get_tick_bboxes
  File "C:\Miniconda3\envs\test-environment\lib\site-packages\matplotlib\axis.py", line 1142 in draw
  File "C:\Miniconda3\envs\test-environment\lib\site-packages\matplotlib\artist.py", line 51 in draw_wrapper
  File "C:\Miniconda3\envs\test-environment\lib\site-packages\matplotlib\image.py", line 132 in _draw_list_compositing_images
  File "C:\Miniconda3\envs\test-environment\lib\site-packages\matplotlib\axes\_base.py", line 2921 in draw
  File "C:\Miniconda3\envs\test-environment\lib\site-packages\matplotlib\_api\deprecation.py", line 431 in wrapper
  File "C:\Miniconda3\envs\test-environment\lib\site-packages\matplotlib\artist.py", line 51 in draw_wrapper
  File "C:\Miniconda3\envs\test-environment\lib\site-packages\matplotlib\image.py", line 132 in _draw_list_compositing_images
  File "C:\Miniconda3\envs\test-environment\lib\site-packages\matplotlib\figure.py", line 2790 in draw
  File "C:\Miniconda3\envs\test-environment\lib\site-packages\matplotlib\artist.py", line 51 in draw_wrapper
  File "C:\Miniconda3\envs\test-environment\lib\site-packages\matplotlib\artist.py", line 74 in draw_wrapper
  File "C:\Miniconda3\envs\test-environment\lib\site-packages\matplotlib\backends\backend_agg.py", line 406 in draw
  File "D:\a\datashader\datashader\datashader\tests\test_mpl_ext.py", line 43 in test_image_update
  File "C:\Miniconda3\envs\test-environment\lib\site-packages\_pytest\python.py", line 182 in pytest_pyfunc_call
  File "C:\Miniconda3\envs\test-environment\lib\site-packages\pluggy\callers.py", line 187 in _multicall
  File "C:\Miniconda3\envs\test-environment\lib\site-packages\pluggy\manager.py", line 84 in <lambda>
  File "C:\Miniconda3\envs\test-environment\lib\site-packages\pluggy\manager.py", line 93 in _hookexec
  File "C:\Miniconda3\envs\test-environment\lib\site-packages\pluggy\hooks.py", line 286 in __call__
  File "C:\Miniconda3\envs\test-environment\lib\site-packages\_pytest\python.py", line 1477 in runtest
  File "C:\Miniconda3\envs\test-environment\lib\site-packages\_pytest\runner.py", line 135 in pytest_runtest_call
  File "C:\Miniconda3\envs\test-environment\lib\site-packages\pluggy\callers.py", line 187 in _multicall
  File "C:\Miniconda3\envs\test-environment\lib\site-packages\pluggy\manager.py", line 84 in <lambda>
  File "C:\Miniconda3\envs\test-environment\lib\site-packages\pluggy\manager.py", line 93 in _hookexec
  File "C:\Miniconda3\envs\test-environment\lib\site-packages\pluggy\hooks.py", line 286 in __call__
  File "C:\Miniconda3\envs\test-environment\lib\site-packages\_pytest\runner.py", line 217 in <lambda>
  File "C:\Miniconda3\envs\test-environment\lib\site-packages\_pytest\runner.py", line 244 in from_call
  File "C:\Miniconda3\envs\test-environment\lib\site-packages\_pytest\runner.py", line 216 in call_runtest_hook
  File "C:\Miniconda3\envs\test-environment\lib\site-packages\_pytest\runner.py", line 186 in call_and_report
  File "C:\Miniconda3\envs\test-environment\lib\site-packages\_pytest\runner.py", line 100 in runtestprotocol
  File "C:\Miniconda3\envs\test-environment\lib\site-packages\_pytest\runner.py", line 85 in pytest_runtest_protocol
  File "C:\Miniconda3\envs\test-environment\lib\site-packages\pluggy\callers.py", line 187 in _multicall
  File "C:\Miniconda3\envs\test-environment\lib\site-packages\pluggy\manager.py", line 84 in <lambda>
  File "C:\Miniconda3\envs\test-environment\lib\site-packages\pluggy\manager.py", line 93 in _hookexec
  File "C:\Miniconda3\envs\test-environment\lib\site-packages\pluggy\hooks.py", line 286 in __call__
  File "C:\Miniconda3\envs\test-environment\lib\site-packages\_pytest\main.py", line 272 in pytest_runtestloop
  File "C:\Miniconda3\envs\test-environment\lib\site-packages\pluggy\callers.py", line 187 in _multicall
  File "C:\Miniconda3\envs\test-environment\lib\site-packages\pluggy\manager.py", line 84 in <lambda>
  File "C:\Miniconda3\envs\test-environment\lib\site-packages\pluggy\manager.py", line 93 in _hookexec
  File "C:\Miniconda3\envs\test-environment\lib\site-packages\pluggy\hooks.py", line 286 in __call__
  File "C:\Miniconda3\envs\test-environment\lib\site-packages\_pytest\main.py", line 247 in _main
  File "C:\Miniconda3\envs\test-environment\lib\site-packages\_pytest\main.py", line 191 in wrap_session
  File "C:\Miniconda3\envs\test-environment\lib\site-packages\_pytest\main.py", line 240 in pytest_cmdline_main
  File "C:\Miniconda3\envs\test-environment\lib\site-packages\pluggy\callers.py", line 187 in _multicall
  File "C:\Miniconda3\envs\test-environment\lib\site-packages\pluggy\manager.py", line 84 in <lambda>
  File "C:\Miniconda3\envs\test-environment\lib\site-packages\pluggy\manager.py", line 93 in _hookexec
  File "C:\Miniconda3\envs\test-environment\lib\site-packages\pluggy\hooks.py", line 286 in __call__
  File "C:\Miniconda3\envs\test-environment\lib\site-packages\_pytest\config\__init__.py", line 124 in main
  File "C:\Miniconda3\envs\test-environment\Scripts\pytest-script.py", line 10 in <module>
datashader/tests/test_mpl_ext.py::test_image_update TaskError - taskid:test_unit_deploy
Command error: 'pytest "datashader"' returned 3221225477

@jbednar have you already seen this kind of error here?

@maximlt
Copy link
Member

maximlt commented Nov 2, 2021

Ok so the tests were failing on Windows due to a problem somewhere between a new version of the package freetype and matplotlib. I ran the tests locally on Windows and they passed. I think this is ready to be merged.

matplotlib/matplotlib#21511
conda-forge/freetype-feedstock#42

@maximlt
Copy link
Member

maximlt commented Nov 3, 2021

Thanks @ianthomas23 for the push on fixing the CI and nbsmoke!

@maximlt maximlt merged commit 6a5e63b into holoviz:master Nov 3, 2021
@jbednar
Copy link
Member

jbednar commented Nov 3, 2021

Thanks to both of you for getting this to the finish line!

@ianthomas23 ianthomas23 deleted the fix_ci branch November 17, 2022 14:32
@ianthomas23 ianthomas23 restored the fix_ci branch November 17, 2022 14:33
@ianthomas23 ianthomas23 deleted the fix_ci branch November 17, 2022 14:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants