Fix linting: address failing workflows on github

[myxie]

Summary 📝

Previous work introduced code not compatible with Python 3.9, we we still support in DALiuGE, which runs these tests.

Details

I've used elements from the typing module for backwards compatibility.

Checks

• Closed #798
• Tested Changes
• Stakeholder Approval

Summary by Sourcery

Bug Fixes:

• Fix compatibility issues with Python 3.9 by replacing modern type hinting syntax with elements from the typing module.

[sourcery-ai]

Reviewer's Guide by Sourcery

This PR fixes linting issues by updating type hints to maintain compatibility with Python 3.9. The changes replace newer Union type syntax (|) with older typing module constructs (Union, Optional, Tuple).

Updated class diagram for DetailedDescription and nodes_from_module

classDiagram
    class DetailedDescription {
        - Optional~str~ descr
        - name
        + __init__(descr: Optional~str~ = None, name=None)
    }

    class nodes_from_module {
        + nodes_from_module(module_path, recursive=True) Tuple~list, Any~
    }

Loading

File-Level Changes

Change	Details	Files
Update type hints for Python 3.9 compatibility	Replace Union[str	None] with Optional[str] Replace tuple[list, Any] with Tuple[list, Any]

Possibly linked issues

• #798: The PR fixes the Union type issue by using Optional from typing for Python 3.9 compatibility.

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time. You can also use
  this command to specify where the summary should be inserted.

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[sourcery-ai]

Hey @myxie - I've reviewed your changes and they look great!

Here's what I looked at during the review

• 🟢 General issues: all looks good
• 🟢 Security: all looks good
• 🟢 Testing: all looks good
• 🟢 Complexity: all looks good
• 🟢 Documentation: all looks good

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 88.28%. Comparing base (10a9e3f) to head (d09dfc8).
Report is 15 commits behind head on main.

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main       #7      +/-   ##
==========================================
+ Coverage   84.92%   88.28%   +3.36%     
==========================================
  Files           6        6              
  Lines        1386     1502     +116     
==========================================
+ Hits         1177     1326     +149     
+ Misses        209      176      -33     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[myxie]

@awicenec I have put together this PR to help the build pass, which will help us pass CI on the DALiuGE workflows as well (they are currently failing due to the paletteGen not being able to run).

On reviewing these failures it also looks like there were a few other issues with the project:

• The only time the unit tests run were with a release, so it's only then we would know if something breaks
• The test_base.py::test_full_numpy test looks like it has been failing for a while, but it was reported as fine on the Ubuntu machine (I think this is because the way GitHub workflows handles the pylint return values is misaligned).
  • Investigating this threw up a can of worms that I have only partially addressed.

Please let me know if you are happy with these changes.

[myxie]

This assumes that what we are getting at mod here is some form of module/function/class, like the following:

mod=<class 'numpy.complexfloating'>
mod=<function histogram2d at 0x754af8827fb0>
mod=<module 'numpy.polynomial' from '/home/00087932/github/daliuge-new/.venv/lib/python3.10/site-packages/numpy/polynomial/__init__.py'>
mod=<function set_default_printstyle at 0x754af28d1a20>
mod=<module 'numpy.polynomial.polynomial' from '/home/00087932/github/daliuge-new/.venv/lib/python3.10/site-packages/numpy/polynomial/polynomial.py'>

The problem here is that during the test_base test, we also get the following pop up as "modules" that are for some reason imported from numpy:

mod=array([0])
mod=array([1])
mod=array([0, 1])

This throws an error because the if not mod check doesn't work for arrays:

E               ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

I don't know why it ends up importing a couple of random arrays, but I suspect somewhere in the numpy code these are declared somewhere. At any rate, quite a confusing bug!

[awicenec]

The test_full_numpy was actually skipped most of the time, because it took very long. Now it is much better and I have re-enabled it.