PyDeequ Stable Release (1.1.0)

[chenliu0831]

This is a meta issue to list major enhancement, features and bug-fixes to bring PyDeequ to a stable release.

Please add a comment with the PRs, issues (create if not exist) or ideas you would like to bring us to stable release.

More to come..

Completed Item

• Support Deequ 2.0.0 (Spark 3.1) and 2.0.1 (Spark 3.2) #100. This closes multiple issues and bring dependency up to date.
• spark 3.3.0 #120

Release Plan

• 1.1.0rc0 PR
• 1.1.0rc0 on PYPI
• 1.1.0rc0 baking, asking for feedback. ETA 1-2 week. On-going, ending ETA 06/30
• 1.1.0 PR - release: 1.1.0 #139
• 1.1.0 on PYPI. ETA: early July or sooner if RC went smooth

[poolis]

hasPattern implementation

Issue: #71
PR: #76

[SemyonSinchenko]

Hello! What is expected date of the release?

[mycaule]

Yes for example in my company, we are using great expectations instead of PyDeequ for Spark/Python because 1.0 isn't available yet. But otherwise, for Spark/Scala project Deequ looked reasonable for that same company.

[lecardozo]

A couple of things I think we should consider before this release:

• Resolve Spark version detection prevents startup of new spark sessions #111, which is breaking the master branch;
• Improve coverage for tests and modernize the test stack;
• Improve CI for PRs (so we can see if tests are running properly before merging)

[marius-xentral]

any news here?

[astrojuanlu]

Notice that, even though it was never tagged on this GitHub repository, there's already pydeequ 1.0.0 and even 1.0.1 on PyPI https://pypi.org/project/pydeequ/1.0.1/ uploaded back in July 2021.

It's encouraging to see that python-deequ got some attention recently, but expectations about release numbering need to be adjusted, and also it would be nice to have some indication on whether such a release is happening ~soon or not.

[lhendrick-t10]

Would really love to see this released!

[chenliu0831]

Notice that, even though it was never tagged on this GitHub repository, there's already pydeequ 1.0.0 and even 1.0.1 on PyPI https://pypi.org/project/pydeequ/1.0.1/ uploaded back in July 2021.
It's encouraging to see that python-deequ got some attention recently, but expectations about release numbering need to be adjusted, and also it would be nice to have some indication on whether such a release is happening ~soon or not.

That's unfortunate - we could still release 1.1.0 though, there does not seem to be breaking changes.

We are working through some administrative issues - expect another update in 2-3 weeks

[runkelcorey]

@chenliu0831 following up on your last message. What's happened in the last few weeks?

[revolutionisme]

@chenliu0831, is there a timeline on this update? We are looking to use pydeequ with glue 4.0 .

[chenliu0831]

We are still blocked on a few administrative (and technical) issues. I have been actively follow-ing up to resolve those. But I do not have ETA at this time.

[chenliu0831]

Update - we resolved the issues. Targeting to release to PYPI by end of Jun.

[chenliu0831]

Publish 1.1.0rc0 to PYPI - we will bake for about 2 weeks. See notes https://github.com/awslabs/python-deequ/releases/tag/v1.1.0rc0

[chenliu0831]

@revolutionisme @runkelcorey @lhendrick-t10 @marius-xentral @lecardozo, tagging just as an FYI. Install instruction in pre-release note https://github.com/awslabs/python-deequ/releases/tag/v1.1.0rc0. Any feedback is welcome if you get chance to try it out

[chenliu0831]

We released to PYPI!

https://pypi.org/project/pydeequ/