Release 0.10.0

[nnansters]

Hello everybody!

It has been a very long time since our last release and we're sorry about that. We've been working on some exciting new stuff behind the covers so the pace for the open source library slowed down a bit.

But happy tidings: we've just released NannyML 0.10.0! This release focuses on improvements and fixes and a lot of contribution. Let's take a look!

Installing / upgrading

You can get this latest version by using pip:

pip install -U nannyml

Or conda:

conda install -c conda-forge nannyml

What’s new?

Multiclass confusion matrix support

We now support estimating and calculating the confusion directly in multiclass classification cases.
This comes with a new parameter for the calculation: normalize_confusion_matrix. Possible values are None for no normalization at all, true to normalize over the true class of the observations, pred to normalize over the predicted class of observations or all to normalize over all observations.

We'll illustrate this with a quick code snippet for performance estimation. Calculating the realized performance is very similar.

import nannyml as nml
from IPython.display import display

reference_df, analysis_df, _ = nml.load_synthetic_multiclass_classification_dataset()

display(reference_df.head(3))

estimator = nml.CBPE(
    y_pred_proba={
        'prepaid_card': 'y_pred_proba_prepaid_card',
        'highstreet_card': 'y_pred_proba_highstreet_card',
        'upmarket_card': 'y_pred_proba_upmarket_card'},
    y_pred='y_pred',
    y_true='y_true',
    timestamp_column_name='timestamp',
    problem_type='classification_multiclass',
    metrics=['confusion_matrix'],
    normalize_confusion_matrix='all',
    chunk_size=6000,
)

estimator.fit(reference_df)

results = estimator.estimate(analysis_df)

You can read more about this in our docs!

Estimating Confusion Matrix Elements for Multiclass Classification
Calculating Confusion Matrix Elements for Multiclass Classification

Big thanks to our alumnus @cartgr for the hard work on this one.

What's changed?

There has been a ton of contributions over the last months and I couldn't be happier about it!

• We didn't realize we were missing database support for the data quality calculators we've added some time ago. @bgalvao
  is a long-time user of our database export functionality and not only spotted the issue but also committed a full implementation for the missing functionality. That was some excellent work @bgalvao , thanks for being a long-time user and supporter! We might still be missing summary statistics in the database export 😉
• We're always trying to improve our error handling but there's still plenty of edge cases with NaN values slipping by. @giodavoli spotted an issue with threshold computation when dealing with NaN values. He was kind enough to propose a fix in his bug report. Credit where credit is due: major thanks to the eagle-eyed @giodavoli !
• @shezadkhan137 noticed that only one out of four confusion matrix metrics was exported to the database and took it into his own hands to right this wrong. I liked the way he did it quite a lot and even changed some of our other things to match. Thanks for the inspring commit @shezadkhan137 !

Some honorary mentions

• "Under the microscope" title goes to @nikml for detecting a minute yet important issue in our specificity calculations!
  "Patience of a saint" title goes to @santiviquez for his laborious efforts on the docs and content (and brother @maciejbalawejder as well)!
• "Rapid fire" title goes to @michael-nml for shooting down bugs and change requests at high speed this last week.

What's next?

We've been working very hard behind the scenes on our commercial offering called NannyML cloud. It is the easiest way to use NannyML, with a custom built web application, fully automated runs of NannyML and some proprietary algorithms like Reverse Concept Shift (RCS) and an improved version of CBPE for performance estimation. This will all be shortly available on the Azure and AWS marketplaces. Do get in touch if you're interested!

The feature set of the NannyML open source library has grown enormously, looking back at our humble "just covariate shift" beginnings. As we grew in functionality, we kept bumping into assumptions and abstractions that were no longer valid. We've decided to start a big refactor of the open source library to align with our new understanding of concepts and challenges that await us. This should allow us to pick up the pace and crank out new functionality far more efficiently. Stay tuned, we're very interested in feedback on our designs!

One reason for the redesign is our research into supporting NLP and computer vision use cases. Stay tuned for more content on those soon!

I'm very happy to finally release a new iteration of NannyML. Sorry for all the contributors that had to wait for a while, you are most appreciated!

As always, any feedback is encouraged! Reach out in our community Slack, log a bug or a feature request our repository or just leave us a star for some end-of-year cheer!

All the best,
Niels

[bgalvao]

tempted to work on it ngl 😄 thanks for your words!