Improve user method of seeing pipelines generated

[eddiebergman]

Currently, the easiest way for a user to see the pipelines included in the ensemble is through estimator.show_models() which just returns a str which needs to be manually parsed and looked through. There could definitely be a nicer format to view any such pipeline and provide easy access.

[eddiebergman]

If anyone would like to give a look at this, it would require looking through how show_models generates the string and then returning the object version of that, maybe with a nicer interface to how the returned object is accessed.

[eddiebergman]

Some relevant issues that explain methods of seeing pipelines:

• [Question] How to get individual models from the ensemble? #1224
• Identify the best model #1206
• [Question] Accessing particular models found by autosklearn #1252
• How to get the classifier with parameters and results of the best models? #1116
• Pipeline Export #952
• Analyzing the top K cross-validated models #709

[eddiebergman]

From #1224, how to access individual pipeline steps

from autosklearn.pipeline.util import get_dataset
from autosklearn.classification import AutoSklearnClassifier

X_train, y_train, X_test, y_test = get_dataset('iris')
automodel = AutoSklearnClassifier(time_left_for_this_task=60)
automodel.fit(X_train, y_train)

# A list of pipelines with their weights [ (ensemble_weight, Pipeline) ]
models_with_weights = automodel.get_models_with_weights() 

# Get the first model with it's weight
weight, model = models_with_weights[0]

# Note that these models and the following are sklearn compatible
# The steps in the models pipeline
# [ 
#    ('data_preprocessing', DataPreprocessor),
#    ('balancing', Balancing ),
#    ('feature_preprocessor', FeaturePreprocessorChoice)
#    ('classifier', ClassifierChoice)
# ]
models_steps = model.steps

# Get the ClassifierChoice wrapper
classifier_str, classifier = model.steps[-1]

# The autosklearn wrapped model 
classifier = classifier.choice 
print(type(classifier)) # autosklearn.pipeline.components.classification.random_forest.RandomForest

# The sklearn model
sklearn_classifier = classifier.estimator

[eddiebergman]

From #1206, how to access a specific model in the ensemble

import sklearn
from sklearn import datasets
from autosklearn.classification import AutoSklearnClassifier

X, y = datasets.load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = sklearn.model_selection.train_test_split(X, y, random_state=1)

clf = AutoSklearnClassifier(time_left_for_this_task=120, per_run_time_limit=30)
clf.fit(X_train, y_train)

# Can find model id's from the `leaderboard()` function
wanted_model_id = ...
wanted_model = None

for (seed, model_id, budget), model in clf.automl_.models_.items():
    if model_id == wanted_model_id:
        wanted_model = model

[sagar-kaushik]

Hello! This is a really great project, I recently used Auto-Sklearn in one of my projects and it really helped me. Thank you!

But I faced the same issue of manually parsing the str output of show_models() method. I am actually looking to start contributing to open source and I would like to work on this issue.

Can you tell me what you think would be a good solution for this? Taking inspiration from the code snippets you have shared, I have come up with the following idea:

• Create a dictionary for the whole ensemble where rank of the model is the key.
• The ensemble dictionary has dictionaries for all the pipelines which contains:
  • ensemble_weight
  • balancing
  • data_preprocessor
  • feature_preprocessor
  • classifer or regressor (auto sklearn wrapped model)
  • sklearn_model

I have implemented it using the code you had shared and for reference, it would look something like this image:

What do you think about this? I will create a PR after editing the show_models function if you approve.

[eddiebergman]

Hi @userfindingself,

Thanks for the kind words and wanting to contribute :) This looks like it would be quite useful for sure! I would be happy to help with a PR for this if you would like to share.

The only thing I would change is perhaps using the run_id as the key to the dictionary instead of rank, as this is more of a unique key to identify a model, leaving the rank to be inside the dictionary. Having the rank as the dict key means that the whole dict may as well be a list where the order identifies rank.

In the future moving forward, where we would like easier access to models, the run_id is a better indicator than rank. In a hypothetical situation where we allow checkpointing and resuming, the rank is likely to change which would lead to issues.

[sagar-kaushik]

Thank you so much for your help! Your suggestion definitely makes more sense.

Can you confirm if run_id is same as model_id in the table created by the leaderboard() method, or something else? I would then create model_id as the key and add rank to the model dictionary instead. If it's something else, can you please point to where I might find the run_id?

Thanks again! :)

[eddiebergman]

Yup that's the one! For context, the optimizer SMAC gives each "run" a number but for us, a "run" corresponds to a model configuration that is trained, hence it makes sense to present it as model_id, sorry for the confusion.

# autosklearn/estimators.py 680
{
    'model_id': rval.additional_info['num_run'],
}

[sagar-kaushik]

Great! Just one more question. Is it okay to use leaderboard() to get 'rank' and 'model_id', or should I write independent code for that inside the show_models() function?