cumulative_dynamic_auc or for Gridseach

[felipe0216]

I could be wrong here, but so far as I have seen, you can only use the concordance_index as a scoring metric when using GridsearchCV, RFECV, or similar wrappers. If you try to specify the "cumulative_dynamic_auc" or "concordance_index_ipcw" you will get an error as more parameters neeed to be specify than your default Scikit learn scoring, hence, getting an error. It is very possible that there might be an existing solution for this, so I apologise if that is the case.

So, according to the documentation you can incorporate GridSeachCV the following way.

from sklearn.exceptions import ConvergenceWarning
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

coxnet_pipe = make_pipeline(
    StandardScaler(),
    CoxnetSurvivalAnalysis(l1_ratio=0.9, alpha_min_ratio=0.01, max_iter=100)
)
warnings.simplefilter("ignore", ConvergenceWarning)
coxnet_pipe.fit(Xt, y)

estimated_alphas = coxnet_pipe.named_steps["coxnetsurvivalanalysis"].alphas_
cv = KFold(n_splits=5, shuffle=True, random_state=0)
gcv = GridSearchCV(
    make_pipeline(StandardScaler(), CoxnetSurvivalAnalysis(l1_ratio=0.9)),
    param_grid={"coxnetsurvivalanalysis__alphas": [[v] for v in estimated_alphas]},
    cv=cv,
    error_score=0.5,
    n_jobs=4).fit(Xt, y)

cv_results = pd.DataFrame(gcv.cv_results_)

Ideally, it would be great to use those metrics like this:

cv = KFold(n_splits=5, shuffle=True, random_state=0)
gcv = GridSearchCV(
    make_pipeline(StandardScaler(), CoxnetSurvivalAnalysis(l1_ratio=0.9)),
    param_grid={"coxnetsurvivalanalysis__alphas": [[v] for v in estimated_alphas]},
    cv=cv,
    scorer = make_scorer(cumulative_dynamic_auc)
    error_score=0.5,
    n_jobs=4).fit(Xt, y)

cv_results = pd.DataFrame(gcv.cv_results_)

[sebp]

cumulative_dynamic_auc or concordance_index_ipcw can currently not be used with GridSearchCV out of the box. I discussed some possible solutions to this in another issue.

[felipe0216]

Hello @sebp ,

Thanks for your response, I have had al look at it, very useful although I didn't fully get it, if in the future you could include an example that would be great.

Lastly, scikit survival doesn't seem to support RFECV either, is that correct?

[sebp]

It is supported, but only for some models. For instance, RandomSurvivalForest does not work, as I explained in #164