Hyperparameter Tuning in Lightning CLI

[ohayonguy]

I wonder how people do Hyperparameter Tuning with Lightning CLI?
Any suggestion of good practices?

Thanks!

[justusschock]

Personally when I tune hyperparameters (e.g. with optuna or nevergrad), I don't use the Lightning CLI much but use the programmatic way to inject the arguments there (since it's easier for communication across different python libs directly in python and not leaving it to os calls).

[ohayonguy]

I ended up using https://github.com/ashleve/lightning-hydra-template and performing Hyperparameter Tuning with the Optuna sweeper in Hydra, in combination with ray aws launcher. I didn't find a clean solution to do the same with Lightning CLI. Thanks!

[tchaton]

Dear @tshu-w,

I was a bit confused with your question. Are you talking using Trainer.tune or doing HPO of some parameters.

In the later case, here is something you could try out.

from pytorch_lightning.utilities.cli import LightningCLI
from unittest import mock
import optuna

config_path = ...

class MyModel(LightningModule):

    def __init__(self, num_layers):
        ...

def objective(trial):
    num_layers = trial.suggest_uniform('num_layers', 10, 100)

    with mock.patch("sys.argv", ["any.py", "--config", str(config_path), "--model.num_layers", str(num_layers)]):
        cli = LightningCLI(MyModel, MyDataModule)

    return cli.trainer.model_checkpoint.best_score

study = optuna.create_study()
study.optimize(objective, n_trials=100)
study.best_params

@carmocca Any thoughts ?

[tshu-w]

@tchaton Sorry, I was trying to answer the @ohayonguy 's question, but I seem to have misunderstood 😂

[carmocca]

I don't see a point in doing that.

If you need to hack the argv, then I don't see the reason why you wouldn't programatically create everything instead.
The CLI is meant to be used through the CLI (command line interface)