lightgbm is killed for extreme parameter values

[geoHeil]

During hyper parameter optimization a wide range of parameters is tried. Some might be considered extreme values. For some of these lightGBM is killed. I think this is a segmentation fault. I would like to catch the thing. It is not an exception. So far I could not catch the signal. I want to handle the signal somehow and send the. failure to the optimization code.

Environment info

Operating System: osx 10.12.5
CPU: MacBook Pro (15-inch, 2017) / 3,1 GHz Intel Core i7
C++/Python/R version: latest from GitHub; python

Error Message:

[LightGBM] [Warning] Unknown parameter categorical_column=
[1]    10157 killed

for suggested parameters of optimization process of:

"bagging_fraction": 0.3,
  "learning_rate": 0.3,
  "max_bin": 20000,
  "max_depth": 50,
  "min_child_weight": 2000,
  "n_estimators": 10,
  "num_leaves": 20000,
  "scale_pos_weight": 0.01,
  "subsample": 1.0

and fixed parameters of:

'boosting_type': 'dart', 'reg_alpha': 0, 'reg_lambda': 2, 'is_unbalance': True,
                             'min_split_gain': 0, 'min_child_samples': 10, 'colsample_bytree': 0.8, 'subsample_freq': 3,
                             'subsample_for_bin': 50000

[guolinke]

@geoHeil
I think it is caused by memory overhead : https://stackoverflow.com/questions/726690/who-killed-my-process-and-why
The memory consumption may is large in some settings.

you can try to set the parameter histogram_pool_size, whose unit is "MB" .

[geoHeil]

I will try this, but assuming it might still fail - is there a way. I can catch this like an exception and report a failure to the optimization process?

edit

The following snippet is trying to handle the signals

for sig in (SIGABRT, SIGILL, SIGINT, SIGSEGV, SIGTERM):
            signal(sig, sig_handler)

Where sig_handler should report the failure to the optimization process

@guolinke how can limiting the memory by the suggested parameter help? Will lightGBM then perform some mini batch operation if the memory is deliberately limited?

[geoHeil]

#562 is an interesting read and histogram_pool_size is the caching for the calculated histogram values.

[Laurae2]

@geoHeil do not use such large parameter value as they are unpractical (and never yielding good performance especially in hyperparameter tuning, other than slowing down tuning):

"num_leaves": 20000

It is so big it will require a massive amount of RAM. Even 4095 (2^12 - 1) is already a lot and too much for most cases. For instance, I expect a 150GB dataset in RAM to require hundred of TBs with such number of leaves.

So far I could not catch the signal. I want to handle the signal somehow and send the. failure to the optimization code.

try:
  my_learning_task()
except: 
  my_fallback_task()

does not work in Python for LightGBM?

[geoHeil]

Thanks.

Simply catching the exception is not enough for me. A combination of

• reducing the search space as suggested
• setting histogram_pool_size
• catching signals
  seems to do the job.

[github-actions]

This issue has been automatically locked since there has not been any recent activity since it was closed. To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues including a reference to this.