[docs] [python] custom metric function in lgb.train() interface should not refer to the the passed dataset a train_data

[jameslamb]

Summary

This issue proposes changing the documentation for feval in lgb.train(), in the Python package, replacing train_data with evaluation_dataset.

Motivation

lgb.train() allows providing custom evaluation metrics defined in Python functions. According to the docs for lgb.train() (here), such functions should have the following signature.

Should accept two parameters: preds, train_data, and return (grad, hess).

train_data: Dataset
The training dataset.

I think it's confusing to refer to this as "The training dataset". In my opinion, that gives the impression that custom evaluation functions will have access to the data being used for training, which is not true. Instead, each custom evaluation function is provided a Dataset that evaluation has been request for, which might be either the training data or a separate validation dataset.

LightGBM/python-package/lightgbm/engine.py

Lines 296 to 299 in da98f24

	if valid_sets is not None:
	if is_valid_contain_train:
	evaluation_result_list.extend(booster.eval_train(feval))
	evaluation_result_list.extend(booster.eval_valid(feval))

LightGBM/python-package/lightgbm/basic.py

Lines 3240 to 3255 in da98f24

	def eval_valid(self, feval=None):
	"""Evaluate for validation data.
	Parameters
	----------
	feval : callable or None, optional (default=None)
	Customized evaluation function.
	Should accept two parameters: preds, valid_data,
	and return (eval_name, eval_result, is_higher_better) or list of such tuples.
	preds : list or numpy 1-D array
	The predicted values.
	If ``fobj`` is specified, predicted values are returned before any transformation,
	e.g. they are raw margin instead of probability of positive class for binary task in this case.
	valid_data : Dataset
	The validation dataset.

LightGBM/python-package/lightgbm/basic.py

Lines 3271 to 3272 in da98f24

	return [item for i in range(1, self.__num_dataset)
	for item in self.__inner_eval(self.name_valid_sets[i - 1], i, feval)]

Description

Specifically, I'm proposing the following changes:

• change the following to evaluation_data, not valid_data

  LightGBM/python-package/lightgbm/basic.py

  Lines 3254 to 3255 in da98f24

  	valid_data : Dataset
  	The validation dataset.

• change the following to evaluation_data, not train_data

  LightGBM/python-package/lightgbm/engine.py

  Lines 88 to 89 in da98f24

  	train_data : Dataset
  	The training dataset.

References

Noticed this while working on #4679 (comment).

[jameslamb]

@StrikerRUS @jmoralez would you support a change like this?

[shiyu1994]

I agree with the changes.
In fact, the description of feval seems just copied from the description of fobj, which is used only by training dataset. And this line should also be changed:

LightGBM/python-package/lightgbm/engine.py

Line 81 in da98f24

Each evaluation function should accept two parameters: preds, train_data,

[StrikerRUS]

Generally, I support this change. But maybe use eval_data name instead of evaluation_data for the consistency with other similar names like eval_valid, eval_train, eval_name, eval_result, etc.?

[akshitadixit]

Is this available to work on?

[jameslamb]

@akshitadixit sure! We'd welcome the help!

Please see @StrikerRUS 's comment above about using the new name eval_data: #4759 (comment)

[akshitadixit]

Sure!

[akshitadixit]

Hi @jameslamb and @StrikerRUS i noticed something I thought of confirming from you.

I have been asked to change valid_data to eval_data in the eval_valid() function

LightGBM/python-package/lightgbm/basic.py

Lines 3240 to 3256 in da98f24

	def eval_valid(self, feval=None):
	"""Evaluate for validation data.
	Parameters
	----------
	feval : callable or None, optional (default=None)
	Customized evaluation function.
	Should accept two parameters: preds, valid_data,
	and return (eval_name, eval_result, is_higher_better) or list of such tuples.
	preds : list or numpy 1-D array
	The predicted values.
	If ``fobj`` is specified, predicted values are returned before any transformation,
	e.g. they are raw margin instead of probability of positive class for binary task in this case.
	valid_data : Dataset
	The validation dataset.
	eval_name : str

whereas there is a similar instance with train_data in the eval_train() function too as was pointed out for engine.py:

LightGBM/python-package/lightgbm/basic.py

Lines 3207 to 3223 in da98f24

	def eval_train(self, feval=None):
	"""Evaluate for training data.
	Parameters
	----------
	feval : callable or None, optional (default=None)
	Customized evaluation function.
	Should accept two parameters: preds, train_data,
	and return (eval_name, eval_result, is_higher_better) or list of such tuples.
	preds : list or numpy 1-D array
	The predicted values.
	If ``fobj`` is specified, predicted values are returned before any transformation,
	e.g. they are raw margin instead of probability of positive class for binary task in this case.
	train_data : Dataset
	The training dataset.
	eval_name : str

Kindly clarify this,

[jameslamb]

@akshitadixit sorry for the delayed response.

Yes, please also change those places to

Should accept two parameters: preds, eval_data

[github-actions]

This issue has been automatically locked since there has not been any recent activity since it was closed.
To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues
including a reference to this.