Reimplement lambdamart ndcg.

* Simplify the implementation for both CPU and GPU.

Fix JSON IO.

Check labels.

Put idx into cache.

Optimize.

File tag.

Weights.

Trivial tests.

Compatibility.

Lint.

Fix swap.

Device weight.

tidy.

Easier to read R failure.

enum.

Fix global configuration.

Tidy.

msvc omp.

dask.

Remove ndcg specific parameter.  Drop label type for smaller PR.

Fix rebase.

Fixes.

Don't mess with includes.

Fixes.

Format.

Use omp util.

Restore some old code.

Revert.

Port changes from the work on quantile loss.

python binding.

param.

Cleanup.

conditional parallel.

types.

Move doc.

fix.

need metric rewrite.

rename ctx.

extract.

Work on metric.

Metric

Init estimation.

extract tests, compute ties.

cleanup.

notes.

extract optional weights.

init.

cleanup.

old metric format.

note.

ndcg cache.

nested.

debug.

fix.

log2.

Begin CUDA work.

temp.

Extract sort and latest cuda.

truncation.

dcg.

dispatch.

try different gain type.

start looking into ub.

note.

consider writing a doc.

check exp gain.

Reimplement lambdamart ndcg.

* Simplify the implementation for both CPU and GPU.

Fix JSON IO.

Check labels.

Put idx into cache.

Optimize.

File tag.

Weights.

Trivial tests.

Compatibility.

Lint.

Fix swap.

Device weight.

tidy.

Easier to read R failure.

enum.

Fix global configuration.

Tidy.

msvc omp.

dask.

Remove ndcg specific parameter.  Drop label type for smaller PR.

Fix rebase.

Fixes.

Don't mess with includes.

Fixes.

Format.

Use omp util.

Restore some old code.

Revert.

Port changes from the work on quantile loss.

python binding.

param.

Cleanup.

conditional parallel.

types.

Move doc.

fix.

need metric rewrite.

rename ctx.

extract.

Work on metric.

Metric

Init estimation.

extract tests, compute ties.

cleanup.

notes.

extract optional weights.

init.

cleanup.

old metric format.

note.

ndcg cache.

nested.

debug.

fix.

log2.

Begin CUDA work.

temp.

Extract sort and latest cuda.

truncation.

dcg.

dispatch.

try different gain type.

start looking into ub.

note.

consider writing a doc.

check exp gain.

Start looking into unbiased.

lambda.

Extract the ndcg cache.

header.

cleanup namespace.

small check.

namespace.

init with param.

gain.

extract.

groups.

Cleanup.

disable.

debug.

remove.

Revert "remove."

This reverts commit ea025f9.

sigmoid.

cleanup.

metric name.

check scores.

note.

check map.

extract utilities.

avoid inline.

fix.

header.

extract more.

note.

note.

note.

start working on map.

fix.

continue map.

map.

matrix.

Remove map.

note.

format.

move check.

cleanup.

use cached discount, use double.

cleanup.

Add position to the Python interface.

pass it into lambda.

Full ratio.

rank.

comment.

some work on GPU.

compile.

move cache initialization.

descending.

Fix arg sort.

basic ndcg score.

metric weight.

config.

extract.

pass position again.

Define a metric decorator.

position.

decorate metric..

return.

note.

irrelevant docs.

fix weights.

header.

Share the bias.

Use position

check info.

use cache for param.

note.

prepare to work on deterministic gpu.

rounding.

Extract op.

cleanup.

Use it.

check label.

ditch launchn.

rounding.

Move rounding into cache.

fix check label.

GPU fixes.

Irrelevant doc.

try to avoid inf.

mad.

Work on metric cache.

Cleanup sort.

use cache.

cache others.

revert.

add test for metric.

fixes.

msg.

note.

remove reduce by key.

comments.

check position.

stream.

min.

small cleanup.

use atomic for now.

fill.

no inline.

norm.

remove op.

start gpu.

cleanup.

use gpu for update.

segmented reduce.

revert.

comments.

comments.

fix.

comments.

fix bounds.

comments.

cache.

pointer.

fixes.

no spark.

revert.

Cleanup.

cleanup.

work on gain type.

fix.

notes.

make metric name.

remove.

revert.

revert.

comment.

revert.

Move back into rank metric.

Set name in objective.

fix.

Don't configure.

note.

merge tests.

accept empty group.

fixes.

float.

revert and fix.

not mutable.

prototype for cache.

extract.

convert to DMatrix.

cache.

Extract the cache.

Port changes.

fix & cleanup.

cleanup.

cleanup.

Rename.

restore.

remove.

header.

revert.

rename.

rename.

doc.

cleanup.

doc.

cleanup.

tests.

tests.

split up.

jvm parameters.

doc.

Fix.

Use cache in cox.

Revert "Use cache in cox."

This reverts commit e1cec37.

Remove pairwise.

iwyu.

rename.

Move.

Merge.

ranking utils.

Fixes.

rename.

Comments.

todos.

Small cleanup.

doc.

Start working on demo.

move some code here.

rename.

Update doc.

Update doc.

Work on demo.

work on demo.

demo.

Demo.

Specify the max rel degree.

remove position.

Fix.

Work on demo.

demo.

Using only one fold.

cache.

demo.

schema.

comments.

Lint.

fix test.

automake.

macos.

schema.

test.

schema.

lint.

fix tests.

Implement MAP and pair sampling.

revert sorting.

Work on ranknet.

remove.

Don't upgrade cost if larger than.

Extract GPU make pairs.

error message.

Remove.

Cleanup some gpu tests.

Move.

Move NDCG test.

fix weights.

Move rest of the tests.

Remove.

Work on tests.

fixes.

Cleanup.

header.

cleanup.

Update document.

update document.

fix build.

cpplint.

rename.

Fixes and cleanup.

Cleanup tests.

lint.

fix tests.

debug macos non-openmp checks.

macos.

fix ndcg test.

Ensure number of threads is smaller than the number of inputs.

fix.

Debug macos.

fixes.

Add weight normalization.

Note on reproducible result.

Don't normalize if it's binary.

old ctk.

Use old objective.

Update doc.

Convert pyspark tests.

black.

Fix rebase.

Fix rebase.

Start looking into CV.

Hacky score function.

extract parsing.

Cleanup and tests.

Lint & note.

test check.

Update document.

Update tests & doc.

Support custom metric as well.

c++-17.

cleanup old metrics.

rename.

Fixes.

Fix cxx test.

test cudf.

start converting tests.

pylint.

fix data load.

Cleanup the tests.

Parameter tests.

isort.

Fix test.

Specify src path for isort.

17 goodies.

Fix rebase.

Start working on ranking cache tests.

Extract CPU impl.

test debiasing.

use index.

ranking cache.

comment.

some work on debiasing.

save the estimated bias.

normalize by default.

GPU norm.

fix gpu unbiased.

cleanup.

cleanup.

Remove workaround.

Default to topk.

Restore.

Cleanup.

Revert change in algorithm.

norm.

Move data generation process in testing for reuse.

Move sort samples as well.

cleanup.

Generate data.

lint.

pylint.

Fix.

Fix spark test.

avoid sampling with unbiased.

Cleanup demo.

Handle single group simulation.

Numeric issues.

More numeric issues.

sigma.

naming.

Simple test.

tests.

brief description.

Revert "brief description."

This reverts commit 0b3817a.

rebase.

symbol.

Rebase.

disable normalization.

Revert "disable normalization."

This reverts commit ef3133d2b4a76714f3514808c6e2ae5937e6a8c2.

R-package/src/Makevars.in

@@ -32,7 +32,6 @@ OBJECTS= \
     $(PKGROOT)/src/objective/objective.o \
     $(PKGROOT)/src/objective/regression_obj.o \
     $(PKGROOT)/src/objective/multiclass_obj.o \
-    $(PKGROOT)/src/objective/rank_obj.o \
     $(PKGROOT)/src/objective/lambdarank_obj.o \
     $(PKGROOT)/src/objective/hinge.o \
     $(PKGROOT)/src/objective/aft_obj.o \

R-package/src/Makevars.win

@@ -32,7 +32,6 @@ OBJECTS= \
     $(PKGROOT)/src/objective/objective.o \
     $(PKGROOT)/src/objective/regression_obj.o \
     $(PKGROOT)/src/objective/multiclass_obj.o \
-    $(PKGROOT)/src/objective/rank_obj.o \
     $(PKGROOT)/src/objective/lambdarank_obj.o \
     $(PKGROOT)/src/objective/hinge.o \
     $(PKGROOT)/src/objective/aft_obj.o \

demo/guide-python/learning_to_rank.py

@@ -0,0 +1,210 @@
+"""
+Getting started with learning to rank
+=====================================
+
+  .. versionadded:: 2.0.0
+
+This is a demonstration of using XGBoost for learning to rank tasks using the
+MSLR_10k_letor dataset. For more infomation about the dataset, please visit its
+`description page <https://www.microsoft.com/en-us/research/project/mslr/>`_.
+
+This is a two-part demo, the first one contains a basic example of using XGBoost to
+train on relevance degree, and the second part simulates click data and enable the
+position debiasing training.
+
+For an overview of learning to rank in XGBoost, please see
+:doc:`Learning to Rank </tutorials/learning_to_rank>`.
+"""
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import pickle as pkl
+
+import numpy as np
+import pandas as pd
+from sklearn.datasets import load_svmlight_file
+
+import xgboost as xgb
+from xgboost.testing.data import RelDataCV, simulate_clicks, sort_ltr_samples
+
+
+def load_mlsr_10k(data_path: str, cache_path: str) -> RelDataCV:
+    """Load the MSLR10k dataset from data_path and cache a pickle object in cache_path.
+
+    Returns
+    -------
+
+    A list of tuples [(X, y, qid), ...].
+
+    """
+    root_path = os.path.expanduser(args.data)
+    cacheroot_path = os.path.expanduser(args.cache)
+    cache_path = os.path.join(cacheroot_path, "MSLR_10K_LETOR.pkl")
+
+    # Use only the Fold1 for demo:
+    # Train,      Valid, Test
+    # {S1,S2,S3}, S4,    S5
+    fold = 1
+
+    if not os.path.exists(cache_path):
+        fold_path = os.path.join(root_path, f"Fold{fold}")
+        train_path = os.path.join(fold_path, "train.txt")
+        valid_path = os.path.join(fold_path, "vali.txt")
+        test_path = os.path.join(fold_path, "test.txt")
+        X_train, y_train, qid_train = load_svmlight_file(
+            train_path, query_id=True, dtype=np.float32
+        )
+        y_train = y_train.astype(np.int32)
+        qid_train = qid_train.astype(np.int32)
+
+        X_valid, y_valid, qid_valid = load_svmlight_file(
+            valid_path, query_id=True, dtype=np.float32
+        )
+        y_valid = y_valid.astype(np.int32)
+        qid_valid = qid_valid.astype(np.int32)
+
+        X_test, y_test, qid_test = load_svmlight_file(
+            test_path, query_id=True, dtype=np.float32
+        )
+        y_test = y_test.astype(np.int32)
+        qid_test = qid_test.astype(np.int32)
+
+        data = RelDataCV(
+            train=(X_train, y_train, qid_train),
+            test=(X_test, y_test, qid_test),
+            max_rel=4,
+        )
+
+        with open(cache_path, "wb") as fd:
+            pkl.dump(data, fd)
+
+    with open(cache_path, "rb") as fd:
+        data = pkl.load(fd)
+
+    return data
+
+
+def ranking_demo(args: argparse.Namespace) -> None:
+    """Demonstration for learning to rank with relevance degree."""
+    data = load_mlsr_10k(args.data, args.cache)
+
+    X_train, y_train, qid_train = data.train
+    sorted_idx = np.argsort(qid_train)
+    X_train = X_train[sorted_idx]
+    y_train = y_train[sorted_idx]
+    qid_train = qid_train[sorted_idx]
+
+    X_test, y_test, qid_test = data.test
+    sorted_idx = np.argsort(qid_test)
+    X_test = X_test[sorted_idx]
+    y_test = y_test[sorted_idx]
+    qid_test = qid_test[sorted_idx]
+
+    ranker = xgb.XGBRanker(
+        tree_method="gpu_hist",
+        lambdarank_pair_method="topk",
+        lambdarank_num_pair_per_sample=13,
+        eval_metric=["ndcg@1", "ndcg@8"],
+    )
+    ranker.fit(
+        X_train,
+        y_train,
+        qid=qid_train,
+        eval_set=[(X_test, y_test)],
+        eval_qid=[qid_test],
+        verbose=True,
+    )
+
+
+def click_data_demo(args: argparse.Namespace) -> None:
+    """Demonstration for learning to rank with click data."""
+    data = load_mlsr_10k(args.data, args.cache)
+    folds = simulate_clicks(data)
+
+    train = [pack[0] for pack in folds]
+    test = [pack[1] for pack in folds]
+
+    X_train, y_train, qid_train, scores_train, clicks_train, position_train = train
+    assert X_train.shape[0] == clicks_train.size
+    X_test, y_test, qid_test, scores_test, clicks_test, position_test = test
+    assert X_test.shape[0] == clicks_test.size
+    assert scores_test.dtype == np.float32
+    assert clicks_test.dtype == np.int32
+
+    X_train, clicks_train, y_train, qid_train = sort_ltr_samples(
+        X_train,
+        y_train,
+        qid_train,
+        clicks_train,
+        position_train,
+    )
+    X_test, clicks_test, y_test, qid_test = sort_ltr_samples(
+        X_test,
+        y_test,
+        qid_test,
+        clicks_test,
+        position_test,
+    )
+
+    class ShowPosition(xgb.callback.TrainingCallback):
+        def after_iteration(self, model, epoch, evals_log) -> bool:
+            config = json.loads(model.save_config())
+            ti_plus = np.array(config["learner"]["objective"]["ti+"])
+            tj_minus = np.array(config["learner"]["objective"]["tj-"])
+            df = pd.DataFrame({"ti+": ti_plus, "tj-": tj_minus})
+            print(df)
+            return False
+
+    ranker = xgb.XGBRanker(
+        n_estimators=512,
+        tree_method="gpu_hist",
+        learning_rate=0.01,
+        reg_lambda=1.5,
+        subsample=0.8,
+        sampling_method="gradient_based",
+        # LTR specific parameters
+        objective="rank:ndcg",
+        # - Enable bias estimation
+        lambdarank_unbiased=True,
+        # - normalization (1 / (norm + 1))
+        lambdarank_bias_norm=1,
+        # - Focus on the top 12 documents
+        lambdarank_num_pair_per_sample=12,
+        lambdarank_pair_method="topk",
+        ndcg_exp_gain=True,
+        eval_metric=["ndcg@1", "ndcg@3", "ndcg@5", "ndcg@10"],
+        callbacks=[ShowPosition()],
+    )
+    ranker.fit(
+        X_train,
+        clicks_train,
+        qid=qid_train,
+        eval_set=[(X_test, y_test), (X_test, clicks_test)],
+        eval_qid=[qid_test, qid_test],
+        verbose=True,
+    )
+    ranker.predict(X_test)
+
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser(
+        description="Demonstration of learning to rank using XGBoost."
+    )
+    parser.add_argument(
+        "--data",
+        type=str,
+        help="Root directory of the MSLR-WEB10K data.",
+        required=True,
+    )
+    parser.add_argument(
+        "--cache",
+        type=str,
+        help="Directory for caching processed data.",
+        required=True,
+    )
+    args = parser.parse_args()
+
+    ranking_demo(args)
+    click_data_demo(args)

doc/contrib/coding_guide.rst

@@ -16,8 +16,10 @@ C++ Coding Guideline
   * Each line of text may contain up to 100 characters.
   * The use of C++ exceptions is allowed.
 
-- Use C++11 features such as smart pointers, braced initializers, lambda functions, and ``std::thread``.
+- Use C++14 features such as smart pointers, braced initializers, lambda functions, and ``std::thread``.
 - Use Doxygen to document all the interface code.
+- We have some comments around symbols imported by headers, some of those are hinted by `include-what-you-use <https://include-what-you-use.org>`_. It's not required.
+- We use clang-tidy and clang-format. You can check their configuration in the root directory of the XGBoost source tree.
 - We have a series of automatic checks to ensure that all of our codebase complies with the Google style. Before submitting your pull request, you are encouraged to run the style checks on your machine. See :ref:`running_checks_locally`.
 
 ***********************

doc/model.schema

@@ -219,6 +219,16 @@
         "num_pairsample": { "type": "string" },
         "fix_list_weight": { "type": "string" }
       }
+    },
+    "lambdarank_param": {
+      "type": "object",
+      "properties": {
+        "lambdarank_num_pair_per_sample": { "type": "string" },
+        "lambdarank_pair_method": { "type": "string" },
+        "lambdarank_unbiased": {"type": "string" },
+        "lambdarank_bias_norm": {"type": "string" },
+        "ndcg_exp_gain": {"type": "string"}
+      }
     }
   },
   "type": "object",
@@ -477,22 +487,22 @@
               "type": "object",
               "properties": {
                 "name": { "const": "rank:pairwise" },
-                "lambda_rank_param": { "$ref": "#/definitions/lambda_rank_param"}
+                "lambda_rank_param": { "$ref": "#/definitions/lambdarank_param"}
               },
               "required": [
                 "name",
-                "lambda_rank_param"
+                "lambdarank_param"
               ]
             },
             {
               "type": "object",
               "properties": {
                 "name": { "const": "rank:ndcg" },
-                "lambda_rank_param": { "$ref": "#/definitions/lambda_rank_param"}
+                "lambda_rank_param": { "$ref": "#/definitions/lambdarank_param"}
               },
               "required": [
                 "name",
-                "lambda_rank_param"
+                "lambdarank_param"
               ]
             },
             {

doc/parameter.rst

@@ -233,7 +233,7 @@ Parameters for Tree Booster
   .. note:: This parameter is working-in-progress.
 
   - The strategy used for training multi-target models, including multi-target regression
-  and multi-class classification. See :doc:`/tutorials/multioutput` for more information.
+    and multi-class classification. See :doc:`/tutorials/multioutput` for more information.
 
     - ``one_output_per_tree``: One model for each target.
     - ``multi_output_tree``:  Use multi-target trees.
@@ -380,9 +380,9 @@ Specify the learning task and the corresponding learning objective. The objectiv
     See :doc:`/tutorials/aft_survival_analysis` for details.
   - ``multi:softmax``: set XGBoost to do multiclass classification using the softmax objective, you also need to set num_class(number of classes)
   - ``multi:softprob``: same as softmax, but output a vector of ``ndata * nclass``, which can be further reshaped to ``ndata * nclass`` matrix. The result contains predicted probability of each data point belonging to each class.
-  - ``rank:pairwise``: Use LambdaMART to perform pairwise ranking where the pairwise loss is minimized
-  - ``rank:ndcg``: Use LambdaMART to perform list-wise ranking where `Normalized Discounted Cumulative Gain (NDCG) <http://en.wikipedia.org/wiki/NDCG>`_ is maximized
-  - ``rank:map``: Use LambdaMART to perform list-wise ranking where `Mean Average Precision (MAP) <http://en.wikipedia.org/wiki/Mean_average_precision#Mean_average_precision>`_ is maximized
+  - ``rank:ndcg``: Use LambdaMART to perform pair-wise ranking where `Normalized Discounted Cumulative Gain (NDCG) <http://en.wikipedia.org/wiki/NDCG>`_ is maximized. This objective supports position debiasing for click data.
+  - ``rank:map``: Use LambdaMART to perform pair-wise ranking where `Mean Average Precision (MAP) <http://en.wikipedia.org/wiki/Mean_average_precision#Mean_average_precision>`_ is maximized
+  - ``rank:pairwise``: Use LambdaRank to perform pair-wise ranking using the `ranknet` objective.
   - ``reg:gamma``: gamma regression with log-link. Output is a mean of gamma distribution. It might be useful, e.g., for modeling insurance claims severity, or for any outcome that might be `gamma-distributed <https://en.wikipedia.org/wiki/Gamma_distribution#Occurrence_and_applications>`_.
   - ``reg:tweedie``: Tweedie regression with log-link. It might be useful, e.g., for modeling total loss in insurance, or for any outcome that might be `Tweedie-distributed <https://en.wikipedia.org/wiki/Tweedie_distribution#Occurrence_and_applications>`_.
 
@@ -395,8 +395,9 @@ Specify the learning task and the corresponding learning objective. The objectiv
 
 * ``eval_metric`` [default according to objective]
 
-  - Evaluation metrics for validation data, a default metric will be assigned according to objective (rmse for regression, and logloss for classification, mean average precision for ranking)
-  - User can add multiple evaluation metrics. Python users: remember to pass the metrics in as list of parameters pairs instead of map, so that latter ``eval_metric`` won't override previous one
+  - Evaluation metrics for validation data, a default metric will be assigned according to objective (rmse for regression, and logloss for classification, `mean average precision` for ``rank:map``, etc.)
+  - User can add multiple evaluation metrics. Python users: remember to pass the metrics in as list of parameters pairs instead of map, so that latter ``eval_metric`` won't override previous ones
+
   - The choices are listed below:
 
     - ``rmse``: `root mean square error <http://en.wikipedia.org/wiki/Root_mean_square_error>`_
@@ -480,6 +481,36 @@ Parameter for using AFT Survival Loss (``survival:aft``) and Negative Log Likeli
 
 * ``aft_loss_distribution``: Probability Density Function, ``normal``, ``logistic``, or ``extreme``.
 
+.. _ltr-param:
+
+Parameters for learning to rank (``rank:ndcg``, ``rank:map``, ``rank:pairwise``)
+================================================================================
+
+These are parameters specific to learning to rank task. See :doc:`Learning to Rank </tutorials/learning_to_rank>` for an in-depth explanation.
+
+* ``lambdarank_pair_method`` [default = ``mean``]
+
+  How to construct pairs for pair-wise learning.
+
+  - ``mean``: Sample ``lambdarank_num_pair_per_sample`` pairs for each document in the query list.
+  - ``topk``: Focus on top-``lambdarank_num_pair_per_sample`` documents. Construct :math:`|query|` pairs for each document at the top-``lambdarank_num_pair_per_sample`` ranked by the model.
+
+* ``lambdarank_num_pair_per_sample`` [range = :math:`[1, \infty]`]
+
+  It specifies the number of pairs sampled for each document when pair method is ``mean``, or the truncation level for queries when the pair method is ``topk``. For example, to train with ``ndcg@6``, set ``lambdarank_num_pair_per_sample`` to :math:`6` and ``lambdarank_pair_method`` to ``topk``.
+
+* ``lambdarank_unbiased`` [default = ``false``]
+
+  Specify whether do we need to debias input click data.
+
+* ``lambdarank_bias_norm`` [default = 2.0]
+
+  :math:`L_p` normalization for position debiasing, default is :math:`L_2`. Only relevant when ``lambdarank_unbiased`` is set to true.
+
+* ``ndcg_exp_gain`` [default = ``true``]
+
+  Whether we should use exponential gain function for ``NDCG``. There are two forms of gain function for ``NDCG``, one is using relevance value directly while the other is using :math:`2^{rel} - 1` to emphasize on retrieving relevant documents. When ``ndcg_exp_gain`` is true (the default), relevance degree cannot be greater than 31.
+
 ***********************
 Command Line Parameters
 ***********************

doc/tutorials/index.rst

@@ -21,6 +21,7 @@ See `Awesome XGBoost <https://github.com/dmlc/xgboost/tree/master/demo>`_ for mo
   monotonic
   rf
   feature_interaction_constraint
+  learning_to_rank
   aft_survival_analysis
   c_api_tutorial
   input_format