Objectives and metrics

The sections below list the supported metrics for various machine learning problems.

  1. Common information
  2. Regression
  3. Classification
  4. Multiclassification
  5. Ranking
Note. The metrics that are not used for optimization can only be output.

Common information

Objectives and metrics can be calculated during the training or separately from the training for a specified model. The calculated values are written to files and can be plotted by visualization tools (both during and after the training) for further analysis.

User-defined parameters

Some objectives and metrics provide user-defined parameters. These parameters are set in the --loss-function (loss_function), --custom-metric (custom_metric) and --eval-metric (eval_metric) training options.

The parameters for each objective and metric are set in the following format:
<Metric>[:<parameter 1>=<value>;..;<parameter N>=<value>]
Usage examples
Quantile:alpha=0.1
Enable, disable and configure metrics calculation

The calculation of metrics can be resource-intensive. It creates a bottleneck in some cases, for example, if many metrics are calculated during the training or the computation is performed on GPU.

The training can be sped up by disabling the calculation of some metrics for the training dataset. Use the hints=skip_train~true parameter to disable the calculation of the specified metrics.

Note.

The calculation of some metrics is disabled by default for the training dataset to speed up the training. Use the hints=skip_train~false parameter to enable the calculation.

Metrics that are not calculated by default for the train dataset
  • PFound
  • YetiRank
  • NDCG
  • YetiRankPairwise
  • AUC
Usage examples
Enable the calculation of the AUC metric:
AUC:hints=skip_train~false
Disable the calculation of the Logloss metric:
Logloss:hints=skip_train~true
Another way to speed up the training is to set up the frequency of iterations to calculate the values of objectives and metrics. Use one of the following parameters:
CLI parametersPython parametersR parameters
--metric-periodmetric_periodmetric_period
For example, use the following parameter in Python or R to calculate metrics once per 50 iterations:
metric_period=50
Variables used
  •  is the label value for the i-th object (from the input data for training).
  •  is the result of applying the model to the i-th object.
  • is the predicted success probability
  •  is the total number of objects.
  • is the number of classes.
  • is the class of the object for binary classification.

  •  is the weight of the i-th object. It is set in the dataset description in columns with the Weight type (if otherwise is not stated) or in the sample_weight parameter of the Python package. The default is 1 for all objects.
  • , , , , are abbreviations for Positive, True Positive, True Negative, False Positive and False Negative.

    By default, , , , , use weights. For example,

  • is the array of pairs specified in the Pair description or in the pairs parameter of the Python package.
  • is the number of pairs for the Pairwise metrics.
  • is the value calculated using the resulting model for the winner object for the Pairwise metrics.
  • is the value calculated using the resulting model for the loser object for the Pairwise metrics.
  • is the weight of the (; ) pair for the Pairwise metrics.
  • is the array of object identifiers from the input dataset with a common GroupId. It is used to calculate theGroupwise metrics.
  • is the set of all arrays of identifiers from the input dataset with a common GroupId. It is used to calculate the Groupwise metrics.

Regression

NameUsed for optimizationUser-defined parametersFormula and/or description
MAE+
use_weights

Use object/group weights to calculate metrics if the specified value is “true” and set all weights to “1” regardless of the input data if the specified value is “false”. The default value is  “true”.

Note. Do not use the use_weights parameter if this metric is set as the loss function for training.
Calculation principles
MAPE+
use_weights

Use object/group weights to calculate metrics if the specified value is “true” and set all weights to “1” regardless of the input data if the specified value is “false”. The default value is  “true”.

Note. Do not use the use_weights parameter if this metric is set as the loss function for training.
Calculation principles
Poisson+
Calculation principles
Quantile+
alpha
The coefficient used in quantile-based losses. The default value is 0.5.
Calculation principles

Depends on the condition for the ratio of the label value and the resulting value:

RMSE+
Calculation principles
LogLinQuantile+
alpha
The coefficient used in quantile-based losses. The default value is 0.5.
Calculation principles

Depends on the condition for the ratio of the label value and the resulting value:

Lq+
q

The power coefficient.

This parameter is obligatory (the default value is not defined). Valid values are real numbers in the following range: 

Calculation principles
SMAPE

use_weights

Use object/group weights to calculate metrics if the specified value is “true” and set all weights to “1” regardless of the input data if the specified value is “false”. The default value is  “true”.

Note. Do not use the use_weights parameter if this metric is set as the loss function for training.
Calculation principles
R2
use_weights

Use object/group weights to calculate metrics if the specified value is “true” and set all weights to “1” regardless of the input data if the specified value is “false”. The default value is  “true”.

Calculation principles

is the average label value:

MSLE
use_weights

Use object/group weights to calculate metrics if the specified value is “true” and set all weights to “1” regardless of the input data if the specified value is “false”. The default value is  “true”.

Calculation principles
MedianAbsoluteError
Calculation principles

Classification

NameUsed for optimizationUser-defined parametersFormula and/or description
Logloss+
border
The label value border. If the value is strictly greater than this threshold, it is considered a positive class. Otherwise it is considered a negative class. The default value is 0.5.
Calculation principles
CrossEntropy+
use_weights

Use object/group weights to calculate metrics if the specified value is “true” and set all weights to “1” regardless of the input data if the specified value is “false”. The default value is  “true”.

Note. Do not use the use_weights parameter if this metric is set as the loss function for training.
Calculation principles
Precision
use_weights

Use object/group weights to calculate metrics if the specified value is “true” and set all weights to “1” regardless of the input data if the specified value is “false”. The default value is  “true”.

Calculation principles
Recall
use_weights

Use object/group weights to calculate metrics if the specified value is “true” and set all weights to “1” regardless of the input data if the specified value is “false”. The default value is  “true”.

Calculation principles

F1
use_weights

Use object/group weights to calculate metrics if the specified value is “true” and set all weights to “1” regardless of the input data if the specified value is “false”. The default value is  “true”.

Calculation principles

BalancedAccuracy
use_weights

Use object/group weights to calculate metrics if the specified value is “true” and set all weights to “1” regardless of the input data if the specified value is “false”. The default value is  “true”.

Calculation principles

BalancedErrorRate
use_weights

Use object/group weights to calculate metrics if the specified value is “true” and set all weights to “1” regardless of the input data if the specified value is “false”. The default value is  “true”.

Calculation principles
MCC
use_weights

Use object/group weights to calculate metrics if the specified value is “true” and set all weights to “1” regardless of the input data if the specified value is “false”. The default value is  “true”.

Calculation principles
Accuracy
use_weights

Use object/group weights to calculate metrics if the specified value is “true” and set all weights to “1” regardless of the input data if the specified value is “false”. The default value is  “true”.

Calculation principles

CtrFactor
use_weights

Use object/group weights to calculate metrics if the specified value is “true” and set all weights to “1” regardless of the input data if the specified value is “false”. The default value is  “true”.

Calculation principles
AUC *
use_weights

Use object/group weights to calculate metrics if the specified value is “true” and set all weights to “1” regardless of the input data if the specified value is “false”. The default value is “false”.

See the Wikipedia article.

BrierScore
Calculation principles
HingeLoss
use_weights

Use object/group weights to calculate metrics if the specified value is “true” and set all weights to “1” regardless of the input data if the specified value is “false”. The default value is  “true”.

Calculation principles
HammingLoss
use_weights

Use object/group weights to calculate metrics if the specified value is “true” and set all weights to “1” regardless of the input data if the specified value is “false”. The default value is  “true”.

Calculation principles
ZeroOneLoss
use_weights

Use object/group weights to calculate metrics if the specified value is “true” and set all weights to “1” regardless of the input data if the specified value is “false”. The default value is  “true”.

Calculation principles

Kappa
Calculation principles

WKappa
Calculation principles
LogLikelihoodOfPrediction
Calculation principles

Multiclassification

NameUsed for optimizationUser-defined parametersFormula and/or description
MultiClass+
use_weights

Use object/group weights to calculate metrics if the specified value is “true” and set all weights to “1” regardless of the input data if the specified value is “false”. The default value is  “true”.

Note. Do not use the use_weights parameter if this metric is set as the loss function for training.
Calculation principles

MultiClassOneVsAll+
use_weights

Use object/group weights to calculate metrics if the specified value is “true” and set all weights to “1” regardless of the input data if the specified value is “false”. The default value is  “true”.

Note. Do not use the use_weights parameter if this metric is set as the loss function for training.
Calculation principles

Precision
use_weights

Use object/group weights to calculate metrics if the specified value is “true” and set all weights to “1” regardless of the input data if the specified value is “false”. The default value is  “true”.

This function is calculated separately for each class k numbered from 0 to M – 1.

Calculation principles

Recall
use_weights

Use object/group weights to calculate metrics if the specified value is “true” and set all weights to “1” regardless of the input data if the specified value is “false”. The default value is  “true”.

This function is calculated separately for each class k numbered from 0 to M – 1.

Calculation principles

F1
use_weights

Use object/group weights to calculate metrics if the specified value is “true” and set all weights to “1” regardless of the input data if the specified value is “false”. The default value is  “true”.

This function is calculated separately for each class k numbered from 0 to M – 1.

Calculation principles

TotalF1
use_weights

Use object/group weights to calculate metrics if the specified value is “true” and set all weights to “1” regardless of the input data if the specified value is “false”. The default value is  “true”.

Calculation principles

is the sum of the weights of the documents which correspond to the k class. If document weights are not specified stands for the number of times the k class, which corresponds to the i-th object, is found among the label values.

MCC
use_weights

Use object/group weights to calculate metrics if the specified value is “true” and set all weights to “1” regardless of the input data if the specified value is “false”. The default value is  “true”.

Calculation principles

This functions is defined in terms of a confusion matrix (where k is the number of classes):

See the Wikipedia article for more details.

Accuracy
use_weights

Use object/group weights to calculate metrics if the specified value is “true” and set all weights to “1” regardless of the input data if the specified value is “false”. The default value is  “true”.

Calculation principles

HingeLoss
use_weights
Use object/group weights to calculate metrics if the specified value is “true” and set all weights to “1” regardless of the input data if the specified value is “false”. The default value is  “true”.
See the Wikipedia article.
HammingLoss
use_weights
Use object/group weights to calculate metrics if the specified value is “true” and set all weights to “1” regardless of the input data if the specified value is “false”. The default value is  “true”.
Calculation principles
ZeroOneLoss
use_weights
Use object/group weights to calculate metrics if the specified value is “true” and set all weights to “1” regardless of the input data if the specified value is “false”. The default value is  “true”.
Calculation principles
Kappa
Calculation principles

is the weighted number of times class k is predicted by the model

is the weighted number of times class k is set as the label for input objects

WKappa
Calculation principles

Ranking

NameUsed for optimizationUser-defined parametersFormula and/or description

Pairwise metrics

All possible pairs in each group are generated without repetition according to the label values if pairs are not given for the PairLogit or PairLogitPairwise loss functions.

Attention.

The object label values from the input dataset are not taken into account when defining the winner object of the pair. Regardless of the label values, the first object in the pair is the winner.

PairLogit+
use_weights

Use object/group weights to calculate metrics if the specified value is “true” and set all weights to “1” regardless of the input data if the specified value is “false”. The default value is  “true”.

Note. Do not use the use_weights parameter if this metric is set as the loss function for training.
max_pairs
The maximum number of generated pairs in each group. Takes effect if no pairs are given and therefore are generated without repetition. By default, all possible pairs are generated in each group.
Calculation principles

This metric may give less accurate results on large datasets compared to PairLogitPairwise but it is calculated significantly faster.

PairLogitPairwise+
use_weights

Use object/group weights to calculate metrics if the specified value is “true” and set all weights to “1” regardless of the input data if the specified value is “false”. The default value is  “true”.

Note. Do not use the use_weights parameter if this metric is set as the loss function for training.
max_pairs
The maximum number of generated pairs in each group. Takes effect if no pairs are given and therefore are generated without repetition. By default, all possible pairs are generated in each group.
Calculation principles

This metric may give more accurate results on large datasets compared to PairLogit but it is calculated significantly slower.

PairAccuracy
use_weights

Use object/group weights to calculate metrics if the specified value is “true” and set all weights to “1” regardless of the input data if the specified value is “false”. The default value is  “true”.

Calculation principles
Groupwise metrics
YetiRank *+
decay
The probability of search continuation after reaching the current object. The default value is 0.99.
permutations
The number of permutations. The default value is 10.
use_weights

Use object/group weights to calculate metrics if the specified value is “true” and set all weights to “1” regardless of the input data if the specified value is “false”. The default value is  “true”.

Note. Do not use the use_weights parameter if this metric is set as the loss function for training.

An approximation of ranking metrics (such as NDCG and PFound). Allows to use ranking metrics for optimization.

The value of this metric can not be calculated. By default, the value of the PFound metric is written to output data if YetiRank is optimized.

This metric gives less accurate results on big datasets compared to YetiRankPairwise but it is significantly faster.
YetiRankPairwise *+
decay
The probability of search continuation after reaching the current object. The default value is 0.99.
permutations
The number of permutations. The default value is 10.
use_weights

Use object/group weights to calculate metrics if the specified value is “true” and set all weights to “1” regardless of the input data if the specified value is “false”. The default value is  “true”.

Note. Do not use the use_weights parameter if this metric is set as the loss function for training.

An approximation of ranking metrics (such as NDCG and PFound). Allows to use ranking metrics for optimization.

The value of this metric can not be calculated. By default, the value of the PFound metric is written to output data if YetiRank is optimized.

This metric gives more accurate results on big datasets compared to YetiRank but it is significantly slower.

QueryRMSE+
use_weights

Use object/group weights to calculate metrics if the specified value is “true” and set all weights to “1” regardless of the input data if the specified value is “false”. The default value is  “true”.

Note. Do not use the use_weights parameter if this metric is set as the loss function for training.
Calculation principles
QuerySoftMax+
use_weights

Use object/group weights to calculate metrics if the specified value is “true” and set all weights to “1” regardless of the input data if the specified value is “false”. The default value is  “true”.

Note. Do not use the use_weights parameter if this metric is set as the loss function for training.
Calculation principles
PFound *
decay

The probability of search continuation after reaching the current object. The default value is 0.85.

top

The number of top label values used to calculate the metric. The default value is –1 (all label values are used).

use_weights

Use object/group weights to calculate metrics if the specified value is “true” and set all weights to “1” regardless of the input data if the specified value is “false”. The default value is  “true”.

Calculation principles

See the PFound section for more details.

NDCG *_
top

The number of top label values used to calculate the metric. The default value is –1 (all label values are used).

use_weights

Use object/group weights to calculate metrics if the specified value is “true” and set all weights to “1” regardless of the input data if the specified value is “false”. The default value is  “true”.

type

Metric calculation principles.

Possible values:
  • Base
  • Exp

The default value is Base.

Refer to the NDCG section for details on the metric calculation principles.

Calculation principles

See the NDCG section for more details.

AverageGain
top

The number of top label values used to calculate the metric.

This is an obligatory parameter (the default value is not defined).

use_weights

Use object/group weights to calculate metrics if the specified value is “true” and set all weights to “1” regardless of the input data if the specified value is “false”. The default value is  “true”.

Represents the average value of the label values for objects with the defined top label values.

See the AverageGain section for more details.

PrecisionAt
top

The number of top label values used to calculate the metric.

The default value is –1 (all label values are used).
border
The label value border. If the value is strictly greater than this threshold, it is considered a positive class. Otherwise it is considered a negative class. The default value is 0.5.
Calculation principles

The calculation of this function consists of the following steps:

  1. The objects are sorted in descending order of predicted relevancies ()

  2. The metric is calculated as follows:

RecallAt
top

The number of top label values used to calculate the metric.

The default value is –1 (all label values are used).
border
The label value border. If the value is strictly greater than this threshold, it is considered a positive class. Otherwise it is considered a negative class. The default value is 0.5.
Calculation principles

The calculation of this function consists of the following steps:

  1. The objects are sorted in descending order of predicted relevancies ()

  2. The metric is calculated as follows:

MAP
top

The number of top label values used to calculate the metric.

The default value is –1 (all label values are used).
border
The label value border. If the value is strictly greater than this threshold, it is considered a positive class. Otherwise it is considered a negative class. The default value is 0.5.
Calculation principles
  1. The objects are sorted in descending order of predicted relevancies ()

  2. The metric is calculated as follows:

    • is the number of groups
    • The value is calculated individually for each j-th group.