mlpack

🔗 LogisticRegression

The LogisticRegression class implements a simple L2-regularized two-class logistic regression classifier for numerical data, by default using L-BFGS to learn the model. The class offers easy configurability, and arbitrary optimizers can be used to learn the model.

Logistic regression is useful for two-class classification (i.e. classes are 0 or 1). For multi-class logistic regression, see SoftmaxRegression.

Simple usage example:

// Train a logistic regression model on random data and predict labels:

// All data and labels are uniform random; 5 dimensional data, 2 classes.
// Replace with a data::Load() call or similar for a real application.
arma::mat dataset(5, 1000, arma::fill::randu); // 1000 points.
arma::Row<size_t> labels =
    arma::randi<arma::Row<size_t>>(1000, arma::distr_param(0, 1));
arma::mat testDataset(5, 500, arma::fill::randu); // 500 test points.

mlpack::LogisticRegression lr;         // Step 1: create model.
lr.Train(dataset, labels);             // Step 2: train model.
arma::Row<size_t> predictions;
lr.Classify(testDataset, predictions); // Step 3: classify points.

// Print some information about the test predictions.
std::cout << arma::accu(predictions == 0) << " test points classified as class "
    << "0." << std::endl;

More examples...

See also:

🔗 Constructors

Constructor Parameters:

name type description default
data arma::mat Column-major training matrix. (N/A)
labels arma::Row<size_t> Training labels, either 0 or 1. Should have length data.n_cols. (N/A)
initialPoint arma::rowvec Initial model weights to start optimization from. Should have length data.n_rows + 1. The first element is the bias. If not specified, a zero vector will be used. zero vector
optimizer any ensmallen optimizer Instantiated ensmallen optimizer for differentiable functions or differentiable separable functions. ens::L_BFGS()
lambda double L2 regularization penalty parameter. Must be nonnegative. 0.0
callbacks... any set of ensmallen callbacks Optional callbacks for the ensmallen optimizer, such as e.g. ens::ProgressBar(), ens::Report(), or others. (N/A)

As an alternative to passing lambda or initialPoint, these can be set with a standalone method. The following functions can be used before calling Train():

Note: Setting lambda too small may cause the model to overfit; however, setting it too large may cause the model to underfit. Automatic hyperparameter tuning can be used to find a good value of lambda instead of a manual setting.

🔗 Training

If training is not done as part of the constructor call, it can be done with one of the following versions of the Train() member function:

Types of each argument are the same as in the table for constructors above.

Notes:

🔗 Classification

Once a LogisticRegression model is trained, the Classify() member function can be used to make class predictions for new data.

Classification Parameters:

usage name type description
single-point point arma::vec Single point for classification.
single-point prediction size_t& size_t to store class prediction into.
single-point probabilitiesVec arma::vec& arma::vec& to store class probabilities into; will have length 2.
       
multi-point data arma::mat Set of column-major points for classification.
multi-point predictions arma::Row<size_t>& Vector of size_ts to store class prediction into; will be set to length data.n_cols.
multi-point probabilities arma::mat& Matrix to store class probabilities into (number of rows will be equal to 2; number of columns will be equal to data.n_cols).
       
all decisionBoundary double If the logistic function value for a point is greater than decisionBoundary, it is classified as class 1. Defaults to 0.5.

🔗 Other Functionality

For complete functionality, the source code can be consulted. Each method is fully documented.

🔗 Simple Examples

See also the simple usage example for a trivial usage of the LogisticRegression class.

Train a logistic regression model using a custom SGD-like optimizer with callbacks.

// See https://datasets.mlpack.org/satellite.train.csv.
arma::mat dataset;
mlpack::data::Load("satellite.train.csv", dataset, true);
// See https://datasets.mlpack.org/satellite.train.labels.csv.
arma::Row<size_t> labels;
mlpack::data::Load("satellite.train.labels.csv", labels, true);

mlpack::LogisticRegression lr;
lr.Lambda() = 0.1;

// Create AMSGrad optimizer with custom step size and batch size.
ens::AMSGrad optimizer(0.01 /* step size */, 16 /* batch size */);
optimizer.MaxIterations() = 100 * dataset.n_cols; // Allow 100 epochs.

// Print a progress bar and an optimization report when training is finished.
lr.Train(dataset, labels, optimizer, ens::ProgressBar(), ens::Report());

// Now predict on test labels and compute accuracy.

// See https://datasets.mlpack.org/satellite.test.csv.
arma::mat testDataset;
mlpack::data::Load("satellite.test.csv", testDataset, true);
// See https://datasets.mlpack.org/satellite.test.labels.csv.
arma::Row<size_t> testLabels;
mlpack::data::Load("satellite.test.labels.csv", testLabels, true);

std::cout << std::endl;
std::cout << "Accuracy on training set: "
    << lr.ComputeAccuracy(dataset, labels) << "\%." << std::endl;
std::cout << "Accuracy on test set:     "
    << lr.ComputeAccuracy(testDataset, testLabels) << "\%." << std::endl;
std::cout << "Objective on training set: "
    << lr.ComputeError(dataset, labels) << "." << std::endl;
std::cout << "Objective on test set:     "
    << lr.ComputeError(testDataset, testLabels) << "." << std::endl;

Train a logistic regression model with SGD and save the model every epoch using a custom ensmallen callback:

// This callback saves the model into "model-<epoch>.bin" after every epoch.
class ModelCheckpoint
{
 public:
  ModelCheckpoint(mlpack::LogisticRegression<>& model) : model(model) { }

  template<typename OptimizerType, typename FunctionType, typename MatType>
  bool EndEpoch(OptimizerType& /* optimizer */,
                FunctionType& /* function */,
                const MatType& /* coordinates */,
                const size_t epoch,
                const double /* objective */)
  {
    const std::string filename = "model-" + std::to_string(epoch) + ".bin";
    mlpack::data::Save(filename, "lr_model", model, true);
    return false; // Do not terminate the optimization.
  }

 private:
  mlpack::LogisticRegression<>& model;
};

With that callback available, the code to train the model is below:

// See https://datasets.mlpack.org/satellite.train.csv.
arma::mat dataset;
mlpack::data::Load("satellite.train.csv", dataset, true);
// See https://datasets.mlpack.org/satellite.train.labels.csv.
arma::Row<size_t> labels;
mlpack::data::Load("satellite.train.labels.csv", labels, true);

mlpack::LogisticRegression lr;

// Create AdaDelta optimizer with a small step size and batch size of 1.
ens::AdaDelta adaDelta(0.001, 1);
adaDelta.MaxIterations() = 100 * dataset.n_cols; // 100 epochs maximum.

// Use the custom callback and an L2 penalty parameter of 0.01.
lr.Train(dataset, labels, adaDelta, 0.01, ModelCheckpoint(lr),
    ens::ProgressBar());

// Now files like model-1.bin, model-2.bin, etc. should be saved on disk.

Load an existing logistic regression model and print some information about it.

mlpack::LogisticRegression lr;
// This assumes that a model called "lr_model" has been saved to the file
// "model-1.bin" (as in the previous example).
mlpack::data::Load("model-1.bin", "lr_model", lr, true);

// Print the dimensionality of the model and some other statistics.
std::cout << "The dimensionality of the model in model-1.bin is "
    << (lr.Parameters().n_elem - 1) << "." << std::endl;
std::cout << "The bias parameter for the model is " << lr.Parameters()[0]
    << "." << std::endl;

arma::vec point(lr.Parameters().n_elem - 1, arma::fill::randu);
std::cout << "The predicted class for a random point, using a decision boundary"
    << " of 0.2, is " << lr.Classify(point, 0.2) << "." << std::endl;

Perform incremental training on multiple datasets with multiple calls to Train().

// Generate two random datasets.
arma::mat firstDataset(5, 1000, arma::fill::randu); // 1000 points.
arma::Row<size_t> firstLabels =
    arma::randi<arma::Row<size_t>>(1000, arma::distr_param(0, 1));

arma::mat secondDataset(5, 1500, arma::fill::randu); // 1500 points.
arma::Row<size_t> secondLabels =
    arma::randi<arma::Row<size_t>>(1500, arma::distr_param(0, 1));

// Train a model on the first dataset with an L2 regularization penalty
// parameter of 0.01.
mlpack::LogisticRegression lr(firstDataset, firstLabels, 0.01);

// Now compute the objective on the second dataset and print it.
std::cout << "Objective on second dataset: "
    << lr.ComputeError(secondDataset, secondLabels) << "." << std::endl;

// Train for a second round on the second dataset.
lr.Train(secondDataset, secondLabels);

// Now compute the objective on the second dataset again and print it.
// (Note that it may not be all that much better because this is random data!)
std::cout << "Objective on second dataset after second training: "
    << lr.ComputeError(secondDataset, secondLabels) << "." << std::endl;

🔗 Advanced Functionality: Different Element Types

The LogisticRegression class has one template parameter that can be used to control the element type of the model. The full signature of the class is:

LogisticRegression<MatType>

MatType specifies the type of matrix used for training data and internal representation of model parameters. Any matrix type that implements the Armadillo API can be used. The example below trains a logistic regression model on sparse 32-bit floating point data.

// Create random, sparse 100-dimensional data.
arma::sp_fmat dataset;
dataset.sprandu(100, 5000, 0.3);
arma::Row<size_t> labels =
    arma::randi<arma::Row<size_t>>(5000, arma::distr_param(0, 1));

// Train with L2 regularization penalty parameter of 0.1.
mlpack::LogisticRegression<arma::sp_fmat> lr(dataset, labels, 0.1);

// Now classify a test point.
arma::sp_fvec point;
point.sprandu(100, 1, 0.3);

size_t prediction;
arma::fvec probabilitiesVec;
lr.Classify(point, prediction, probabilitiesVec);

std::cout << "Prediction for random test point: " << prediction << "."
    << std::endl;
std::cout << "Class probabilities for random test point: "
    << probabilitiesVec.t();

Note: if MatType is a sparse object (e.g. sp_fmat), the internal parameter representation will be a dense vector containing elements of the same type (e.g. frowvec). This is because L2-regularized logistic regression, even when training on sparse data, does not necessarily produce sparse models.