• C++ API Documentation

  • Quickstart
  • Matrices and data
  • Loading and saving
  • Citation
  • Core library functions
  • Classification

    • AdaBoost
    • DecisionTree
    • HoeffdingTree
    • LinearSVM
    • LogisticRegression
    • NaiveBayesClassifier
    • Perceptron
    • RandomForest
    • SoftmaxRegression
  • Regression

    • BayesianLinearRegression
    • DecisionTreeRegressor
    • LARS
    • LinearRegression
  • Clustering
  • Geometry
  • Preprocessing
  • Transformations

    • NMF
    • PCA
  • Modeling

    • Cross-validation
    • Hyperparameter tuning
  • Examples
  • Developers

• Binding API

  • Python quickstart
  • Python binding documentation
  • Julia quickstart
  • Julia binding documentation
  • R quickstart
  • R binding documentation
  • CLI quickstart
  • CLI binding documentation
  • Go quickstart
  • Go binding documentation

• mlpack Julia binding documentation [top]
  • Overview
  • Julia Quickstart
  • Data Formats
  • Classification

    • decision_tree()
    • linear_svm()
    • logistic_regression()
    • nbc()
    • perceptron()
    • random_forest()
    • softmax_regression()
    • adaboost()
  • Regression

    • bayesian_linear_regression()
    • lars()
    • linear_regression()
  • Clustering

    • dbscan()
    • gmm_train()
    • gmm_generate()
    • gmm_probability()
    • hoeffding_tree()
    • kmeans()
    • mean_shift()
  • Geometry

    • approx_kfn()
    • emst()
    • fastmks()
    • lsh()
    • knn()
    • kfn()
    • krann()
  • Preprocessing

    • preprocess_split()
    • preprocess_binarize()
    • preprocess_describe()
    • preprocess_scale()
    • preprocess_one_hot_encoding()
    • image_converter()
  • Misc. / Other

    • cf()
    • det()
    • hmm_train()
    • hmm_generate()
    • hmm_loglik()
    • hmm_viterbi()
    • kde()
    • nmf()
  • Transformations

    • kernel_pca()
    • lmnn()
    • local_coordinate_coding()
    • nca()
    • pca()
    • radical()
    • sparse_coding()

mlpack Julia binding documentation

🔗 mlpack overview

mlpack is an intuitive, fast, and flexible header-only C++ machine learning library with bindings to other languages. It aims to provide fast, lightweight implementations of both common and cutting-edge machine learning algorithms.

This reference page details mlpack’s bindings to Julia.

Further useful mlpack documentation links are given below.

• mlpack homepage
• mlpack on Github
• mlpack main documentation page

See also the quickstart guide for Julia:

• Julia Quickstart

🔗 Data Formats

🔗 approx_kfn()

Approximate furthest neighbor search

julia> using mlpack: approx_kfn
julia> distances, neighbors, output_model = approx_kfn( ;
          algorithm="ds", calculate_error=false, exact_distances=zeros(0, 0),
          input_model=nothing, k=0, num_projections=5, num_tables=5,
          query=zeros(0, 0), reference=zeros(0, 0), verbose=false)

An implementation of two strategies for furthest neighbor search. This can be used to compute the furthest neighbor of query point(s) from a set of points; furthest neighbor models can be saved and reused with future query point(s). Detailed documentation.

🔗 Input options

🔗 Output options

Results are returned as a tuple, and can be unpacked directly into return values or stored directly as a tuple; undesired results can be ignored with the _ keyword.

🔗 Detailed documentation

This program implements two strategies for furthest neighbor search. These strategies are:

• The ‘qdafn’ algorithm from “Approximate Furthest Neighbor in High Dimensions” by R. Pagh, F. Silvestri, J. Sivertsen, and M. Skala, in Similarity Search and Applications 2015 (SISAP).
• The ‘DrusillaSelect’ algorithm from “Fast approximate furthest neighbors with data-dependent candidate selection”, by R.R. Curtin and A.B. Gardner, in Similarity Search and Applications 2016 (SISAP).

These two strategies give approximate results for the furthest neighbor search problem and can be used as fast replacements for other furthest neighbor techniques such as those found in the mlpack_kfn program. Note that typically, the ‘ds’ algorithm requires far fewer tables and projections than the ‘qdafn’ algorithm.

Specify a reference set (set to search in) with reference, specify a query set with query, and specify algorithm parameters with num_tables and num_projections (or don’t and defaults will be used). The algorithm to be used (either ‘ds’—the default—or ‘qdafn’) may be specified with algorithm. Also specify the number of neighbors to search for with k.

Note that for ‘qdafn’ in lower dimensions, num_projections may need to be set to a high value in order to return results for each query point.

If no query set is specified, the reference set will be used as the query set. The output_model output parameter may be used to store the built model, and an input model may be loaded instead of specifying a reference set with the input_model option.

Results for each query point can be stored with the neighbors and distances output parameters. Each row of these output matrices holds the k distances or neighbor indices for each query point.

🔗 Example

For example, to find the 5 approximate furthest neighbors with reference_set as the reference set and query_set as the query set using DrusillaSelect, storing the furthest neighbor indices to neighbors and the furthest neighbor distances to distances, one could call

julia> using CSV
julia> query_set = CSV.read("query_set.csv")
julia> reference_set = CSV.read("reference_set.csv")
julia> distances, neighbors, _ = approx_kfn(algorithm="ds", k=5,
            query=query_set, reference=reference_set)

and to perform approximate all-furthest-neighbors search with k=1 on the set data storing only the furthest neighbor distances to distances, one could call

julia> using CSV
julia> reference_set = CSV.read("reference_set.csv")
julia> distances, _, _ = approx_kfn(k=1, reference=reference_set)

A trained model can be re-used. If a model has been previously saved to model, then we may find 3 approximate furthest neighbors on a query set new_query_set using that model and store the furthest neighbor indices into neighbors by calling

julia> using CSV
julia> new_query_set = CSV.read("new_query_set.csv")
julia> _, neighbors, _ = approx_kfn(input_model=model, k=3,
            query=new_query_set)

🔗 See also

• k-furthest-neighbor search
• k-nearest-neighbor search
• Fast approximate furthest neighbors with data-dependent candidate selection (pdf)
• Approximate furthest neighbor in high dimensions (pdf)
• QDAFN class documentation
• DrusillaSelect class documentation

🔗 bayesian_linear_regression()

BayesianLinearRegression

julia> using mlpack: bayesian_linear_regression
julia> output_model, predictions, stds = bayesian_linear_regression( ;
          center=false, input=zeros(0, 0), input_model=nothing,
          responses=Float64[], scale=false, test=zeros(0, 0), verbose=false)

An implementation of the bayesian linear regression. Detailed documentation.

🔗 Input options

🔗 Output options

Results are returned as a tuple, and can be unpacked directly into return values or stored directly as a tuple; undesired results can be ignored with the _ keyword.

🔗 Detailed documentation

An implementation of the bayesian linear regression. This model is a probabilistic view and implementation of the linear regression. The final solution is obtained by computing a posterior distribution from gaussian likelihood and a zero mean gaussian isotropic prior distribution on the solution. Optimization is AUTOMATIC and does not require cross validation. The optimization is performed by maximization of the evidence function. Parameters are tuned during the maximization of the marginal likelihood. This procedure includes the Ockham’s razor that penalizes over complex solutions.

This program is able to train a Bayesian linear regression model or load a model from file, output regression predictions for a test set, and save the trained model to a file.

To train a BayesianLinearRegression model, the input and responsesparameters must be given. The centerand scale parameters control the centering and the normalizing options. A trained model can be saved with the output_model. If no training is desired at all, a model can be passed via the input_model parameter.

The program can also provide predictions for test data using either the trained model or the given input model. Test points can be specified with the test parameter. Predicted responses to the test points can be saved with the predictions output parameter. The corresponding standard deviation can be save by precising the stds parameter.

🔗 Example

For example, the following command trains a model on the data data and responses responseswith center set to true and scale set to false (so, Bayesian linear regression is being solved, and then the model is saved to blr_model:

julia> using CSV
julia> data = CSV.read("data.csv")
julia> responses = CSV.read("responses.csv")
julia> blr_model, _, _ = bayesian_linear_regression(center=1,
            input=data, responses=responses, scale=0)

The following command uses the blr_model to provide predicted responses for the data test and save those responses to test_predictions:

julia> using CSV
julia> test = CSV.read("test.csv")
julia> _, test_predictions, _ =
            bayesian_linear_regression(input_model=blr_model, test=test)

Because the estimator computes a predictive distribution instead of a simple point estimate, the stds parameter allows one to save the prediction uncertainties:

julia> using CSV
julia> test = CSV.read("test.csv")
julia> _, test_predictions, stds =
            bayesian_linear_regression(input_model=blr_model, test=test)

🔗 See also

• Bayesian Interpolation
• Bayesian Linear Regression, Section 3.3
• BayesianLinearRegression C++ class documentation

🔗 cf()

Collaborative Filtering

julia> using mlpack: cf
julia> output, output_model = cf( ;
                                 algorithm="NMF",
                                 all_user_recommendations=false,
                                 input_model=nothing, interpolation="average",
                                 iteration_only_termination=false,
                                 max_iterations=1000, min_residue=1e-05,
                                 neighbor_search="euclidean", neighborhood=5,
                                 normalization="none", query=zeros(Int, 0, 0),
                                 rank=0, recommendations=5, seed=0,
                                 test=zeros(0, 0), training=zeros(0, 0),
                                 verbose=false)

An implementation of several collaborative filtering (CF) techniques for recommender systems. This can be used to train a new CF model, or use an existing CF model to compute recommendations. Detailed documentation.

🔗 Input options

🔗 Output options

Results are returned as a tuple, and can be unpacked directly into return values or stored directly as a tuple; undesired results can be ignored with the _ keyword.

🔗 Detailed documentation

This program performs collaborative filtering (CF) on the given dataset. Given a list of user, item and preferences (the training parameter), the program will perform a matrix decomposition and then can perform a series of actions related to collaborative filtering. Alternately, the program can load an existing saved CF model with the input_model parameter and then use that model to provide recommendations or predict values.

The input matrix should be a 3-dimensional matrix of ratings, where the first dimension is the user, the second dimension is the item, and the third dimension is that user’s rating of that item. Both the users and items should be numeric indices, not names. The indices are assumed to start from 0.

A set of query users for which recommendations can be generated may be specified with the query parameter; alternately, recommendations may be generated for every user in the dataset by specifying the all_user_recommendations parameter. In addition, the number of recommendations per user to generate can be specified with the recommendations parameter, and the number of similar users (the size of the neighborhood) to be considered when generating recommendations can be specified with the neighborhood parameter.

For performing the matrix decomposition, the following optimization algorithms can be specified via the algorithm parameter:

• ‘RegSVD’ – Regularized SVD using a SGD optimizer
• ‘NMF’ – Non-negative matrix factorization with alternating least squares update rules
• ‘BatchSVD’ – SVD batch learning
• ‘SVDIncompleteIncremental’ – SVD incomplete incremental learning
• ‘SVDCompleteIncremental’ – SVD complete incremental learning
• ‘BiasSVD’ – Bias SVD using a SGD optimizer
• ‘SVDPP’ – SVD++ using a SGD optimizer
• ‘RandSVD’ – RandomizedSVD learning
• ‘QSVD’ – QuicSVD learning
• ‘BKSVD’ – Block Krylov SVD learning

The following neighbor search algorithms can be specified via the neighbor_search parameter:

• ‘cosine’ – Cosine Search Algorithm
• ‘euclidean’ – Euclidean Search Algorithm
• ‘pearson’ – Pearson Search Algorithm

The following weight interpolation algorithms can be specified via the interpolation parameter:

• ‘average’ – Average Interpolation Algorithm
• ‘regression’ – Regression Interpolation Algorithm
• ‘similarity’ – Similarity Interpolation Algorithm

The following ranking normalization algorithms can be specified via the normalization parameter:

• ‘none’ – No Normalization
• ‘item_mean’ – Item Mean Normalization
• ‘overall_mean’ – Overall Mean Normalization
• ‘user_mean’ – User Mean Normalization
• ‘z_score’ – Z-Score Normalization

A trained model may be saved to with the output_model output parameter.

🔗 Example

To train a CF model on a dataset training_set using NMF for decomposition and saving the trained model to model, one could call:

julia> using CSV
julia> training_set = CSV.read("training_set.csv")
julia> _, model = cf(algorithm="NMF", training=training_set)

Then, to use this model to generate recommendations for the list of users in the query set users, storing 5 recommendations in recommendations, one could call

julia> using CSV
julia> users = CSV.read("users.csv"; type=Int)
julia> recommendations, _ = cf(input_model=model, query=users,
            recommendations=5)

🔗 See also

• Collaborative Filtering on Wikipedia
• Matrix factorization on Wikipedia
• Matrix factorization techniques for recommender systems (pdf)
• CFType class documentation

🔗 dbscan()

DBSCAN clustering

julia> using mlpack: dbscan
julia> assignments, centroids = dbscan(input; epsilon=1, min_size=5,
          naive=false, selection_type="ordered", single_mode=false,
          tree_type="kd", verbose=false)

An implementation of DBSCAN clustering. Given a dataset, this can compute and return a clustering of that dataset. Detailed documentation.

🔗 Input options

🔗 Output options

Results are returned as a tuple, and can be unpacked directly into return values or stored directly as a tuple; undesired results can be ignored with the _ keyword.

🔗 Detailed documentation

This program implements the DBSCAN algorithm for clustering using accelerated tree-based range search. The type of tree that is used may be parameterized, or brute-force range search may also be used.

The input dataset to be clustered may be specified with the input parameter; the radius of each range search may be specified with the epsilon parameters, and the minimum number of points in a cluster may be specified with the min_size parameter.

The assignments and centroids output parameters may be used to save the output of the clustering. assignments contains the cluster assignments of each point, and centroids contains the centroids of each cluster.

The range search may be controlled with the tree_type, single_mode, and naive parameters. tree_type can control the type of tree used for range search; this can take a variety of values: ‘kd’, ‘r’, ‘r-star’, ‘x’, ‘hilbert-r’, ‘r-plus’, ‘r-plus-plus’, ‘cover’, ‘ball’. The single_mode parameter will force single-tree search (as opposed to the default dual-tree search), and ‘naive will force brute-force range search.

🔗 Example

An example usage to run DBSCAN on the dataset in input with a radius of 0.5 and a minimum cluster size of 5 is given below:

julia> using CSV
julia> input = CSV.read("input.csv")
julia> _, _ = dbscan(input; epsilon=0.5, min_size=5)

🔗 See also

• DBSCAN on Wikipedia
• A density-based algorithm for discovering clusters in large spatial databases with noise (pdf)
• DBSCAN class documentation

🔗 decision_tree()

Decision tree

julia> using mlpack: decision_tree
julia> output_model, predictions, probabilities = decision_tree( ;
          input_model=nothing, labels=Int[], maximum_depth=0,
          minimum_gain_split=1e-07, minimum_leaf_size=20,
          print_training_accuracy=false, print_training_error=false,
          test=zeros(0, 0), test_labels=Int[], training=zeros(0, 0),
          verbose=false, weights=zeros(0, 0))

An implementation of an ID3-style decision tree for classification, which supports categorical data. Given labeled data with numeric or categorical features, a decision tree can be trained and saved; or, an existing decision tree can be used for classification on new points. Detailed documentation.

🔗 Input options

🔗 Output options

Results are returned as a tuple, and can be unpacked directly into return values or stored directly as a tuple; undesired results can be ignored with the _ keyword.

🔗 Detailed documentation

Train and evaluate using a decision tree. Given a dataset containing numeric or categorical features, and associated labels for each point in the dataset, this program can train a decision tree on that data.

The training set and associated labels are specified with the training and labels parameters, respectively. The labels should be in the range [0, num_classes - 1]. Optionally, if labels is not specified, the labels are assumed to be the last dimension of the training dataset.

When a model is trained, the output_model output parameter may be used to save the trained model. A model may be loaded for predictions with the input_model parameter. The input_model parameter may not be specified when the training parameter is specified. The minimum_leaf_size parameter specifies the minimum number of training points that must fall into each leaf for it to be split. The minimum_gain_split parameter specifies the minimum gain that is needed for the node to split. The maximum_depth parameter specifies the maximum depth of the tree. If print_training_error is specified, the training error will be printed.

Test data may be specified with the test parameter, and if performance numbers are desired for that test set, labels may be specified with the test_labels parameter. Predictions for each test point may be saved via the predictions output parameter. Class probabilities for each prediction may be saved with the probabilities output parameter.

🔗 Example

For example, to train a decision tree with a minimum leaf size of 20 on the dataset contained in data with labels labels, saving the output model to tree and printing the training error, one could call

julia> using CSV
julia> data = CSV.read("data.csv")
julia> labels = CSV.read("labels.csv"; type=Int)
julia> tree, _, _ = decision_tree(labels=labels,
            minimum_gain_split=0.001, minimum_leaf_size=20,
            print_training_accuracy=1, training=data)

Then, to use that model to classify points in test_set and print the test error given the labels test_labels using that model, while saving the predictions for each point to predictions, one could call

julia> using CSV
julia> test_set = CSV.read("test_set.csv")
julia> test_labels = CSV.read("test_labels.csv"; type=Int)
julia> _, predictions, _ = decision_tree(input_model=tree,
            test=test_set, test_labels=test_labels)

🔗 See also

• Random forest
• Decision trees on Wikipedia
• Induction of Decision Trees (pdf)
• DecisionTree C++ class documentation

🔗 det()

Density Estimation With Density Estimation Trees

julia> using mlpack: det
julia> output_model, tag_counters_file, tag_file, test_set_estimates,
          training_set_estimates, vi = det( ; folds=10, input_model=nothing,
          max_leaf_size=10, min_leaf_size=5, path_format="lr",
          skip_pruning=false, test=zeros(0, 0), training=zeros(0, 0),
          verbose=false)

An implementation of density estimation trees for the density estimation task. Density estimation trees can be trained or used to predict the density at locations given by query points. Detailed documentation.

🔗 Input options

🔗 Output options

Results are returned as a tuple, and can be unpacked directly into return values or stored directly as a tuple; undesired results can be ignored with the _ keyword.

🔗 Detailed documentation

This program performs a number of functions related to Density Estimation Trees. The optimal Density Estimation Tree (DET) can be trained on a set of data (specified by training) using cross-validation (with number of folds specified with the folds parameter). This trained density estimation tree may then be saved with the output_model output parameter.

The variable importances (that is, the feature importance values for each dimension) may be saved with the vi output parameter, and the density estimates for each training point may be saved with the training_set_estimates output parameter.

Enabling path printing for each node outputs the path from the root node to a leaf for each entry in the test set, or training set (if a test set is not provided). Strings like ‘LRLRLR’ (indicating that traversal went to the left child, then the right child, then the left child, and so forth) will be output. If ‘lr-id’ or ‘id-lr’ are given as the path_format parameter, then the ID (tag) of every node along the path will be printed after or before the L or R character indicating the direction of traversal, respectively.

This program also can provide density estimates for a set of test points, specified in the test parameter. The density estimation tree used for this task will be the tree that was trained on the given training points, or a tree given as the parameter input_model. The density estimates for the test points may be saved using the test_set_estimates output parameter.

🔗 See also

• Density estimation on Wikipedia
• Density estimation trees (pdf)
• DTree class documentation

🔗 emst()

Fast Euclidean Minimum Spanning Tree

julia> using mlpack: emst
julia> output = emst(input; leaf_size=1, naive=false,
                     verbose=false)

An implementation of the Dual-Tree Boruvka algorithm for computing the Euclidean minimum spanning tree of a set of input points. Detailed documentation.

🔗 Input options

🔗 Output options

Results are returned as a tuple, and can be unpacked directly into return values or stored directly as a tuple; undesired results can be ignored with the _ keyword.

🔗 Detailed documentation

This program can compute the Euclidean minimum spanning tree of a set of input points using the dual-tree Boruvka algorithm.

The set to calculate the minimum spanning tree of is specified with the input parameter, and the output may be saved with the output output parameter.

The leaf_size parameter controls the leaf size of the kd-tree that is used to calculate the minimum spanning tree, and if the naive option is given, then brute-force search is used (this is typically much slower in low dimensions). The leaf size does not affect the results, but it may have some effect on the runtime of the algorithm.

🔗 Example

For example, the minimum spanning tree of the input dataset data can be calculated with a leaf size of 20 and stored as spanning_tree using the following command:

julia> using CSV
julia> data = CSV.read("data.csv")
julia> spanning_tree = emst(data; leaf_size=20)

The output matrix is a three-dimensional matrix, where each row indicates an edge. The first dimension corresponds to the lesser index of the edge; the second dimension corresponds to the greater index of the edge; and the third column corresponds to the distance between the two points.

🔗 See also

• Minimum spanning tree on Wikipedia
• Fast Euclidean Minimum Spanning Tree: Algorithm, Analysis, and Applications (pdf)
• DualTreeBoruvka class documentation

🔗 fastmks()

FastMKS (Fast Max-Kernel Search)

julia> using mlpack: fastmks
julia> indices, kernels, output_model = fastmks( ; bandwidth=1,
          base=2, degree=2, input_model=nothing, k=0, kernel="linear",
          naive=false, offset=0, query=zeros(0, 0), reference=zeros(0, 0),
          scale=1, single=false, verbose=false)

An implementation of the single-tree and dual-tree fast max-kernel search (FastMKS) algorithm. Given a set of reference points and a set of query points, this can find the reference point with maximum kernel value for each query point; trained models can be reused for future queries. Detailed documentation.

🔗 Input options

🔗 Output options

Results are returned as a tuple, and can be unpacked directly into return values or stored directly as a tuple; undesired results can be ignored with the _ keyword.

🔗 Detailed documentation

This program will find the k maximum kernels of a set of points, using a query set and a reference set (which can optionally be the same set). More specifically, for each point in the query set, the k points in the reference set with maximum kernel evaluations are found. The kernel function used is specified with the kernel parameter.

🔗 Example

For example, the following command will calculate, for each point in the query set query, the five points in the reference set reference with maximum kernel evaluation using the linear kernel. The kernel evaluations may be saved with the kernels output parameter and the indices may be saved with the indices output parameter.

julia> using CSV
julia> reference = CSV.read("reference.csv")
julia> query = CSV.read("query.csv")
julia> indices, kernels, _ = fastmks(k=5, kernel="linear",
            query=query, reference=reference)

The output matrices are organized such that row i and column j in the indices matrix corresponds to the index of the point in the reference set that has j’th largest kernel evaluation with the point in the query set with index i. Row i and column j in the kernels matrix corresponds to the kernel evaluation between those two points.

This program performs FastMKS using a cover tree. The base used to build the cover tree can be specified with the base parameter.

🔗 See also

• k-nearest-neighbor search
• Dual-tree Fast Exact Max-Kernel Search (pdf)
• FastMKS class documentation

🔗 gmm_train()

Gaussian Mixture Model (GMM) Training

julia> using mlpack: gmm_train
julia> output_model = gmm_train(gaussians,
                                input; diagonal_covariance=false,
                                input_model=nothing, kmeans_max_iterations=1000,
                                max_iterations=250, no_force_positive=false,
                                noise=0, percentage=0.02, refined_start=false,
                                samplings=100, seed=0, tolerance=1e-10,
                                trials=1, verbose=false)

An implementation of the EM algorithm for training Gaussian mixture models (GMMs). Given a dataset, this can train a GMM for future use with other tools. Detailed documentation.

🔗 Input options

🔗 Output options

Results are returned as a tuple, and can be unpacked directly into return values or stored directly as a tuple; undesired results can be ignored with the _ keyword.

🔗 Detailed documentation

This program takes a parametric estimate of a Gaussian mixture model (GMM) using the EM algorithm to find the maximum likelihood estimate. The model may be saved and reused by other mlpack GMM tools.

The input data to train on must be specified with the input parameter, and the number of Gaussians in the model must be specified with the gaussians parameter. Optionally, many trials with different random initializations may be run, and the result with highest log-likelihood on the training data will be taken. The number of trials to run is specified with the trials parameter. By default, only one trial is run.

The tolerance for convergence and maximum number of iterations of the EM algorithm are specified with the tolerance and max_iterations parameters, respectively. The GMM may be initialized for training with another model, specified with the input_model parameter. Otherwise, the model is initialized by running k-means on the data. The k-means clustering initialization can be controlled with the kmeans_max_iterations, refined_start, samplings, and percentage parameters. If refined_start is specified, then the Bradley-Fayyad refined start initialization will be used. This can often lead to better clustering results.

The ‘diagonal_covariance’ flag will cause the learned covariances to be diagonal matrices. This significantly simplifies the model itself and causes training to be faster, but restricts the ability to fit more complex GMMs.

If GMM training fails with an error indicating that a covariance matrix could not be inverted, make sure that the no_force_positive parameter is not specified. Alternately, adding a small amount of Gaussian noise (using the noise parameter) to the entire dataset may help prevent Gaussians with zero variance in a particular dimension, which is usually the cause of non-invertible covariance matrices.

The no_force_positive parameter, if set, will avoid the checks after each iteration of the EM algorithm which ensure that the covariance matrices are positive definite. Specifying the flag can cause faster runtime, but may also cause non-positive definite covariance matrices, which will cause the program to crash.

🔗 Example

As an example, to train a 6-Gaussian GMM on the data in data with a maximum of 100 iterations of EM and 3 trials, saving the trained GMM to gmm, the following command can be used:

julia> using CSV
julia> data = CSV.read("data.csv")
julia> gmm = gmm_train(6, data; trials=3)

To re-train that GMM on another set of data data2, the following command may be used:

julia> using CSV
julia> data2 = CSV.read("data2.csv")
julia> new_gmm = gmm_train(6, data2; input_model=gmm)

🔗 See also

• gmm_generate()
• gmm_probability()
• Gaussian Mixture Models on Wikipedia
• GMM class documentation

🔗 gmm_generate()

GMM Sample Generator

julia> using mlpack: gmm_generate
julia> output = gmm_generate(input_model, samples;
                             seed=0, verbose=false)

A sample generator for pre-trained GMMs. Given a pre-trained GMM, this can sample new points randomly from that distribution. Detailed documentation.

🔗 Input options

🔗 Output options

Results are returned as a tuple, and can be unpacked directly into return values or stored directly as a tuple; undesired results can be ignored with the _ keyword.

🔗 Detailed documentation

This program is able to generate samples from a pre-trained GMM (use gmm_train to train a GMM). The pre-trained GMM must be specified with the input_model parameter. The number of samples to generate is specified by the samples parameter. Output samples may be saved with the output output parameter.

🔗 Example

The following command can be used to generate 100 samples from the pre-trained GMM gmm and store those generated samples in samples:

julia> samples = gmm_generate(gmm, 100)

🔗 See also

• gmm_train()
• gmm_probability()
• Gaussian Mixture Models on Wikipedia
• GMM class documentation

🔗 gmm_probability()

GMM Probability Calculator

julia> using mlpack: gmm_probability
julia> output = gmm_probability(input,
                                input_model; verbose=false)

A probability calculator for GMMs. Given a pre-trained GMM and a set of points, this can compute the probability that each point is from the given GMM. Detailed documentation.

🔗 Input options

🔗 Output options

Results are returned as a tuple, and can be unpacked directly into return values or stored directly as a tuple; undesired results can be ignored with the _ keyword.

🔗 Detailed documentation

This program calculates the probability that given points came from a given GMM (that is, P(X | gmm)). The GMM is specified with the input_model parameter, and the points are specified with the input parameter. The output probabilities may be saved via the output output parameter.

🔗 Example

So, for example, to calculate the probabilities of each point in points coming from the pre-trained GMM gmm, while storing those probabilities in probs, the following command could be used:

julia> using CSV
julia> points = CSV.read("points.csv")
julia> probs = gmm_probability(points, gmm)

🔗 See also

• gmm_train()
• gmm_generate()
• Gaussian Mixture Models on Wikipedia
• GMM class documentation

🔗 hmm_train()

Hidden Markov Model (HMM) Training

julia> using mlpack: hmm_train
julia> output_model = hmm_train(input_file;
                                batch=false, gaussians=0, input_model=nothing,
                                labels_file="", seed=0, states=0,
                                tolerance=1e-05, type="gaussian",
                                verbose=false)

An implementation of training algorithms for Hidden Markov Models (HMMs). Given labeled or unlabeled data, an HMM can be trained for further use with other mlpack HMM tools. Detailed documentation.

🔗 Input options

🔗 Output options

Results are returned as a tuple, and can be unpacked directly into return values or stored directly as a tuple; undesired results can be ignored with the _ keyword.

🔗 Detailed documentation

This program allows a Hidden Markov Model to be trained on labeled or unlabeled data. It supports four types of HMMs: Discrete HMMs, Gaussian HMMs, GMM HMMs, or Diagonal GMM HMMs

Either one input sequence can be specified (with input_file), or, a file containing files in which input sequences can be found (when input_fileandbatch are used together). In addition, labels can be provided in the file specified by labels_file, and if batch is used, the file given to labels_file should contain a list of files of labels corresponding to the sequences in the file given to input_file.

The HMM is trained with the Baum-Welch algorithm if no labels are provided. The tolerance of the Baum-Welch algorithm can be set with the toleranceoption. By default, the transition matrix is randomly initialized and the emission distributions are initialized to fit the extent of the data.

Optionally, a pre-created HMM model can be used as a guess for the transition matrix and emission probabilities; this is specifiable with output_model.

🔗 See also

• hmm_generate()
• hmm_loglik()
• hmm_viterbi()
• Hidden Mixture Models on Wikipedia
• HMM class documentation

🔗 hmm_generate()

Hidden Markov Model (HMM) Sequence Generator

julia> using mlpack: hmm_generate
julia> output, state = hmm_generate(length, model; seed=0,
          start_state=0, verbose=false)

A utility to generate random sequences from a pre-trained Hidden Markov Model (HMM). The length of the desired sequence can be specified, and a random sequence of observations is returned. Detailed documentation.

🔗 Input options

🔗 Output options

Results are returned as a tuple, and can be unpacked directly into return values or stored directly as a tuple; undesired results can be ignored with the _ keyword.

🔗 Detailed documentation

This utility takes an already-trained HMM, specified as the model parameter, and generates a random observation sequence and hidden state sequence based on its parameters. The observation sequence may be saved with the output output parameter, and the internal state sequence may be saved with the state output parameter.

The state to start the sequence in may be specified with the start_state parameter.

🔗 Example

For example, to generate a sequence of length 150 from the HMM hmm and save the observation sequence to observations and the hidden state sequence to states, the following command may be used:

julia> observations, states = hmm_generate(150, hmm)

🔗 See also

• hmm_train()
• hmm_loglik()
• hmm_viterbi()
• Hidden Mixture Models on Wikipedia
• HMM class documentation

🔗 hmm_loglik()

Hidden Markov Model (HMM) Sequence Log-Likelihood

julia> using mlpack: hmm_loglik
julia> log_likelihood = hmm_loglik(input,
                                   input_model; verbose=false)

A utility for computing the log-likelihood of a sequence for Hidden Markov Models (HMMs). Given a pre-trained HMM and an observation sequence, this computes and returns the log-likelihood of that sequence being observed from that HMM. Detailed documentation.

🔗 Input options

🔗 Output options

Results are returned as a tuple, and can be unpacked directly into return values or stored directly as a tuple; undesired results can be ignored with the _ keyword.

🔗 Detailed documentation

This utility takes an already-trained HMM, specified with the input_model parameter, and evaluates the log-likelihood of a sequence of observations, given with the input parameter. The computed log-likelihood is given as output.

🔗 Example

For example, to compute the log-likelihood of the sequence seq with the pre-trained HMM hmm, the following command may be used:

julia> using CSV
julia> seq = CSV.read("seq.csv")
julia> _ = hmm_loglik(seq, hmm)

🔗 See also

• hmm_train()
• hmm_generate()
• hmm_viterbi()
• Hidden Mixture Models on Wikipedia
• HMM class documentation

🔗 hmm_viterbi()

Hidden Markov Model (HMM) Viterbi State Prediction

julia> using mlpack: hmm_viterbi
julia> output = hmm_viterbi(input, input_model;
                            verbose=false)

A utility for computing the most probable hidden state sequence for Hidden Markov Models (HMMs). Given a pre-trained HMM and an observed sequence, this uses the Viterbi algorithm to compute and return the most probable hidden state sequence. Detailed documentation.

🔗 Input options

🔗 Output options

Results are returned as a tuple, and can be unpacked directly into return values or stored directly as a tuple; undesired results can be ignored with the _ keyword.

🔗 Detailed documentation

This utility takes an already-trained HMM, specified as input_model, and evaluates the most probable hidden state sequence of a given sequence of observations (specified as ‘input, using the Viterbi algorithm. The computed state sequence may be saved using the output output parameter.

🔗 Example

For example, to predict the state sequence of the observations obs using the HMM hmm, storing the predicted state sequence to states, the following command could be used:

julia> using CSV
julia> obs = CSV.read("obs.csv")
julia> states = hmm_viterbi(obs, hmm)

🔗 See also

• hmm_train()
• hmm_generate()
• hmm_loglik()
• Hidden Mixture Models on Wikipedia
• HMM class documentation

🔗 hoeffding_tree()

Hoeffding trees

julia> using mlpack: hoeffding_tree
julia> output_model, predictions, probabilities = hoeffding_tree( ;
          batch_mode=false, bins=10, confidence=0.95, info_gain=false,
          input_model=nothing, labels=Int[], max_samples=5000, min_samples=100,
          numeric_split_strategy="binary", observations_before_binning=100,
          passes=1, test=zeros(0, 0), test_labels=Int[], training=zeros(0, 0),
          verbose=false)

An implementation of Hoeffding trees, a form of streaming decision tree for classification. Given labeled data, a Hoeffding tree can be trained and saved for later use, or a pre-trained Hoeffding tree can be used for predicting the classifications of new points. Detailed documentation.

🔗 Input options

🔗 Output options

Results are returned as a tuple, and can be unpacked directly into return values or stored directly as a tuple; undesired results can be ignored with the _ keyword.

🔗 Detailed documentation

This program implements Hoeffding trees, a form of streaming decision tree suited best for large (or streaming) datasets. This program supports both categorical and numeric data. Given an input dataset, this program is able to train the tree with numerous training options, and save the model to a file. The program is also able to use a trained model or a model from file in order to predict classes for a given test set.

The training file and associated labels are specified with the training and labels parameters, respectively. Optionally, if labels is not specified, the labels are assumed to be the last dimension of the training dataset.

The training may be performed in batch mode (like a typical decision tree algorithm) by specifying the batch_mode option, but this may not be the best option for large datasets.

When a model is trained, it may be saved via the output_model output parameter. A model may be loaded from file for further training or testing with the input_model parameter.

Test data may be specified with the test parameter, and if performance statistics are desired for that test set, labels may be specified with the test_labels parameter. Predictions for each test point may be saved with the predictions output parameter, and class probabilities for each prediction may be saved with the probabilities output parameter.

🔗 Example

For example, to train a Hoeffding tree with confidence 0.99 with data dataset, saving the trained tree to tree, the following command may be used:

julia> using CSV
julia> dataset = CSV.read("dataset.csv")
julia> tree, _, _ = hoeffding_tree(confidence=0.99,
            training=dataset)

Then, this tree may be used to make predictions on the test set test_set, saving the predictions into predictions and the class probabilities into class_probs with the following command:

julia> using CSV
julia> test_set = CSV.read("test_set.csv")
julia> _, predictions, class_probs =
            hoeffding_tree(input_model=tree, test=test_set)

🔗 See also

• decision_tree()
• random_forest()
• Mining High-Speed Data Streams (pdf)
• HoeffdingTree class documentation

🔗 kde()

Kernel Density Estimation

julia> using mlpack: kde
julia> output_model, predictions = kde( ; abs_error=0,
          algorithm="dual-tree", bandwidth=1, initial_sample_size=100,
          input_model=nothing, kernel="gaussian", mc_break_coef=0.4,
          mc_entry_coef=3, mc_probability=0.95, monte_carlo=false,
          query=zeros(0, 0), reference=zeros(0, 0), rel_error=0.05,
          tree="kd-tree", verbose=false)

An implementation of kernel density estimation with dual-tree algorithms. Given a set of reference points and query points and a kernel function, this can estimate the density function at the location of each query point using trees; trees that are built can be saved for later use. Detailed documentation.

🔗 Input options

🔗 Output options

Results are returned as a tuple, and can be unpacked directly into return values or stored directly as a tuple; undesired results can be ignored with the _ keyword.

🔗 Detailed documentation

This program performs a Kernel Density Estimation. KDE is a non-parametric way of estimating probability density function. For each query point the program will estimate its probability density by applying a kernel function to each reference point. The computational complexity of this is O(N^2) where there are N query points and N reference points, but this implementation will typically see better performance as it uses an approximate dual or single tree algorithm for acceleration.

Dual or single tree optimization avoids many barely relevant calculations (as kernel function values decrease with distance), so it is an approximate computation. You can specify the maximum relative error tolerance for each query value with rel_error as well as the maximum absolute error tolerance with the parameter abs_error. This program runs using an Euclidean metric. Kernel function can be selected using the kernel option. You can also choose what which type of tree to use for the dual-tree algorithm with tree. It is also possible to select whether to use dual-tree algorithm or single-tree algorithm using the algorithm option.

Monte Carlo estimations can be used to accelerate the KDE estimate when the Gaussian Kernel is used. This provides a probabilistic guarantee on the the error of the resulting KDE instead of an absolute guarantee.To enable Monte Carlo estimations, the monte_carlo flag can be used, and success probability can be set with the mc_probability option. It is possible to set the initial sample size for the Monte Carlo estimation using initial_sample_size. This implementation will only consider a node, as a candidate for the Monte Carlo estimation, if its number of descendant nodes is bigger than the initial sample size. This can be controlled using a coefficient that will multiply the initial sample size and can be set using mc_entry_coef. To avoid using the same amount of computations an exact approach would take, this program recurses the tree whenever a fraction of the amount of the node’s descendant points have already been computed. This fraction is set using mc_break_coef.

🔗 Example

For example, the following will run KDE using the data in ref_data for training and the data in qu_data as query data. It will apply an Epanechnikov kernel with a 0.2 bandwidth to each reference point and use a KD-Tree for the dual-tree optimization. The returned predictions will be within 5% of the real KDE value for each query point.

julia> using CSV
julia> ref_data = CSV.read("ref_data.csv")
julia> qu_data = CSV.read("qu_data.csv")
julia> _, out_data = kde(bandwidth=0.2, kernel="epanechnikov",
            query=qu_data, reference=ref_data, rel_error=0.05, tree="kd-tree")

the predicted density estimations will be stored in out_data. If no query is provided, then KDE will be computed on the reference dataset. It is possible to select either a reference dataset or an input model but not both at the same time. If an input model is selected and parameter values are not set (e.g. bandwidth) then default parameter values will be used.

In addition to the last program call, it is also possible to activate Monte Carlo estimations if a Gaussian kernel is used. This can provide faster results, but the KDE will only have a probabilistic guarantee of meeting the desired error bound (instead of an absolute guarantee). The following example will run KDE using a Monte Carlo estimation when possible. The results will be within a 5% of the real KDE value with a 95% probability. Initial sample size for the Monte Carlo estimation will be 200 points and a node will be a candidate for the estimation only when it contains 700 (i.e. 3.5200) points. If a node contains 700 points and 420 (i.e. 0.6700) have already been sampled, then the algorithm will recurse instead of keep sampling.

julia> using CSV
julia> ref_data = CSV.read("ref_data.csv")
julia> qu_data = CSV.read("qu_data.csv")
julia> _, out_data = kde(bandwidth=0.2, initial_sample_size=200,
            kernel="gaussian", mc_break_coef=0.6, mc_entry_coef=3.5,
            mc_probability=0.95, monte_carlo=, query=qu_data,
            reference=ref_data, rel_error=0.05, tree="kd-tree")

🔗 See also

• knn()
• Kernel density estimation on Wikipedia
• Tree-Independent Dual-Tree Algorithms
• Fast High-dimensional Kernel Summations Using the Monte Carlo Multipole Method
• KDE C++ class documentation

🔗 kernel_pca()

Kernel Principal Components Analysis

julia> using mlpack: kernel_pca
julia> output = kernel_pca(input, kernel;
                           bandwidth=1, center=false, degree=1, kernel_scale=1,
                           new_dimensionality=0, nystroem_method=false,
                           offset=0, sampling="kmeans", verbose=false)

An implementation of Kernel Principal Components Analysis (KPCA). This can be used to perform nonlinear dimensionality reduction or preprocessing on a given dataset. Detailed documentation.

🔗 Input options

🔗 Output options

Results are returned as a tuple, and can be unpacked directly into return values or stored directly as a tuple; undesired results can be ignored with the _ keyword.

🔗 Detailed documentation

This program performs Kernel Principal Components Analysis (KPCA) on the specified dataset with the specified kernel. This will transform the data onto the kernel principal components, and optionally reduce the dimensionality by ignoring the kernel principal components with the smallest eigenvalues.

For the case where a linear kernel is used, this reduces to regular PCA.

The kernels that are supported are listed below:

• ‘linear’: the standard linear dot product (same as normal PCA): K(x, y) = x^T y

• ‘gaussian’: a Gaussian kernel; requires bandwidth: K(x, y) = exp(-(\|\| x - y \|\| ^ 2) / (2 * (bandwidth ^ 2)))

• ‘polynomial’: polynomial kernel; requires offset and degree: K(x, y) = (x^T y + offset) ^ degree

• ‘hyptan’: hyperbolic tangent kernel; requires scale and offset: K(x, y) = tanh(scale * (x^T y) + offset)

• ‘laplacian’: Laplacian kernel; requires bandwidth: K(x, y) = exp(-(\|\| x - y \|\|) / bandwidth)

• ‘epanechnikov’: Epanechnikov kernel; requires bandwidth: K(x, y) = max(0, 1 - \|\| x - y \|\|^2 / bandwidth^2)

• ‘cosine’: cosine distance: K(x, y) = 1 - (x^T y) / (\|\| x \|\| * \|\| y \|\|)

The parameters for each of the kernels should be specified with the options bandwidth, kernel_scale, offset, or degree (or a combination of those parameters).

Optionally, the Nystroem method (“Using the Nystroem method to speed up kernel machines”, 2001) can be used to calculate the kernel matrix by specifying the nystroem_method parameter. This approach works by using a subset of the data as basis to reconstruct the kernel matrix; to specify the sampling scheme, the sampling parameter is used. The sampling scheme for the Nystroem method can be chosen from the following list: ‘kmeans’, ‘random’, ‘ordered’.

🔗 Example

For example, the following command will perform KPCA on the dataset input using the Gaussian kernel, and saving the transformed data to transformed:

julia> using CSV
julia> input = CSV.read("input.csv")
julia> transformed = kernel_pca(input, "gaussian")

🔗 See also

• Kernel principal component analysis on Wikipedia
• Kernel Principal Component Analysis (pdf)
• KernelPCA class documentation

🔗 kmeans()

K-Means Clustering

julia> using mlpack: kmeans
julia> centroid, output = kmeans(clusters,
                                 input; algorithm="naive",
                                 allow_empty_clusters=false, in_place=false,
                                 initial_centroids=zeros(0, 0),
                                 kill_empty_clusters=false,
                                 kmeans_plus_plus=false, labels_only=false,
                                 max_iterations=1000, percentage=0.02,
                                 refined_start=false, samplings=100, seed=0,
                                 verbose=false)

An implementation of several strategies for efficient k-means clustering. Given a dataset and a value of k, this computes and returns a k-means clustering on that data. Detailed documentation.

🔗 Input options

🔗 Output options

Results are returned as a tuple, and can be unpacked directly into return values or stored directly as a tuple; undesired results can be ignored with the _ keyword.

🔗 Detailed documentation

This program performs K-Means clustering on the given dataset. It can return the learned cluster assignments, and the centroids of the clusters. Empty clusters are not allowed by default; when a cluster becomes empty, the point furthest from the centroid of the cluster with maximum variance is taken to fill that cluster.

Optionally, the strategy to choose initial centroids can be specified. The k-means++ algorithm can be used to choose initial centroids with the kmeans_plus_plus parameter. The Bradley and Fayyad approach (“Refining initial points for k-means clustering”, 1998) can be used to select initial points by specifying the refined_start parameter. This approach works by taking random samplings of the dataset; to specify the number of samplings, the samplings parameter is used, and to specify the percentage of the dataset to be used in each sample, the percentage parameter is used (it should be a value between 0.0 and 1.0).

There are several options available for the algorithm used for each Lloyd iteration, specified with the algorithm option. The standard O(kN) approach can be used (‘naive’). Other options include the Pelleg-Moore tree-based algorithm (‘pelleg-moore’), Elkan’s triangle-inequality based algorithm (‘elkan’), Hamerly’s modification to Elkan’s algorithm (‘hamerly’), the dual-tree k-means algorithm (‘dualtree’), and the dual-tree k-means algorithm using the cover tree (‘dualtree-covertree’).

The behavior for when an empty cluster is encountered can be modified with the allow_empty_clusters option. When this option is specified and there is a cluster owning no points at the end of an iteration, that cluster’s centroid will simply remain in its position from the previous iteration. If the kill_empty_clusters option is specified, then when a cluster owns no points at the end of an iteration, the cluster centroid is simply filled with DBL_MAX, killing it and effectively reducing k for the rest of the computation. Note that the default option when neither empty cluster option is specified can be time-consuming to calculate; therefore, specifying either of these parameters will often accelerate runtime.

Initial clustering assignments may be specified using the initial_centroids parameter, and the maximum number of iterations may be specified with the max_iterations parameter.

🔗 Example

As an example, to use Hamerly’s algorithm to perform k-means clustering with k=10 on the dataset data, saving the centroids to centroids and the assignments for each point to assignments, the following command could be used:

julia> using CSV
julia> data = CSV.read("data.csv")
julia> centroids, assignments = kmeans(10, data)

To run k-means on that same dataset with initial centroids specified in initial with a maximum of 500 iterations, storing the output centroids in final the following command may be used:

julia> using CSV
julia> data = CSV.read("data.csv")
julia> initial = CSV.read("initial.csv")
julia> final, _ = kmeans(10, data; initial_centroids=initial,
            max_iterations=500)

🔗 See also

• dbscan()
• k-means++
• Using the triangle inequality to accelerate k-means (pdf)
• Making k-means even faster (pdf)
• Accelerating exact k-means algorithms with geometric reasoning (pdf)
• A dual-tree algorithm for fast k-means clustering with large k (pdf)
• KMeans class documentation

🔗 lars()

LARS

julia> using mlpack: lars
julia> output_model, output_predictions = lars( ; input=zeros(0, 0),
          input_model=nothing, lambda1=0, lambda2=0, no_intercept=false,
          no_normalize=false, responses=zeros(0, 0), test=zeros(0, 0),
          use_cholesky=false, verbose=false)

An implementation of Least Angle Regression (Stagewise/laSso), also known as LARS. This can train a LARS/LASSO/Elastic Net model and use that model or a pre-trained model to output regression predictions for a test set. Detailed documentation.

🔗 Input options

🔗 Output options

Results are returned as a tuple, and can be unpacked directly into return values or stored directly as a tuple; undesired results can be ignored with the _ keyword.

🔗 Detailed documentation

An implementation of LARS: Least Angle Regression (Stagewise/laSso). This is a stage-wise homotopy-based algorithm for L1-regularized linear regression (LASSO) and L1+L2-regularized linear regression (Elastic Net).

This program is able to train a LARS/LASSO/Elastic Net model or load a model from file, output regression predictions for a test set, and save the trained model to a file. The LARS algorithm is described in more detail below:

Let X be a matrix where each row is a point and each column is a dimension, and let y be a vector of targets.

The Elastic Net problem is to solve

min_beta 0.5 || X * beta - y ||_2^2 + lambda_1 ||beta||_1 + 0.5 lambda_2 ||beta||_2^2

If lambda1 > 0 and lambda2 = 0, the problem is the LASSO. If lambda1 > 0 and lambda2 > 0, the problem is the Elastic Net. If lambda1 = 0 and lambda2 > 0, the problem is ridge regression. If lambda1 = 0 and lambda2 = 0, the problem is unregularized linear regression.

For efficiency reasons, it is not recommended to use this algorithm with lambda1 = 0. In that case, use the ‘linear_regression’ program, which implements both unregularized linear regression and ridge regression.

To train a LARS/LASSO/Elastic Net model, the input and responses parameters must be given. The lambda1, lambda2, and use_cholesky parameters control the training options. A trained model can be saved with the output_model. If no training is desired at all, a model can be passed via the input_model parameter.

The program can also provide predictions for test data using either the trained model or the given input model. Test points can be specified with the test parameter. Predicted responses to the test points can be saved with the output_predictions output parameter.

🔗 Example

For example, the following command trains a model on the data data and responses responses with lambda1 set to 0.4 and lambda2 set to 0 (so, LASSO is being solved), and then the model is saved to lasso_model:

julia> using CSV
julia> data = CSV.read("data.csv")
julia> responses = CSV.read("responses.csv")
julia> lasso_model, _ = lars(input=data, lambda1=0.4, lambda2=0,
            responses=responses)

The following command uses the lasso_model to provide predicted responses for the data test and save those responses to test_predictions:

julia> using CSV
julia> test = CSV.read("test.csv")
julia> _, test_predictions = lars(input_model=lasso_model,
            test=test)

🔗 See also

• linear_regression()
• Least angle regression (pdf)
• LARS C++ class documentation

🔗 linear_svm()

Linear SVM is an L2-regularized support vector machine.

julia> using mlpack: linear_svm
julia> output_model, predictions, probabilities = linear_svm( ;
          delta=1, epochs=50, input_model=nothing, labels=Int[], lambda=0.0001,
          max_iterations=10000, no_intercept=false, num_classes=0,
          optimizer="lbfgs", seed=0, shuffle=false, step_size=0.01,
          test=zeros(0, 0), test_labels=Int[], tolerance=1e-10,
          training=zeros(0, 0), verbose=false)

An implementation of linear SVM for multiclass classification. Given labeled data, a model can be trained and saved for future use; or, a pre-trained model can be used to classify new points. Detailed documentation.

🔗 Input options

🔗 Output options

Results are returned as a tuple, and can be unpacked directly into return values or stored directly as a tuple; undesired results can be ignored with the _ keyword.

🔗 Detailed documentation

An implementation of linear SVMs that uses either L-BFGS or parallel SGD (stochastic gradient descent) to train the model.

This program allows loading a linear SVM model (via the input_model parameter) or training a linear SVM model given training data (specified with the training parameter), or both those things at once. In addition, this program allows classification on a test dataset (specified with the test parameter) and the classification results may be saved with the predictions output parameter. The trained linear SVM model may be saved using the output_model output parameter.

The training data, if specified, may have class labels as its last dimension. Alternately, the labels parameter may be used to specify a separate vector of labels.

When a model is being trained, there are many options. L2 regularization (to prevent overfitting) can be specified with the lambda option, and the number of classes can be manually specified with the num_classesand if an intercept term is not desired in the model, the no_intercept parameter can be specified.Margin of difference between correct class and other classes can be specified with the delta option.The optimizer used to train the model can be specified with the optimizer parameter. Available options are ‘psgd’ (parallel stochastic gradient descent) and ‘lbfgs’ (the L-BFGS optimizer). There are also various parameters for the optimizer; the max_iterations parameter specifies the maximum number of allowed iterations, and the tolerance parameter specifies the tolerance for convergence. For the parallel SGD optimizer, the step_size parameter controls the step size taken at each iteration by the optimizer and the maximum number of epochs (specified with epochs). If the objective function for your data is oscillating between Inf and 0, the step size is probably too large. There are more parameters for the optimizers, but the C++ interface must be used to access these.

Optionally, the model can be used to predict the labels for another matrix of data points, if test is specified. The test parameter can be specified without the training parameter, so long as an existing linear SVM model is given with the input_model parameter. The output predictions from the linear SVM model may be saved with the predictions parameter.

🔗 Example

As an example, to train a LinaerSVM on the data ‘data’ with labels ‘labels’ with L2 regularization of 0.1, saving the model to ‘lsvm_model’, the following command may be used:

julia> using CSV
julia> data = CSV.read("data.csv")
julia> labels = CSV.read("labels.csv"; type=Int)
julia> lsvm_model, _, _ = linear_svm(delta=1, labels=labels,
            lambda=0.1, num_classes=0, training=data)

Then, to use that model to predict classes for the dataset ‘test’, storing the output predictions in ‘predictions’, the following command may be used:

julia> using CSV
julia> test = CSV.read("test.csv")
julia> _, predictions, _ = linear_svm(input_model=lsvm_model,
            test=test)

🔗 See also

• random_forest()
• logistic_regression()
• LinearSVM on Wikipedia
• LinearSVM C++ class documentation

🔗 lmnn()

Large Margin Nearest Neighbors (LMNN)

julia> using mlpack: lmnn
julia> centered_data, output, transformed_data = lmnn(input;
          batch_size=50, center=false, distance=zeros(0, 0), k=1, labels=Int[],
          linear_scan=false, max_iterations=100000, normalize=false,
          optimizer="amsgrad", passes=50, print_accuracy=false, range=1, rank=0,
          regularization=0.5, seed=0, step_size=0.01, tolerance=1e-07,
          verbose=false)

An implementation of Large Margin Nearest Neighbors (LMNN), a distance learning technique. Given a labeled dataset, this learns a transformation of the data that improves k-nearest-neighbor performance; this can be useful as a preprocessing step. Detailed documentation.

🔗 Input options

🔗 Output options

Results are returned as a tuple, and can be unpacked directly into return values or stored directly as a tuple; undesired results can be ignored with the _ keyword.

🔗 Detailed documentation

This program implements Large Margin Nearest Neighbors, a distance learning technique. The method seeks to improve k-nearest-neighbor classification on a dataset. The method employes the strategy of reducing distance between similar labeled data points (a.k.a target neighbors) and increasing distance between differently labeled points (a.k.a impostors) using standard optimization techniques over the gradient of the distance between data points.

To work, this algorithm needs labeled data. It can be given as the last row of the input dataset (specified with input), or alternatively as a separate matrix (specified with labels). Additionally, a starting point for optimization (specified with distancecan be given, having (r x d) dimensionality. Here r should satisfy 1 <= r <= d, Consequently a Low-Rank matrix will be optimized. Alternatively, Low-Rank distance can be learned by specifying the rankparameter (A Low-Rank matrix with uniformly distributed values will be used as initial learning point).

The program also requires number of targets neighbors to work with ( specified with k), A regularization parameter can also be passed, It acts as a trade of between the pulling and pushing terms (specified with regularization), In addition, this implementation of LMNN includes a parameter to decide the interval after which impostors must be re-calculated (specified with range).

Output can either be the learned distance matrix (specified with output), or the transformed dataset (specified with transformed_data), or both. Additionally mean-centered dataset (specified with centered_data) can be accessed given mean-centering (specified with center) is performed on the dataset. Accuracy on initial dataset and final transformed dataset can be printed by specifying the print_accuracyparameter.

This implementation of LMNN uses AdaGrad, BigBatch_SGD, stochastic gradient descent, mini-batch stochastic gradient descent, or the L_BFGS optimizer.

AdaGrad, specified by the value ‘adagrad’ for the parameter optimizer, uses maximum of past squared gradients. It primarily on six parameters: the step size (specified with step_size), the batch size (specified with batch_size), the maximum number of passes (specified with passes). Inaddition, a normalized starting point can be used by specifying the normalize parameter.

BigBatch_SGD, specified by the value ‘bbsgd’ for the parameter optimizer, depends primarily on four parameters: the step size (specified with step_size), the batch size (specified with batch_size), the maximum number of passes (specified with passes). In addition, a normalized starting point can be used by specifying the normalize parameter.

Stochastic gradient descent, specified by the value ‘sgd’ for the parameter optimizer, depends primarily on three parameters: the step size (specified with step_size), the batch size (specified with batch_size), and the maximum number of passes (specified with passes). In addition, a normalized starting point can be used by specifying the normalize parameter. Furthermore, mean-centering can be performed on the dataset by specifying the centerparameter.

The L-BFGS optimizer, specified by the value ‘lbfgs’ for the parameter optimizer, uses a back-tracking line search algorithm to minimize a function. The following parameters are used by L-BFGS: max_iterations, tolerance(the optimization is terminated when the gradient norm is below this value). For more details on the L-BFGS optimizer, consult either the mlpack L-BFGS documentation (in lbfgs.hpp) or the vast set of published literature on L-BFGS. In addition, a normalized starting point can be used by specifying the normalize parameter.

By default, the AMSGrad optimizer is used.

🔗 Example

Example - Let’s say we want to learn distance on iris dataset with number of targets as 3 using BigBatch_SGD optimizer. A simple call for the same will look like:

julia> using CSV
julia> iris = CSV.read("iris.csv")
julia> iris_labels = CSV.read("iris_labels.csv"; type=Int)
julia> _, output, _ = lmnn(iris; k=3, labels=iris_labels,
            optimizer="bbsgd")

An another program call making use of range & regularization parameter with dataset having labels as last column can be made as:

julia> using CSV
julia> letter_recognition = CSV.read("letter_recognition.csv")
julia> _, output, _ = lmnn(letter_recognition; k=5, range=10,
            regularization=0.4)

🔗 See also

• nca()
• Large margin nearest neighbor on Wikipedia
• Distance metric learning for large margin nearest neighbor classification (pdf)
• LMNN C++ class documentation

🔗 local_coordinate_coding()

Local Coordinate Coding

julia> using mlpack: local_coordinate_coding
julia> codes, dictionary, output_model = local_coordinate_coding( ;
          atoms=0, initial_dictionary=zeros(0, 0), input_model=nothing,
          lambda=0, max_iterations=0, normalize=false, seed=0, test=zeros(0, 0),
          tolerance=0.01, training=zeros(0, 0), verbose=false)

An implementation of Local Coordinate Coding (LCC), a data transformation technique. Given input data, this transforms each point to be expressed as a linear combination of a few points in the dataset; once an LCC model is trained, it can be used to transform points later also. Detailed documentation.

🔗 Input options

🔗 Output options

Results are returned as a tuple, and can be unpacked directly into return values or stored directly as a tuple; undesired results can be ignored with the _ keyword.

🔗 Detailed documentation

An implementation of Local Coordinate Coding (LCC), which codes data that approximately lives on a manifold using a variation of l1-norm regularized sparse coding. Given a dense data matrix X with n points and d dimensions, LCC seeks to find a dense dictionary matrix D with k atoms in d dimensions, and a coding matrix Z with n points in k dimensions. Because of the regularization method used, the atoms in D should lie close to the manifold on which the data points lie.

The original data matrix X can then be reconstructed as D * Z. Therefore, this program finds a representation of each point in X as a sparse linear combination of atoms in the dictionary D.

The coding is found with an algorithm which alternates between a dictionary step, which updates the dictionary D, and a coding step, which updates the coding matrix Z.

To run this program, the input matrix X must be specified (with -i), along with the number of atoms in the dictionary (-k). An initial dictionary may also be specified with the initial_dictionary parameter. The l1-norm regularization parameter is specified with the lambda parameter.

🔗 Example

For example, to run LCC on the dataset data using 200 atoms and an l1-regularization parameter of 0.1, saving the dictionary dictionary and the codes into codes, use

julia> using CSV
julia> data = CSV.read("data.csv")
julia> codes, dict, _ = local_coordinate_coding(atoms=200,
            lambda=0.1, training=data)

The maximum number of iterations may be specified with the max_iterations parameter. Optionally, the input data matrix X can be normalized before coding with the normalize parameter.

An LCC model may be saved using the output_model output parameter. Then, to encode new points from the dataset points with the previously saved model lcc_model, saving the new codes to new_codes, the following command can be used:

julia> using CSV
julia> points = CSV.read("points.csv")
julia> new_codes, _, _ =
            local_coordinate_coding(input_model=lcc_model, test=points)

🔗 See also

• sparse_coding()
• Nonlinear learning using local coordinate coding (pdf)
• LocalCoordinateCoding C++ class documentation

🔗 logistic_regression()

L2-regularized Logistic Regression and Prediction

julia> using mlpack: logistic_regression
julia> output_model, predictions, probabilities = logistic_regression(
          ; batch_size=64, decision_boundary=0.5, input_model=nothing,
          labels=Int[], lambda=0, max_iterations=10000, optimizer="lbfgs",
          print_training_accuracy=false, step_size=0.01, test=zeros(0, 0),
          tolerance=1e-10, training=zeros(0, 0), verbose=false)

An implementation of L2-regularized logistic regression for two-class classification. Given labeled data, a model can be trained and saved for future use; or, a pre-trained model can be used to classify new points. Detailed documentation.

🔗 Input options

🔗 Output options

Results are returned as a tuple, and can be unpacked directly into return values or stored directly as a tuple; undesired results can be ignored with the _ keyword.

🔗 Detailed documentation

An implementation of L2-regularized logistic regression using either the L-BFGS optimizer or SGD (stochastic gradient descent). This solves the regression problem

y = (1 / 1 + e^-(X * b)).

In this setting, y corresponds to class labels and X corresponds to data.

This program allows loading a logistic regression model (via the input_model parameter) or training a logistic regression model given training data (specified with the training parameter), or both those things at once. In addition, this program allows classification on a test dataset (specified with the test parameter) and the classification results may be saved with the predictions output parameter. The trained logistic regression model may be saved using the output_model output parameter.

The training data, if specified, may have class labels as its last dimension. Alternately, the labels parameter may be used to specify a separate matrix of labels.

When a model is being trained, there are many options. L2 regularization (to prevent overfitting) can be specified with the lambda option, and the optimizer used to train the model can be specified with the optimizer parameter. Available options are ‘sgd’ (stochastic gradient descent) and ‘lbfgs’ (the L-BFGS optimizer). There are also various parameters for the optimizer; the max_iterations parameter specifies the maximum number of allowed iterations, and the tolerance parameter specifies the tolerance for convergence. For the SGD optimizer, the step_size parameter controls the step size taken at each iteration by the optimizer. The batch size for SGD is controlled with the batch_size parameter. If the objective function for your data is oscillating between Inf and 0, the step size is probably too large. There are more parameters for the optimizers, but the C++ interface must be used to access these.

For SGD, an iteration refers to a single point. So to take a single pass over the dataset with SGD, max_iterations should be set to the number of points in the dataset.

Optionally, the model can be used to predict the responses for another matrix of data points, if test is specified. The test parameter can be specified without the training parameter, so long as an existing logistic regression model is given with the input_model parameter. The output predictions from the logistic regression model may be saved with the predictions parameter.

This implementation of logistic regression does not support the general multi-class case but instead only the two-class case. Any labels must be either 0 or 1. For more classes, see the softmax regression implementation.

🔗 Example

As an example, to train a logistic regression model on the data ‘data’ with labels ‘labels’ with L2 regularization of 0.1, saving the model to ‘lr_model’, the following command may be used:

julia> using CSV
julia> data = CSV.read("data.csv")
julia> labels = CSV.read("labels.csv"; type=Int)
julia> lr_model, _, _ = logistic_regression(labels=labels,
            lambda=0.1, print_training_accuracy=1, training=data)

Then, to use that model to predict classes for the dataset ‘test’, storing the output predictions in ‘predictions’, the following command may be used:

julia> using CSV
julia> test = CSV.read("test.csv")
julia> _, predictions, _ = logistic_regression(input_model=lr_model,
            test=test)