Module elastic_net

Expand description

§Elastic Net

Elastic net is an extension of linear regression that adds regularization penalties to the loss function during training. Just like in ordinary linear regression you assume a linear relationship between input variables and the target variable. Unlike linear regression elastic net adds regularization penalties to the loss function during training. In particular, the elastic net coefficient estimates \(\beta\) are the values that minimize

\[L(\alpha, \beta) = \vert \boldsymbol{y} - \boldsymbol{X}\beta\vert^2 + \lambda_1 \vert \beta \vert^2 + \lambda_2 \vert \beta \vert_1\]

where \(\lambda_1 = \alpha l_{1r}\), \(\lambda_2 = \alpha (1 - l_{1r})\) and \(l_{1r}\) is the l1 ratio, elastic net mixing parameter.

In essense, elastic net combines both the L1 and L2 penalties during training, which can result in better performance than a model with either one or the other penalty on some problems. The elastic net is particularly useful when the number of predictors (p) is much bigger than the number of observations (n).

Example:

use smartcore::linalg::basic::matrix::DenseMatrix;
use smartcore::linear::elastic_net::*;

// Longley dataset (https://www.statsmodels.org/stable/datasets/generated/longley.html)
let x = DenseMatrix::from_2d_array(&[
              &[234.289, 235.6, 159.0, 107.608, 1947., 60.323],
              &[259.426, 232.5, 145.6, 108.632, 1948., 61.122],
              &[258.054, 368.2, 161.6, 109.773, 1949., 60.171],
              &[284.599, 335.1, 165.0, 110.929, 1950., 61.187],
              &[328.975, 209.9, 309.9, 112.075, 1951., 63.221],
              &[346.999, 193.2, 359.4, 113.270, 1952., 63.639],
              &[365.385, 187.0, 354.7, 115.094, 1953., 64.989],
              &[363.112, 357.8, 335.0, 116.219, 1954., 63.761],
              &[397.469, 290.4, 304.8, 117.388, 1955., 66.019],
              &[419.180, 282.2, 285.7, 118.734, 1956., 67.857],
              &[442.769, 293.6, 279.8, 120.445, 1957., 68.169],
              &[444.546, 468.1, 263.7, 121.950, 1958., 66.513],
              &[482.704, 381.3, 255.2, 123.366, 1959., 68.655],
              &[502.601, 393.1, 251.4, 125.368, 1960., 69.564],
              &[518.173, 480.6, 257.2, 127.852, 1961., 69.331],
              &[554.894, 400.7, 282.7, 130.081, 1962., 70.551],
         ]).unwrap();

let y: Vec<f64> = vec![83.0, 88.5, 88.2, 89.5, 96.2, 98.1, 99.0,
          100.0, 101.2, 104.6, 108.4, 110.8, 112.6, 114.2, 115.7, 116.9];

let y_hat = ElasticNet::fit(&x, &y, Default::default()).
                and_then(|lr| lr.predict(&x)).unwrap();

§References:

Structs§

ElasticNet: Elastic net
ElasticNetParameters: Elastic net parameters
ElasticNetSearchParameters: ElasticNet grid search parameters
ElasticNetSearchParametersIterator: ElasticNet grid search iterator