Multivariate adaptive regression spline

This section introduces MARS using a few examples. We start with a set of data: a matrix of input variables x, and a vector of the observed responses y, with a response for each row in x. For example, the data could be:

Here there is only one independent variable, so the x matrix is just a single column. Given these measurements, we would like to build a model which predicts the expected y for a given x.

A linear model

A linear model for the above data is

${\displaystyle {\widehat {y}}=-37+5.1x}$

The hat on the ${\displaystyle {\widehat {y}}}$ indicates that ${\displaystyle {\widehat {y}}}$ is estimated from the data. The figure on the right shows a plot of this function: a line giving the predicted ${\displaystyle {\widehat {y}}}$ versus x, with the original values of y shown as red dots.

The data at the extremes of x indicates that the relationship between y and x may be non-linear (look at the red dots relative to the regression line at low and high values of x). We thus turn to MARS to automatically build a model taking into account non-linearities. MARS software constructs a model from the given x and y as follows

${\displaystyle {\begin{aligned}{\widehat {y}}=&\ 25\\&{}+6.1\max(0,x-13)\\&{}-3.1\max(0,13-x)\end{aligned}}}$

A simple MARS model of the same data

The figure on the right shows a plot of this function: the predicted ${\displaystyle {\widehat {y}}}$ versus x, with the original values of y once again shown as red dots. The predicted response is now a better fit to the original y values.

MARS has automatically produced a kink in the predicted y to take into account non-linearity. The kink is produced by hinge functions. The hinge functions are the expressions starting with ${\displaystyle \max }$ (where ${\displaystyle \max(a,b)}$ is ${\displaystyle a}$ if ${\displaystyle a>b}$, else ${\displaystyle b}$). Hinge functions are described in more detail below.

In this simple example, we can easily see from the plot that y has a non-linear relationship with x (and might perhaps guess that y varies with the square of x). However, in general there will be multiple independent variables, and the relationship between y and these variables will be unclear and not easily visible by plotting. We can use MARS to discover that non-linear relationship.

An example MARS expression with multiple variables is

${\displaystyle {\begin{aligned}\mathrm {ozone} =&\ 5.2\\&{}+0.93\max(0,\mathrm {temp} -58)\\&{}-0.64\max(0,\mathrm {temp} -68)\\&{}-0.046\max(0,234-\mathrm {ibt} )\\&{}-0.016\max(0,\mathrm {wind} -7)\max(0,200-\mathrm {vis} )\end{aligned}}}$

Variable interaction in a MARS model

This expression models air pollution (the ozone level) as a function of the temperature and a few other variables. Note that the last term in the formula (on the last line) incorporates an interaction between ${\displaystyle \mathrm {wind} }$ and ${\displaystyle \mathrm {vis} }$.

The figure on the right plots the predicted ${\displaystyle \mathrm {ozone} }$ as ${\displaystyle \mathrm {wind} }$ and ${\displaystyle \mathrm {vis} }$ vary, with the other variables fixed at their median values. The figure shows that wind does not affect the ozone level unless visibility is low. We see that MARS can build quite flexible regression surfaces by combining hinge functions.

To obtain the above expression, the MARS model building procedure automatically selects which variables to use (some variables are important, others not), the positions of the kinks in the hinge functions, and how the hinge functions are combined.

MARS builds models of the form

${\displaystyle {\widehat {f}}(x)=\sum _{i=1}^{k}c_{i}B_{i}(x).}$

The model is a weighted sum of basis functions ${\displaystyle B_{i}(x)}$. Each ${\displaystyle c_{i}}$ is a constant coefficient. For example, each line in the formula for ozone above is one basis function multiplied by its coefficient.

Each basis function ${\displaystyle B_{i}(x)}$ takes one of the following three forms:

1) a constant 1. There is just one such term, the intercept. In the ozone formula above, the intercept term is 5.2.

2) a hinge function. A hinge function has the form ${\displaystyle \max(0,x-{\text{constant}})}$ or ${\displaystyle \max(0,{\text{constant}}-x)}$. MARS automatically selects variables and values of those variables for knots of the hinge functions. Examples of such basis functions can be seen in the middle three lines of the ozone formula.

3) a product of two or more hinge functions. These basis functions can model interaction between two or more variables. An example is the last line of the ozone formula.

A mirrored pair of hinge functions with a knot at x=3.1

A key part of MARS models are hinge functions taking the form

${\displaystyle \max(0,x-c)}$

or

${\displaystyle \max(0,c-x)}$

where ${\displaystyle c}$ is a constant, called the knot. The figure on the right shows a mirrored pair of hinge functions with a knot at 3.1.

A hinge function is zero for part of its range, so can be used to partition the data into disjoint regions, each of which can be treated independently. Thus for example a mirrored pair of hinge functions in the expression

${\displaystyle 6.1\max(0,x-13)-3.1\max(0,13-x)}$

creates the piecewise linear graph shown for the simple MARS model in the previous section.

One might assume that only piecewise linear functions can be formed from hinge functions, but hinge functions can be multiplied together to form non-linear functions.

Hinge functions are also called ramp, hockey stick, or rectifier functions. Instead of the ${\displaystyle \max }$ notation used in this article, hinge functions are often represented by ${\displaystyle [\pm (x_{i}-c)]_{+}}$ where ${\displaystyle [\cdot ]_{+}}$ means take the positive part.

MARS builds a model in two phases: the forward and the backward pass. This two-stage approach is the same as that used by recursive partitioning trees.

No regression modeling technique is best for all situations. The guidelines below are intended to give an idea of the pros and cons of MARS, but there will be exceptions to the guidelines. It is useful to compare MARS to recursive partitioning and this is done below. (Recursive partitioning is also commonly called regression trees, decision trees, or CART; see the recursive partitioning article for details).

• MARS models are more flexible than linear regression models.
• MARS models are simple to understand and interpret.[5] Compare the equation for ozone concentration above to, say, the innards of a trained neural network or a random forest.
• MARS can handle both continuous and categorical data.[6][7] MARS tends to be better than recursive partitioning for numeric data because hinges are more appropriate for numeric variables than the piecewise constant segmentation used by recursive partitioning.
• Building MARS models often requires little or no data preparation.[5] The hinge functions automatically partition the input data, so the effect of outliers is contained. In this respect MARS is similar to recursive partitioning which also partitions the data into disjoint regions, although using a different method.
• MARS (like recursive partitioning) does automatic variable selection (meaning it includes important variables in the model and excludes unimportant ones). However, there can be some arbitrariness in the selection, especially when there are correlated predictors, and this can affect interpretability.[5]
• MARS models tend to have a good bias-variance trade-off. The models are flexible enough to model non-linearity and variable interactions (thus MARS models have fairly low bias), yet the constrained form of MARS basis functions prevents too much flexibility (thus MARS models have fairly low variance).
• MARS is suitable for handling large datasets, and implementations run very quickly. However, recursive partitioning can be faster than MARS.
• With MARS models, as with any non-parametric regression, parameter confidence intervals and other checks on the model cannot be calculated directly (unlike linear regression models). Cross-validation and related techniques must be used for validating the model instead.
• The earth, mda, and polspline implementations do not allow missing values in predictors, but free implementations of regression trees (such as rpart and party) do allow missing values using a technique called surrogate splits.
• MARS models can make predictions very quickly, as they only require evaluating a linear function of the predictors.
• The resulting fitted function is continuous, unlike recursive partitioning, which can give a more realistic model in some situations. (However, the model is not smooth or differentiable).

• Generalized linear models (GLMs) can be incorporated into MARS models by applying a link function after the MARS model is built. Thus, for example, MARS models can incorporate logistic regression to predict probabilities.
• Non-linear regression is used when the underlying form of the function is known and regression is used only to estimate the parameters of that function. MARS, on the other hand, estimates the functions themselves, albeit with severe constraints on the nature of the functions. (These constraints are necessary because discovering a model from the data is an inverse problem that is not well-posed without constraints on the model.)
• Recursive partitioning (commonly called CART). MARS can be seen as a generalization of recursive partitioning that allows for continuous models, which can provide a better fit for numerical data.
• Generalized additive models. Unlike MARS, GAMs fit smooth loess or polynomial splines rather than hinge functions, and they do not automatically model variable interactions. The smoother fit and lack of regression terms reduces variance when compared to MARS, but ignoring variable interactions can worsen the bias.
• TSMARS. Time Series Mars is the term used when MARS models are applied in a time series context. Typically in this set up the predictors are the lagged time series values resulting in autoregressive spline models. These models and extensions to include moving average spline models are described in "Univariate Time Series Modelling and Forecasting using TSMARS: A study of threshold time series autoregressive, seasonal and moving average models using TSMARS".
• Bayesian MARS (BMARS) uses the same model form, but builds the model using a Bayesian approach. It may arrive at different optimal MARS models because the model building approach is different. The result of BMARS is typically an ensemble of posterior samples of MARS models, which allows for probabilistic prediction.[8]

• Linear regression
• Local regression
• Rational function modeling
• Segmented regression
• Spline interpolation
• Spline regression

• Hastie T., Tibshirani R., and Friedman J.H. (2009) The Elements of Statistical Learning, 2nd edition. Springer, ISBN 978-0-387-84857-0 (has a section on MARS)
• Faraway J. (2005) Extending the Linear Model with R, CRC, ISBN 978-1-58488-424-8 (has an example using MARS with R)
• Heping Zhang and Burton H. Singer (2010) Recursive Partitioning and Applications, 2nd edition. Springer, ISBN 978-1-4419-6823-4 (has a chapter on MARS and discusses some tweaks to the algorithm)
• Denison D.G.T., Holmes C.C., Mallick B.K., and Smith A.F.M. (2004) Bayesian Methods for Nonlinear Classification and Regression, Wiley, ISBN 978-0-471-49036-4
• Berk R.A. (2008) Statistical learning from a regression perspective, Springer, ISBN 978-0-387-77500-5

Several free and commercial software packages are available for fitting MARS-type models.

Free software

• R packages:
  • earth function in the earth package
  • mars function in the mda package
  • polymars function in the polspline package. Not Friedman's MARS.
  • bass function in the BASS package for Bayesian MARS.
• Matlab code:
  • ARESLab: Adaptive Regression Splines toolbox for Matlab
  • Code from the book Bayesian Methods for Nonlinear Classification and Regression[1] for Bayesian MARS.
• Python
  • Earth – Multivariate adaptive regression splines
  • py-earth
  • pyBASS for Bayesian MARS.

Commercial software

• MARS from Salford Systems. Based on Friedman's implementation.
• STATISTICA Data Miner from StatSoft
• ADAPTIVEREG from SAS.