Penalized linear models
Penalized linear models introduce an penalty term to the minimization in addition to the squares of the residuals. These are typically of the form, λ∑i|βi|k, for different numbers k. The motivation for this extra term is to introduce some stability of the β, and requires picking a parameter λ which tunes how much the least squares part and how much the penalty affect the solution. When k=2, this is referred to as ridge regression, Tikhonov regularization, or L2 regularization. When k=1, this is referred to as LASSO or L1 regularization. A good reference for these penalized linear models is the Elements of Statistical Learning textbook, which is available as a free pdf. Some R packages which implement penalized linear models are the lm.ridge function in the MASS package, the lars package, and the glmnet package.