Simple linear Regression

Simple Linear Regression Simple linear regression provides a model of the relationship between the magnitude of one variable and that of a second for example, as X increases, Y also increases. Or as X increases, Y decreases.1 Correlation is another way to measure how two variables are related see. Stimates how much Y will change when X changes by a certain amount. With the correlation coefficient, the variables X and Y are interchangeable, With regression, we are trying to predict the Y variable from X using a linear relationship

Y_{i}=b_{0}+b_{1}X_{i}+e_{i} Y_{i}=b_{0}+b_{1}X_{i}+e_{i}

  • Response The variable we are trying to predict. Synonyms dependent variable, Y variable, target, outcome

  • Independent variable The variable used to predict the response. Synonyms X variable, feature, attribute, predictor

  • Record The vector of predictor and outcome values for a specific individual or case. Synonyms row, case, instance, example

  • Intercept The intercept of the regression line—that is, the predicted value when X = 0. Synonyms b0, β0

  • Regression coefficient The slope of the regression line. Synonyms slope, b1, β1, parameter estimates, weights

  • Fitted values The estimates Yi obtained from the regression line. Synonym predicted values

  • Residuals The difference between the observed values and the fitted values. Synonym errors

  • Least squares The method of fitting a regression by minimizing the sum of squared residuals. Synonym Ordinary least squares, OLS

Multiple linear Regression

  • Least squares The method of fitting a regression by minimizing the sum of squared residuals.

  • Root mean squared error The square root of the average squared error of the regression (this is the most widely used metric to compare regression models). Synonym RMSE

  • Residual standard error The same as the root mean squared error, but adjusted for degrees of freedom. Synonym RSE

  • R-squared The proportion of variance explained by the model, from 0 to 1. Synonyms coefficient of determination, R2

  • t-statistic The coefficient for a predictor, divided by the standard error of the coefficient, giving a metric to compare the importance of variables in the model. See “t-Tests” on page 110.

  • Weighted regression Regression with the records having different weights.

The primary purpose of regression in data science is prediction. This is useful to keep in mind, since regression, being an old and established statistical method, comes with baggage that is more relevant to its traditional role as a tool for explanatory modeling

Factor Variables

  • Dummy variables Binary 0–1 variables derived by recoding factor data for use in regression and other models.

  • Reference coding The most common type of coding used by statisticians, in which one level of a factor is used as a reference and other factors are compared to that level. Synonym treatment coding

  • One hot encoder A common type of coding used in the machine learning community in which all factor levels are retained. While useful for certain machine learning algorithms, this approach is not appropriate for multiple linear regression.

  • Deviation coding A type of coding that compares each level against the overall mean as opposed to the reference level. Synonym sum contrasts Dummy Variables Representation

Interpreting Regression Equation

  • Correlated variables When the predictor variables are highly correlated, it is difficult to interpret the individual coefficients.

  • Multicollinearity When the predictor variables have perfect, or near-perfect, correlation, the regression can be unstable or impossible to compute. Synonym collinearity

  • Confounding variables An important predictor that, when omitted, leads to spurious relationships in a regression equation.

  • Main effects The relationship between a predictor and the outcome variable, independent of other variables.

  • Interactions An interdependent relationship between two or more predictors and the

Regression Diagnostics

  • Standardized residuals Residuals divided by the standard error of the residuals.

  • Outliers Records (or outcome values) that are distant from the rest of the data (or the pre‐ dicted outcome).

  • Influential value A value or record whose presence or absence makes a big difference in the regression equation.

  • Leverage The degree of influence that a single record has on a regression equation. Synonym hat-value

  • Non-normal residuals Non-normally distributed residuals can invalidate some technical requirements of regression but are usually not a concern in data science.

  • Heteroskedasticity When some ranges of the outcome experience residuals with higher variance (may indicate a predictor missing from the equation).

  • Partial residual plots A diagnostic plot to illuminate the relationship between the outcome variable

Polynomial and Spline Regression

Polynomial regression involves including polynomial terms in a regression equation. The use of polynomial regression dates back almost to the development of regression itself with a paper by Gergonne in 1815. For example, a quadratic regression between

  • Polynomial regression Adds polynomial terms (squares, cubes, etc.) to a regression.
  • Spline regression Fitting a smooth curve with a series of polynomial segments.
  • Knots Values that separate spline segments.
  • Generalized additive models Spline models with automated selection of knots. Synonym GAM
predictors = ['SqFtTotLiving', 'SqFtLot', 'Bathrooms',  'Bedrooms', 'BldgGrade'] 
outcome = 'AdjSalePrice'
X = house_98105[predictors].values 
y = house_98105[outcome] 

gam = LinearGAM(s(0, n_splines=12) + l(1) + l(2) + l(3) + l(4))   
gam.gridsearch(X, y)