Regression Analysis - Study Notes
Chapter Summary
Regression analysis finds the mathematical functional relationship between a dependent (response) variable and one or more independent variables (regressors or predictors). Correlation describes the type and intensity of a linear relationship but gives no function for prediction; regression supplies it and is widely used for prediction.
Background
- Sir Francis Galton (1822-1911) coined the terms correlation and regression, and also quartile, decile and percentile.
- The idea comes from genetics: Galton's "regression towards mediocrity in hereditary stature" showed that extreme characteristics such as height in parents are not passed on completely to their offspring.
- Galton's work on the inherited characteristics of sweet peas led to the first idea of linear regression; Galton and Karl Pearson later developed multiple regression and the product-moment correlation coefficient.
Types of Regression
| Type | Meaning |
|---|---|
| Simple linear regression | Linear relationship with one independent variable: \( Y = a + bX + e \) |
| Multiple linear regression | Linear relationship with two or more independent variables: \( Y = a + b_1X_1 + b_2X_2 + \dots + b_tX_t + e \) |
| Non-linear (curvilinear) regression | The relationship is not a straight line |
In \( Y = a + bX + e \), \( X \) is the independent variable, \( Y \) the dependent variable, \( a \) the intercept, \( b \) the slope and \( e \) the error term. Multiple and non-linear regression are outside the syllabus and are given only for knowledge.
Uses of Regression
- It gives the mathematical relationship between \( X \) and \( Y \) (model construction).
- It shows the strength of impact \( b \) of the independent variable on the dependent variable.
- It estimates (interpolates) the response for values of the regressor within the range of the data; extrapolation is not generally permissible.
- With several independent variables, it sorts out which variables matter most and which can be ignored.
Why Two Regression Lines?
When \( X \) and \( Y \) are interchangeable, we get the regression line of \( Y \) on \( X \) and that of \( X \) on \( Y \). Both are valid, but one must choose the one suited to the situation. If \( X \) only causes \( Y \), there is only one line, of \( Y \) on \( X \).
Assumptions
- The relationship between \( Y \) and \( X \) is (approximately) linear.
- The error term \( e \) is a random variable with mean zero and constant variance.
- The variables are measured on an interval scale; linear regression is very sensitive to outliers.
Method of Least Squares
The line of best fit minimises the error in estimating \( Y \) for any \( X \) in its range. The residual is \( e_i = y_i - \hat{y}_i \). The method chooses \( a \) and \( b \) so that
- the sum of the residuals is zero: \( \sum (y_i - \hat{y}_i) = 0 \);
- the sum of the squares of the residuals, \( E(a,b) = \sum (y_i - a - bx_i)^2 \), is the least.
Setting the partial derivatives of \( E(a,b) \) to zero gives the normal equations:
- \( na + b\sum x_i = \sum y_i \)
- \( a\sum x_i + b\sum x_i^2 = \sum x_i y_i \)
Their solution is \( \hat{a} = \bar{y} - \hat{b}\bar{x} \) and \( \hat{b} = \frac{Cov(X,Y)}{V(X)} = r_{XY}\frac{SD(Y)}{SD(X)} \).
Regression Coefficients
| Y on X | X on Y | |
|---|---|---|
| Coefficient | \( b_{YX} = r_{XY}\frac{SD(Y)}{SD(X)} \) | \( b_{XY} = r_{XY}\frac{SD(X)}{SD(Y)} \) |
| From sums | \( \frac{n\sum xy - \sum x \sum y}{n\sum x^2 - (\sum x)^2} \) | \( \frac{n\sum xy - \sum x \sum y}{n\sum y^2 - (\sum y)^2} \) |
| Slope-point form | \( \hat{Y} - \bar{y} = b_{YX}(x - \bar{x}) \) | \( \hat{X} - \bar{x} = b_{XY}(y - \bar{y}) \) |
The slope \( b_{YX} \) is the change in the mean of \( Y \) for every unit change in \( X \). When the means are decimals, deviations from assumed values \( u = x - A \), \( v = y - B \) can be used, since the coefficients do not change with origin.
Properties of Regression Coefficients
- \( r_{XY} = \sqrt{b_{XY} \times b_{YX}} \): correlation is the geometric mean of the coefficients.
- Both coefficients have the same sign.
- If one coefficient is greater than unity, the other is less than unity.
- \( r_{XY} \) has the same sign as the coefficients.
- \( \frac{b_{XY} + b_{YX}}{2} \ge r_{XY} \).
- They are independent of change of origin but not of scale.
Properties of Regression Equations
- If \( r = 0 \), the lines are perpendicular to each other.
- If \( r = \pm 1 \), the lines coincide (the book says coincide or parallel).
- Angle between them: \( \theta = \tan^{-1}\left(\frac{m_1 - m_2}{1 + m_1 m_2}\right) \); it shows the degree of dependence.
- They intersect at \( (\bar{X}, \bar{Y}) \), so solving the two equations gives the means.
Correlation versus Regression
| Correlation | Regression |
|---|---|
| Shows only the nature and extent of linear relationship | Studies the impact of the independent variable on the dependent variable; used for prediction |
| Either variable can be x or y | Independent and dependent variables must be chosen with care |
| Symmetric: \( r_{XY} = r_{YX} \) | Not symmetric: \( b_{XY} \) and \( b_{YX} \) differ in meaning |
Worked Example (book Example 5.7)
Yield: mean 10, SD 8; rainfall: mean 8, SD 2; \( r = 0.5 \). \( b_{YX} = 0.5 \times \frac{8}{2} = 2 \), so at rainfall 9, \( \hat{Y} = 10 + 2(9 - 8) = 12 \) kg per unit area.