Correlation and Regression Analysis

Do students who study more score more? Does server load rise with users? Correlation measures the strength and direction of a linear relationship between two variables; regression gives an equation to predict one from the other.

3.1 Correlation

Two variables are correlated if they change together: positive (both rise: study hours and marks), negative (one rises as the other falls: price and demand), or zero/no correlation.

Correlation is not causation: ice-cream sales and drowning both rise in summer without either causing the other.

Scatter diagram: plot each pair (x ,\ y) as a point. Points rising from lower left to upper right indicate positive correlation, falling points negative, a shapeless cloud none; points close to a straight line indicate strong linear correlation.

Karl Pearson's coefficient of correlation r = Cov(x, y)/(σₓσᵧ) = Σ(x − x̄)(y − ȳ)/√[Σ(x − x̄)² · Σ(y − ȳ)²]. Computational form:

Properties: - 1 \leq r \leq + 1; r = + 1 perfect positive, −1 perfect negative, 0 no linear correlation; r is unchanged by a change of origin or positive scale; r has no units.

Rough interpretation: |r| < 0.3 weak, 0.3–0.7 moderate, > 0.7 strong. The coefficient of determination r^{2} is the proportion of variation in y explained by x.

Spearman's rank correlation for ordered data is r_{s} = 1 - \frac{6 \sum d^{2}}{n (n^{2} - 1)}.

3.2 Regression analysis

Regression finds the line of best fit used to estimate the dependent variable (y, the response) from the independent variable (x, the predictor).

The least squares method chooses the line y = a + b x that minimises Σ(y − ŷ)², the sum of squared vertical errors. Setting derivatives to zero gives the normal equations:

Solution: b = Sxy/Sxx = \frac{n \sum x y - \sum x \sum y}{n \sum x^{2} - \left( \sum x \right)^{2}}, a = ȳ − b \bar{x}. The slope b is the regression coefficient of y on x (b_{y} x): the change in y per unit x.

If we regress x on y we get a different line, x = a' + b' y, with b_{x} y = Sxy/Syy. The two lines intersect at (\bar{x}, ȳ), and r^{2} = b_{y} x \times b_{x} y (r takes the sign of the b's).

If r = \pm 1 the two lines coincide; if r = 0 they are perpendicular (horizontal and vertical).

Dependent and independent variables: choose which to predict before fitting; the y-on-x line minimises vertical errors, and is wrong for predicting x from y.

Assumptions: linear relationship, errors independent with roughly constant variance, no influential outliers. Using software: Excel =CORREL(x,y), =SLOPE(y,x), =INTERCEPT(y,x); SPSS Analyze → Correlate / Regression; Python numpy.polyfit.

(a) Karl Pearson, English statistician

(b) Least squares: residuals of the worked example

3.4 Eight uses of correlation and regression

1. Predicting a number from another. A simple regression can estimate a student's final mark from internal marks. The line gives a quick, honest estimate with a known error. A college can then warn students who are likely to fail early.
2. Forecasting demand and sales. Shops link sales to price, festival season or rainfall. Before Dashain, a Kathmandu electronics shop fits a line of past sales on advertising spend to plan stock. Better forecasts reduce both lost sales and unsold goods.
3. Machine learning baseline. Linear regression is the first model every data scientist tries. It is fast, easy to explain and a yardstick for complex models. Slope and intercept show directly which factor matters.
4. Finding related features. Correlation between columns of a data set shows duplicated information. If two sensors have r = 0.99, one can be dropped to save storage. Analysts also use r to pick features that relate to the target.
5. Performance modelling in computing. Run time often grows with input size, and regression estimates by how much. A developer fits time against file size to predict when a script will become too slow. Server planners do the same with traffic and CPU load.
6. Economics and public policy. Economists relate income to spending, and rainfall to paddy yield. Nepal Rastra Bank studies relationships such as remittance and consumption this way. Policy makers still remember that correlation alone does not prove cause.
7. Health and agriculture. Doctors relate dose to response, and farmers relate fertiliser to yield. The slope tells how much extra yield one more kilogram of fertiliser gives. The r^{2} value shows how much of the yield variation fertiliser explains.
8. Quality checks of other methods. Calibration of a cheap instrument against an accurate one uses a regression line. An r close to 1 shows the cheap instrument is trustworthy. The residuals reveal where it fails.

(a) Sir Francis Galton (regression, correlation)

(b) Regression to the mean in 1,000 simulated pairs

(a) An ice-cream cart on the beach

(b) A confounder creates a correlation

Formula and fact sheet

• r = \frac{\mathrm{Sxy}}{\sqrt{\mathrm{Sxx} \cdot \mathrm{Syy}}} ;\ - 1 \leq r \leq 1 ;\ r^{2} = proportion explained.
• Least squares: b = Sxy/Sxx, a = ȳ − b \bar{x}; ŷ = a + b x; lines meet at (\bar{x}, ȳ).
• b_{y} x \times b_{x} y = r^{2}. Spearman r_{s} = 1 - \frac{6 \sum d^{2}}{n (n^{2} - 1)}.

Chapter summary

• Correlation measures linear association with r from −1 to +1; correlation does not prove causation.
• Least squares regression gives the best-fitting line for predicting y from x; the two regression lines meet at the means and their slopes multiply to r^{2}.

Practice

1. Differentiate correlation and regression.
2. For x = 2 ,\ 4 ,\ 6, 8 and y = 3 ,\ 7 ,\ 5, 10 compute r and the regression line of y on x; predict y for x = 10.
3. Interpret r = - 0.85 and r = 0.1.
4. If b_{y} x = 0.8 and b_{x} y = 0.45, find r.
5. Why must the regression line of y on x not be used to predict x?
6. State three properties of r.

Answers and worked solutions

1. Correlation measures strength and direction of association (symmetric); regression gives an equation to predict a dependent variable from an independent one (asymmetric).

2. \bar{x} = 5, ȳ = 6.25.

Sxx = 9 + 1 + 1 + 9 = 20; Syy = 26.75; Sxy = (- 3) (- 3.25) + (- 1) (0.75) + (1) (- 1.25) + (3) (3.75) = 9.75 - 0.75 - 1.25 + 11.25 = 19. r = \frac{19}{\sqrt{20 \times 26.75}} = 19 / 23.13 = 0.821. b = \frac{19}{20} = 0.95; a = 6.25 - 0.95 \times 5 = 1.5; ŷ = 1.5 + 0.95x; at x = 10, ŷ = 11.0.

3. −0.85: strong negative linear relationship; 0.1: practically no linear relationship.

4. r^{2} = 0.8 \times 0.45 = 0.36 ,\ r = +0.6 (both coefficients positive).

5. That line minimises errors in y, not in x; the x-on-y line is the correct one for estimating x.

6. Between −1 and 1; independent of origin and scale; unitless.

Model answers: medium (5 marks) and long (10 marks) questions

Q1 (5 marks). A shop records advertising spend x (in thousand rupees) and weekly sales y (in tens of thousands): x = 2 ,\ 4 ,\ 6 ,\ 8, 10 and y = 5 ,\ 7 ,\ 10 ,\ 12 ,\ 16. Find Karl Pearson's coefficient of correlation and the regression line of y on x.

Introduction. Karl Pearson's r measures the strength and direction of a linear relationship between two quantitative variables. Regression then gives the line for prediction.

Working. n = 5 ,\ \sum x = 30 ,\ \sum y = 50 ,\ \sum x y = 354 ,\ \sum x^{2} = 220 ,\ \sum y^{2} = 574.

Correlation. r = \frac{n \sum x y - \sum x \sum y}{\sqrt{\{\left[ n \sum x^{2} - \left( \sum x \right)^{2} \right] \left[ n \sum y^{2} - \left( \sum y \right)^{2} \right]\}}} = \frac{5 \times 354 - 30 \times 50}{\sqrt{(1100 - 900) (2870 - 2500)}} = \frac{270}{\sqrt{200 \times 370}} = 270 / 272.03 = 0.9925.

This is a very strong positive correlation.

Regression. b = Sxy/Sxx = 270/200 = 1.35. \bar{x} = 6, ȳ = 10, so a = 10 - 1.35 \times 6 = 1.9. The line is ŷ = 1.9 + 1.35x.

Interpretation. Every extra thousand rupees of advertising goes with 1.35 tens of thousands of extra sales, and r^{2} = 0.985 means 98.5% of the variation in sales is associated with advertising.

Conclusion Spend and sales move together very closely, but this data alone does not prove that advertising caused the sales.

Q2 (5 marks). Six students have marks in Mathematics: 86, 72, 65, 91, 58, 77 and in Statistics: 80, 70, 60, 88, 62, 75. Calculate Spearman's rank correlation coefficient and say when it is preferred to Pearson's r.

Introduction. Spearman's rank correlation r_{s} = 1 - \frac{6 \sum d^{2}}{n (n^{2} - 1)} measures association between the ranks of two variables, where d is the difference of ranks.

*Ranking (highest = 1 ) . \ast

Maths	86	72	65	91	58	77
Rank R_{1}	2	4	5	1	6	3
Statistics	80	70	60	88	62	75
Rank R_{2}	2	4	6	1	5	3
d = R_{1} - R_{2}	0	0	−1	0	1	0

\sum d^{2} = 0 + 0 + 1 + 0 + 1 + 0 = 2, and n = 6. So r_{s} = 1 - \frac{6 \times 2}{6 \times 35} = 1 − 12/210 = 0.943.

Interpretation. The two rankings agree almost perfectly: students strong in Mathematics are strong in Statistics.

When preferred Spearman is used when data are only ordinal (ranks, grades), when the relation is monotonic but not a straight line, or when outliers would distort Pearson's r.

Conclusion r_{s} = 0.943 shows a very strong positive association.

Q3 (10 marks). Explain correlation and regression. The CPU load x (%) and response time y (ms) of a server were: x = 10 ,\ 20 ,\ 30 ,\ 40 ,\ 50, 60 and y = 12 ,\ 15 ,\ 21 ,\ 23 ,\ 30 ,\ 31. Find r, both regression lines, verify r^{2} = b_{y} x \times b_{x} y, and predict y at x = 45.

Introduction. Correlation measures how closely two variables move together; regression fits a line to predict one variable (y, dependent) from the other (x, independent). Figure 3.1 shows the patterns of r and Figure 3.2 the two regression lines.

Key ideas.

1. Scatter diagram. Plot the pairs first; a curve or an outlier makes r misleading.
2. Pearson's r = \frac{\mathrm{Sxy}}{\sqrt{\mathrm{Sxx} \cdot \mathrm{Syy}}}, with - 1 \leq r \leq + 1, unit-free and unchanged by a change of origin or scale.
3. Coefficient of determination r^{2} is the share of variation in y explained by x.
4. Least squares chooses ŷ = a + b x to minimise Σ(y − ŷ)²; b = Sxy/Sxx and a = ȳ − b \bar{x}.
5. Two lines. y on x has slope b_{y} x = Sxy/Sxx; x on y has slope b_{x} y = Sxy/Syy. They intersect at (\bar{x}, ȳ).
6. Link. r^{2} = b_{y} x \times b_{x} y, and r has the sign of both coefficients.

Calculation. n = 6 ,\ \sum x = 210 ,\ \sum y = 132 ,\ \sum x^{2} = 9100 ,\ \sum y^{2} = 3200 ,\ \sum x y = 5330. \bar{x} = 35, ȳ = 22. Sxx = 9100 - 210^{2} / 6 = 1750. Syy = 3200 - 132^{2} / 6 = 296. Sxy = 5330 - \frac{210 \times 132}{6} = 710.

Correlation. r = \frac{710}{\sqrt{1750 \times 296}} = 710 / 719.7 = 0.9865, r^{2} = 0.973.

*Regression of y on x . \ast b_{y} x = 710/1750 = 0.4057; a = 22 - 0.4057 \times 35 = 7.80. So ŷ = 7.80 + 0.4057x.

*Regression of x on y . \ast b_{x} y = 710/296 = 2.3986, so x̂ = 35 + 2.3986 (y - 22).

Verification. b_{y} x \times b_{x} y = 0.4057 \times 2.3986 = 0.9732 = r^{2} ✓.

Prediction. At x = 45%: ŷ = 7.80 + 0.4057 \times 45 = 26.06 ms. This is interpolation (45 lies inside 10-60) and is reliable; predicting at 90% load would be extrapolation and may fail.

Comparison (Table).

Point	Correlation	Regression
Purpose	strength of association	prediction of y from x
Variables	symmetric, x and y interchangeable	asymmetric, dependent and independent
Result	one number r	an equation a + b x
Units	none	units of y per unit of x

Conclusion. Response time rises by about 0.41 ms for every extra 1% of CPU load, and 97% of the variation is explained by load. The relation is strong, but the model is valid only within the observed range.