Regression analysis: a complete guide

Regression fits a line, or a plane, through your data so that the total squared distance from the observed points to the line is as small as possible. The result is an equation that predicts the outcome from the predictors.

For one predictor, the model is y = b₀ + b₁x + e. The intercept b₀ is the predicted outcome when the predictor is zero, the slope b₁ is how much the outcome changes per unit of the predictor, and e is the residual — the part of each observation the model does not explain.

The family is larger than most courses cover, and choosing the wrong member is a common and consequential error. The decision is driven almost entirely by the nature of the outcome: continuous outcomes take linear regression, binary outcomes take logistic, counts take Poisson or negative binomial, and any outcome measured repeatedly on the same units needs a multilevel or mixed model regardless of its type.

Six students report weekly study hours (x) and achieve exam scores (y). We want the line predicting score from hours.

Step 1 — the means

x̄ = 30 / 6 = 5 and ȳ = 384 / 6 = 64

Step 2 — the slope

b₁ = Σ(x−x̄)(y−ȳ) / Σ(x−x̄)² = 111 / 28 = 3.964

Step 3 — the intercept

b₀ = ȳ − b₁x̄ = 64 − (3.964 × 5) = 64 − 19.82 = 44.18

Step 4 — the equation

score = 44.18 + 3.964 × hours

So each additional weekly study hour is associated with about 3.96 more exam marks, and a student studying zero hours is predicted to score 44.18 — a figure that lies outside the observed range of the data and should be treated with caution.

Step 5 — how well does it fit?

Total sum of squares SSₜₒₜ = Σ(y−ȳ)² = 450. Regression sum of squares SSₕₑₔ = b₁ × Σ(x−x̄)(y−ȳ) = 3.964 × 111 = 440.04.

R² = 440.04 / 450 = 0.978 — the model accounts for about 98% of the variance in scores.

One further quantity is worth knowing about even though it rarely appears in student write-ups: the semi-partial correlation, sometimes called the part correlation. Squared, it gives the proportion of variance in the outcome uniquely attributable to that predictor — the amount R² would fall by if the predictor were removed. It is often more informative than the standardised coefficient for answering “how much does this variable actually add?”, because it is expressed on the same scale as R² and is not distorted by the predictor's own variability. Most software reports it alongside the coefficients; it is simply overlooked.

When to use which

Report the unstandardised coefficient when the units are meaningful — “each additional study hour is worth 3.96 marks” is directly interpretable and is what a reader can act on. Report standardised coefficients when comparing the relative importance of predictors measured on different scales, since the raw coefficients are not comparable across units.

Note also that a coefficient's significance answers a narrower question than most readers assume. It tests whether that predictor adds anything once the others are in the model. A predictor strongly related to the outcome on its own can appear non-significant simply because another predictor already captures the same information.

R² increases mechanically with every predictor added, including random noise, which is why adjusted R² exists and why it should be the one reported whenever there is more than one predictor. If adding a variable raises R² but lowers adjusted R², that variable is not earning its place.

In a multiple regression, each coefficient describes the association between that predictor and the outcome with the other predictors held constant. This is the phrase that does most of the work in a discussion section, and it deserves precision.

“Holding constant” is a statistical operation, not an experimental one. The model estimates what the relationship would look like among cases that were identical on the other measured variables. It cannot hold constant anything you did not measure, and it cannot make an observational design causal.

A related and more subtle problem is that adding a variable can change a coefficient's sign entirely. This is not a bug: it happens when the added variable was suppressing or confounding the original relationship, and the reversed sign may be the more accurate estimate. But it also means a coefficient reported from one model cannot be quoted as though it were a property of the predictor itself. It is a property of that predictor in that model, and changing the covariates changes it. Always state which variables were in the model alongside any coefficient you report.

Multicollinearity

When predictors are strongly correlated with each other, the model struggles to separate their contributions. Coefficients become unstable, swinging in size or even sign when another variable is added, and standard errors inflate. Check the variance inflation factor; values above about 5 warrant attention and above 10 are usually taken as serious.

The residuals-versus-fitted plot is the single most informative diagnostic and takes seconds to produce. A healthy plot shows a formless band of points around zero. A funnel widening to the right means the variance grows with the prediction, so your standard errors and therefore your p-values are wrong. A curve means the relationship is not linear and the model is systematically wrong at both ends.

Rules of thumb are widely quoted and mostly too crude to rely on, but they are useful as a floor.

With 4 predictors, the second rule gives a minimum of 108 participants for individual coefficients to be tested with reasonable power. A proper power analysis is better than any of these, because the requirement depends on the size of effect you want to detect, and a small expected effect can demand several times the rule-of-thumb figure.

1. Claiming causation from observational data

Regression describes association. “Predicted” is a statistical term, not a causal one, and readers will take it literally if you are not careful.

2. Interpreting the intercept when zero is impossible

If no participant has a predictor value of zero, the intercept is an extrapolation. Centre the predictors to make it meaningful.

3. Reporting R² rather than adjusted R² in multiple regression

R² rises with every predictor added, including irrelevant ones.

4. Not plotting the residuals

One plot checks three assumptions. Omitting it is the most common reason a violated assumption goes undetected.

5. Ignoring clustering in the data

Pupils within schools or repeated measures within people need a multilevel model. Ordinary regression will give p-values that are too small.

6. Too many predictors for the sample

The model will fit noise and will not replicate. Check the sample size rules before fitting.

7. Extrapolating beyond the observed range

A model fitted on 2 to 8 study hours says nothing reliable about 20 hours.