Guides / self-selection bias
Self-selection bias
When people decide for themselves whether to take part, and that decision is connected to what you are measuring, your results describe the people who volunteered rather than the population you meant to study. This guide covers how to recognise it, why a bigger sample makes it worse rather than better, how to check how far your sample has drifted, and what can honestly be done about it.
Written and reviewed by Sidra Khan, Lead BiostatisticianSelf-selection bias occurs when the people who choose to take part in a study differ from those who do not, in ways that affect what you are measuring. The problem is not that the sample is unusual. It is that the reason people joined is connected to the outcome, so the result is tilted and you cannot tell by how much. Collecting more responses does not help, because every extra response comes from the same self-selecting group.
Definition
How it happens
Every study has a point where someone decides whether to be in it. Self-selection bias appears when that decision is not independent of the thing you are measuring.
A survey about workload is answered by people who feel strongly about workload. A symptom-tracking app is downloaded by people whose symptoms bother them enough to track. A trial recruits volunteers who are healthier, better informed and more motivated than the patients the treatment is meant for. In each case nobody has done anything wrong, and in each case the sample has quietly stopped representing the population.
Ask whether the reason someone joined could also affect their answer. If yes, you have self-selection bias and should report it. If the decision to take part is genuinely unrelated to the outcome, your sample may be unusual but your estimate is not biased.
The dangerous part
Why a bigger sample makes it worse
This is the point that most often gets missed. Random error shrinks as a sample grows. Bias does not. A larger self-selected sample gives you a tighter confidence interval around the wrong number.
Ten thousand responses from an opt-in web panel does not beat four hundred from a properly drawn random sample. The ten thousand look authoritative, carry narrow intervals and impressive p-values, and are centred in the wrong place. Precision and accuracy are different things, and a large biased sample buys the first while destroying the second.
The most persuasive wrong answers in applied research come from very large self-selected datasets. The size itself is what stops people asking who is missing.
Terminology
Related biases, and the differences
These names get used loosely. They describe different points at which a sample goes wrong.
| Name | What happens | Example |
|---|---|---|
| Self-selection bias | People choose whether to join, and the choice relates to the outcome | An online poll on a topic people feel strongly about |
| Sampling bias | The researcher's method systematically misses part of the population | A phone survey that reaches only landlines |
| Non-response bias | People are selected properly but those who reply differ from those who do not | A staff survey answered mainly by the dissatisfied |
| Volunteer bias | Volunteers differ systematically from non-volunteers | Trial participants healthier than typical patients |
| Survivorship bias | Only those who remained are observed | Studying firms that are still trading |
| Attrition bias | People leave during the study for reasons related to the outcome | Patients dropping out when treatment is not working |
Self-selection and non-response are two sides of the same coin: one is about who opts in, the other about who fails to opt out. Both leave you reasoning about people you cannot see.
Honesty about limits
Which way does it push?
Usually you cannot know, and claiming to know is where write-ups overreach. You can reason about likely direction, and that reasoning belongs in the discussion, but it stays an argument rather than a correction.
Sometimes the direction is clear enough to state. App-based symptom studies tend to recruit people with more troublesome symptoms, which inflates apparent severity. Service feedback forms attract the delighted and the furious, which tends to hollow out the middle. Clinical trial volunteers are typically healthier than the eventual patient group, which flatters outcomes.
"If anything this would bias us towards the null" is one of the most common unsupported claims in applied papers. Unless you can demonstrate the direction, say the direction is unknown. Reviewers notice.
Worked example
What weighting fixes, and what it leaves behind
A satisfaction survey. The population is evenly split by age; the sample is not, because younger people responded far more readily.
| Group | Population | Sample | Satisfied | Weight |
|---|---|---|---|---|
| Under 45 | 50% | 80% | 40% | 0.625 |
| 45 and over | 50% | 20% | 70% | 2.500 |
Step 1. The unweighted estimate
(0.80 × 0.40) + (0.20 × 0.70) = 0.32 + 0.14 = 0.46
The survey reports 46% satisfied.
Step 2. The truth
(0.50 × 0.40) + (0.50 × 0.70) = 0.20 + 0.35 = 0.55
The population value is 55%. The survey is out by 9 percentage points, entirely because older people, who are more satisfied, are under-represented.
Step 3. Post-stratification weights
Each weight is the population share divided by the sample share.
under 45: 0.50 / 0.80 = 0.625
45 and over: 0.50 / 0.20 = 2.500
Each older respondent now counts as 2.5 people. Reweighting returns 55% exactly. The bias is removed completely, because age was the only thing driving it and age was measured.
Step 4. Now add something you did not measure
Suppose that within the under-45 group, the half who responded are 40% satisfied and the half who did not would have been 20% satisfied. The true under-45 figure is then 30%, not 40%, and the survey can never see it.
true population value = (0.50 × 0.30) + (0.50 × 0.70) = 0.50
The weighted estimate is still 55%. It was right a moment ago and is now wrong by 5 percentage points in the other direction, and nothing in the output has changed to warn you.
It corrects imbalance on the variables you measured and have population figures for. It does nothing about why people chose to respond, unless that reason happens to be one of those variables. Report weighted figures by all means, and describe them as adjusted for age and region rather than as corrected for non-response.
Diagnosis
How to check your sample
You cannot measure the bias directly, because the people who would tell you about it are the ones who did not take part. You can gather evidence about how far the sample has drifted.
Remedies
What can be done about it
Prevention beats correction by a wide margin, and every correction rests on an assumption you cannot test.
| Approach | What it does | Honest limitation |
|---|---|---|
| A proper sampling frame | Draws from a defined population at random, so joining is not a choice | The only real fix, and only available before you collect |
| Chase non-responders | Reminders and mixed modes pull in people unlike the early responders | Costly, and never reaches everyone |
| Post-stratification weighting | Reweights the sample to match population figures | Only corrects on variables you measured and hold population data for |
| Inverse probability weighting | Weights by modelled probability of participating | Needs a model of why people took part; wrong model, wrong answer |
| Heckman selection models | Models participation and outcome jointly | Leans heavily on an exclusion restriction that is hard to justify |
| Report it clearly | Lets readers judge how far the result travels | Not a fix, but better than an unstated assumption |
Weighting is worth doing where you can, but it should never be presented as having removed the problem. It narrows a gap on the variables you happen to have measured.
A hard case
Apps, panels and online surveys
Digital data is where self-selection does the most damage, because the samples are large enough to look authoritative and the filtering happens in several stages at once.
A health app user has chosen to own a smartphone, to look for an app, to install it, to register, and to keep using it. Each step filters. The people still logging after six months are a small and unusual subset of the people who downloaded, who were already an unusual subset of the condition. Worse, continued use often depends on the outcome itself, which also makes the missing observations non-random rather than merely sparse.
None of this makes the data worthless. Real-world data of this kind answers questions no trial can. It does mean the population has to be described as what it is, which is engaged users of this product, and conclusions kept inside that boundary.
Self-selection is one of the first things a statistical reviewer tests, along with the unit of analysis and how missing data was handled. A fixed-price review checks all of it before your work reaches a journal or a board.
About independent statistical reviewDoing it
Weighting in R, SPSS and Stata
R
The survey package handles weights properly, including standard errors that account for them. Weighted estimates with unweighted standard errors are a common and invisible error.
library(survey)
d <- svydesign(ids = ~1, weights = ~wt, data = df)
svymean(~satisfied, d)
svyby(~satisfied, ~region, d, svymean)
To build the weights rather than supply them, postStratify() takes the population totals directly, and rake() handles the case where you have marginal totals for several variables but not the full cross-tabulation.
SPSS
Compute the weight, then declare it.
COMPUTE wt = 0.
IF (agegroup = 1) wt = 0.625.
IF (agegroup = 2) wt = 2.500.
WEIGHT BY wt.
Plain WEIGHT BY adjusts the point estimates but treats the weighted counts as though they were real observations, so the standard errors come out too small. For honest intervals use the Complex Samples module, which is a separate licence.
Stata
Declare the design once and every subsequent command respects it.
svyset [pweight = wt]
svy: mean satisfied
svy: proportion satisfied, over(region)
pweight is the right weight type for sampling probabilities. Using fweight instead tells Stata each row represents that many identical observations, which gives the same point estimate and standard errors that are far too narrow.
Give the range of weights and the effective sample size after weighting. A handful of respondents carrying weights of 15 or 20 can drive an estimate on their own, and the design effect from weighting alone can cut your effective sample substantially. A weighted result from a badly unbalanced sample is more fragile than its interval suggests.
Avoid these
Six mistakes that cost marks
| Mistake | Why it matters |
|---|---|
| Citing sample size as evidence of representativeness | Size reduces random error, not bias. Ten thousand opt-in responses are less trustworthy than four hundred drawn at random. |
| Saying weighting corrected for non-response | It corrected imbalance on the variables you weighted by. Everything else is untouched, as the worked example shows. |
| Claiming a bias is conservative without evidence | "If anything this biases us towards the null" is usually asserted rather than demonstrated. If you cannot show the direction, say it is unknown. |
| Not reporting the response rate | Or reporting one without stating the denominator. If it cannot be calculated, say so, because that is itself informative. |
| Weighting without adjusting the standard errors | SPSS's plain WEIGHT BY and Stata's fweight both produce intervals that are far too narrow. |
| Generalising beyond the recruited population | App data describes engaged users of that app. That is a legitimate population; it is just not the condition in general. |
Writing it up
How to report it
For observational studies the STROBE statement asks for most of this directly, and answering it properly in the methods removes a whole category of reviewer objection. For survey work the same information belongs with your survey analysis write-up rather than buried in a limitations paragraph at the end.
Common questions
Frequently asked questions
What is self-selection bias?
It occurs when the people who choose to take part in a study differ from those who do not, in ways connected to what is being measured. The result then describes volunteers rather than the intended population, and the size of the distortion cannot be determined from the data itself.
Does a larger sample fix self-selection bias?
No, and it makes the problem more convincing. Larger samples reduce random error but not bias, so you get a tighter confidence interval around the wrong value. Ten thousand opt-in responses are less trustworthy than four hundred drawn at random.
What is the difference between self-selection bias and sampling bias?
Self-selection is caused by the participants, who decide whether to join. Sampling bias is caused by the method, which systematically misses part of the population. A poll people opt into shows self-selection; a landline-only phone survey shows sampling bias.
Can weighting remove it?
Only partly. Post-stratification and inverse probability weighting adjust on variables you measured and have population figures for. If people joined for reasons you did not measure, weighting leaves that untouched. It narrows the gap rather than closing it, and should be described that way.
How do I know which way the bias goes?
Usually you cannot. You can argue a likely direction from how people were recruited, and that argument belongs in the discussion, but it remains an argument. Claiming without evidence that a bias is conservative is a common and easily spotted overreach.
Is data from a health app usable for research?
Yes, for questions suited to it. App data captures change over time that clinic visits miss. The population is engaged users of that product, not everyone with the condition, and conclusions must stay inside that boundary. Recruitment, retention and missingness all need to be described rather than assumed away.
Keep reading
Related
Clustered data and the unit of analysis
The other half of why large real-world datasets are smaller than they look.
Survey and insight analysis
Survey data analysed and reported so the findings hold up to scrutiny.
Independent statistical review
A fixed-price check of analysis already done, before it reaches a journal or a board.