Guides / clustered data
Clustered data and the unit of analysis
Most applied datasets are clustered and most analyses pretend otherwise. Patients sit within wards, pupils within schools, and every repeated measurement sits within the person who provided it. This guide explains how much information you really have, what happens to your results if you ignore the structure, and how to choose between a mixed effects model, GEE, cluster-robust standard errors and simply averaging.
Written and reviewed by Sidra Khan, Lead BiostatisticianClustered data is data in which observations fall into groups whose members are more alike than two observations drawn at random. The consequence is that observations are not independent, so every method that assumes independence — the t-test, ordinary regression, chi-square — produces standard errors that are too small, confidence intervals that are too narrow and p-values that are too low. The size of the error is captured by the design effect, 1 + (m − 1) × ICC, and the honest sample size is your raw count divided by it.
Definition
Where clustering comes from
Clustering arises whenever something shared sits between the observations. Patients on the same ward share staff, protocols and a case mix. Pupils in the same class share a teacher. Repeated blood pressure readings from one patient share that patient's physiology. In each case, knowing which cluster an observation belongs to tells you something about its value, and that is exactly what statistical independence rules out.
It helps to recognise the three forms it usually takes.
| Form | Cluster | Observation | Typical ICC |
|---|---|---|---|
| Natural grouping | Ward, practice, school, household, village | Patient, pupil, resident | 0.01 to 0.2 |
| Repeated measures | One person | Each measurement from that person | 0.3 to 0.7 |
| Multiple levels | Pupils in classes in schools | Each pupil, each test | varies by level |
You cannot tell whether data is clustered by looking at the spreadsheet. A file with 50,000 rows and no participant identifier may still be 50,000 measurements from 3,000 people. If the identifier is missing you have not escaped the clustering, you have only lost the ability to account for it. That is a data problem to fix before analysis, not a licence to proceed.
The core question
The unit of analysis
The unit of analysis is the entity your conclusions are about, and the level at which observations can reasonably be treated as independent. Choosing it wrongly is the single most consequential decision in an applied analysis, and it is usually made without being noticed.
The classic failure is an intervention delivered to groups but analysed on individuals. A training programme is rolled out to twelve clinics, outcomes are measured on nine hundred patients, and the analysis compares nine hundred patients as though each had been allocated independently. The intervention was allocated twelve times, not nine hundred. The effective sample size for the comparison is far closer to twelve than to nine hundred, and an analysis at patient level will report a confident result that the design cannot support.
There is a mirror-image error. Aggregating everything to cluster level and then drawing conclusions about individuals is the ecological fallacy: a relationship that holds between schools need not hold between pupils, and can even run the other way. Both errors come from the same root, which is treating one level of the data as if it were another.
Measuring it
The intraclass correlation
The intraclass correlation, usually written ICC or the Greek letter rho, is the proportion of total variation that sits between clusters rather than within them. Formally it is the between-cluster variance divided by the sum of the between-cluster and within-cluster variances, and you obtain it from a model with a random intercept for cluster.
Read it as a measure of redundancy. At an ICC of 0, cluster membership tells you nothing and every observation is a fresh piece of information. At an ICC of 1, every observation within a cluster is identical, so a cluster of fifty tells you no more than a cluster of one. Real data sits between, and where it sits determines how much of your dataset is real.
| Setting | Typical ICC | What it means in practice |
|---|---|---|
| Repeated symptom or mood reports from one person | 0.4 to 0.7 | Severe. Most of the rows repeat what you already knew. |
| Repeated clinical measurements on one patient | 0.3 to 0.6 | Severe, and often underestimated by researchers. |
| Pupils within a school | 0.1 to 0.2 | Moderate, and enough to change conclusions. |
| Patients within a GP practice or ward | 0.01 to 0.1 | Small per observation, but large clusters magnify it. |
| Households within a region | 0.01 to 0.05 | Usually minor unless clusters are very large. |
The design effect depends on cluster size as well as ICC. An ICC of 0.02 sounds negligible, but with 200 patients per practice the design effect is about 5, so your effective sample is a fifth of your row count. Large clusters make small correlations expensive.
The arithmetic
Design effect and effective sample size
The design effect is how much your variance is inflated by the clustering:
Design effect = 1 + (m − 1) × ICC, where m is the average number of observations per cluster.
Divide your raw observation count by the design effect and you have the effective sample size, the number of genuinely independent observations your data is worth. Standard errors are inflated by the square root of the design effect, and confidence intervals widen by the same factor.
A worked example. Three thousand app users contribute an average of seventeen symptom entries each, giving 51,000 rows. At an ICC of 0.5 the design effect is 9, so the effective sample size is about 5,700. Roughly 11% of the rows carry new information, and any analysis treating all 51,000 as independent produces confidence intervals three times narrower than they should be.
As observations per cluster increase, the effective sample size approaches the number of clusters divided by the ICC and never passes it. With 3,000 users and an ICC of 0.5 you cannot exceed 6,000 effective observations no matter how long the app runs or how diligently people log. Collecting more entries per person buys almost nothing once you are near the ceiling. Only recruiting more people raises it. This is the most important practical consequence of clustering and it is routinely missed in study planning.
When cluster sizes vary a lot, using the plain average understates the design effect. The more accurate cluster size is the sum of the squared cluster sizes divided by the sum of the cluster sizes, which gives greater weight to the large clusters that do the damage.
Put in your clusters, observations each and ICC, and the free calculator returns the design effect, your effective sample size, how far out your confidence intervals are, and the ceiling you cannot pass. It also writes the sentence for your methods section.
Open the effective sample size calculatorConsequences
What ignoring it does to your results
Ignoring clustering is not a conservative simplification. It biases your results in the direction of finding things, which is the worst possible direction to be wrong in.
| Quantity | What happens | At a design effect of 9 |
|---|---|---|
| Point estimate | Often roughly unbiased, which is why the problem hides | Looks fine |
| Standard error | Too small by the square root of the design effect | Three times too small |
| Confidence interval | Too narrow by the same factor | Three times too narrow |
| p-value | Too small, sometimes by orders of magnitude | p = 0.04 might truly be p = 0.4 |
| Type I error rate | Far above the nominal 5% | Can exceed 30% |
Because the coefficient itself usually looks reasonable, nothing in the output announces the error. The table of results appears sensible, the effect sizes are plausible, and only the uncertainty is wrong. This is why the fault survives internal review and is then caught by a statistical reviewer at a journal, which is the most expensive possible moment to find it.
Doing it properly
Four ways to analyse clustered data
There is no single correct method. There are four defensible ones, and which suits you depends on what your question is and how many clusters you have.
| Approach | Use it when | Watch out for |
|---|---|---|
| Aggregate to cluster level | You want simplicity and your question is about clusters. Reduce each cluster to one summary, then use ordinary methods. | Loses within-cluster relationships and some precision when cluster sizes vary. Often the safest choice with few clusters. |
| Cluster-robust standard errors | You are happy with your model but need honest uncertainty. Keeps the coefficients, corrects the variance. | Needs a reasonable number of clusters, commonly quoted as around 40. Below that the correction is unreliable without a small-sample adjustment. |
| Mixed effects model | You care about individual clusters, want to quantify variation between them, or need the within-cluster effect separated from the between-cluster one. | Random effect variances are poorly estimated with few clusters. Decide deliberately whether slopes vary, not just intercepts. |
| GEE | You want the population-averaged effect and regard clustering as a nuisance to correct rather than a structure to model. | Also wants a decent number of clusters. The working correlation structure is a choice you should state and justify. |
For continuous outcomes the two usually produce similar coefficients. For binary outcomes they do not, and the difference is real rather than a numerical artefact. A mixed effects model estimates the effect for a given cluster, holding its random effect fixed. GEE estimates the average effect across the population of clusters. The mixed model coefficients are larger in magnitude, and reporting one while describing the other is a genuine misinterpretation rather than a stylistic preference.
Doing it
In R, SPSS and Stata
Four approaches, four sets of syntax. The ICC comes out of the first one and feeds the design effect.
R
library(lme4); library(geepack)
m <- lmer(outcome ~ treatment + (1 | ward), data = df) # mixed model
performance::icc(m) # the ICC
g <- geeglm(outcome ~ treatment, id = ward, corstr = "exchangeable", data = df)
lmtest::coeftest(fit, vcov = sandwich::vcovCL(fit, cluster = ~ward)) # robust SEs
For the aggregate approach, aggregate(outcome ~ ward + treatment, df, mean) then an ordinary test on the result.
SPSS
MIXED outcome BY treatment
/FIXED = treatment
/RANDOM = INTERCEPT | SUBJECT(ward)
/PRINT = SOLUTION TESTCOV.
TESTCOV prints the variance components you need for the ICC, which SPSS otherwise omits. GENLIN with a REPEATED subcommand gives GEE. There is no built-in cluster-robust option for ordinary regression, so use one of the other three approaches.
Stata
mixed outcome treatment || ward:
estat icc
xtset ward
xtgee outcome treatment, corr(exchangeable) vce(robust)
regress outcome treatment, vce(cluster ward)
The last line is the quickest honest answer available: it keeps your original model and corrects only the standard errors. estat icc after mixed gives the intraclass correlation directly.
Cluster-robust standard errors and GEE both need a reasonable number of clusters, commonly quoted as around forty, before they behave. Below that, a mixed model or simple aggregation is safer. Check how many clusters the software reports, not how many rows.
A hard case
Repeated measures, apps and wearables
Digital health data is the setting where clustering does the most damage, because the row counts are enormous, the ICCs are high and the structure is easy to forget. A symptom-tracking app with three thousand users and fifty thousand entries looks like a large dataset. At a realistic ICC it is a moderate one.
Two further problems ride along with it, and both are more serious than the clustering itself.
None of these makes the data unusable. All of them need to be stated, and the analysis has to be built with them in view rather than corrected afterwards.
Writing it up
How to report it
Reviewers are not looking for sophistication, they are looking for evidence that you noticed. Five sentences are usually enough.
For observational studies the STROBE statement is the reporting standard most journals expect, and it asks directly how correlated data were handled. Answering it properly in the methods section removes an entire category of reviewer objection before it is raised.
Avoid these
Six common mistakes
| Mistake | Why it matters |
|---|---|
| Treating rows as the sample | The row count is a description of the file, not a measure of the information in it. |
| Assuming a small ICC is safe | With large clusters even an ICC of 0.02 can divide your effective sample by five. |
| Collecting more observations per person | Beyond a point it adds almost nothing. Recruiting more people is what raises the ceiling. |
| Using GEE or robust errors with few clusters | Below roughly 40 clusters the variance estimator is unreliable and understates uncertainty, which defeats the purpose. |
| Reporting mixed model results as population effects | For binary outcomes these are different quantities, and the mistake changes what the paper claims. |
| Dropping the cluster identifier | Losing the identifier does not remove the clustering, it removes your ability to account for it, and the analysis becomes unfixable. |
Common questions
Frequently asked questions
What is clustered data?
Data in which observations fall into groups whose members are more alike than two observations picked at random. Patients within a ward, pupils within a school, households within a street, and repeated measurements from one person are all clustered. The defining consequence is that observations are not independent, so standard methods that assume independence give the wrong answer.
What is the unit of analysis?
The entity your conclusions are about and the level at which observations can be treated as independent. If an intervention was delivered to whole clinics, the clinic is the unit even though the outcomes were measured on patients. Analysing patients as if they were independent when the clinic was randomised is the unit of analysis error, and it is one of the most common serious faults in applied research.
What is the intraclass correlation?
The proportion of total variation that lies between clusters rather than within them. An ICC of 0 means cluster membership tells you nothing about an observation; an ICC of 1 means every observation within a cluster is identical and the cluster contributes only one piece of information. Repeated self-reported measures from one person often sit between 0.4 and 0.7; patients within a GP practice typically between 0.01 and 0.1.
How does clustering reduce my sample size?
Through the design effect, 1 + (m − 1) × ICC, where m is the average observations per cluster. Divide your raw count by it to get the effective sample size. Seventeen observations each from 3,000 people with an ICC of 0.5 gives 51,000 rows but an effective sample of about 5,700, so roughly 11% of the rows carry new information.
Should I use a mixed effects model or GEE?
Use a mixed effects model when you care about individual clusters, want to quantify variation between them, or need predictions for a specific cluster. Use GEE when you want the population-averaged effect and treat the clustering as a nuisance to be corrected for. For continuous outcomes the two usually give similar coefficients; for binary outcomes they estimate genuinely different quantities and the mixed model coefficients are larger in magnitude.
Can I just average within each cluster?
Yes, and it is a valid and underrated option. Reduce each cluster to a single summary and analyse those summaries with ordinary methods. You lose the ability to examine within-cluster relationships and some precision when cluster sizes vary, but the result is simple, transparent and almost impossible to get wrong. For a small number of clusters it is often the safest choice.
How many clusters do I need?
More clusters are worth far more than more observations per cluster. As observations per cluster rise, the effective sample size approaches the number of clusters divided by the ICC and never passes it. For cluster-robust standard errors and GEE, roughly 40 clusters is a common minimum before the robust variance estimator becomes unreliable; with fewer, small-sample corrections or a mixed model are safer.
Keep reading
Related
Effective sample size calculator
Design effect, effective sample size and the ceiling you cannot pass.
Reliability and validity
What a measure has to do before its numbers mean anything.
Independent statistical review
A fixed-price check of analysis already done, before it reaches a journal or a board.