Guides / employee surveys
Employee engagement survey analysis
Most engagement reports quote a single percentage, rank the departments underneath it, and act on the bottom three. Every step of that is shakier than it looks. This guide covers what the response rate does to your headline number, how to compare teams without manufacturing false findings, what driver analysis can and cannot tell you, and how to report a result a board can act on.
Start with who did not answer, because that decides what the headline number means. Report the engagement score as a proportion with a confidence interval rather than a bare percentage. Compare teams using differences with intervals, not a ranked table, and account for how many comparisons you are making. Use regression to identify drivers, while being clear that it shows association and not cause. Then report the uncertainty alongside the number, because a board acting on a two point movement needs to know whether two points means anything.
Definitions
What the score actually measures
Engagement is not a thing you can weigh. It is a construct, inferred from how people answer a set of questions, and the number you report depends entirely on which questions you asked and how you combined them.
Most instruments use a five point agreement scale and summarise in one of two ways.
| Method | What it does | Trade-off |
|---|---|---|
| Percent favourable | Counts the share choosing agree or strongly agree | Easy to explain. Throws away the difference between a 3 and a 1, so a workforce drifting from neutral to hostile can look unchanged. |
| Mean score | Averages the numeric responses | Uses the whole scale. Treats a Likert scale as though the gap from 1 to 2 equals the gap from 4 to 5, which is an assumption rather than a fact. |
Neither is wrong. What matters is picking one, saying which, and never switching between them across waves, because a change of method can produce a movement larger than anything your organisation actually did.
If you are combining several items into one engagement score, they should be measuring the same underlying thing. Cronbach's alpha is the usual check, and anything below about 0.7 means the items are not cohering and the composite is hiding more than it shows. An alpha above 0.95 is the opposite problem: the questions are near-duplicates and you are paying people to answer the same thing five times.
The part that decides everything
The response rate, and the number nobody reports
An engagement survey is voluntary. People who feel strongly, in either direction, are more likely to answer than people who feel nothing much. That makes it a textbook case of self-selection, and it is why the response rate is not an administrative footnote but the single most important figure in the report.
Typical rates for a whole-organisation survey run between 40% and 70%. Below 40% the result describes a subgroup you cannot characterise. Above 85%, check nobody felt obliged.
Worked example one
The score and its interval
1,200 staff invited. 684 responded. Of those, 431 answered favourably on the engagement index.
Step 1. The response rate
684 / 1,200 = 0.570, so 57.0%. Respectable, and it also means 516 people said nothing.
Step 2. The headline score
431 / 684 = 0.630, so 63.0% favourable.
Step 3. The interval around it
Use a Wilson interval rather than the ordinary normal approximation, because it behaves properly near the ends of the scale where department-level figures often sit.
95% CI: 59.3% to 66.5%, a width of 7.2 points.
That width is the useful part. If last year's figure was 61%, this year's 63% sits comfortably inside the range of ordinary sampling variation. Reporting "engagement rose two points" would be describing noise.
Step 4. The number nobody reports
The 63% describes respondents. The board will hear it as a statement about the workforce. Those are different claims, and you can bound the gap.
if every non-responder were unfavourable: 431 / 1,200 = 35.9%
if every non-responder were favourable: (431 + 516) / 1,200 = 78.9%
The true workforce figure lies somewhere between 36% and 79%. That is a 43 point span, and it is not a trick: it is the honest consequence of 516 people declining to answer.
Not report it as the finding, which would be useless. Report the 63% as what it is, a figure describing respondents, and use the bounds once in the methodology to establish why the response rate matters. It stops anyone treating the headline as a census, and it makes the case for improving the response rate next wave far better than any amount of encouragement.
Worked example two
Comparing two departments
Operations and Finance are reported side by side. Operations looks poor.
| Department | Favourable | Responded | Score | 95% CI |
|---|---|---|---|---|
| Operations | 118 | 210 | 56.2% | 49.4% to 62.7% |
| Finance | 47 | 62 | 75.8% | 63.8% to 84.8% |
Step 1. Notice the interval widths
Operations spans 13 points, Finance spans 21. Finance looks better and is measured far less precisely, because 62 responses cannot pin a percentage down. Any ranking that ignores this will put small teams at both the top and the bottom.
Step 2. Test the difference properly
Do not eyeball whether the intervals overlap, which is a conservative and slightly wrong test. Compute the interval around the difference itself.
difference = 75.8% − 56.2% = 19.6 points
SE = √[ (0.562 × 0.438 / 210) + (0.758 × 0.242 / 62) ] = 0.0643
95% CI = 19.6 ± 1.96 × 6.43 = 7.0 to 32.2 points
Step 3. Read it honestly
The interval excludes zero, so there is a real difference here. But it runs from 7 points to 32, which is the difference between a mild gap worth monitoring and a serious problem worth intervening in. The point estimate of 19.6 is the middle of a wide range, not a measurement.
That is the sentence to put in the report: there is a difference, and its size is uncertain.
Worked example three
Comparing with last year, and with other organisations
Every engagement report answers two comparison questions. Both are harder than they look.
Year on year, worked through
Last year: 61% favourable from 652 responses. This year: 63% from 684. The slide deck says engagement is up two points.
change = 63.0% − 61.0% = +2.0 points
SE = √[ (0.61 × 0.39 / 652) + (0.63 × 0.37 / 684) ] = 2.66 points
95% CI = 2.0 ± 1.96 × 2.66 = −3.2 to +7.2 points
The interval includes zero. The honest reading is that engagement is not distinguishably different from last year. It may have risen seven points. It may have fallen three.
The smallest change this survey can actually detect
At these sample sizes anything under about 5.2 points is inside the noise. That single number is worth knowing before the survey goes out, because it tells you what movements are worth building a narrative around.
And the uncomfortable arithmetic behind it
To detect a two point change reliably, with 80% power, you would need roughly 9,242 responses in each wave. At a 57% response rate that means inviting about 16,214 people, every year.
An organisation of 1,200 staff cannot detect a two point change. Not with a better survey, not with more reminders, not ever. The sample is the organisation. That is a limit of the method rather than a failure of execution, and saying so plainly in the report protects everyone from chasing movements that are not there.
Set a threshold in advance. Agree with the board that movements under five points will be reported as "no detectable change", and that anything you act on has to clear it. Done before the data arrives, that is good governance. Done afterwards it looks like excuse-making.
External benchmarks
Comparing against an industry benchmark is the weakest comparison in the report, and it is usually given the most prominence.
Use benchmarks for orientation, never as a target. Your own trend, measured consistently with the same instrument and reported with its uncertainty, is far more informative than a comparison with organisations you cannot inspect.
The league table problem
Why ranking departments manufactures findings
Comparing two departments is fine. Comparing all of them against each other and then acting on whichever gaps look significant is not, and it is what almost every engagement report does.
| Departments | Pairwise comparisons | Chance of at least one false positive at p<.05 |
|---|---|---|
| 6 | 15 | 54% |
| 12 | 66 | 97% |
| 20 | 190 | >99% |
With twelve departments you are almost certain to find at least one "significant" gap that is pure chance. Act on it and you will restructure a team that was performing normally.
There is a second, quieter problem. Employees sit within teams, teams within departments, and people in the same team share a manager and a workload. Their answers are not independent, which is the clustering problem, and it means the effective sample size is smaller than the response count suggests.
The analysis boards ask for
Driver analysis, and what it cannot tell you
Driver analysis regresses the overall engagement score on the individual survey items to see which ones move with it. It is genuinely useful and routinely over-interpreted.
What it does
It ranks items by how strongly they relate to overall engagement, usually with standardised coefficients or relative weights. Items high on that list are candidates for attention.
Three reasons it is not a to-do list
Report drivers as "areas most associated with engagement in this survey" and never as "what to fix to raise engagement". The second claim needs a change over time, or better still something that was actually changed in one area and not another.
Doing it
In R, SPSS and Stata
R
Score, interval and comparison in four lines.
library(DescTools)
BinomCI(431, 684, method = "wilson")
BinomDiffCI(47, 62, 118, 210, method = "scorecc")
psych::alpha(items)
For driver analysis use relative weights rather than plain coefficients, because the items overlap: relaimpo::calc.relimp(model, type = "lmg"). If you need to respect team clustering, lme4::lmer(engagement ~ driver + (1 | team)).
SPSS
Reliability first, then the proportions.
RELIABILITY VARIABLES = q1 TO q12 /MODEL = ALPHA /STATISTICS = SCALE CORR.
FREQUENCIES VARIABLES = favourable /STATISTICS = MEAN.
CROSSTABS TABLES = department BY favourable /STATISTICS = CHISQ /CELLS = COUNT ROW.
SPSS gives a Wald interval for a proportion by default, which misbehaves near 0% and 100%. For department-level figures, compute Wilson intervals yourself or use the bootstrap module.
Stata
ci proportions favourable, wilson
prtest favourable, by(department)
alpha q1-q12, item
If the survey was weighted to match the workforce profile, declare it once with svyset [pweight = wt] and prefix everything with svy:, otherwise the intervals ignore the weighting and come out too narrow.
Survey platforms vary in whether "percent favourable" excludes neutral responses, excludes non-answers to that specific item, or uses the full respondent count. Three defensible choices, three different numbers. Establish which your platform uses before comparing anything with last year.
Avoid these
Six mistakes that cost you credibility
| Mistake | Why it matters |
|---|---|
| A headline percentage with no interval | 63% has a range of 59% to 67% here. Without it, a two point movement looks like a result. |
| Treating the score as the workforce view | It describes respondents. With 57% responding, the workforce figure could sit anywhere from 36% to 79%. |
| Ranking departments | With twelve departments there is a 97% chance of at least one false positive. Small teams fill both ends of the ranking. |
| Quoting tiny teams | A department with 8 responses has an interval roughly 35 points wide. Suppress below a stated floor. |
| Changing the method between waves | Switching from mean score to percent favourable can move the number more than anything the organisation did. |
| Presenting drivers as causes | Cross-sectional, overlapping predictors, single method. It identifies associations, not levers. |
A 54% response rate is not 54% of a view, it is the view of the people who chose to reply, and that group is rarely neutral. Send the export and you get the index, the drivers that actually move it, and what the non-response pattern does to the result.
See survey and insight analysis
Saad TariqPrincipal StatisticianWriting it up
How to report it
One sentence is worth drafting carefully, because it is the one a chief executive repeats: "Engagement among respondents was 63% favourable (95% CI 59% to 67%), based on a 57% response rate. The two point movement from last year is within the range expected from sampling alone."
Common questions
Frequently asked questions
What is a good response rate for an employee survey?
Between 40% and 70% is typical for a whole-organisation survey. Below 40% the result describes a subgroup you cannot characterise, and department-level figures become unreliable well before the overall one does. Report the rate with its denominator and by department, because an overall 57% can hide a department at 22%.
Should I report percent favourable or the mean score?
Either, provided you pick one and keep it. Percent favourable is easier to explain but discards the difference between neutral and hostile. The mean uses the whole scale but assumes equal spacing between points. Switching between the two across waves can produce a movement larger than anything your organisation actually did.
Is a two point rise in engagement meaningful?
Usually not. With 684 responses a score of 63% carries an interval of roughly 59% to 67%, so a two point movement sits well inside ordinary sampling variation. Report the interval alongside the score and the question answers itself.
Can I rank departments by engagement?
You can, but it manufactures findings. Twelve departments produce 66 pairwise comparisons and a 97% chance of at least one false positive at the conventional threshold. Small teams also have wide intervals and so fill both ends of any ranking. Plot each department with its interval instead.
What minimum team size should I report?
Twenty responses is a common floor, for anonymity as much as for precision. Below that the interval is so wide the number carries almost no information, and individuals can start to be identifiable from the breakdown.
Does driver analysis tell me what to fix?
No. It identifies items most strongly associated with overall engagement in a single cross-sectional survey, where the predictors overlap heavily and everything was measured by one person on one questionnaire. It generates candidates worth investigating, not a list of levers.
Do I need to weight the results?
If responders differ from the workforce on something you can measure, such as grade or site, weighting brings the estimate closer. It corrects only on the variables you weight by, so describe the result as adjusted for those variables rather than corrected for non-response. Declare the weights in the software or the intervals will be too narrow.
Keep reading
Related
Related services
If the findings have to persuade a board
Survey and insight analysis
Survey data analysed and reported so the findings hold up to scrutiny.
Fixed quoteStatistical consultancy
Design, analysis and reporting for charities, universities, the NHS and business.
From £950Data cleaning & preparation
Messy data made analysis-ready, with a documented log of every change.
Fixed quoteEvery piece of work is done and signed by a named statistician. Send the data or describe the project and you get a price and a deadline, usually within one working day.