Sample Size, Explained: How Much Data Is Enough?

Sample size is the number of observations a study needs to answer its question at a stated level of confidence. It is the most consequential decision made before any data are collected, and the most commonly skipped. Too small a sample and real effects hide in the noise; too large and time and money are spent purchasing certainty nobody needed. The right size is not a folk number — it is a calculation.

How it works

Four quantities drive the calculation. First, the variation in the process: noisier data demand more observations. Second, the smallest effect worth detecting: seeing a large shift takes little data, while seeing a subtle one takes a great deal. Third, the confidence level — how rarely you will tolerate a false alarm. Fourth, the power — how rarely you will tolerate missing an effect that is really there. Fix those four and the required sample size follows from standard formulas or any statistical package. The relationship is unforgiving in one direction: cutting an interval’s width in half, or detecting an effect half as large, requires roughly four times the data.

The habit the calculation replaces is convenience sampling — “we measured thirty because we always measure thirty.” Thirty may be far too few for a subtle effect in a noisy process, and wastefully many for a crude check. The number deserves the same rigor as the analysis it will feed.

A worked example

Imagine an e-commerce operation piloting a simplified checkout intended to reduce abandoned orders. Before launching, the analyst asks the four questions: how much does the abandonment rate fluctuate day to day, what reduction would justify the engineering work, and what false-alarm and miss rates are tolerable? The arithmetic then reports how many sessions the pilot must include to give a real improvement a fair chance of being seen. Run the pilot shorter than that, and a verdict of “no significant difference” would be meaningless — the study never had the sensitivity to detect the improvement it was looking for. The scenario is generic; the pre-commitment is the discipline.

• Compute the sample size before collecting data — a study sized by convenience answers only by accident.
• Base the calculation on the smallest effect that matters to the business, not the largest you hope for.
• Do not keep collecting until significance appears; a stopping rule invented mid-study corrupts the result.
• A large sample fixes imprecision, not bias — data gathered the wrong way stays wrong at any volume.
• Subgroup questions need their own size math; a study sized for the whole rarely supports verdicts about the parts.

Guessed sample sizes produce guessed conclusions with confident formatting.

The logic of sample size enters with confidence intervals in our Green Belt program ($299, 35 hours). The full calculations, tied to statistical power, are Black Belt material — 60 hours for $499, with a 150-question proctored exam and one free retake included.