One-Factor-at-a-Time Testing: What It Is and Why It Fails

One-factor-at-a-time experimentation — OFAT — changes a single input, holds everything else fixed, records the result, then moves to the next input. It feels rigorous, it is how most people were taught to test things, and it is quietly one of the least efficient and most misleading ways to study a process with more than one important variable.

How it fails

OFAT has three structural defects. First, it cannot see interactions: if temperature’s effect depends on pressure, a series of runs that never varies them together will never notice, and every conclusion holds only at the exact settings that happened to be frozen. Second, it stalls at false optima. Tuning factor A to its best value, then B, then C traces one staircase path through the settings space and can end far from the true best combination, with every individual step looking locally sensible. Third, it wastes information: each OFAT run informs a single comparison, while each run of a factorial design contributes to estimating every effect at once — more precision from fewer runs.

Add the usual absence of randomization and replication, and OFAT results are also fragile against drift and noise — differences get credited to the factor when they belong to the afternoon. The people who practice OFAT are rarely careless; they are usually careful people applying a rule that works for troubleshooting a single fault but breaks down as a strategy for studying a system.

A worked example

A food producer tunes an oven-drying step by OFAT. Holding time fixed, the team varies temperature and picks the best; holding that temperature, it varies time and picks again. The process improves, and everyone stops — unaware that a hotter-and-shorter combination, never visited by the staircase, performs better still. A later eight-run factorial covering both factors together finds it in one afternoon, along with the interaction that explains why the OFAT path could not: the best drying time depends strongly on the temperature chosen.

Common rationalizations

• “We change one thing at a time to be scientific” — control is not the same as blindness to interactions
• “Factorials are too complicated” — an eight-run factorial is arithmetic a spreadsheet handles
• “We don’t have time for a designed experiment” — OFAT typically needs more runs for weaker answers
• “It worked” — OFAT often improves things; the question is what it left on the table

Changing one thing at a time guarantees you will never learn what things do together.

The contrast between OFAT and factorial thinking opens the experimental-design material in our Black Belt program ($499, about 60 hours, 150-question proctored exam). And the free White Belt program — six hours, with a 30-question exam — plants the seed early: a conclusion is only as good as the comparison that produced it.