A/B testing.
A/B testing splits users between variants and measures the difference in a predefined metric. Done properly it establishes causation rather than correlation, which is why it is the only reliable way to know whether a change helped.
Where A/B testing fits on a long engagement.
The statistics are less forgiving than the tooling suggests. Sample size must be calculated before the test starts from the effect size worth detecting; stopping when the result looks significant — peeking — inflates the false positive rate substantially, because a random walk crosses a threshold eventually. A test stopped early on a promising number is closer to a coin flip than to evidence.
Most product changes produce no detectable effect, and teams are consistently unprepared for that. A realistic programme expects a majority of tests to be inconclusive, which is itself valuable information: it says the change did not matter enough to build on.
What an assigned team does with A/B testing.
Experimentation is a capability rather than a tool. It needs an assignment process that is genuinely random, metrics defined before the test, a decision rule agreed in advance, and someone who will hold the line when a stakeholder wants to stop early.
Building that discipline takes time on the same product rather than a tooling rollout, which is what a dedicated development team provides.
What we use A/B testing for.
- Changes evaluated on evidence A measured causal effect rather than a before-and-after comparison that confounds everything.
- Sample size fixed before the start A stopping rule set in advance, so the result is not manufactured by peeking.
- Inconclusive treated as an answer Accepting that most changes do not move the metric, and acting on that.
How A/B testing capacity is assigned.
Experimentation capacity is assigned under managed data services where the constraint is the measurement pipeline, and alongside product engineering where it is the assignment mechanism.
Tell us what your roadmap needs A/B testing for.
A service delivery manager replies with the disciplines we would assign, the monthly capacity and what the first month looks like.
Loading the contact form… You can also email hello@azendo.co.
We reply within one working day. No obligation, and no newsletter.