experimentation design.
Experimentation design is the work before a test runs: choosing the hypothesis, the unit of assignment, the primary metric, the guardrails, the duration and the decision rule. It determines whether the result will mean anything.
Where experimentation design fits on a long engagement.
The unit of assignment is the decision that most often invalidates a test quietly. Randomising by session when the change affects behaviour across sessions, or by user when the product has network effects between users, breaks the independence the statistics assume. The test still produces a number; the number just does not mean what it appears to.
Guardrail metrics are what stop a local win from being an overall loss. A change that lifts conversion while increasing refunds or support contacts has not improved anything, and without guardrails declared in advance nobody looks. Declaring them before the result is known is the part that matters.
What an assigned team does with experimentation design.
The decision rule has to be agreed before the data exists. Once a result is visible, the discussion about what threshold counts is no longer neutral, and whoever wanted the change will find a reading that supports it.
Holding that sequence is a practice rather than a policy document, and it depends on specialists who have run it repeatedly. Developing that depth is what the Talent Success and Academy programme is for.
What we use experimentation design for.
- Assignment units that preserve independence Randomisation chosen so interference between units does not invalidate the estimate.
- Guardrails declared in advance Refunds, support load and latency watched, so a local win is not an overall loss.
- Decision rules fixed before data Thresholds agreed while the outcome is still unknown and the discussion is neutral.
How experimentation design capacity is assigned.
Experiment design capacity is assigned alongside the analytics capacity that measures it, under data engineering outsourcing where the pipeline is the limiting factor.
Tell us what your roadmap needs experimentation design for.
A service delivery manager replies with the disciplines we would assign, the monthly capacity and what the first month looks like.
Loading the contact form… You can also email hello@azendo.co.
We reply within one working day. No obligation, and no newsletter.