Experimentation design specialists for your dedicated team.
Experimentation design is the work before a test runs: choosing the hypothesis, the unit of assignment, the primary metric, the guardrails, the duration and the decision rule. It determines whether the result will mean anything.
Where experimentation design fits in a long-term product team.
The unit of assignment is the decision that most often invalidates a test quietly. Randomising by session when the change affects behaviour across sessions, or by user when the product has network effects between users, breaks the independence the statistics assume. The test still produces a number; the number just does not mean what it appears to.
Guardrail metrics are what stop a local win from being an overall loss. A change that lifts conversion while increasing refunds or support contacts has improved nothing, and guardrails declared in advance are what catch it. Declaring them before the result is known is the part that matters.
What your dedicated team does with experimentation design.
The decision rule has to be agreed before the data exists. Once a result is visible, the discussion about what threshold counts is no longer neutral, and whoever wanted the change will find a reading that supports it.
Holding that sequence is a practice, and it depends on specialists who have run it many times. Developing that depth is what the Talent Success and Academy programme is for.
Common experimentation design use cases.
- Assignment units that preserve independence Randomisation chosen so interference between units does not invalidate the estimate.
- Guardrails declared in advance Refunds, support load and latency watched, so a local win is not an overall loss.
- Decision rules fixed before data Thresholds agreed while the outcome is still unknown and the discussion is neutral.
Adding experimentation design skills to your team.
Experiment design capacity is assigned alongside the analytics capacity that measures it, in a dedicated data engineering team where the pipeline is the limiting factor.
You can add experimentation design capacity as one dedicated specialist or as part of a wider remote team, managed by a service delivery manager from our offices in Thailand. Companies hire dedicated developers this way under one agreement and one monthly fee.
Tell us what your roadmap needs experimentation design for.
A service delivery manager will reply with the specialists we'd suggest, the monthly capacity and what the first month looks like.
Loading the form…
We reply within one business day.
