---
title: "A/B testing | Skills We Assign For | Azendo"
description: "A/B testing in product work — sample size, peeking, and what invalidates a result, plus the Azendo roles assigned for it."
url: "https://azendo.co/skills/a-b-testing/"
---

[Skills](https://azendo.co/skills/) Analytics and BI 

# A/B testing.

A/B testing splits users between variants and measures the difference in a predefined metric. Done properly it establishes causation rather than correlation, which is why it is the only reliable way to know whether a change helped.

## Where A/B testing fits on a long engagement.

The statistics are less forgiving than the tooling suggests. Sample size must be calculated before the test starts from the effect size worth detecting; stopping when the result looks significant — peeking — inflates the false positive rate substantially, because a random walk crosses a threshold eventually. A test stopped early on a promising number is closer to a coin flip than to evidence.

Most product changes produce no detectable effect, and teams are consistently unprepared for that. A realistic programme expects a majority of tests to be inconclusive, which is itself valuable information: it says the change did not matter enough to build on.

## What an assigned team does with A/B testing.

Experimentation is a capability rather than a tool. It needs an assignment process that is genuinely random, metrics defined before the test, a decision rule agreed in advance, and someone who will hold the line when a stakeholder wants to stop early.

Building that discipline takes time on the same product rather than a tooling rollout, which is what a [dedicated development team](https://azendo.co/services/dedicated-development-team/) provides.

## What we use A/B testing for.

* Changes evaluated on evidence A measured causal effect rather than a before-and-after comparison that confounds everything.
* Sample size fixed before the start A stopping rule set in advance, so the result is not manufactured by peeking.
* Inconclusive treated as an answer Accepting that most changes do not move the metric, and acting on that.

## How A/B testing capacity is assigned.

Experimentation capacity is assigned under [managed data services](https://azendo.co/services/data-engineering/) where the constraint is the measurement pipeline, and alongside product engineering where it is the assignment mechanism.

## Roles we assign A/B testing for

* [Data Scientist Data science](https://azendo.co/services/data-science/data-scientist/)

## Service lines it sits in

* [Data science](https://azendo.co/services/data-science/)

Capacity is agreed as a committed monthly capacity across a discipline, not per skill.

## Related in analytics and bi

* [Looker — skill we assign for](https://azendo.co/skills/looker/)
* [Power BI — skill we assign for](https://azendo.co/skills/power-bi/)
* [Tableau — skill we assign for](https://azendo.co/skills/tableau/)
* [semantic layers — skill we assign for](https://azendo.co/skills/semantic-layers/)
* [experimentation design — skill we assign for](https://azendo.co/skills/experimentation-design/)
* [causal inference — skill we assign for](https://azendo.co/skills/causal-inference/)
* [cohort analysis — skill we assign for](https://azendo.co/skills/cohort-analysis/)
* [segmentation — skill we assign for](https://azendo.co/skills/segmentation/)

## Tell us what your roadmap needs A/B testing for.

A service delivery manager replies with the disciplines we would assign, the monthly capacity and what the first month looks like.

[All skills we assign for](https://azendo.co/skills/)