Experiments + Test Results

The Test Gate

Do not experiment with a question the business should simply govern.

Testing is useful when uncertainty blocks a real choice and a fair comparison can resolve it. It is a poor substitute for judgment when the answer is already set by law, ethics, customer trust, or an obvious quality standard.

ConditionExperiment whenDecide directly when

Consequence

At least two feasible actions would create materially different outcomes.

Only one option meets the company's legal, ethical, or trust standard.

Uncertainty

The team does not know which action causes the better result.

The issue is a known defect, a broken promise, or missing operating discipline.

Comparison

Eligible units can receive stable conditions without harmful spillover.

Treatment changes the control group or the comparison cannot remain fair.

Timing

The outcome can mature before the decision loses value.

Delay costs more than the uncertainty, or exposure creates unacceptable risk.

Entry rule: run the test only when at least one plausible result would change what the company does.

Precommit the Choice

Write the action rule before the result can influence it.

A hypothesis is not enough. The business needs a compact contract that says which result matters, how much change is worthwhile, what harm blocks the win, and when the evidence is mature enough to read.

Experiment decision contractFreeze before the first eligible unit is exposed.

DecisionName the choices and owner

State what the company may advance, change, stop, or leave alone after the test.

OutcomeChoose one primary measure

Set the smallest improvement worth acting on and the window in which it must appear.

ProtectionDefine harm before upside

Set customer, revenue, quality, legal, and operating guardrails with clear stop rules.

ValidityLock the comparison

Record the assignment unit, eligibility, exposure, exclusions, and maturity rule.

Change rule: if the team changes the primary measure, population, or success bar after seeing the data, label that analysis exploratory and confirm it with a new test.

Four Result States

Inconclusive and invalid are different business answers.

Before reading an effect, verify assignment, treatment delivery, metric integrity, and outcome maturity. Then report the state that the evidence supports. A test does not become positive because the team needs a launch story.

StateWhat the evidence supportsOperating response

PositiveA worthwhile improvement remains credible and every guardrail holds.

Advance only within the tested population, treatment, and conditions.

NegativeHarm is credible or meaningful upside is no longer plausible.

Stop the treatment and preserve why the belief failed.

InconclusiveThe evidence still allows meaningful upside, little difference, and possible harm.

Gather more evidence only if the remaining uncertainty is worth its cost.

InvalidThe implementation did not preserve the comparison the decision required.

Repair and rerun. Do not issue a verdict on the underlying idea.

Interpretation rule: no clear difference is not proof of no meaningful difference. The range of plausible outcomes must be narrow enough for the business choice.

Rollout Authority

A win authorizes the next controlled move, not unlimited scale.

The experiment estimates what happened under specific conditions. Broader rollout changes traffic, service load, customer mix, staff behavior, economics, and competitive response. Treat scale as a new risk decision.

Tested scope

Keep the claim inside the evidence.

Name the population, channel, treatment, timing, and outcome window the result can actually support.

Commercial value

Price the gain after delivery.

Subtract implementation cost, operating load, downside exposure, and any deterioration in customer quality.

Next exposure

Expand in a reversible step.

Keep the control path available, monitor guardrails, and state who can pause or reverse the rollout.

Shipping rule: a statistically credible lift can still be too small, expensive, fragile, or risky to ship.

AI + Human Boundary

AI can inspect the test. A person owns the exposure and the claim.

AI is useful for protocol checks, anomaly review, result explanation, and record preparation. It should not choose a favorable metric after launch, hide a guardrail breach, or convert an exploratory pattern into a settled conclusion.

AI may support

Prepare and challenge the evidence

Check whether the belief, treatment, outcome, and decision align.Flag sample imbalance, missing exposure, metric changes, and incomplete windows.Compare declared segments and label unplanned cuts as exploratory.Draft the result with uncertainty, limits, and sources attached.

A person must own

Risk, meaning, and authority

Choose the question, population, meaningful effect, and harm boundaries.Approve exposure and decide whether a compromised test can continue.Judge whether the evidence and commercial value justify action.Authorize rollout, repair, rejection, retest, or no change.

Human rule: the system may surface a promising pattern. It cannot grant that pattern business authority.

Experiment System Health

More tests can create more noise without creating better decisions.

Do not reward experiment volume or win rate. Those measures encourage trivial questions and weak controls. Judge the system by the value, integrity, speed, safety, and durability of the decisions it produces.

ValueDecision-ready test rate

Tests launched with a real choice, meaningful effect threshold, action rule, and accountable owner.

IntegrityInvalid or compromised rate

Tests blocked by assignment, exposure, interference, instrumentation, or maturity failures.

SpeedTime from question to defensible action

The full delay from an approved belief through exposure, maturity, analysis, and decision.

SafetyDamage contained before scale

Guardrail failures detected and reversed while customer and business exposure remained bounded.

DurabilityValue that survives rollout

Results that remain commercially worthwhile after broader exposure, delivery cost, and downstream effects appear.

Portfolio rule: a test that cannot change a decision is measurement debt before it begins.

Experiments + Test Results Record

Six lines protect the decision from hindsight.

The record preserves what the company believed, what it allowed the test to change, whether the comparison held, and why the final action was justified.

01Decision

Which real choice can this test change, who owns it, and what is outside its authority?

02Belief

What should change, for whom, through which mechanism, and why is the uncertainty worth resolving?

03Design

Which units are eligible, how are conditions assigned, and where could treatment spill into control?

04Thresholds

Which primary outcome, meaningful effect, guardrails, stop rules, and maturity window govern the read?

05Result

Did integrity hold, which state applies, what remains uncertain, and which limits constrain the claim?

06Action

What happens next, who authorizes it, how far may it go, and what triggers review or reversal?

Handoff test: another leader can reconstruct the original choice and reach the same evidence boundary without hearing the team's preferred story.

Current Tools

Choose the testing surface after defining the decision contract.

Web optimization, open experimentation infrastructure, and feature delivery solve different operating problems. The platform should preserve assignment, exposure, metric versions, health checks, and rollback control.

VWOWeb + feature experimentation

Combines visual and feature tests with reusable metrics, guardrail checks, sample-ratio monitoring, and warnings when a running campaign changes in ways that can bias the result.

Best fitGrowth and product teams that need accessible web testing with stronger experiment-health controls than a simple page editor.

GrowthBookOpen-source experimentation

Runs feature flags and experiment analysis on existing data, exposes the SQL behind results, and can be self-hosted when the company needs infrastructure and privacy control.

Best fitTechnical teams that want transparent analysis and flexible deployment without handing product data to a closed testing stack.

LaunchDarklyFeature delivery + guarded rollout

Connects experiments to feature flags, versioned metrics, warehouse-backed measures, health checks, and controlled releases that can detect regressions during rollout.

Best fitProduct and engineering teams that need the test, release control, and reversal path in one operating surface.

Tools and links reviewed Q3 2026. Verify fit, data, privacy, AI terms, and pricing before use.

Examples Worth Studying

The hardest experiment problem is often the comparison, not the calculation.

Microsoft and DoorDash publish useful practices for blocking invalid reads and changing the assignment unit when users affect one another. These are company-published operating examples, not independent performance audits.

Microsoft-published experimentation practice

Microsoft blocks the effect until assignment passes

Microsoft says every A/B test in its Experimentation Platform must pass a sample-ratio mismatch check before the effect is revealed. In one MSN test, a treatment changed engagement enough to confuse bot filtering, so the missing users reversed the apparent result.

Lesson: inspect whether the comparison survived before debating the lift. Missing participants are often related to the treatment, not random data loss.

Study Microsoft's integrity gate

DoorDash-published engineering practice

DoorDash changes the unit when treatment spills over

DoorDash explains that a delivery assignment can affect other deliveries in the same marketplace, which makes a simple user-level A/B test unreliable. Its switchback design assigns treatment by region and time window so connected marketplace activity receives one condition.

Lesson: randomization does not fix interference. Choose an assignment unit large enough to contain the treatment's real operating effects.

Study DoorDash's switchback design

Tool sources: official product material from VWO, GrowthBook, and LaunchDarkly.

Operating examples: company-published material from Microsoft Research and DoorDash Engineering.

Measurement 03

Make the result strong enough to survive an inconvenient answer.

Test only a real choice. Freeze the success and harm rules before exposure. Reject a broken comparison, keep the claim inside the evidence, and treat rollout as a separate grant of authority.

Back to Measurement Get in touch