Oblongix logoOblongixEnterprise AI transformation

Working example 03 Bayesian A/B/n testing

Uncertain impact

“We invest in interventions, but do not know which ones actually work.”

A subscription business spends on discounts, proactive calls and loyalty emails to keep customers. Each looks effective in isolation; none has been tested against doing nothing.

How it works

The method in three steps

  1. RandomiseEach week, 200 at-risk customers per arm receive a discount, a call, an email or nothing.arm ∈ {control, discount, call, email}
  2. Update beliefsRetention in each arm has a Beta posterior that sharpens as evidence accumulates.θ | data ~ Beta(1 + retained, 1 + lost)
  3. Decide on value20,000 posterior draws give the probability each option has the highest net value after its cost, and the expected regret of choosing it now.net = 1,000 × ((θ_arm − θ_control) × value − cost)

Try it

The working example

Experiment settings

Let the experiment run longer, or change what a retained customer is worth.

Customers in the test–
Best-value option–
Expected regret–Per 1,000 customers if we choose it now
Decision–

What we believe about each retention rate

Posterior distributions. Narrow, separated curves mean clear differences.

Net value after cost

Expected net value per 1,000 customers treated, with 95% credible intervals.

Results by intervention

InterventionRetainedRateUplift (95% interval)P(beats control)P(best value)Verdict

Interpretation

From example to engagement

What the example shows

The intervention with the highest retention rate is not always the one to fund: its cost can outweigh the extra customers kept. Some interventions are indistinguishable from doing nothing. Expected regret gives a principled stopping rule instead of waiting for an arbitrary significance threshold.

When an experiment is not possible

  • Difference-in-differences or synthetic controls using regions or periods that did not receive the intervention.
  • Propensity-score matching or inverse weighting on rich customer data.
  • Each relies on assumptions we would state and test explicitly.