Working example 06 Bayesian change-point analysis

Repeatedly unsuccessful approaches

“We have tried several solutions, but the problem keeps returning.”

An online retailer has made three fixes to complaint handling in two years. Each seemed to work, then complaints returned. The team is planning a fourth fix.

How it works

The method in three steps

Re-examine the measureRaw complaint counts move with order volume. Dividing by orders tests whether the underlying problem has really changed.y_t ~ Poisson(λ · e_t), e_t = 1 or orders_t
Test each fixFor each fix, compare a model where the rate changed on that date with one where it did not, using the weeks either side.BF = p(y_before) · p(y_after) / p(y_before ∪ y_after)
Search for the real changeThe conjugate Gamma prior gives an exact posterior over the week in which the level changed.p(τ | y) ∝ p(y₁…y_τ) · p(y_τ₊₁…y_T)

Try it

The working example

Analysis settings

Measure Complaints per week Per 1,000 orders

Weeks either side of each fix

Operational event log

Week 14	Price list updated
Week 40	Warehouse system upgrade
Week 66	Switched to a new courier
Week 90	New packaging supplier

Most likely change–

Evidence for a change–Bayes factor against no change at all

Fixes that worked–

Two years of complaints

Fixes are marked in purple; the detected change point in navy.

When did the level change?

Posterior probability for each possible change week.

Did each fix work?

Comparison of the weeks before and after each fix.

Fix	Change	Bayes factor	Evidence

Interpretation

From example to engagement

What the example shows

When fixes are triggered by spikes, the next measurement is almost always lower: spikes are, by definition, unusual. That makes every fix look effective. Measuring the right thing removes the illusion, and a change-point search points to a cause nobody was investigating.

With real data

Incident logs, volumes and an honest timeline of operational changes.
Multiple change points, seasonality and covariates through Bayesian structural time-series models.
Process walk-throughs to test whether the statistical signal makes operational sense.