A/B тестирование ML моделей

A/B testing — золотой стандарт measurement impact новых моделей и features.

Design

Randomization: users, sessions, or requests. Sample size calculation: statistical power >80%. Metrics: primary (revenue), guardrails (churn, satisfaction). Duration: cover business cycles (weekend, promotions). Multiple testing correction для one experiment.

ML specifics

Champion vs Challenger: baseline vs new model. Multi-armed bandit: dynamic traffic allocation. Shadow deployment: new model scores без action, comparison offline. Interleaving для search/recommendations. Uplift measurement: не just accuracy, но business impact.

Common pitfalls

Sample ratio mismatch: bad randomization. Novelty effect: initial bump fades. Peeking: multiple looks inflate false positives. Segmentation bias: user-level variance overlooked. Interaction effects: multiple experiments interfering. Regulatory: gambling constraints on experimentation.

Инструменты

Optimizely, Split.io: platforms. Statsig, LaunchDarkly: modern. GrowthBook: open-source. Custom implementations: server-side (Netflix XPS). Statistical library: scipy.stats, statsmodels. Frequentist vs Bayesian frameworks. Interim analysis rules определены заранее.