From 917e0d052c8316be6dd146d8a6d975a93c36f4c6 Mon Sep 17 00:00:00 2001 From: Daniel Bachhuber Date: Wed, 22 Jan 2025 13:38:59 -0800 Subject: [PATCH] Win probability is calculated against control, not all variants --- contents/docs/experiments/funnels-statistics.mdx | 2 +- contents/docs/experiments/statistics.mdx | 2 +- contents/docs/experiments/trends-continuous-statistics.mdx | 2 +- contents/docs/experiments/trends-count-statistics.mdx | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/contents/docs/experiments/funnels-statistics.mdx b/contents/docs/experiments/funnels-statistics.mdx index c82f4f913b6d..4512503a83d9 100644 --- a/contents/docs/experiments/funnels-statistics.mdx +++ b/contents/docs/experiments/funnels-statistics.mdx @@ -26,7 +26,7 @@ One more thing worth noting: Bayesian inference starts with an initial guess tha ## Win probabilities -The **win probability** tells you how likely it is that a given variant has the highest conversion rate compared to all other variants. It helps you determine whether the metric shows a **statistically significant** real effect vs. simply random chance. +The **win probability** tells you how likely it is that a given variant produces a higher conversion rate compared to the control. It helps you determine whether the metric shows a **statistically significant** real effect vs. simply random chance. Let's say you're testing a new way of presenting pineapple on the website and have these results: diff --git a/contents/docs/experiments/statistics.mdx b/contents/docs/experiments/statistics.mdx index b3f87644ed27..2b8954f27044 100644 --- a/contents/docs/experiments/statistics.mdx +++ b/contents/docs/experiments/statistics.mdx @@ -23,7 +23,7 @@ Say you started an experiment a few hours ago and see these results: The first two values are pure math: dividing the number of successes by the total number of users gives us the raw success rates. It's not enough to just compare these conversion rates, however. -The last two values are derived using Bayesian statistics and describe our confidence in the results. The **win probability** tells you how likely it is that a given variant has the highest conversion rate compared to all other variants in the experiment. The **credible interval** tells you the range where the true conversion rate lies with 95% probability. It is displayed _relative to the control conversion rate_, which makes it easier to understand the likelihood of a variant being better than the control (e.g. the test variant performs somewhere between 69.89% worse and 158.88% better than the control). +The last two values are derived using Bayesian statistics and describe our confidence in the results. The **win probability** tells you how likely it is that a given variant produces a higher conversion rate compared to the control. The **credible interval** tells you the range where the true conversion rate lies with 95% probability. It is displayed _relative to the control conversion rate_, which makes it easier to understand the likelihood of a variant being better than the control (e.g. the test variant performs somewhere between 69.89% worse and 158.88% better than the control).