Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Win probability is calculated against control, not all variants #10437

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion contents/docs/experiments/funnels-statistics.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ One more thing worth noting: Bayesian inference starts with an initial guess tha

## Win probabilities

The **win probability** tells you how likely it is that a given variant has the highest conversion rate compared to all other variants. It helps you determine whether the metric shows a **statistically significant** real effect vs. simply random chance.
The **win probability** tells you how likely it is that a given variant produces a higher conversion rate compared to the control. It helps you determine whether the metric shows a **statistically significant** real effect vs. simply random chance.

Let's say you're testing a new way of presenting pineapple on the website and have these results:

Expand Down
2 changes: 1 addition & 1 deletion contents/docs/experiments/statistics.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ Say you started an experiment a few hours ago and see these results:

The first two values are pure math: dividing the number of successes by the total number of users gives us the raw success rates. It's not enough to just compare these conversion rates, however.

The last two values are derived using Bayesian statistics and describe our confidence in the results. The **win probability** tells you how likely it is that a given variant has the highest conversion rate compared to all other variants in the experiment. The **credible interval** tells you the range where the true conversion rate lies with 95% probability. It is displayed _relative to the control conversion rate_, which makes it easier to understand the likelihood of a variant being better than the control (e.g. the test variant performs somewhere between 69.89% worse and 158.88% better than the control).
The last two values are derived using Bayesian statistics and describe our confidence in the results. The **win probability** tells you how likely it is that a given variant produces a higher conversion rate compared to the control. The **credible interval** tells you the range where the true conversion rate lies with 95% probability. It is displayed _relative to the control conversion rate_, which makes it easier to understand the likelihood of a variant being better than the control (e.g. the test variant performs somewhere between 69.89% worse and 158.88% better than the control).

<ProductScreenshot
imageLight = "https://res.cloudinary.com/dmukukwp6/image/upload/wide_credible_interval_light_v2_fa81c3a2ee.png"
Expand Down
2 changes: 1 addition & 1 deletion contents/docs/experiments/trends-continuous-statistics.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ One more thing worth noting: Bayesian inference starts with an initial guess tha

## Win probabilities

The **win probability** tells you how likely it is that a given variant has the highest value compared to all other variants. It helps you determine whether the metric shows a **statistically significant** real effect vs. simply random chance.
The **win probability** tells you how likely it is that a given variant produces a higher value compared to the control. It helps you determine whether the metric shows a **statistically significant** real effect vs. simply random chance.

Let's say you're testing a new pricing page and have these results:

Expand Down
2 changes: 1 addition & 1 deletion contents/docs/experiments/trends-count-statistics.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ One more thing worth noting: Bayesian inference starts with an initial guess tha

## Win probabilities

The **win probability** tells you how likely it is that a given variant has the highest rate compared to all other variants. It helps you determine whether the metric shows a **statistically significant** real effect vs. simply random chance.
The **win probability** tells you how likely it is that a given variant produces a higher rate compared to the control. It helps you determine whether the metric shows a **statistically significant** real effect vs. simply random chance.

Let's say you're testing a new menu design and have these results:

Expand Down
Loading