A/B testing is an experimental method used in Conversion Rate Optimization (CRO) to improve user experience. It compares two versions of the same thing, such as a website, app, ad, or email, to determine which one is more effective in improving conversion rates. CRO aims to maximize the number of users who take desired actions, such as increasing sales or filling out forms. In A/B testing, user traffic is divided into two groups—one sees version A, and the other sees version B. Results such as conversion rate , clicks, and time on page are analyzed to determine which version is more effective at achieving the intended goals, providing an effective way to optimize results in the CRO process.
Form A and B variants
We will promote your company on the Internet
Order a free quote!
Your company email *
Phone number
Page address
Content of the inquiry
* I consent to KS Sp. z o. o. (NIP: 6852338589) sending cooperation proposals to the email address I provided and to using it for email communication for marketing purposes. I also consent to being contacted by phone in order to process my offer application.
(read more)
How to prepare for A/B testing?
For A/B testing to yield valuable and reliable results, proper preparation is important. This includes not only choosing the element to test, but also planning the experiment in detail, from setting a hypothesis to verifying the data. Below are the steps to help you prepare for A/B testing:
1. Identifying areas that require change and affect the effectiveness of the website
The first step in setting up an A/B test is to identify elements on your page that may need to be optimized. These could be call-to-action (CTA ) buttons, headings, form layout, colors, images, or other interactive elements. Consider which areas of your page could impact conversions , such as button clicks or time spent on the page.
2. Preparation of version B (alternative)
Once you’ve identified the element you want to test, it’s time to create an alternate version of it. Version B might differ from A in just one element, such as the button content, color, or location on the page. It’s important to change only one variable in each version so you can accurately assess what affected the result.
3. Creating a test scenario
A/B testing requires a clear plan. Prepare a scenario that includes a few key steps:
What will be changed? Define what element is being modified job seekers database and what are the differences between version A and version B.
Hypothesis: Define how the change will affect user behavior. For example: “Changing the color of the button to a more visible one will increase clicks by 10%.”
Metrics: Set key success indicators that will allow you to evaluate the effectiveness of both versions. These could be conversion rates, clicks, time on page, form submissions, etc.
Custom Events: If you’re testing more complex interactions, like clicks on dynamic elements, define the custom events you want to track. These could include clicks on drop-down menus, scrolling through a page, or browsing a product gallery.
4. Estimating website traffic and test duration
For A/B testing to provide reliable results, it is crucial to conduct the test on a large enough sample of users to ensure that the results are statistically significant. Statistical significance allows you to assess whether the differences between version A and version B are the result of an actual change and not random fluctuations. Here are the steps to help you calculate an appropriate sample:
How to calculate a sample for A/B testing?
Determine the conversion rate for the current version of the page (version A): First, you need to know the current conversion rate, which is the percentage of users who complete the desired action (e.g., click a button, fill out a form). This metric will be the baseline for evaluating version B.
Define the Minimum Detectable Effect (MDE): Think about the minimum improvement in results you think is worth noting. For example, do you want version B to improve conversion rates by 5%, 10%, or more? The MDE is the difference in results you want to detect. The smaller the MDE, the larger the sample size you will need to achieve statistical significance.
Determine the significance level (α) and the power of the test (1-β):
The significance level (α) is the probability of obtaining a difference result that does not actually exist (a so-called type I error). It is typically set at 5% (0.05), which means 95% certainty that the difference detected is not accidental.
The power of a test (1-β) is the probability of detecting a true difference between versions A and B. Typically, the power of a test is set to 80% (0.8), which means that there is 80% confidence that the test will detect a true difference.
Use a calculator to calculate your sample size: There are many online calculators for calculating your A/B test sample size. To use one, you will need information about:
Current conversion rate (for version A),
Minimum Detectable Difference (MDE),
Significance level (α),
Power of the test (1-β).
Example:
If your current conversion rate is 10% and you want to detect a 20% increase (up to 12%), with a significance level of 5% and a test power of 80%, the calculator will show how many users must complete the test (for both versions) for the difference to be statistically significant. For the given data, at least 3,623 users should see each version.
5. Website traffic and test duration
Once you have calculated the number of users you need for your test, you should estimate how long it will take to reach that trial based on your site traffic. For example, if your site averages 1,000 visitors per day and the calculator says you need 10,000 users for each version (A and B), your test should run for at least 20 days. Just make sure the test is long enough to account for different days of the week and natural fluctuations in traffic.
6. Implementation of an alternative version
Once everything is ready, it’s time to deploy the alternate version to your site. Make sure your A/B testing system is working properly and that traffic is being split between both versions.
7. Event measurement in analytical tools
An important step is to configure your analytics carefully. Install and configure tools such as Google Analytics , Microsoft Clarity, Hotjar, or other solutions for monitoring website events. Make sure all custom events are tracked correctly so you can later analyze how both versions affected user behavior. Monitor interactions that are important to the experiment, such as button clicks or form fills.
Measuring Page Scrolling with Clarity A/B Testing
Measuring page scrolling using A/B testing in Clarity. Source: clarity
