VWO Logo
Dashboard
VWO Testing
A/B Testing

A/B Testing Solutions to Maximize Your Website Conversions

Create and A/B test different versions of your website and its elements to continuously discover the best-performing versions that improve your conversions.

Know more
Complies with:
VWO GDPR Ready Badge
VWO CCPA Ready Badge
VWO G2Crowd Leader Spring Badge
VWO BS 10012 Certification Badge

We tweet useful stuff daily

Related content:

How to Create a Strong A/B Testing Hypothesis?

9 Min Read

Experimenting without a hypothesis is akin to getting lost in a labyrinth—it may appear that it’s leading you somewhere, but such pointless wandering can seldom lead you to your desired destination.

Whether it’s reaching your destination or hitting a conversion goal on your website, the process involves numerous baby steps that you take in the form of iterative testing and hypothecation. Along with the process, they keep evolving to find an optimal route to desired conversions. 

In this post, you will learn about constructing strong A/B testing hypotheses, which will ensure you are moving in the right direction to reach your destination in time.

What is a hypothesis?

A prediction that you make before running a test is called a hypothesis. It is a bold statement that clearly states what change you want to make, why you want to do so, and its expected impact. 

We have emphasized enough why constructing a hypothesis is vital before running any test. Now, let us make it more clear by following an example here.

One day you wake up and want to run a test for the color of the CTA button on your website. You rush and achieve conversion lifts with this change, but you do not know exactly what you wanted from this test? 

You did not pay any heed to your conversion funnel. That is, you did not analyze the test fully before executing it! 

Blog Banner How To Create A Strong Ab Testing Hypothesis

There is a difference between running a test scientifically to prove something and generating random results.Now, this something is your hypothesis.

Components of a hypothesis

As you would notice here, in the world of user experience optimization, a strong hypothesis is made up of three components: defining a problem, describing a proposed solution, and measuring results.

Components of Ab Testing Hypothesis

For instance, you could hypothesize that adding trust badges to your payment page could fix low conversion rates on that page. You can find out why that happens by identifying the right metrics for success.

But how does one begin to start formulating a hypothesis?

But how does one begin to start formulating a hypothesis?

How to formulate a winning hypothesis?

It is not wise to blindly follow best practices crafted by someone else for your business. Every business is unique, so are its strategies. It is always recommended to construct your own best practices. 

Below are some essential elements that make a solid hypothesis:

1. They aim to alter customer behavior, either positively or negatively.

A psychological principle often forms the basis of a hypothesis that triggers a reaction from prospects. 

It changes the way they perceive your offer/brand, which alters their behavior during the test. Like in the headline example below, the urgency of the new message is why the variation headline is expected to perform better than the original headline.

Changing the headline from ‘Grab your tickets now!’ to 

‘Tickets filling out soon – only last 50 left!’ could increase ticket sales online. 

Only because you follow the above syntax to formulate a hypothesis doesn’t mean that you’ve got the winning hypothesis.

2. They focus on deriving customer learning from tests.

When pros develop a hypothesis, their focus is on the big picture. Dr. Flint McGlaughlin reveals their secret sauce:

So your test might give you any result. What’s important is for you to analyze ‘why’ your visitors behaved in a certain way? 

As data-driven marketers, it might seem difficult sometimes to make your peace with negative lifts. 

But if the test reveals significant customer learning, it can pave the way for colossal conversion lifts in the future. 

For example, when Michael Aagaard, the conversion copywriter of Content Verve, conducted a CTA copy test on his client’s website. He learned that changing the CTA copy from ‘Start your 30 day free trial’ to ‘Start my 30-day free trial’ resulted in a 90% increase in sign-ups.

screenshot of the A/B test run by Michael Aagaard of Content Verve
Image source: Marketing Experiments

However, he cautioned that applying this technique would not work across the linguistic globe though.

3. They are derived from at least some evidence.

You can explore many avenues to construct good hypotheses. 

The image below by the popular Conversion Expert, Craig Sullivan, shows several ways to collect meaningful insights.

ways to collect meaningful insights to construct a hypothesis
Image source: Slideshare

a. Usability testing

In simple words, you can sit and observe how your test participants use your website. 

Make notes where they get stuck or confused. Form your hypotheses to improve these situations and let A/B tests decide if they work for you.

This method gives you exceptional insights into your customers’ workarounds and struggles in using your website. 

Write down the exact questions you would like to ask during your research. Asking questions related to the homepage, checkout, pricing page, and navigation reveal great insights. 

Some sample questions you can ask are: 

  • Was it easy to find what you were looking for? 
  • Were the words/vocab used to define categories/sub-categories clear to you? 
  • Do you have any suggestions to improve our website navigation? 
  • Does our website look credible to you? 
  • Is our pricing transparent? 
  • Is there anything else you’d like to know before signing up with us? 
  • Will you shop with us again? Why/why not? 
  • Do you think the form has any confusing/unnecessary input fields?

b. Customer surveys

Surveys are an effective tool for understanding your customer hesitations and knowing their intent/pain points or concerns. They help you identify the optimization opportunities on your websites.

Here are some sample questions to use for on-site surveys:

a screenshot of the sample questions used by twilio for on-site surveys
Image source: Qualaroo

Visiting the pricing page shows the intent of buying. See how Qualaroo leverages targeted traffic on their pricing page to understand customer pain points:

screenshot of the survey run by Qualaroo on their pricing page

Ask these survey questions during cancellation of a subscription:

screenshot of a survey asking questions to inquire about the reasons for cancelling subscription

To existing customers, you can ask: 

Have you ever criticized/praised us to someone in the past few months? 

What did you say? 

If you are made the owner of [insert your company name], what would you change? 

For those who have just signed up with you, you can send an auto-generated mail asking: 

Did you have any doubts or concerns about our product/service before you signed up? 

What made you overcome those doubts? How did you hear about us? 

Too many questions can be annoying, especially in on-site surveys. Make sure you respect your prospects’ choice if they choose not to answer your questions. 

Similarly, off-site surveys also serve the same purpose of gathering feedback via email or third-party survey websites. 

Apart from SurveyMonkey, you could also use Fluid Surveys and Confirmit for designing off-site feedback surveys.

c. Heatmaps

The data in your testing tool can tell you what your visitors are doing but not why they are doing it. 

Heatmaps can help you identify interest areas of your prospects as well as what areas they choose to ignore. Sometimes this can help you identify great insights when a vital page element goes unnoticed by visitors because some other element on the page is stealing its thunder. 

The heatmap below shows how a non-clickable element in the image takes away all the attention from the CTA on the page:

screenshot of a heatmap that highlights the distraction of a non-clickable element in an image
Image source: Techwyse

Later, when this element was removed, the heatmap shows a clear emphasis on the call-to-action of the page (as it should be):

screenshot of the heatmap of the web-page when the distracting element has been removed

Without referring to their heatmap, TechWyse had never understood the reason for their dropping conversion numbers.

If not this, heatmaps can sometimes also help you figure out a page element/navigation item that may be taking too much of important online real estate while being completely ignored by users. 

You can consider replacing it with something more relevant to your conversion goal and conduct an A/B test to see how it performs. 

d. Website analytics

You might know this already, but exploring data in your website analytics can give you some great insights to get started. 

For instance, the hypothesis of “adding trust badges on payment page” we formed earlier, could have originated from a “high exit rate of the page.” The exit rate of the page — along with other metrics — can be found within your website analytics. 

Website analytics tools like Google Analytics can show you quantitative data on how visitors navigate your website on a site architecture level. 

Some of the important metrics that you could track to validate an idea and build a hypothesis are:

  • Traffic report: Metrics like total traffic, the total number of visitors (overall and on individual pages) could help you track how many people the test will impact and how long it would take to finish it. 
  • Acquisition report: This could help you determine where your visitors are coming from (your best traffic sources) and how the performance differs between different channels. 
  • Landing page report: Your top landing and exit pages show how visitors enter and leave the site. 
  • Funnel report: This would give you insights into questions like where your visitors enter into or exit from your marketing funnel and how they navigate between the different pages.
  • Device type: This will help you decide whether you should focus on optimizing the user experience on a particular device on priority. 

For any observation that you come across from analyzing these, ask yourself enough ‘why’s to form a solid hypothesis.

Test with confidence

Now since the grunt work has been done, nothing should stop you. It is time to take a plunge:

  • Every outcome should be hypothesized: Remember that every test is an opportunity to learn. Think one step ahead of your experiment as to what you would learn if your hypothesis is proven correct or incorrect.
  • High-level goals mapping: You must keep yourself acquainted with company-wide goals in order to construct robust tests. You need to ensure that the tests you are conducting are backed by data. They will provide you with the insights that would help grow your business.
  • Documentation: This is a very crucial step. Make it a habit to document all of the tests that you run. This will serve as a reference for future tests and can be utilized as a forum wherein marketers can share the context for the tests conducted so far. 

Forming a well-structured hypothesis is a critical piece in the conversion optimization puzzle. It helps you identify and remove the friction along your conversion funnel.

You might find that you have multiple hypotheses to try. But instead of running in all directions, prioritize your tests and monitor them for conversions and learning.

Banner How To Create A Strong Ab Testing Hypothesis 1
Nida Zehra
Nida Zehra Content Marketer at VWO
More from VWO on A/B Testing
Top 4 Methods for Finding Test Ideas for Your eCommerce Store

Top 4 Methods for Finding Test Ideas for Your eCommerce Store

Are you looking to continuously improve your eCommerce store? If so, you’ve probably wondered whether…

Read More
Daniel Escobar

Daniel Escobar

11 Min Read
Learn How Experts Derive Insights from A/B Test Results

Learn How Experts Derive Insights from A/B Test Results

You conducted an A/B test—great! But what next? How would you derive valuable insights from…

Read More
Vaishali Jain

Vaishali Jain

7 Min Read
Why Random A/B Testing is a One Night Stand

Why Random A/B Testing is a One Night Stand

So you run a test or two, have your moments of virtual euphoria and promise yourself…

Read More
Mohita Nagpal

Mohita Nagpal

5 Min Read

Scale your A/B testing and experimentation with VWO.

Start Free Trial Request Demo
Shanaz Khan from VWO

Hi, I am Shanaz from the VWO Research Desk.

Join our community of 10,000+ Marketing, Product & UX Folks today & never miss the latest from the world of experience optimization.

A value for this field is required.

Thank you!

Check your inbox for the confirmation mail