A/B Testing Your SaaS Website: Tiny Changes & Big Results

If you have a SaaS website that gets moderate traffic on a monthly basis (~5,000 total visits per month), you really need to consider A/B testing different aspects of your website in order to increase the amount of conversions and leads generated throughout it.

In this post, I’ll share the results from a recent A/B test we conducted for one of our clients, and explain how to set up your very own test.

Now you may be thinking, “Splendiferous, Matt! But what in the heck is A/B testing?”

I’m so glad you asked because I prepped the next section just for you.

What is A/B testing?

A/B testing (also referred to as split testing) is the method of testing two different versions of something on your website while only changing one variable between the two versions. Typically, the “A Version” (or control) is what’s currently living on your site and the “B Version” (or variation) is something new you’re trying to test.

For example, let’s say you wanted to test the color of the “Demo Request” button on your website in order to see if your audience reacts any differently to other colors. Does one color drive more clicks than the other? Or, does one color lead to a higher rate of contact form submissions? You won’t know unless you test it out!

For this A/B test example, the “A Version” would be the current color of your “Demo Request” button, and the “B Version” would be whatever new color you want to test out.

This is similar to an experiment we shared earlier this year where changing one word on our client’s website led to a 42% increase in software demos. If interested, you can learn more about that experiment here.

Client A/B Test: The CTA Button Experiment

The Overview

Recently, we performed an A/B test for one of our clients who wanted to get more conversions from one of their inbound marketing offers.

The goal of this test was to determine a more effective way to increase the number of web visitors submitting their contact info in order to download this offer.

To perform the test, we decided to slightly tweak the CTA associated with this offer to see if it would have any positive impact on our submission rate.

Our “A Version” had the phrase “Download Now” on the CTA.

For our “B Version,” we decided to test how website visitors would react to the phrase “Download Free Template” on the CTA.

Our hypothesis for this A/B test was the following:

Including the word “Free” in our CTA will convert more web visitors because of the positive psychological connotation associated with this word.

(Who doesn’t love free stuff?!)

The Results

Running this test for a few months produced the following results:

Even though the data seems fairly close to one another, when we calculated the difference between these two submission rates we found the difference to be statistically significant.

The Conclusion

With these results, we concluded that our initial hypothesis was correct—version B drove more submissions. Despite version B having a lower click rate, we determined with our client that having a higher rate of conversions was more important than having a higher rate of web visitors just clicking the CTA. Plus, the difference in click rates was not statistically significant. As a result, we decided to turn off version A and continue running only version B on the website.

How To Run An A/B Test On Your SaaS Website

Chomping at the bit to try your own A/B test? Here’s a quick step-by-step guide to help you set it up:

Step 1: Identify your test subject.

Look for anything on your website that receives a fair amount of views on a monthly basis. This way, you’re not testing it for a full year (or longer) before you collect enough data to reach a level of statistical significance between the two versions.

Typically, A/B tests will need to run at least 2-3 months before you have enough data to reach a level of statistical significance, but this number will always depend on the volume of traffic reaching your website.

Also, you need to make sure whatever you’re testing has the ability to convert your traffic into leads. This can include anything from your main website CTA (like a “Free Demo” or “Demo Request” button on your site’s header menu) to a contact form on one of your most popular pages.

Step 2: Create a hypothesis.

You need some sort of reasoning to back up your experiment; this is where the hypothesis comes in handy!

There are different methods for creating a hypothesis, but for the sake of this mini guide here’s a simple template you can apply to your own test:

Step 3: Create your “B variation” and start testing.

Next, you’re going to create your “B variation” and launch your test. If you’re using HubSpot, creating a “B variation” is as simple as clicking into the “Content Details” of a specific CTA or page, clicking “More” in the upper right corner, and then clicking “Create A/B Variation.”

From there, you’ll just need to follow the prompts from HubSpot in order to launch your test.

If your website doesn’t use HubSpot, you can also check out these third-party A/B testing tools to help create your “B variation” and launch your test:

• VWO
• Optimizely
• Omniconvert

Step 4: Analyze the results

Finally, once you’ve let your A/B test run for a month or two, you’ll want to compare the results between your two variations.

When performing your analysis, make sure the differences between the two variations are statistically significant. This is how you’ll know you’ve collected enough data to complete your A/B test.

HubSpot automatically tells you when one version is performing better than the other with a level of statistical significance, so you don’t have to worry much about this step if you’re using them.

While other third-party tools may have this feature included, you can also refer to this A/B Test Calculator to help you with identifying statistical significance in your data.

Once you’ve finalized your results and selected your winning variation, think about how you can apply your findings to other aspects of your website. For example, based on the conclusion from our client’s A/B test, we plan on incorporating the word “Free” on CTA buttons for their other inbound marketing offers.

If you truly want to continue improving your website performance, you have to keep testing and trying different things. Just make sure whatever you’re testing has enough traffic on a monthly basis and will provide enough of an impact for your website performance.

The reason you shouldn’t A/B test unless you have around 5,000 visitors per month has to do with collecting a large enough sample size within a reasonable amount of time.

You could always try A/B testing something on your website with a smaller amount of traffic, but your results might not be as noticeable (or significant) with that lower sample size. Also, you might need to test it for an absurd amount of time (a year or longer) before your test results even reach a level of statistical significance, at which point your audience tendencies could be completely different from when you first started your test.

Need help driving more traffic to your website?

Nectafy specializes in creating growth content for B2B SaaS websites in order to drive organic traffic and quality leads to them.

If you don’t have enough traffic coming to your SaaS website to perform an A/B test, see how Nectafy can help by talking to our founder Lance about your goals. He doesn’t bite—and even if he did you’d be meeting him behind the safety of your own computer screen!

Also, if you’re hungry for more information about SaaS website best practices, check out some of these other Nectafy blog posts:

• What Does B2B Content Strategy Look Like?
• 11 Items To Check On Your Website To See Why It's Not Generating Leads
• How To Generate $2 Million In Sales With Growth Content [Case Study]