How to Choose the Right Testing Software For Your Business

by 9 06/25/2014

If you’ve spent any amount of time on The Daily Egg or similarly focused blogs, you know that thorough testing is paramount to your web business’ success.

Every customer base is unique, and winning marketing victories depends on understanding the users visiting your website each and every day. You might be intuitive (or lucky) enough to correctly read your customers on any given campaign, but over the long run, success WILL depend on data analysis.

Despite the hundreds of voices claiming that marketing is an art, you’ve heard of Neil Patel, Rand Fishkin, Avinash Kaushik, Danny Sullivan, and Brian Clark for one reason: They understand marketing is a science.

Unfortunately for the the small business owner, data analysis is decidedly technical. Simply reading a few blogs won’t be enough to turn data into dollars. You’ll need a handful of powerful tools at your disposable.

Fortunately, there are an abundance of such tools available.


source: Placeit.net

If you have a marketing team ready to pounce on this, these programs will provide an effective setup for efficient testing. If the burden falls on you, the business owner, there will be a slight learning curve. But don’t worry, each of the programs listed here are highly successful, fully supported, and come with a dedicated team invested in making your experience flawless.

Keep reading for our review of six A/B testing software.

1. Google Analytics

google_analytics_review
As the only free option on our list, Google Analytics is the baseline of all Internet data analysis. It offers users the rare opportunity to draw directly from Google’s extensive data pool. If you’ve never used Google Analytics before, or if you’ve wandered around the dashboard a few times without really grasping its potential, here’s a great place to start.

Once you’ve learned the basics, it’s time to use this new-found data to drive revenue for your business. This requires actionable steps, and you can begin here.

So what about A/B testing?

Analytics offers a feature called “Content Experiments,” which allows users to split-test page variants within the program. It’s a useful feature with advanced reporting options but unfortunately lacks a visual editor, posing a major stumbling block for many would-be users.

The main drawback to Google Analytics is the steep learning curve.

There’s a wealth of information there, but you’ll need to be a pretty proficient analyst to take advantage. Advanced use of Google Analytics should be an aspiration of every business owner, but it’s definitely not the easiest place to start.

The Pros

  • It’s free!
  • Simple interface
  • “Goals” feature allows for easy tracking of business targets
  • Offers quicker, more efficient multi-armed bandit testing
  • Directly integrated with other analysis features and huge data pool

The Cons

  • Raw data – interpretation and use are on you
  • Lacks multivariate testing and other testing options offered by premium competitors
  • Difficult to use on dynamic pages
  • Lacks a visual-based editor
  • Unable to integrate additional data sources

2. Optimizely

optimizely-screen-cap

Optimizely is a great first stop for business owners wanting to start testing. Installation is remarkably simple, and the WYSIWYG interface is highly intuitive, even for newbies. With plans starting at $17, Optimizely is a very cost-efficient entry point.

But don’t mistake simplicity and budget pricing for an inferior product. Optimizely’s testing software is good. Very good. With split, multivariate, and mobile testing options, you’ll be teaching data anlysis by the time you’ve outgrown this product’s utility.

Plus, as your company grows and you become more familiar with testing, advanced features at the higher price point allow for continued use without needing to learn completely new software.

If you’re starting completely from scratch, Optimizely might not be your best option. You’ll need passable landing pages to test in the first place, and you are entirely responsible for creating your own A/B tests. This is a pretty big prerequisite for someone whose marketing knowledge consists of nothing more than a few blog post reads.

The Pros

  • Cost-efficient premium option for entry-level users
  • Easy, intuitive interface
  • Effective A/B, multivariate, and mobile-focused testing options
  • Solid basic offering with quality advanced features at higher price point

The Cons

  • Lacks outside data integration
  • No algorithm alternatives for judging results, such as bandit or adaptive testing
  • Multivariate testing requires expensive Gold membership ($359 per month)

2nd Opinion

“To start, Optimizely is very competitively priced and dead simple to set-up and install, so the bar to start getting a return on your investment starts off low. From there, it’s up to you. What ideas do you have to improve your digital business? Optimizely is a tool that will help you test and measure them, and quantifiably tell you if they work or not. Your return will be as good as your testing ideas.”

- Ryan Garner, Co-founder Clearhead

3. Visual Website Optimizer

visual-website-optimizer-screen

Visual Website Optimizer (VWO) is a full-featured testing suite that’s achieved tremendous popularity in the last two years. Overall, it’s very similar to Optmizely but with a few key differences.

At $49 per month for the starting price point, it’s not exactly the “Why not?” pricing of Optimizely, but you can expect a much wider feature set for the money. VWO boasts a simple, easy-to-use interface and comes with an impressive 7 different testing options:

  1. A/B Test
  2. Multivariate Test
  3. Split URL Test
  4. Targeting
  5. Usability Test
  6. Conversion Tracking
  7. Heatmap and Clicks

The software also includes a handful of  idea-generation tools. This is a unique selling point for business owners unsure of where to start in creating and running A/B tests.

Overall, VWO’s benefit to your company depends on your needs. If you need something dirt cheap to get started, or something very advanced, Optimizely is probably the way to go. If you’re needing beginner- to intermediate-level testing options at a traffic level greater than 1,000 visitors per month, VWO is the perfect tool for your business.

The Pros

  • Feature-rich for the price – full feature set available at all price points
  • Intuitive, easy-to-use interface
  • Idea-generation tools

The Cons

  • Lacks same level of real-time support options compared to competitors
  • Lacks multi-page testing options
  • Hit or miss online documentation

2nd Opinion

“Overall, Visual Website Optimizer has the slight edge for casual A/B testers, but its more expensive than the basic version of Optimizely. For serious A/B testers, I definitely recommend Optimizely – more comprehensive offering, even though you have to pay extra for premium features.”

- Rich Page, Website Optmization Consultant

4. Unbounce

unbounce-screen

Unbounce has carved out a niche for itself by focusing solely on landing pages. That’s it. Just landing pages. And they’ve managed to make themselves somewhat of an authority on the topic as a result.

Unbounce’s unique distinction is that they offer 80+ pre-designed, optimized landing page templates. This is a fantastic resource for business owners starting from scratch or who have seen little success with their own designs. Each template is engineered by the Unbounce team for the purpose of conversions and can be quickly customized for your company’s use.

In addition to templates, an Unbounce subscription will allow you to perform A/B testing on your landing page. You’ll be given a few metrics for interpreting the data, and that’s it. Other than a few periphery extras for the higher-priced options, the full Unbounce experience consists of landing page templates and A/B split testing.

If you’re a true marketing rookie and want results fast, this is probably your best bet. Few other programs cater to ground-zero quite like Unbounce, and you’ll be able to publish conversion-optimized landing pages in a matter of hours. To sum things up, Unbounce is limited in its goals but accomplishes those goals admirably.

The Pros

  • Landing page templates provide great start-from-scratch resource
  • Solid split testing capablities
  • Simple, easy-to-use interface
  • Designed to easily integrate with other marketing and testing software

The Cons

  • Limited to simple A/B split testing
  • Relatively high cost for a limited feature set

2nd Opinion

When creating a new landing page, Unbounce is our company’s go-to for A/B testing conversions. It is quick, gets the job done, and the analytics are actionable. We can’t live without its confidence score rating, which tells you how likely a good conversion rate is, or whether it is mere luck.

 –  Brian CurlissCEO MailLift

5. KISSmetrics

kissmetrics-screen

Unless you’ve been living in a box for the last few years, you’ve heard of KISSmetrics. This popular testing tool focuses on the human side of data. Reports and testing data are connected to real people, who are tracked across their interactions with your website.

At $150 per month to start (with full-year commitment), KISSmetrics is definitely a premium service, but it comes with the advanced abilities you’d expect at that price point. You won’t find software offering deeper or more diverse sets of data. KISSmetrics lets you keep a pulse on the entire conversion funnel, from start to finish, and identify where and how customers are disconnecting from your website.

If you are serious about data, KISSmetrics is one of the best tools around. Its ability to connect data to actual users might also be a great alternative for business owners struggling to wrap their heads around more traditional reporting styles.

The Pros

  • Highly flexible data reporting options
  • Simple, intuitive interface
  • Can trace data to real people
  • Excels  at funnels with extremely accurate funnel-based data
  • Integrates well with complementary software

The Cons

  • Lower powered A/B testing compared to other testing software
  • High cost – won’t be affordable for many small businesses
  • More advanced data reporting requires a higher learning curve

2nd Opinion

“I use other split-testing tools, such as Optimizely, but KISSmetrics is a must if you’re serious about getting accurate results. The problem with many of the tools out there is they don’t measure the impact of the test throughout your entire funnel. That’s why I love KISSmetrics; it will show you exactly what the impact is for a test at every step of the buying process (not just the next step).”

- Ruben Gamez, Founder Bidsketch

6. Crazy Egg

sample-heatmap-crazyegg

Of course, our list wouldn’t be complete without the home team’s offering. Crazy Egg specializes in click-tracking. The goal of all testing is to understand user behavior, and Crazy Egg’s software allows business owners to identify exactly what users are doing when they visit your website.

If you’re a visual learner, Crazy Egg will be your go-to source for data. The six different available reports frame data in an easily digestible way that is intuitive and provides immediate insight into user activity, regardless of your analytical skills.

If you’re at least capable of building a landing page, Crazy Egg is the ultimate entry-level option. At $9 per month, it is the cheapest testing software you’ll find anywhere.

While it is decidedly limited in scope and test variety, you’ll find it much easier to use than Google Analytics and most other options on this list. It is, perhaps, the only offering on this list that allows business owners to immediately read and understand test data with zero context for web analytics.

The Pros

  • Highly visual data – enjoyable to view
  • The benchmark of interface simplicity
  • Extremely low learning curve with intuitive reporting
  • Cheapest pricing you’ll find for testing software

The Cons

  • Limited feature sets and testing options
  • Limited utility for dynamic pages

2nd Opinion

“It’s a great tool considering how inexpensive it is. If used correctly and you have a plan for tracking your websites, this tool can make a world of a difference.”

- Amber Steinert, Marketing Analyst 3M

Read other Crazy Egg articles by Jacob McMillen.

About 

Jacob McMillen is an expert copywriter, blogger, and small business marketing specialist. In addition to increasing revenue for quality products, he enjoys reading exceptional books and pretending to think in his spare time. Connect at JacobMcMillen.com

Get our Daily Newsletter

Get conversion optimization, design and copywriting articles delivered to your inbox FREE

9 COMMENTS

Zac Aghion

Nice review Jacob, thanks!

Google Content Experiments can also be used to run automated A/B tests. From a high level, the way this approach works is to start showing variations that work better more often, automatically. It’s a much better solution to the question of ‘What works better: A or B?”

In the most basic example of an A/B test, you have a variation A and a variation B each shown to 50% of your user base. By definition, this approach will be sending half of your users to a worse performing version during the entire duration of the test!

The automated approach is based on a bandit algorithm that updates the percentage of your user base which is shown a given variation dynamically. With each new piece of data that you collect on the conversion rate and statistical confidence of the variations in a test, it adjusts the percentages automatically so that better performing variations are promoted and worse performers are pruned away.

This leads to:

1) faster results, because your directing test resources (i.e.: users and their data) to validate what you actually care about (i.e.: confidence in the best variation’s performance)

2) a higher average conversion rate during the test itself, because relatively more users are being sent to the better performing variation automatically, and

3) less time and effort required to actively manage your experiments.

Though the math behind this approach is slightly more complex than a traditional A/B test, it’s worth talking about for audiences that are interested in making data-driven decisions because of how much better the results are that it produces.

For anyone interested, here’s a post we put together on how it works: http://splitforce.com/resources/auto-optimization/

June 25, 2014 Reply

    Kathryn Aragon

    Great insights, Zac. Thanks for sharing your expertise.

    June 25, 2014 Reply

    Brian Lang

    Great comment Zac – I’m convinced minimizing regret is the way to go, but I think there are limitations/concerns with existing offerings – Auto-optimization sounds great in theory, but the reality is it isn’t as simple.

    For example, most of the available tools haven’t implemented contextually aware auto-optimization methods, so you ultimately end up creating a lot more work implementing the same experiment for many different user segments (a time consuming pain point), or you settle for, “what’s the best option for most”.

    Separately, overlapping experiments with an auto-optimization setup, where one experiments better-performing arm can inadvertently skew (positively) the results of another experiments inferior arm, is a real problem, and most tools available don’t offer a solution to it.

    There are also issues with traffic quality and returning customer bias (through things like the recency/novelty effect) in-advertently skewing results of one arm, when it in fact might be the inferior one – in this situation, you would in-advertently be maximizing, not minimizing regret, and depending on the method of your “stopping point criteria”, might lead to wrong conclusions being made.

    A frequentist approach *typically* doesn’t suffer the above issues, since you are typically doing a 50/50 split of traffic, and can then segment results. There are, of course, ways to address these issues in an auto-optimization setup, but as of yet I haven’t really found a tool that does so.

    Thanks again!

    Brian

    June 26, 2014 Reply

Tia Kelly

Hey Jacob! Really appreciate you mentioning Unbounce in this list. Have to give props to the other 5 you chose to highlight here :) All do a great job in their specialty. I also liked the “second opinion” quotes—something different that I haven’t really seen done before in a top tools type of post that adds credibility. Thanks!

Social Strategist, Unbounce

June 26, 2014 Reply

    Jacob McMillen

    Thanks Tia. My goal is that it will make choosing software easier for business owners.

    June 26, 2014 Reply

      Neil Patel

      Jacob, Im sure it will. Thanks for the feedback :)

      June 26, 2014 Reply

Dennis van der Heijden

Hi Jacob,

Hope Convert.com makes it to the next list.

Dennis

June 26, 2014 Reply


Leave comment

Some HTML allowed

Get conversion optimization & A/B testing articles FREE >>>