How do you simultaneously help boost a test-driven culture, conveniently monitor test results and measure more KPIs than just the main conversion variable you set up in your testing tool?

Simple, build an A/B Test Dashboard. With the help of your developer and account manager for the A/B Testing tool you use, you can implement a custom dimension to allow everything I mentioned to be achieved in a dashboard.
First, the technical part:

Implement a Treatment Version Custom Dimension

The first element required is to coordinate with the account manager for your A/B Testing tool and have them work with your dev team to surface a custom dimension for each test participant.

This way you can delineate which treatment each user is being bucketed into. Once you have this fitted and you know it’s firing properly, you have endless data you can compare across your site via custom segmentation. But, in order to segment your way to the bonus, you have to know what to segment by.


A Plethora of Neglected KPIs for A/B Testing

The average A/B Test and subsequent analysis can be a bit narrow-focused, as it has a single conversion metric that you are focusing on in order to determine if your results are statistically significant.

This is because A/B Testing tools are set up around one key success event. Yet, as an analyst or product team, you need to know how this test you are performing could affect any of your KPIs downstream.

Are you testing a new Homepage design? Well, what if the new design is beating the control? Some would call it a success, but then, being the inquisitive analyst you are, you need to determine the impact your shiny newly reduced Bounce Rate had on your AdSense campaigns, email sign-ups and revenue!

Beyond this scenario are far more complex A/B Tests where you are checking complete end-to-end site redesign experiences or multi-page occurrences of a particular flow that is crucial to your Macro-conversion KPIs.

Yet, you still want to know the effect the test has on other Micro-conversion KPIs. If you are testing a new site redesign, you want to know more than just if it leads to more purchases (your main A/B Test KPI). You want to understand if the newly redesigned prominent search bar led to more search engagement if your updated navigational header assisted in more product views.

Testing a new checkout flow? You will want to know the impact on your Return Rate or subsequent purchases of premium features or packages. None of these scenarios will be an option without knowing where else to measure.


Measurement Plan, Segmentation and A/B Test Dashboard Utopia

Maybe you haven’t actually gone through a process similar to what I’ve been explaining. You are actually more used to the Wild-West, shoot ‘em up, lone-ranger, A/B Test culture, where you hypothesize and test without much planning, deeper analysis of test participants behavior, or you haven’t even promoted your tests to the wider team yet. If that is the case, then here is a simple process to follow.

1. Create a measurement plan and actually show it to people. You want to sit down with the analysts and product owners on your team and ask them what is important to them. If you don’t have all the KPIs you should measure in an A/B Test Dashboard, then ask your team and compile a list. Then determine whether these are actually worth measuring. Another element you shouldn’t move on without is goals. Many times, if goals aren’t already in place for the desired impact from your A/B Test, then make some up! Draw a line in the sand so you can at least say you failed or succeeded, not just in A vs. B, but also in B vs. what you want for your business.
2.  Build your segments. Do this by taking these mind-bending new KPIs you’ve just discovered with your team and build some segments around them at visit or visitor level, depending on your KPI. Do this for each of your treatment versions that you can now determine from the newly implemented custom dimension.
3.Build yourself a dashboard. Use whatever data visualization platform you have (Tableau, Excel or Domo) and create your dashboard comparing two equal sides of matching metrics, so your team can look at version A performance on the left and version B performance on the right.
Or you can choose to trend them out on a chart with each treatment version, as one of your series on the graph. This way your team can look anytime they want and see how the treatments are performing, and not just on the big kahuna KPI, but even the little ones. This means George, who owns bottom funnel, can watch how your A/B Test is impacting the exit rate on step 4 of his checkout funnel. This can lead to deeper analysis, further insights and even more testing hypothesis from the team that they now want to run (ahem, job security!).

Now Show Off A Little

If you take all these steps and build the A/B Test dashboard of your dreams, then you must use it to publicize the impact of your tests. This will engender a culture where A/B Testing isn’t this obscure practice you do with your special toy you won’t let anyone touch.

Instead, you have complete transparency and buy-in from everyone affected by your practice. We at Cognetik have done this for one of our clients and found the dashboard to be extremely helpful for this purpose. By having a big-screen TV up in the office, each team-member can watch results play out and put forth new test ideas and get a chance to shine in glory when they stomp on the control!

It’s also a method of weeding out superfluous tests that people want to run because their results will be evident for all to see. They will take more care in proposing A/B Tests when it’s their butt on the line for all to see. Take a look at the screenshot of the dashboard we have below as an example. The best part about these Cognetik dashboards is we just update segments for each test and refresh these dashboards and voila! Each team has their own opportunity to shine in a test-driven culture.


About the author

Dan Saugstad

Dan Saugstad
Dan enjoys nothing more than learning and analyzing something new, which is why he loves digital analytics so much. To him, there are always more ways to add depth and context to any data point and gain valuable insight from that intersection. His passion lies in pulling stories together to help others succeed.

eBook: Understand Your Customers

Cognetik eBook: Guide to User Journey Analysis


Related Articles