Current Issues Facing Clickstream Analytics QA

For many years, clickstream analytics quality assurance has meant that engineering and QA resources have to manually spot-check clickstream beacon calls for errors or unexpected data values. 
 
Some clickstream tools also have no data transformation layer, meaning the data that gets sent to the data collection server from the user’s browser is pretty much the same data that shows up in the clickstream reporting tools.
 
These tools also have ways of manipulating data once it’s on the collection servers (i.e., processing rules and vista processing rules), but most of those interfaces and capabilities haven’t been updated in more than a decade and are not robust enough to identify, intercept, and correct data errors across an entire user experience. 
 
That is why manual quality assurance efforts with clickstream analytics solutions are being done in most organizations.  
 
Unfortunately, this manual process is often extremely time-consuming, hard to scale (without increasing your headcount), and is typically supported by resources that know less about how the technology works and the user experience than the business people in charge of managing them. 
 
Many believe that QA resources should be experts when it comes to understanding the user experience and how the application functions, but in most organizations, that is simply not the case.
 

Embracing a Data-Driven QA Approach

This is why using a data-driven QA approach is critical. By using the same data the business stakeholders and analysts rely on each day to drive their business decisions, organizations can ensure the data is credible and reliable in the first place. 
 
This approach guarantees that the engineers, QA resources, and analysts all work from the same version of the truth.  
 
Historically, when analysts conduct an analysis, they uncover inconsistencies in their data that they either have to exclude or explain to business stakeholders. This takes their attention away by interrupting the data and having them make recommendations, which ultimately slows down the speed to answers and the organization’s success in the market. 
 

Fixing Dirty & Inconsistent Data

When leading with a data-driven QA solution, defects and inconsistencies in the data are caught and often resolved before analysts and stakeholders have time to discover them on their own.  
 
So what kinds of defects can data-driven QA solutions find?

  • Duplication of clickstream beacon calls.
  • Missing clickstream beacon calls.
  • Missing e-commerce and experience success counters, such as cart views, cart adds, and cart removals.
  • Missing global variables, such as browser and device type.
  • Missing contextual information, such as product names, page names, and other user-friendly descriptors.

 
There are some tools in the industry that can handle these issues, but many tools are not able to keep up with the exponential amount of user interfaces. The different combinations of operating systems, screen sizes, page layouts, and geographical variations explode every time a new user device enters the market. 
 

Key Advantages

A key advantage of implementing a data-driven QA approach in this ever-evolving marketplace is that all the data ends up in a single reporting tool, like Adobe Analytics or Google Analytics. By having the data quality process in a reporting tool centered on data from all devices, you no longer need different QA processes for different interfaces. You must standardize your approach across interfaces when measuring it. 
 
Keep in mind that you still have to dedicate resources to designing and implementing the tagging solution, but once the data is collected, it makes sense to use the data to validate itself. You can then take the software development mindset and think of data quality as an iterative approach instead of a one and done approach.
 
For example, with the introduction of Adobe Analytics Analysis Workspace, users of the tool can now build any report or visual they want. First, you’d have to check the data to find out what percentage of the time data is captured and if a specific global variable was also captured. 
 

How to Begin Implementing a Data-Driven QA Approach

To do this, you’ll need to create calculated metrics that compare the number of data records with the global variable present to the total number of data records captured by Adobe. We like to call this the Data Completeness Quality Score. This measurement can be built for every variable captured by your implementation.  
 
Now, let’s assume that 10% of the time when this global variable is captured, the value is actually “not applicable.” This speaks to data accuracy and the usability of the data that gets captured. 
 
In this case, you’ll need to take the same Completeness metric, apply a segment that excludes “not applicable” from the denominator, and now you have what we call the Data Accuracy Quality Score. This too can be built for every variable captured by your implementation.
 
These quality score metrics can now be applied across any number of segments and interfaces.  For example, say your organization had a new version of the mobile app released last week. You can compare your quality scores across the different app versions to quickly confirm your tracking defects were fixed with the new app version, and that new defects didn’t pop up unexpectedly.
 

Summary

At the end of the day, the goal of any data capturing and maintenance organization has to be to ensure the data is free from errors and usable by the business stakeholders. The industry standard of capturing data and letting the analyst say when something is wrong weeks down the road has to evolve like all other aspects of all organizations. 
 
The data-driven QA approach is the next step of that evolution.
 
If you want to learn more about how to lead with data or to implement this data-driven QA approach, Cognetik can help. Contact us today so we can help streamline your workflow, blast through data silos, and get your data to tell the whole story.

About the author

Jeremy Linhardt

Jeremy is a strategic thinker and innovative business leader with more than 16 years of experience in managing and analyzing customer online and offline behavioral and business operational trends in existing and emerging markets.

New eBook Release: A Guide to User Journey Analysis

Related Articles