How to avoid errors in web metrics after every new release

One of the main challenges for any large project is the correct metrics collection from a constantly changing website. As patches or updates are released, checking the operation and settings of web metrics takes a long time, and there is always a chance of an error due to human factor. Of course, the more updates occur, the higher the probability of errors in data collection, which leads to the inability to evaluate the success of marketing as a whole.

In this case study, we explain how to save all web metrics after new releases with the example of a large e-commerce project that regularly updates the website search. We describe how to face such a challenge and how to avoid manual checking of every single parameter every now and then.

With over 5 million items in its product range, the e-commerce enterprise regularly updates its website, improving product cards, favorites, and so on. Thus, the company needs to implement new metrics and test user behavior scenarios, to properly measure the efficiency of the recent updates.

Each time a new feature was released, the company’s specialists had to manually test the new metrics and make sure that all the data is properly collected in Google Analytics. Moreover, there was always a risk to ruin the metrics that had been already implemented. For large projects with millions of website visitors, all these issues could seriously influence the decisions taken, leading to money and time losses.

Manual tests of the implemented and new metrics consumed too much time and effort. Each time a new release came out, developers, testers, and analysts needed to check if everything was okay with the info in dataLayer, and if the transactions or add-to-cart events worked fine. This could take from 0.15 to 2 hours, depending on the release.

The company decided to cut the time and money expenses and implement automatic tests for the website metrics to reduce risks when adding new functions to the website.

Analysts from OWOX BI had years of experience in creating automated tests for website metrics for big e-commerce projects, so they suggested a three-staged solution:

• Collect data for testing.
• Set up testing and process data according to the testing results.
• Build reports and use the obtained data.

Let’s give in some more details about each of the stages.

As the store keeps regularly adding new features to its website, there are more and more metrics to test. The company’s analysts created a table in Google Sheets with all of the important dimensions and metrics. This way they can update or edit it any time necessary, and let the other colleagues know what KPIs are tracked on the website.

The analysts started the table by adding the dimensions and metrics that are sent to Google Analytics and are used in reports on sales performance and micro conversions. These very dimensions and metrics are used for evaluating the main business KPIs and for exchanging data with ad services, influencing the decisions made by the CMO and analysts.

As a result, there are three datasets for testing (the list of dimensions and metrics in all the datasets can be changed or updated at any time):

1. Page data.
2. Event data.
3. User scenario data.

The page dataset contains the following:

• Definition — name of the parameter in dataLayer, like clientID, pageType, categoryName, prodName, etc.
• Type — numeric or string parameter.
• False pattern — the format that parameter turns in on the pages where it’s not filled.
• True pattern — the format that parameter turns in on the pages where it’s filled.
• Rules for testing — description of the type of page and user (authenticated or not) the value for which should or shouldn’t be sent. For example, the TotalOrderCount parameter is only sent for authenticated users, and there should be no value sent if a user is not authenticated.

Below is the sample of such a dataset:

The event dataset includes:

• Event name — such as product view, click on the product, adding a product to the cart.
• Parameter — to test action, list, id, position, name, price, etc.
• Description — for the acceptable parameter values, like any value greater than 0.
• Type — numeric or string parameter.
• Comment — required or not required parameter
• Page Type — for testing if there are any parameters on a certain page.

Here’s an example of such a dataset:

The user scenario dataset contains:

• Scenario name — new user and online payment, new user and offline payment, etc.
• Event — a set of events in a scenario.
• Parameters to test — ecommerce.purchase.actionField.action (id, revenue, affiliation), ecommerce.purchase.products.id, ecommerce.purchase.products.name, ecommerce.purchase.products.price, etc.

You can find an example of this dataset below:

Once the company’s analysts provided the list of metrics and dimensions, the OWOX BI team created and set up a script to check if the variables in dataLayer on website pages match the ones provided in the list. The testing script can be launched regularly at any time necessary.

Here’s how the script works:

1. Imitates user action on the website, like opening a page, clicking on the product card, adding a product to the cart, etc.
2. Checks if all the received parameters in DataLayer match the ones that are sent to Google Analytics after the user actions.
3. Imports the testing results to Google BigQuery for creating error reports.

The OWOX BI analysts used Google BigQuery to collect data because this cloud storage meets the highest security standards. Moreover, the BigQuery set of connectors and data export to Google Sheets allows proceeding any necessary data to dashboards or tables.

The testing results are saved in a Google BigQuery table in the following structure:

• Time for testing.
• Page for testing.
• PageType for the page tested.
• Status of testing (Ok or Error).
• Type of data tested (pageview, event, or scenario).
• EventName to define the event tested.
• ErrorDescription to give more details about the error.

The analysts export data from Google BigQuery to Data Studio to easily use the testing results for work. The OWOX BI specialists prepared a dashboard to monitor the following data:

• Tested Pages, Events, and Scenarios — the exact number.
• Share of Pages/Events/Scenarios with Error — the percentage of completed tests with errors across pages, events, and scenarios.
• Details about errors:
  • Internal — script errors.
  • Redirect — sitemap errors.
  • Implementation — dataLayer errors.

Below is a dashboard with the testing results, with filters to apply to the obtained data. For example, you can find out about the errors on the catalog page, by using such filters: Page Type = catalog list, Test type = pageview, Error Type = Implementation.

From the graph of changes in the test launch above, you can see that starting from January 16th there appeared much more tests with errors due to the redesign of the product card (DetailProductView). The reason for that was that analysts hadn’t considered that the banner “Name1” was deleted in the updated website version. Therefore, when testing the click on the product card banner, there was an error with the text “ValueError — no elements to click.”

In another dashboard, we can see that such parameters as position and list weren’t sent in the productView event. It happened because the position parameter disappeared from dataLayer, while the list and name parameters were confused. Therefore, the names for the product lists were sent to the wrong parameter.

Also, at any time it’s possible to check if the important KPIs are tracked properly to fix errors quickly. The OWOX BI team has also set up automatic emails to provide the company’s team with short reports on errors across page types and events, on a daily basis.

Here’s an example of such a letter:

With this workflow, every time the website is updated, developers and analysts can run automated tests for website metrics. The solution from OWOX BI allowed to:

• Significantly increase testing speed and accuracy, excluding the human factor.
• Cut the testing costs. Running tests on a daily basis, saved around 1.5 work hours with every new release.
• Discover the missed out 2,700 data discrepancies on 37 types of pages. For example, recommendation blocks sent the wrong values of product prices or simply sent incorrect values.
• Reports also showed that the catalogs could place unavailable products in the first positions. The redirect functions didn’t work properly, sending users to pages that don’t exist.