How to Set Up Longitudinal Experiments: A Data Scientist’s Guide

A Data Scientist’s Guide to Longitudinal Experiments for Personalization Programs

Unlocking rapid “test-and-learn” and capturing full-scaled personalization value from longitudinal experimentation

A/B testing vs. longitudinal experiment

Experimentation does not always need to be complex; simple A/B test framework could be just excellent in situations with manageable marketing levers. The design and implementation of experimentation should always go hand in hand with marketing learning agenda, marketing technology (MarTech) maturity, and creative design capability.

Let’s take grocery shopping as an example. To understand impacts of one-time promotions and offerings on online grocery shoppers, a simple A/B test framework of control and test variants will do. It matters less if these shoppers are assigned to consistent control and test groups throughout their customer life journey, nor if a few of them dropped out midway.

Longitudinal experiments, also known as panel studies, provide a framework of studying causal relationships over time. Unlike one-time experiments, they allow for the examination of evolving patterns and trends within a population or sample group. Traditionally prominent in fields like medical sciences and economics, longitudinal experiments have found increasing use cases in sectors including tech, retail, banking, and insurance.

Longitudinal experiments offer distinct advantages in complex personalization scenarios. They enable a deeper understanding of the cumulative impact of personalized marketing strategies and help determine when to scale these efforts.

A case study — longitudinal experiment at a bike part supplier

Consider a hypothetical scenario with AvidBikers, a leading supplier of premium bike parts for mountain cyclists to customize and upgrade their bikes. They recently launched a personalization program to send weekly best offerings and promotions to their loyal cyclist customer base.

Contrary to one-time grocery trips, typical shopper journeys at AvidBikers consists of a series of online shopping trips to get all parts needed to DIY bikes and upgrade biking equipments.

As personalization program is rolling out, AvidBikers’ marketing data science team would like to understand both the individual campaign effectiveness and the overall program-level incrementality from combined personalized marketing strategies.

Program vs. campaign experiment

AvidBikers implements a dual-layered longitudinal experimentation framework to track the overall personalization program-wide impacts as well as impacts from individual campaigns. Here program-wide effects refer to the impacts from running the personalization program, sometime consists of up to thousands of individual campaigns, whereas campaign-level impacts refer to that of sending personalized weekly best offerings vs. promotions to most relevant customers.

To implement the framework, test and control groups are created on both global level and campaign level. Global test group is the customer base who receives personalized offerings and promos when eligible, whereas global control is carved out as “hold-out” group. Within the global test group, we further carve out campaign-level test and control groups to measure impacts of different personalization strategies.

Addressing dynamic customer in-and-out

Challenges arise, however, from new and departing customers as they could disrupt test-control group balance. For one, customer attrition likely has an uneven impact on test and control groups, creating uncontrolled differences that could not be attributed to the personalization treatment / interventions.

To address such bias, new customers are assigned into program-level and campaign level test and control groups, followed by a statistical test to validate groups are balanced. In addition, a longitudinal quality check will be run to ensure audience assignment is consistent week over week.

Measure, iterate, and repeat

Measurement is often (mistakenly) used interchangeably with experimentation. The difference, in simple terms, is that experimentations are frameworks to test hypotheses and identify causal relationship whereas measurement is the collection and quantification of observed data points.

Measurement is key to capturing learnings and financial impacts of company endeavors. Similarly to experimentation, AvidBikers prepared program and campaign-level measurement files to run statistical tests to understand program and campaign-level performance and impacts. Program-level measurement results indicate the overall success of AvidBikers personalization program. On the other hand, campaign-level measurement tells us which specific personalization tactic (personalized offering or promo) is the winning strategy for which subset of the customer base.

With measurement results, AvidBiker data scientists could work closely with their marketing and pricing teams to find the best personalization tactics through numerous fast “test-and-learn” cycles.

Implementing longitudinal experiment at scale

Implementing longitudinal experiments at scale demands a balance of technological infrastructure and methodological rigor. Tools like Airflow and Databricks streamline workflow management and data processing, facilitating the orchestration of complex experiments. Nevertheless, the cornerstone of success remains the meticulous design and execution of the experimentation framework tailored to the specific business context.

In my personal experience, complexities such as cold-start, customer dropouts, and overlapping strategies could arise, which require case-by-case evaluation and customization in experiment design and implementation. However, as customer needs continue to evolve, strategic implementation of longitudinal experiments stands as a pivotal foundation in the evolution of customer-centric personalization.

Thanks for reading and stay tuned for more data science and AI topics in the future 🙂

How to Set Up Longitudinal Experiments: A Data Scientist’s Guide was originally published in Towards Data Science on Medium, where people are continuing the conversation by highlighting and responding to this story.