Evolving Churn Predictions: Navigating Interventions and Retraining

Retraining churn models presents unique challenges that need special attention

Retraining machine learning models, especially those focused on customer churn prediction, is an essential step to ensure their relevancy and accuracy over time. However, retraining churn models presents unique challenges that need special attention. Among the most notable is distinguishing between causal effects of interventions — identify customers who stayed due to the proactive retention program to target them exclusively.

Intervention Impact on Churn Retraining

Consider the following sequence of events:

1. Initial Model Training: A model is trained using historical customer data.
2. Model Inference: Certain customers are flagged as likely to churn.
3. Intervention by Business: Interacting with these customers to persuade them to stay or using measures like promotions and personalized offers to encourage retention.
4. Retraining with New Data: When the model’s performance degradates, it’s likely time for retraining — the model is updated with more recent data, which includes the outcomes of these interventions.

Imagine a scenario: A customer is predicted to churn, they are treated by a retention representative, and then they stay. The challenge arises when trying to tell the reason behind their decision — Did the intervention change their mind, or were they misclassified by the model in the first place?
When retraining the model on such ambiguous data, there’s a risk of distorting the model’s future predictions — labeling the above customer as “stayed” might be misleading, as they might have left had we not persuaded them to stay.
As well as for churn labels — an intervention could be the only trigger for churn, making some churn labels unreliable.

Possible ways to deal with the bias:

1. Control Groups and Synthetic Data: Create a subset of “likely-to-churn” customers to whom no interventions are applied. By comparing outcomes between this control group and the intervened group, one can tell the true impact of the interventions — if a certain segment of customers churns more when it is treated, its exclusion from the intervention should be examined.
   When it’s time to retrain, one can utilize data from the control group and exclude data from the intervened group, ensuring the model relies on reliable churn labels.
   The disadvantage of this method is the loss of critical data, so to compensate the exclution of intervened customers, try generating synthetic samples of the control group to represent those customers. This can be done by SMOTE among other oversampling strategies.
2. Feedback Surveys: Directly engage with customers post-intervention to understand their reasons for staying/churning. Insights gathered can provide clarity on the effectiveness of interventions and help differentiate between genuine stayers and those swayed by the efforts.
3. Merge Models: Try combining the initial training model with the new one. Averaging predictions or using ensemble methods can reduce the risk of any single model’s biases dominating the overall prediction.
   Notice that as time passes, the initial training data may be less relevant.

Uplift Modeling: The Gold Standard for Refining Churn Predictions

Unlike traditional churn models that predict who might leave, Uplift modeling identify customers whose behavior changes directly due to an intervention.
By comparing the treated group with the control group, these models predict which customers stay because of the intervention and on the other hand which ones leave because of an intervention.
This targeted approach helps businesses optimize resources and maximize customer value.

Customers can be divided into four theoretical categories given they were treated / not treated for retention efforts:
Sure Things: Customers who won’t churn. Targeting them doesn’t offer extra returns but adds costs, such as communication efforts and potential financial incentives.
Lost Causes: Customers who will churn regardless of interventions. They don’t add revenue and may result in reduced costs compared to Sure Things, as they don’t exploit offered incentives.
Persuadables: Customers who remain only after the retention effort. They contribute additional revenue.
Do-Not-Disturbs: Customers who churn only if targeted. Leaving them undisturbed is beneficial, while targeting them adds significant costs without revenue gains, making them “sleeping dogs”.

The goal of uplift modeling is to exclusively target the persuadables.
The challenge is that we can’t determine which category individuals fall into. We can’t simultaneously treat them and have them in the control group. So, how could we identify them? how could we know if they were persuaded or did not intend to churn in the first place? This is where uplift modeling steps in.

There are several uplift approaches for this challenge, we’ll take a look at the ‘Transformed Outcome’ method. this method requires data from both a control and treatment groups, and it shifts our focus from a classification task to a regression one.
Labels are allocated based on a specific formula, and for a random treatment assignment where treatment propensity equals 0.5, the target variable translates to these values:

We can use a loss function such as mean squared error (MSE) as a metric to solve this regression problem:
For Persuadables, control is labeled 0 and treated is 2. The lowest MSE between them will be where the score is 1, representing the uplift of Persuadables.
For Do-Not-Disturbs, control is -2 and treated is 0, with the optimal prediction being -1, signifying the uplift.
For both Lost Causes and Sure Cases, the best prediction is 0.
ideally, one should target the highest scores to try to retain Persuadables and to avoid Do-Not-Disturbs and others as much as possible.

Navigating the Future of Churn Predictions

As we’ve explored, the landscape of churn prediction, with its complexities of interventions and evolving data, poses challenges for businesses.
Retraining models is not merely a technical exercise but a part of understanding customer behavior and make sense of genuine retention. Leveraging tools like control groups, feedback mechanisms, and uplift modeling.
But perhaps most crucial is the recognition that data isn’t static. Our understanding of customer behavior must adapt. Embracing this dynamic situation, continually refining models, and staying attuned to the shifting patterns will be the key of successful churn prediction and management in the future.

Further Reading:

• Uplift modeling using the Transformed Outcome Approach
• Causal Inference and Uplift Modeling: A Review of the Literature
• Machine Learning Methods for Estimating Heterogeneous Causal Effects
• Pylift python package for Uplift modeling

Evolving Churn Predictions: Navigating Interventions and Retraining was originally published in Towards Data Science on Medium, where people are continuing the conversation by highlighting and responding to this story.