Predicting Churn of Veteran Users in Free to Play Games

1. Introduction

As in all businesses, free-to-play-games being no exception, retention is critical.

The goal is two-fold: Increase return on investment (ROI) and reduce cost per install (CPI). 
The method: Reduce player churn to a minimum.

Paper Structure

2. Churn Defined

While building and testing the models in R, it was clear from the outset that the churn point will prove difficult to pinpoint. One strives to catch churn tendency in time for action, while on the other hand not ‘jumping the gun’ on a false positive. We identified that users reduce their activity before they actually churn (the last session). This time period in users’ lifecycle is what we aim to predict.

Definition of Churn – 70% reduction in play-day activity in the last 14 days, compared to regular day-to-day activity.

3. Predicting Churn

a. The Panel

Using Sql Server and R-studio, a panel was created based on historical data. The data included approximately 100,000 active users with longevity of 30 days and above, of which 18% churned within 4 weeks. Many calculated explanatory variables were included for each of the users per different time frames (last 0-7 days, 7-14 days, 14-30 days etc).
The idea of calculating for different time frames was to capture the corresponding change in user activity.
Activity monitor features:

b. Methods

Different prediction techniques were tested, including: logistic regression, decision trees and neural networks – providing very similar results. Since this was the case, we opted for Logistic Regression, for simplicity and ease of use.

4. Taking Action upon Churn Model Scores

a. Model Results

i. Overall: The prediction exercise yielded good results. All the methods applied produced good AUC: ranging from 0.78 to 0.91, they were all good enough to take action upon.

ii. Methodical Conclusions: We proceeded with Logistic Regression for performance and ease of use.

iii. Coverage & Error: Coverage was above 80% of churners and false-positive under 20%.

The features (variables) that turned out significant (p-value < 0.05) used in prediction: last 0-7 days , 7-14 days, 14-30 days, 7-14 days, weekend/weekday/both.

iv. Measuring Methods: To compare the impact of the actions applied to group A vs. the control group B, data regarding the intervention and response was collected for both groups, including: churn rates, click rates and time in-app.

b. The Prediction Exercise

The app in question is a freemium model app, thus the intervention decided upon at first was giving away free coins in the form of a bonus, sent via notification or email. The aim was to entice players to continue playing by granting them a one-time bonus.

Approximately 17,000 active players, who were identified as bound to churn according to the Churn model score, were randomly assigned to two groups A (70%) and a control group, B (30%).
The actions applied to the two groups were as follows:

- Group A -
Players identified as about to churn received notifications via Facebook and mobile notifications, informing the user of the special bonus. This is an improved “win-back" approach, where as opposed to the classic “win-back”, the player has not yet churned but is about to churn. The bonus is designed to “win-back” the previous high activity. At a later stage we employed in-app gifting.

- Group B -
This was the control group, with no action taken.

c. Results

Per Intervention Type:

i) Bonus via Facebook or Mobile Notification:
No increase was found in user time in app between group A and group B. Impact was minimal or non-existent
(P-value = 0.31)

ii) In-app Bonus Gifting:
Increase was found in time in app for group A, compared to group B. The difference was significant
(P-value = 0.05)

5. Points for Discussion and Takeaways

a. Churn Predictability
One of the insights gained by this work was that churn is a user behavior that is predictable rather accurately.  Interestingly enough, the predictive features that ultimately turned out to be significant were not game descriptive variables but mostly time app usage variables. This is an important insight due to the vast amount of available data and variables (features) in games, specifically in free to play games. Searching for the most useful predictive features can prove very time consuming.

b. Intervention Type
The intervention was initially performed via emails and notification. After a short period, it became apparent that the response is very low and even nonexistent. A more aggressive channel was then tested, using a bonus pop-up within the app. Almost 100% of users who were offered the bonus, collected it (who refuses free money?). We therefore proceeded with this method for the veteran users.

c. ROI on Churn Reducing Activity
Understanding the impact of a 1% increase in long-term retention is a good impetus for building churn models of this kind. From our findings in our user base, increasing long-term retention has a bigger impact on LTV compared to improving short-term retention, but this is not necessarily true of all user bases. It depends on where there is more potential for improvement. We address short-term retention differently and in a more specific fashion.

d. Further Experimentation with Additional Intervention Types
This analysis tested one intervention method based on churn model score. In order to improve churn rates, many more interventions/actions will have to be tested in order to improve veteran user retention. A list of possible actions and interventions one should consider testing include: opening special game features, good experience sessions, bonuses at different points in the game, faster level ups and the like.

e. Final Notes
In the worlds of CRM and loyalty programs, the emphasis in on building an ongoing relationship with the user. Incorporating predictive analysis and A/B testing have a direct effect on user behavior and churn, but cannot replace a comprehensive long-term communication strategy.

This analysis and model focused on retaining paying and non-paying veteran players. Bringing about a small improvement in retaining non-paying as well as paying users (taking into account ~2% of installs in the free-to-play game) may lead to more paying users and therefore to higher ROI and lower CPI.

(Many thanks to Micky Daniels for his linguistic and editorial advice)