Historical Context. The A/B testing methodology, which has become the gold standard of digital analytics since the 2010s, loses effectiveness during global changes in the business model that affect the entire user base simultaneously. In such conditions, product analysts turn to quasi-experimental econometric methods: Difference-in-Differences, Synthetic Control Method, and Propensity Score Matching, designed for policy evaluation in social sciences. These approaches allow for the isolation of causal effects in the presence of endogenous self-selection and lack of randomization.
Problem Statement. The implementation of the subscription option faces a fundamental self-selection problem: the most loyal users with a high product usage frequency are the ones who switch to the subscription. A simple comparison of the LTV of subscribers and one-time buyers gives a biased estimate, as it ignores the basic differences in behavioral patterns. Additional distortions are introduced by macroeconomic shocks and seasonality, which correlate with the timing of the feature launch and affect the audience's purchasing power regardless of the business model.
Detailed Solution. The optimal strategy combines Propensity Score Matching to balance observed characteristics between cohorts before and after the launch with Difference-in-Differences to control for temporal trends. For building the subscription propensity score, Gradient Boosting is used instead of logistic regression, allowing for non-linear interactions between behavioral features. Macroeconomic variability is absorbed by fixed effects for time periods or Google Trends indices as control variables, while seasonality is eliminated through STL decomposition of time series prior to applying the main model.
An online education platform launched an "Unlimited Subscription" plan alongside the purchase of individual courses through a catalog. The business feared that users would switch to the cheap subscription instead of expensive one-time purchases, leading to a revenue decline. The release coincided with the onset of economic instability, further complicating the clean comparison with historical data and necessitating the isolation of external shocks.
Option 1: Direct comparison of subscribers and non-subscribers. We collect data on current subscribers and compare their LTV with historical one-time buyers of similar age. Pros: extremely quick implementation in one day, intuitively clear to the business. Cons: completely ignores the fact of self-selection of motivated users into the subscriber group and the external economic crisis, which reduces base demand, giving an inflated estimate of the subscription effect.
Option 2: Cohort analysis before/after without control. We compare the LTV of cohorts of users who joined in the three months before the launch with cohorts after the launch, considering the difference as the effect of the subscription. Pros: simplicity of calculation and no need for propensity modeling. Cons: impossible to separate the influence of the subscription from the degradation in purchasing power due to the crisis and seasonal spikes during holiday sales, resulting in a biased estimate with unknown sign.
Option 3: Combined PSM + DiD with Synthetic Control approach. We build a subscription propensity model on pre-launch cohorts, find twins for actual subscribers, then apply DiD with synthetic control weighing historical cohorts to simulate the counterfactual. Pros: isolates the subscription effect from macroeconomic shocks through temporal contrasts and eliminates self-selection bias through covariate balancing. Cons: requires strong assumptions about parallel trends and is computationally complex for interpretation by non-technical stakeholders.
Option 3 was chosen using Causal Forest to assess the heterogeneity of the effect by segments since it was the only one that allowed separation of the true incremental effect from the noise of the crisis and self-selection. This approach provided the necessary accuracy for making a strategic decision about subscription targeting despite implementation complexity.
The final result demonstrated that the subscription increases LTV by 40% for users with a purchase history of more than three courses but decreases it by 15% for casual buyers. The recommendation to introduce an activity threshold for access to the subscription was implemented via an A/B test of gating, resulting in a +12% increase in portfolio revenue without decline in the first quarter.
How to validate the assumption of parallel trends in DiD when the treatment timing varies among users (staggered adoption)?
It is necessary to conduct placebo tests, artificially shifting the "treatment" to historical periods and checking for significant effects in the pre-treatment era. It is critically important to construct event-study plots to visualize the dynamics of coefficients before and after the event. Candidates often ignore the violation of SUTVA (Stable Unit Treatment Value Assumption): one user's subscription may influence the behavior of others through a learning effect or cannibalization of one-time purchases, requiring clustering of standard errors at the level of geography or cohort.
Why does standard logistic regression for Propensity Score fail in high-dimensional product data and what should it be replaced with?
Classical logistic regression suffers from the curse of dimensionality with hundreds of behavioral features and is unable to capture non-linear interactions between features that are critical for predicting self-selection. It is advisable to use Generalized Random Forest for propensity estimation or Coarsened Exact Matching (CEM), which ensures balance on key metrics without assumptions about functional form. Junior analysts overlook the necessity of checking covariate balance through Standardized Mean Differences (SMD), requiring values of less than 0.1 for all key covariates after matching.
How to correctly handle right-censoring in LTV analysis when subscriber cohorts are „fresh“ and have not undergone a full life cycle?
One should not compare realized revenue, as new subscribers simply have not had time to make all possible payments. It is necessary to apply Kaplan-Meier survival curves or Cox proportional hazards models to estimate churn intensity, followed by discounting future cash flows. A key mistake is ignoring differences in churn patterns between subscribers and one-time buyers, leading to an overestimation of the LTV of subscriptions in the early months due to the "honeymoon period" effect.