Historical Context
From 2010 to 2015, the approach of full compatibility dominated, where companies supported native apps for iOS and Android, covering 95% of the market by OS versions. However, as the complexity of functionality grew and the transition to modern APIs (such as Biometric API, Camera2, Jetpack Compose) took place, the cost of supporting legacy code exceeded the margins of retained users. By the 2020s, the “n-2” policy became standard, requiring the development of methodologies to assess the true effect of disconnection rather than just a correlational analysis of metrics.
Problem Statement
Forced disconnection creates endogeneity of self-selection: users with outdated devices physically cannot upgrade to the required version of iOS or Android, while the active audience with modern smartphones updates promptly. The observed decline in MAU can result from actual churn, or migration to the PWA (Progressive Web App) or mobile web. Classic A/B testing is impossible since disconnection is technical and applies to all users of a specific OS version simultaneously, and tracking identity between the native app and the web version is complicated due to the limitations of Safari and Chrome in working with cookies.
Detailed Solution
The optimal methodology is based on a combination of Regression Discontinuity Design (RDD) and Synthetic Control Method. Firstly, RDD is used with a threshold by OS version (e.g., cutoff at Android 8.0 versus Android 9.0), where users with versions just below the threshold serve as a control group for users just above the threshold, with adjustment for smooth characteristics (device model, historical usage frequency).
Secondly, to assess migration to the web channel, a synthetic control is built based on historical DAU data for cohorts of users comparable by behavioral patterns before disconnection. A weighted combination of unaffected cohorts (e.g., users from other regions with a similar device structure) is created that models the counterfactual trajectory of metrics.
Thirdly, difference-in-differences with Propensity Score Matching is applied to match users who could have upgraded but did not with those who upgraded, adjusting for the technical specifications of devices. It is important to track cross-device migration through a Customer Data Platform (CDP), linking the device_id of the mobile app with the cookie_id of the web version through a single user_id upon authentication. Additionally, survival analysis (Cox models) is used to assess time to churn depending on the OS version and availability of web alternatives.
Context: A large marketplace decided to stop supporting Android 7.0 and below (about 8% of the user base) in order to implement the Biometric API for secure payments. The project budget assumed a loss of no more than 3% of the active audience, compensated by conversion growth in newer versions.
Option 1: Simple comparison of MAU before and after disconnection by dates with calculation of the loss percentage. Pros: simplicity of calculation, speed of result acquisition, does not require complex infrastructure. Cons: completely ignores seasonality, migration to the web, and self-selection based on devices; high risk of false positives regarding churn when users have simply transitioned to m.site.
Option 2: Constructing cohort analysis with matching by devices with Android 8.0 (which could have stayed but updated) against Android 7.0 (which could not update). Pros: takes technical limitations into account, allows isolation of the impact of inability to update from unwillingness. Cons: difficulty in obtaining clean data due to fragmentation of OEM manufacturers (Samsung, Xiaomi), differences in behavior of users from different brands, and geographical heterogeneity.
Option 3: A comprehensive approach with Synthetic Control at the level of geographical regions with a high share of old devices (comparison of region A, where 30% are on Android 7, with region B, where only 5% are, before and after disconnection) with adjustment to overall market trends. Pros: considers macroeconomic factors and seasonality, allows to assess the overall effect on the business. Cons: requires larger samples and assumes no other simultaneous interventions in the regions.
Chosen Solution: Option 3 was implemented in combination with cohort analysis of authenticated users (to track migration to the web via SSO logins). The choice was determined by the need to separate actual churn from cannibalization of web traffic, which is critical for assessing unit-economics (web users have a 15% lower AOV).
Outcome: The analysis showed that only 40% of the "lost" MAU actually churned, 35% migrated to PWA, and 25% upgraded their devices within the quarter. The true negative effect was found to be 2.5 times less than predicted, allowing the continuation of the API update strategy for the remaining 92% of the audience with an 8% growth in GMV due to new payment features.
How to distinguish between technical inability to upgrade and behavioral refusal to upgrade if both groups remain on old versions of the application?
The answer must be built on the analysis of device_change events in the CDP. Users with behavioral reluctance (lazy updaters) often have a pattern of "delayed updates" in their history (e.g., skipping several minor versions but installing critical security patches), while technically limited users never change the OS version throughout the device lifecycle. Additionally, hardware fingerprint is analyzed through WebGL or Canvas in the web version: if a user appears in the PWA with the same device (according to User-Agent and screen resolution) as in the native application before disconnection, this confirms migration rather than churn. It is also important to segment by app_version history: if a user regularly updated within Android 7 (between patches 7.0->7.1) but did not switch to 8.0, this indicates a technical limit rather than unwillingness.
Why can standard Propensity Score Matching give biased estimates when evaluating the effect of forced upgrades in the presence of a strong correlation between user income and device model?
Standard PSM is based on conditional independence, assuming that observed covariates fully explain self-selection. However, in the case of sunset policy, there exists a hidden variable — disposable income — that correlates simultaneously with the smartphone model (flagship vs. budget) and with LTV of the user. Budget devices are less frequently updated, and their owners have lower purchasing power. Standard PSM will underestimate the negative effect as it “weights” wealthy users with new devices as counterparts to the poor with old ones, even though their behavioral patterns differ fundamentally. The solution is to use Coarsened Exact Matching (CEM) with precise stratification by price segments of devices (low-end, mid-range, flagship) before applying PSM, or instrumental variables (IV) using OEM upgrade policies as exogenous shocks.
How to properly account for network effects between users of different versions of the application when assessing churn, if the functionality of the "share product" feature works differently in the old and new versions?
Network effects create spillover between treatment and control groups: if an active user of the new version (treatment) cannot share content with a friend on the old version (which does not support the new format of deep link), it worsens the experience for both and may accelerate the churn of the control group not due to the sunset policy, but due to UX degradation. To correct this, network-based DID (Difference-in-Differences with network weights) must be applied. A graph of social ties is built (through the analysis of referral codes, joint orders, or messages in the app chat). The “contagion” of metrics is assessed: if a user in the control group (old version) has many connections with the treatment group (new version), their behavior will be distorted. An interaction term Treatment x Network_Exposure is added to the model, where Network_Exposure is the share of connections in the network using the new version. At high levels of network effects, the true effect of the sunset policy will be underestimated, as part of the “control” users were actually subjected to indirect exposure, requiring adjustment for intention-to-treat (ITT) considering this contamination.