Historical context of digital product evolution shows a gradual shift from isolated native applications to hybrid interaction models. Progressive Web App is a technology that allows a website to function as a full-fledged application with offline access and push notifications, historically blurring the line between mobile web and native experience. Classic methods of evaluating the effectiveness of such implementations, including simple cohort analyses or A/B testing, face fundamental limitations such as the inability to isolate a control group from SEO effects and the technical infeasibility of randomization at the user level without violating UX.
Problem statement requires solving a multidimensional task of identifying causal relationships in the conditions of endogeneity of platform self-selection. Users decide independently whether to use PWA or a native app, creating self-selection bias that correlates with technical literacy and engagement. At the same time, launching PWA generates organic traffic through improved Core Web Vitals and service worker indexing, distorting the baseline traffic level in the analyzed cohorts. Cannibalization between platforms appears as a session outflow from the native app to PWA, requiring separation of the migration effect from the true increase in engagement.
Detailed solution is based on the synthesis of Synthetic Control Method (SCM) and difference-in-differences analysis with propensity score matching. In the first stage, a synthetic control is constructed from geographical regions or user segments with a delayed launch of PWA, allowing modeling of the counterfactual trajectory of metrics without intervention. Next, Causal Impact analysis is applied to isolate temporal effects with adjustments for covariates, including seasonality and marketing activities. To assess cannibalization, an instrumental variables approach is used, with the technical availability of PWA (the version of the browser that supports service workers) serving as the instrument, providing quasi-experimental variation independent of user preferences. Cross-platform retention is modeled using survival analysis with competing risks, distinguishing the outflow risks within the platform from cross-platform migration.
In the largest electronics marketplace, there was a need to launch PWA to reduce entry barriers for new users, but there was a critical business hypothesis about potential cannibalization of high-value users of the native app. The team faced a choice of evaluation methodology that would allow them to separate the true increment from the redistribution of the existing audience between channels without conducting traditional A/B testing, which was impossible due to the technical specifics of automatic PWA installation via browser banners.
The first considered option involved using a simple comparison of key metrics (conversion rate, retention day 7) between users who visited the site before and after the PWA launch. The pros of this approach included the speed of obtaining results and minimal data infrastructure requirements. The cons included critical vulnerability to seasonal fluctuations in electronics demand and the inability to separate the PWA effect from a simultaneously launched advertising campaign on television, rendering the results statistically insignificant and business risky.
The second option included geographical A/B testing with a gradual rollout through Google Optimize and geographical segmentation by IP addresses, where the test regions were given access to PWA while the control regions were not. The pros consisted of the reproducibility of experimental logic and clarity for stakeholders. The cons appeared in the inability to isolate the SEO effect since the improvement of Core Web Vitals extended to the indexing of the entire domain, regardless of user geography, creating a spillover effect and contaminating the control group with organic traffic from positive selection.
The third option, ultimately implemented, combined Synthetic Control Method with Regression Discontinuity Design based on the threshold of the mobile browser version (Chrome 90+). The team created a synthetic control by weighting the behavior of Safari users and outdated versions of Chrome before the PWA launch for the test group, which allowed modeling counterfactual retention without intervention. The pros included internal validity of the quasi-experiment and the ability to isolate cannibalization through analysis of the overlap of device IDs between platforms. The cons required significant computational resources to construct synthetic weights and complexity in interpretation for the product team.
The result was a net incremental effect of +8.3% in retention day 30 for the mobile web after adjusting for cannibalization, which amounted to -2.1% from native app activity. The neutral effect on the overall LTV of the user allowed for a strategic decision on a full rollout of PWA with optimal deeplinks to minimize cross-platform friction.
How to differentiate true cannibalization from complementarity effect between PWA and native application when a user can use both platforms within a single day?
The answer requires understanding the concept of incrementality at the user-journey level instead of a device-centric analysis. It is necessary to build a multi-touch attribution model with time windows, where sessions are coded as sequences of states (Web → PWA → App), and the effect is evaluated through the analysis of changes in total time spent in the product and frequency of key events. A key nuance is the use of surge analysis to identify complementarity: if the introduction of PWA increases the frequency of native sessions within 24 hours (cross-platform triggering), this indicates a synergistic effect rather than cannibalization. Junior analysts often aggregate metrics by the last attribution channel, losing critical temporal granularity.
How to adjust the effect assessment in the presence of network effects in a two-sided marketplace, where the launch of PWA for one segment of users affects the experience of other segments?
Here, it is necessary to apply panel data methods with fixed effects to isolate the direct effect from spillovers. SUTVA (Stable Unit Treatment Value Assumption) is violated, so modeling of peer effects is required through spatial autoregressive models or two-stage least squares (2SLS), where the technical availability of PWA in the region serves as the instrument, and the dependent variable is the activity of non-PWA users in the same region. A nuance lies in building exposure mapping, determining the intensity of interaction between market segments through a transaction graph, allowing for the quantitative assessment of indirect network effects and adjusting the direct estimate for the magnitude of externalities.
How to account for self-selection bias in the PWA adoption rate when early adopters systematically differ in engagement from the average user, distorting the average treatment effect (ATE) assessment?
It is critically important to apply Heckman correction or Inverse Probability Weighting (IPW) to adjust for observed and unobserved characteristics. The self-selection process should be modeled through a probit model, where the dependent variable is the fact of PWA installation, and the predictors include technical device characteristics, interaction history with the product, and demographic variables. The Inverse Mills ratio from the first equation is included in the second outcome equation to adjust for bias. Alternatively, coarsened exact matching (CEM) can be used to balance covariates between adopter and non-adopter groups. Junior specialists often overlook this bias, interpreting high adopter metrics as a causal effect of the technology when, in fact, they reflect heterogeneity in the audience's technological readiness.