From Logs to Metrics: The Impact of Researcher Degrees of Freedom on Smartphone Usage Metrics

Smartphone usage metrics derived from event log data involve numerous researcher degrees of freedom, yet the consequences of these analytical decisions for resulting measures remain poorly understood. We conducted a multiverse analysis examining 16,128 unique analytical specifications derived from 10 methodological dimensions and applied them to four usage metrics: duration, frequency, fragmented smartphone use, and sticky smartphone use. Results revealed substantial variability in absolute metric values, with estimates of duration, frequency, and fragmentation varying by up to nearly threefold depending on the specification.

Session threshold emerged as the dominant factor, explaining 51–78% of the variance in absolute metric values. Despite this variability, participant rankings remained stable across specifications (mean Spearman r = .87–.93). Notably, the dimension most influential for absolute values (session threshold) differed from the dimension most consequential for ranking stability (window length).

Correlations of smartphone usage duration with self-reported usage (r = .28–.45) and age (r = −.36 to −.16) varied across specifications in strength but not in direction, with longer measurement windows and stricter data availability criteria generally yielding stronger associations. These findings demonstrate that while analytical decisions substantially affect absolute smartphone usage estimates—posing challenges for cross-study comparisons—relative comparisons between individuals remain relatively robust, which is reassuring for individual-differences research.