Title Comparing errors from non-coverage to other errors in a mobile Web survey
Workshop Workshop 2014
Year 2014

Purpose of the study: The quality of mobile-only Web surveys is up for discussion: some argue that surveys requiring a smartphone to participate should be avoided; others tout their usefulness for health and travel research. We help inform this discussion by estimating the magnitude of three sources of error in a mobile Web survey. Design/methodology/approach: 1385 members of the Longitudinal Internet Studies for the Social Sciences (LISS) panel were asked to answer the same questions, once using a smartphone and once using a personal computer (PC). We use the PC Web survey as a benchmark, and deviations from the benchmark are regarded as error. To estimate coverage errors in the mobile Web survey, we compared those with their own smartphones (71%) to the full sample. To estimate nonresponse errors, we compared those who responded on their smartphone (73%) to the covered sample. Finally, to estimate measurement differences, we compared how the same people responded when using their smartphone and when using their PC. Findings: We find large non-coverage error errors relative to nonresponse and measurement errors. Furthermore, the non-coverage errors were not consistently offset by the other sources of error. This suggests that limiting Web surveys to mobile Web users only is risky for general population surveys. We repeated the analyses, but this time counted those who were uncovered as covered because in reality they were sent a borrowed phone to participate. This strategy of providing phones appeared to be effective: coverage errors were eliminated (by definition), and there were no unintended effects on nonresponse or measurement errors. Originality/value: This study provides an update on earlier estimates by Fuchs and Busse (2009) of coverage errors for mobile-only surveys. We utilized several measures that one might want to measure in a smartphone survey related to health, technology, and travel. Since these measures were collected in a parallel PC Web survey, we know the characteristics of those who did or did not participate in the mobile Web survey. Research limitations/implications: This research is a first step towards understanding the effect of mobile-only Web surveys on data quality using total survey error framework. Future research should consider other question domains and survey contexts. Practical implications: These findings help practitioners answer the important question of whether mobile-only Web surveys are becoming a viable alternative to traditional PC-based methods for their research topics.