The behaviour of the mobile generation is changing and market research methodologies need to keep up. Survey chunking on mobile is our answer to the need for more in-depth studies on mobile.
Survey chunking - the process of dividing a survey into shorter sections which respondent can answer in one go or in several phases.
We believe that the age of 30-minute surveys is over. However, the need for more in-depth approach for certain projects remains. Our experience with over 10 million mobile surveys shows that drop-off rates start to increase after the 15th question. We were sure there was a way around that limitation.
<odr:mediaImg filename="20130922215942_survey-chunking-flow.png" alt="Survey chunking flow" width="600"/> Same person answers both chunks
This lead us to research and validate a hypothesis that survey chunking on mobile will not just give researchers enough data for their research but in fact improves the data quality of the answers.
Survey chunking - does it work?
We conducted a pilot research project in the UK to test the hypothesis that survey chunking is the answer for doing longer research on mobile. The experiment proved what we instinctively knew after having run over 10 million surveys in 57 countries - taking surveys in shorter "bursts" improves data quality.
<odr:mediaImg filename="20130922215927_drop-off-comparison.png" alt="Drop-off comparison chart" width="600"/>
Source: UK data from On Device Research study, August 2013; n=2324
- Short, 15-question surveys, have the highest completion rate. However, there's no significant difference in the drop-off rates between a long 30-question survey and a 30-question chunked survey.
- 97% of respondents who start the 2nd chunk complete the survey.
- Chunked surveys are especially relevant to the male audience who are less likely to complete a 30-question survey.
Battling survey fatigue with success
Survey fatigue is a well-known problem with long surveys - respondents start to lose attention. They skim question and answer texts, select random answers and generally try to get to the end of the survey as fast as possible.
Chunking is an excellent solution to this problem in mobile surveys. Chunked surveys clearly outperform straight long surveys - far fewer people fall for the trap questions and the location of the questions in the surveys stops affecting data quality.
<odr:mediaImg filename="20130922215955_trap-question-results.png" alt="Trap question results comparison" width="600"/>
- Over twice as many respondents fell for the trap question in the long survey vs. the chunked survey - a marked difference.
- The number of mentions in a multi-code question is consistent in a chunked survey regardless of the location of the question in the survey. In a 30-question survey the position - either in the beginning of the long survey or the end of it - introduces a variability in the number of mentions given.
- As question position does not affect the answer quality there is no added complexity in the survey design phase.
Consumers have changed and the mobile generation do things at their convenience. Research methodologies need to change with them while allowing researchers to collect enough data. - Alistair Hill, CEO of On Device Research
You can <odr:mediaHref filename="20141105103601_mobile-survey-chunking-whitepaper.pdf" title="Whitepaper: Survey chunking on mobile">download a printable version</odr:mediaHref> with the research-on-research results or email us at info@ondeviceresearch.com if you're interested in discussing mobile research methodologies further.
