Questions, questions, and more questions — as a survey interview goes on questions start to sound or look too much alike. The respondent may gradually pay less attention to the next question and its response options, replying more automatically and less reliably. If the respondent is truly fed up she may quit before the end of the questionnaire — abandoning a survey is even easier in self-administered online surveys than hanging up on an interviewer in a phone survey. It is a key problem researchers in marketing and other fields of social sciences are grappling with in the past decade, next to the non-response problem (i.e., whereby individuals included in the original sample don’t even get to start the questionnaire).

Researchers are struggling to return somehow to the days (1960s to 1980s) when surveys in many countries were still something new and intriguing, a unique and attractive channel for people to voice their opinions, preferences and attitudes. But since then surveys have become far more frequent, and many consumers have become more experienced answering them. This is an advantage when a respondent is more familiar with a type of question and understands better its logic to give a correct, candid or reliable answer. It is a clear disadvantage if the respondent loses interest in survey questions because they seem too familiar and predictable, whereof he or she may not read questions fully, may give answers haphazardly, and sometimes may try to outsmart the question. New media channels may also be perceived now as more attractive for consumers wherein to express their views. Truly, researchers have been concerned by this issue for many years and have always worked on devising techniques to increase respondents’ interest and engagement through the process of completing a survey in different modes. But the approach of “gamification” imported into marketing research lately provokes controversy and doubts: How far should researchers go in their effort to make surveys more likeable, appealling or entertaining to consumers-respondents?

The concept of “gamification” means the introduction of custom features or characteristics of games into survey questionnaires, applying particularly to self-administered questionnaires on the Internet. Deterding and his colleagues define “gamification” as the “use of game design elements in non-game contexts” (1), as in the contexts of advertising and research. The technology-driven inspiration for this approach comes from video games, as directly addressed by Deterding et al., yet gamification borrows its elements also from the broader framework of games. Gaming in general is different from playing, by setting goals to achieve and rules to follow, and creating a competitive challenge for the players.

The game elements to be utilised can range from interactive graphic features, scenarios and scenes, objectives and rewards, to explicit rules that have to be followed. However, a gamified application (e.g., survey question or task) adopts only a subset of design elements shared by games, that makes the applications look and feel like a game but is usually not a full-fledged game (2). The problem starts with ambiguity about the choice of elements to be imported into survey research and how to implement them. It is a concept still in formation and researchers appear to give it different interpretations:

• Using interactive visual response options and features for a playful design (rather than gameful);
• Creating scenarios (story-like) that set a more interesting and realistic context in which respondents should answer the actual question;
• Formulating questions as problems or challenges using the logic of games;
• Designing a questionnaire  in a set-up that mimics a video game (e.g., the respondent chooses a personal avatar, screens appear like scenes from an adventure game, awards like bonus points are given as the respondent progresses through the questionnaire).

In the first option, playful features include for instance sliders instead of standard  scales (i.e., ticking circles), animation, and drag-and-drop response items (e.g., for ranking or sorting them into categories). While such features may resemble isolated interactive elements that appear in video games, they lack other fundamental components of a game. In the second and third options we may find more game logic without the visual appearance of a video game (perhaps evoking instead ‘visualization’ by the respondent). Using an alternative terminology in the field of gaming, some of the methods employ more ‘game mechanism’ while others emphasise more ‘game thinking’. Which of these aspects is more relevant and important with respect to research? And how closely should we stick to the ‘video game’ as a reference for gamification? The fourth option for implementation of gamification appears much less compatible with the objectives of marketing research projects than the preceding options . The methods that use scenarios, game-like problems and ‘playful’ interactive response tools exhibit potential to improve (online) surveys but their planning and implementation need to be carefully considered and thought-out because of the ways in which they may alter the context, meaning, and scale of measurement of responses.

In order to illustrate benefits to be gained vs. pitfalls at risk when ‘gamifying’ survey questions, I refer to an instructive award-winning research paper by Rintoul and Puleston (3). They tested the effects of ‘gamified’ question formats against standard formats in an online survey, while comparing response patterns between Western countries (US, Australia) and Asian countries (China, India, Japan, Korea, and Singapore). Let us consider just two examples here:

• When examining what olympic sport events people like to watch, a standard question asks “What are your favourite Olympic events?” But respondents may be requested to imagine a scenario in which they play a more important role that induces them to be more engaged in choosing sport events: “Imagine you were in charge of the TV coverage of the London 2012 Olympics and they broadcast only the events You liked to watch, can you draw a list of all the events you would show on TV?” Australians listed in the scenario version more than double the number of events on average than in the standard version (6.8 versus 3.3). Large lifts were also found in Asian countries where the “base” number of events was the smallest (e.g., China: from 2 to 8 events). The scenario proposed in this study seems legitimate as it merely adds some flare of interest in a situation that is not truly likely to happen in reality. There is the danger, however, that when the scenario is too peculiar or defines a condition too specific the researcher would no longer be safe to infer the validity of those consumer responses in other conditions. (Note: Scenarios opening with keywords like “suppose” and “imagine” have been used in surveys before the age of ‘gamification’, for instance in conjoint studies).
• Rintoul and Puleston test a ‘gamified’ scale with a ‘smooth’ slider in two visual versions: one version used just the slider, the other version included an additional animated feature, a person figure that changes his posture according to the position of the ‘slider’ handle on the scale — the person rises and applauds when the highest level is chosen on one end to that person remaining seated leaning backwards ‘asleep’ when the lowest level is chosen on the other end. The researchers show that the animated version reduces differences in response patterns between countries. The concern is that respondents would be affected by animation, trying to experiment with the figure or ‘please’ it. In this study the ‘applauding person’ did not lead to higher frequency of the top-level; yet, conspicuously, more responses in the US and Australia were attracted by the person ‘asleep’ at the low end of the scale than in the simple version. (Note: The ‘rating’ scale did not show numeric levels.)

To the credit of Rintoul and Puleston, their approach is well-reasoned, moderate in its application of ‘gamified’ elements, and realistic. But as advocates of ‘gamification’ techniques they tend to play down the potential problems with those techniques. Some of their visual instruments seem to alter the type of scale or mix between measurement scales where the properties of those new instruments are not properly understood (e.g., sorting items into Likert-type categories, the “water tap” scale, rating scales using faces and text with no numeric levels). A critical question should be raised with regard to every gamified feature: how relevant is it to the mental construct, behaviour, or decision situation studied (e.g., a scenario, a time constraint instructsing “answer within 2 minutes”); at the very least it should not conflict with the focal construct. There is also concern whether additional items chosen when using pictorial images truly reflect respondents’ preferences or interests or were they chosen just as part of the ‘game’? Utilisation of ‘gaming’ features in an artificial way and of little relevance to the original issue studied can seriously hurt the attainment of the research objectives.

Another study that compared four conditions of design format found just modest support for the application of gamifying elements (4). The formats tested were text-only, decoratively visual (DV) , functionally visual (FV, i.e., interactive visual features), and gamified questionnaire. The researchers, Downes-Le Guin and colleagues, address the argument that sliders and other visual features help to combat the problem of ‘straightlining’ in Likert-type scales by increasing response variation between items (an issue addressed also by Rintoul and Puleston). They found limited support for such improvement in the FV condition and none in the DV condition. The FV and gamified designs showed some benefits in increasing enjoyment and interest but no improvement in data quality. The gamified version adds in fact a kind of back-stage view from a game-video to a question display similar to FV. Even when some gaming features like choice of avatar are added, it still functions not substantially more than as a stage decoration while bearing no relevance to the content of questions. It cannot be too surprising that this design has demonstrated little effectiveness.

If researchers wish to introduce characteristics of games right into the content of questions, they may borrow concepts from the fields of game theory, decision making, particularly through the prism of behavioural economics, and simulations. These areas have aspects that are particularly relevant to marketing (competition) and consumer behaviour which can be used to make questions more interesting, intriguing (e.g., as a puzzle), relating more closely to realistic market situations, and being of greater self-relevance (e.g., by quoting or bidding prices for products). There are also limitations to this line of questioning. Questions posed as game-like problems are usually more difficult to comprehend and answer, and take more time to do so. In addition, problems presented in experiments of behavioural economics often reveal biases in judgement that need to be addressed or accounted for. Therefore, the level of sophistication has to be balanced against the ability and willingness of respondents to “solve” the problem-questions.

This course of “gamified” research should have more reason to look for examples and ideas in the domain of “serious games” (e.g., applied in training and education) than video games for fun and entertainment. There are important areas of marketing research where advanced models and techniques of gamification can indeed contribute such as: (1) studying reactions to scenarios of customer experiences in service delivery, or (2) simulation of shopping trips in stores and malls.

Introducing gamified elements and techniques to marketing survey questionnaires can help to improve respondent behaviour, receiving a more favourable attitude towards the survey and smoothing their progress through the questionnaire. The benefits, however, are not clear-cut and are subject to much debate. The contribution of gamification depends greatly on how it is implemented — not all interpretations are welcome from a research perspective. There is justified concern that an entertainment-driven approach will lead to superficial game-like applications that only distract consumers-respondents from answering questionnaires accurately and sincerely. From playful interactive features to more advanced mechanisms like scenarios and problems with game-logic that also encourage more thinking, it is important to keep the gamified design sensibly relevant and related to the domain of the question.

May you all my readers have a Joyful and Successful New Year 2013

Ron Ventura, Ph.D. (Marketing)

Notes:

(1) “From Game Design Elements to Gamefulness: Defining Gamification”; Sebastian Deterding, Dan Dixon, Rilla Khaled, & Lennart Nacke, 2011; Conference Proceeding MindTrek ’11 (ACM)http://85.214.46.140/niklas/bach/MindTrek_Gamification_PrinterReady_110806_SDE_accepted_LEN_changes_1.pdf

(2)  Ibid. 1

(3) “Beyond Colour and Movement: Measuring the Impact of Dynamic Answer Formats on Respondent Behaviour”; Duncan Rintoul and Jon Puleston, 2012; AMSRS Conference (Australian Marketing and Social Research Society) [See another paper related to this research presented at ESOMAR Asia ’12 Conference). http://iibsor.uow.edu.au/consulting/UOW109801.html (Step down the page of Rintoul to “Downloads”) / See also an article with more examples by Deborah Sleep, “Improving Online Market Research through Gamification”, The Guardian: Media Network blog, posted 15 Aug., 2012  http://www.guardian.co.uk/media-network/media-network-blog/2012/aug/15/online-market-research-gamification

 (4) “Myths and Realities of Respondent Engagement in Online Surveys”; Theo Downes-Le Guin, Reg Baker, Joannne Menchling, & Eric Ruylea, 2012, International Journal of Market Research, 54 (5), pp. 1-21 http://www.marketstrategies.com/user_area/content_media/Online_Survey_Engagement.pdf