Posts Tagged ‘Surveys’

‘Experience’ has gained a prime status in the past decade — everything seems to revolve around experience in the universe of management, marketing, and even more specifically with respect to relationship marketing. It has become like a sine qua non of operating in this universe. There can be multiple contexts for framing experience — customer experience, brand experience, user (or product) experience, and also employee experience. Nevertheless, these concepts are inter-linked, and customer experience could be the central point-of-reference just because all other forms of experience eventually contribute to the customer’s experience. After all, this is the age of experience economy (cf. Pine and Gilmore).

This focus on the role of experience and primarily customer experience (CX) in contemporary marketing surely has not escaped the attention of companies involved with data-based marketing particularly on the service side (e.g., technology, research, consulting). In mid-November 2018 enterprise information technology company SAP announced a stark move of acquiring research technology firm Qualtrics for the sum of $8 billion in cash (deal expected to materialise during the first half of 2019). Qualtrics started in 2002 by specialising in survey technology for conducting consumer and customer surveys online, and has later on broadened the spectrum of its software products and tools to address a range of experience domains, put in a framework entitled Experience Management (XM).

However, less visible to the public, Qualtrics made an acquisition of its own of Temkin Group — an expert company specialising in customer experience research, training and consulting — about two weeks before announcing the SAP-Qualtrics deal. Qualtrics was reportedly engaged at the time of these deals in preparations for its IPO. Adding the knowledge and capabilities of Temkin Group to those of Qualtrics could fairly be viewed as a positive enforcement of the latter prior to its IPO, and eventually the selling of Qualtrics to SAP. Therefore, it would be right to say that Qualrtics + Temkin Group and SAP are effectively joining forces in domain knowledge, research capabilities and data technologies. Yet since the original three entities (i.e., as before November 2018) were so unequal in size and power, it raises some major questions about how their union under the umbrella of SAP will work out.

SAP specialises in enterprise software applications for organisational day-to-day functions across-the-board, and supporting software-related services (SAP was established in 1972, based in Germany). It operates today in 130 countries with 100+ innovation and development centres; its revenue in the 2017 financial year was $23.46 billion. Many of the company’s software applications can be deployed on premises, in the cloud, or hybrid (SAP reports 150 million subscribers in the cloud service user base). The two product areas of highest relevance to this story are CRM & Customer Experience solutions and the Enterprise Resource Planning (ERP) solutions & Digital Core (featuring its flagship platform HANA). The two areas of solutions correspond with each other.

The S4/HANA platform is described as an intelligent ERP software, a real-time solution suite . It enables, for example, delivering personally customised products ordered online (e.g., bicycles). For marketing activities and customer-facing services it should require data from the CRM and CX applications. The ERP platform supports, however, the financial planning and execution of overall activities of a client organisation. The CRM & Customer Experience suite of solutions includes five key components: Customer Data Cloud (enabled actually by Gigya, another acquisition by SAP in 2017); Marketing Cloud; Commerce Cloud; Sales Cloud; and Service Cloud. The suite covers a span of activities and functions: profiling and targeting at segment-level and individual level, applicable, for instance, in campaigns or tracking customer journeys (Marketing); product order and content management (Commerce); comprehensive self-service processes plus field service management and remote service operations by agents (Service). In all these sub-areas we may find potential links to the kinds of data that can be collected and analysed with the tools of Qualtrics while SAP’s applications are run on operational data gathered within its system apparatus. The key strengths offered in the Customer Data Cloud are integrating data, securing customer identity and access to digital interfaces across channels and devices, and data privacy protection. SAP highlights that its marketing and customer applications are empowered by artificial intelligence (AI) and machine learning (ML) capabilities to personalise and improve experiences.

  • At the technical and analytic level, SAP’s Digital Platform is in charge of the maintenance of solutions and databases (e.g., ERP HANA) and management of data processes, accompanied by the suite of Business Analytics that includes the Analytics Cloud, Business Analytics, Predictive Analytics and Collaborative Enterprise Planning. Across platforms SAP makes use of intelligent technologies and tools organised in its Leonardo suite.

Qualtrics arrives from quite a different territory, nestled much closer to the field of marketing and customer research as a provider of technologies for data collection through surveys of consumers and customers, and data analytic tools. The company has gained acknowledgement thanks to its survey software for collecting data online whose use has so expanded to make it one of the more popular among businesses for survey research. Qualtrics now focuses on four domains for research: Customer Experience, Brand Experience, Product Experience, and Employee Experience.

  • The revenue of Qualtrics in 2018 is expected to exceed $400 million (in first half of 2018 revenue grew 42% to $184m); the company forecast that revenue will continue to grow at an annual rate of 40% before counting its benefits from synergies with SAP (CNBC; TechCrunch on 11 November 2018).

Qualtrics organises its research methodologies and tools by context under the four experience domains aforementioned. The flagship survey software, PER, allows for data collection through multiple digital channels (e.g., e-mail, web, mobile app, SMS and more), and is accompanied by a collection of techniques and tools for data analysis and visualisation. The company emphasises that its tools are so designed that use of them does not require one to be a survey expert or a statistician.

Qualtrics provides a range of intelligent assistance and automation capabilities; they can aid, guide and support the work of users according to their level of proficiency. Qualtrics has developed a suite of intelligent tools, named iQ, among them Stats iQ for statistical analysis, Text iQ for text analytics and sentiment scoring, and Predict iQ + Driver iQ for advanced statistical analysis and modelling. Additionally, it offers ExpertReview for helping with questionnaire composition (e.g., by giving AI-expert ‘second opinion’). In a marketing context, the company offers techniques for ad testing, brand tracking, pricing research, market segmentation and more. Some of these research methodologies and tools would be of less relevance and interest to SAP unless they can be connected directly to customer experiences that SAP needs to understand and account for through the services it offers.

The methods and tools by Qualtrics are dedicated to bringing the subjective perspective of customers about their experiences. Under the topic of Customer Experience Qualtrics covers customer journey mapping, Net Promoter Score (NPS), voice of the customer, and digital customer experience; user experience is covered in the domain of Product Experience, and various forms of customer-brand interactions are addressed as part of Brand Experience. The interest of SAP especially in Qualtrics, as stated by the firm, is  complementing or enhancing its operational data (O-data) with customer-driven experience data (X-data) produced by Qualtrics (no mention is made of Temkin Group). The backing and wide business network of SAP should create new opportunities for Qualtrics to enlarge its customer base, as suggested by SAP. The functional benefits for Qualtrics are less clear; possible gains may be achieved by combining operational metrics in customer analyses as benchmarks or by making comparisons between objective and subjective evaluations of customer experiences, assuming clients will subscribe to some of the services provided by the new parent company SAP.

Temkin Group operated as an independent firm for eight years (2010-2018), headed by Bruce Temkin (with wife Karen), until its acquisition by Qualtrics in late October 2018. It provided consulting, research and training activities on customer experience (at its core was customer experience but it dealt with various dimensions of experience beyond and in relation to customers). A key asset of Temkin Group is its blog / website Experience Matters, a valued resource of knowledge; its content remains largely in place (viewed January 2018), and hopefully will stay on.

Bruce Temkin developed several strategic concepts and constructs of experience. The Temkin Experience Rating metric is based on a three-component construct of experience: Success, Effort and Emotion. The strategic model of experience includes four required competencies: (a) Purposeful Leadership; (b) Compelling Brand Values; (c) Employee Engagement; and (d) Customer Connectedness. He made important statements in emphasising the essence of employee engagement to deliver superior customer experience, and in including Emotion as one of the pillars of customer experience upon which it should be evaluated. The more prominent of the research reports published by Temkin Group were probably the annual series of Temkin Experience Rating reports, covering 20 industries or markets with a selection of companies competing in each.

Yet Temkin apparently has come to a realisation that he should not go it alone any longer. In a post blog on 24 October 2018, entitled “Great News: Temkin Group Joins Forces With Qualtrics“, Temkin explained as the motivation to his deal with Qualtrics a recognition he had reached during the last few years: “it’s become clear to me that Qualtrics has the strongest momentum in CX and XM“. Temkin will be leading the Qualtrics XM Institute, built on the foundations of Temkin CX Institute dedicated to training. The new institute will be sitting on top of Qualtrics XM platform. In his blog announcement Temkin states that the Qualtrics XM Institute will “help shape the future of experience management, establish and publish best practices, drive product innovation, and enable certification and training programs that further build the community of XM professionals” — a concise statement that can be viewed as the charter of the institute Temkin will be in charge of at Qualtrics. Temkin has not taken long to adopt the framework of Experience Management and support it in writing for the blog.

The teams of Temkin and Qualtrics (CEO and co-founder Ryan Smith) may co-operate more closely in developing research plans on experience for clients and initiating research reports similar to the ones Temkin Group produced so far. Bruce Temkin should have easy and immediate access to the full range of tools and technologies of Qualtrics to continue with research projects and improve on them. Qualtrics should have much to benefit from the knowledge and training experience of Temkin in the new XM institute at Qualtrics. It seems easier to foresee beneficial synergies between Temkin Group and Qualtrics than their expected synergies with SAP.

However, there is a great question arising now, how all this vision and plans for Temkin and Qualtrics working together, and particularly their project of Qualtrics XM Institute, will be sustained following the acquisition of Qualtrics by SAP. One cannot overlook the possibility that SAP will develop its own expectations and may require changes to plans only recently made or modifications to Qualtrics CX Platform and XM Solutions so as to satisfy the needs of SAP. According to TechCrunch (11 Nov. 2018) Qualtrics will continue to function as a subsidiary company and will retain its branding and personnel (note: it may be gradually assimilated into SAP while keeping Qualtrics associated names, as seems to be the case of Israel-based Gigya). Much indeed can depend on giving Qualtrics + Temkin Group autonomy to pursue with their specialisations and vision on XM while they share knowledge, data and technologies with SAP.

Bill McDermott, CEO of SAP, is looking high in the sky: as quoted in the company’s news release from 11 November 2018, he describes bringing together SAP and Qualtrics as “a new paradigm, similar to market-making shifts in personal operating systems, smart devices and social networks“. But it is also evident that SAP still sees the move through the prism of technology: “The combination of Qualtrics and SAP reaffirms experience management as the ground-breaking new frontier for the technology industry“.

Temkin’s viewpoint is much more customer-oriented and marketing-driven vis-à-vis the technology-driven view of McDermott and SAP, which may put them in greater conflict with time about priorities and future direction for XM. Qualtrics headed by Ryan Smith will have to decide how it prefers to balance between the marketing-driven view and technology-driven view on experience. Temkin, for example, has reservations about the orientation of the technology known as Enterprise Feedback Management (EFM), suggesting instead a different focus by naming this field “Customer Insight and Action (AIC) Platforms”. In his comments on the acquisition of Qualtrics by SAP (16 November 2018) he explains that organisations “succeed by taking action on insights that come from many sources, combining experience data (X-data) and operational data (O-data)“. In his arguments in favour of joining SAP with Qualtrics, Temkin recollects an observation he made in an award-winning report from 2002 while at Forrester Research: he argued then that “widespread disappointing results of CRM were a result of a pure technology-orientation and that companies needed to focus more on developing practices and perspectives that used the technology to better serve customers”; he claims that much has changed in the field since that time. Yet it is hard to be convinced that technology has much less influence now in shaping organisational, managerial and marketing processes, on both service side (e.g., SAP) and client side.

  • As a note aside, if SAP gets the upper hand in setting the agenda and does not give sufficient autonomy to Qualtrics as suggested earlier, the first sector at risk of having most to lose from this deal would be ‘marketing and customer research’.

SAP and Qualtrics are both involved in development and implementation of technology, yet SAP is focused on information technology enabling overall day-to-day operations of an organisation, whereas Qualtrics is focused on technology enabling experience and marketing research. Qualtrics and Temkin Group are both engaged in domains of experience: Qualtrics specialises in the technology that enables the research, while Temkin Group brought strengths in conducting research plus strategic thinking and training (education) on customer experience. In order for their joint forces to succeed they all will have to find ways to bridge gaps between their viewpoints, to ‘live and let live’, and at the same time complement one another in areas of shared understanding and expertise.

Ron Ventura, Ph.D. (Marketing)


Read Full Post »

Questions, questions, and more questions — as a survey interview goes on questions start to sound or look too much alike. The respondent may gradually pay less attention to the next question and its response options, replying more automatically and less reliably. If the respondent is truly fed up she may quit before the end of the questionnaire — abandoning a survey is even easier in self-administered online surveys than hanging up on an interviewer in a phone survey. It is a key problem researchers in marketing and other fields of social sciences are grappling with in the past decade, next to the non-response problem (i.e., whereby individuals included in the original sample don’t even get to start the questionnaire).

Researchers are struggling to return somehow to the days (1960s to 1980s) when surveys in many countries were still something new and intriguing, a unique and attractive channel for people to voice their opinions, preferences and attitudes. But since then surveys have become far more frequent, and many consumers have become more experienced answering them. This is an advantage when a respondent is more familiar with a type of question and understands better its logic to give a correct, candid or reliable answer. It is a clear disadvantage if the respondent loses interest in survey questions because they seem too familiar and predictable, whereof he or she may not read questions fully, may give answers haphazardly, and sometimes may try to outsmart the question. New media channels may also be perceived now as more attractive for consumers wherein to express their views. Truly, researchers have been concerned by this issue for many years and have always worked on devising techniques to increase respondents’ interest and engagement through the process of completing a survey in different modes. But the approach of “gamification” imported into marketing research lately provokes controversy and doubts: How far should researchers go in their effort to make surveys more likeable, appealling or entertaining to consumers-respondents?

The concept of “gamification” means the introduction of custom features or characteristics of games into survey questionnaires, applying particularly to self-administered questionnaires on the Internet. Deterding and his colleagues define “gamification” as the “use of game design elements in non-game contexts” (1), as in the contexts of advertising and research. The technology-driven inspiration for this approach comes from video games, as directly addressed by Deterding et al., yet gamification borrows its elements also from the broader framework of games. Gaming in general is different from playing, by setting goals to achieve and rules to follow, and creating a competitive challenge for the players.

The game elements to be utilised can range from interactive graphic features, scenarios and scenes, objectives and rewards, to explicit rules that have to be followed. However, a gamified application (e.g., survey question or task) adopts only a subset of design elements shared by games, that makes the applications look and feel like a game but is usually not a full-fledged game (2). The problem starts with ambiguity about the choice of elements to be imported into survey research and how to implement them. It is a concept still in formation and researchers appear to give it different interpretations:

  • Using interactive visual response options and features for a playful design (rather than gameful);
  • Creating scenarios (story-like) that set a more interesting and realistic context in which respondents should answer the actual question;
  • Formulating questions as problems or challenges using the logic of games;
  • Designing a questionnaire  in a set-up that mimics a video game (e.g., the respondent chooses a personal avatar, screens appear like scenes from an adventure game, awards like bonus points are given as the respondent progresses through the questionnaire).

In the first option, playful features include for instance sliders instead of standard  scales (i.e., ticking circles), animation, and drag-and-drop response items (e.g., for ranking or sorting them into categories). While such features may resemble isolated interactive elements that appear in video games, they lack other fundamental components of a game. In the second and third options we may find more game logic without the visual appearance of a video game (perhaps evoking instead ‘visualization’ by the respondent). Using an alternative terminology in the field of gaming, some of the methods employ more ‘game mechanism’ while others emphasise more ‘game thinking’. Which of these aspects is more relevant and important with respect to research? And how closely should we stick to the ‘video game’ as a reference for gamification? The fourth option for implementation of gamification appears much less compatible with the objectives of marketing research projects than the preceding options . The methods that use scenarios, game-like problems and ‘playful’ interactive response tools exhibit potential to improve (online) surveys but their planning and implementation need to be carefully considered and thought-out because of the ways in which they may alter the context, meaning, and scale of measurement of responses.

In order to illustrate benefits to be gained vs. pitfalls at risk when ‘gamifying’ survey questions, I refer to an instructive award-winning research paper by Rintoul and Puleston (3). They tested the effects of ‘gamified’ question formats against standard formats in an online survey, while comparing response patterns between Western countries (US, Australia) and Asian countries (China, India, Japan, Korea, and Singapore). Let us consider just two examples here:

  • When examining what olympic sport events people like to watch, a standard question asks “What are your favourite Olympic events?” But respondents may be requested to imagine a scenario in which they play a more important role that induces them to be more engaged in choosing sport events: “Imagine you were in charge of the TV coverage of the London 2012 Olympics and they broadcast only the events You liked to watch, can you draw a list of all the events you would show on TV?” Australians listed in the scenario version more than double the number of events on average than in the standard version (6.8 versus 3.3). Large lifts were also found in Asian countries where the “base” number of events was the smallest (e.g., China: from 2 to 8 events). The scenario proposed in this study seems legitimate as it merely adds some flare of interest in a situation that is not truly likely to happen in reality. There is the danger, however, that when the scenario is too peculiar or defines a condition too specific the researcher would no longer be safe to infer the validity of those consumer responses in other conditions. (Note: Scenarios opening with keywords like “suppose” and “imagine” have been used in surveys before the age of ‘gamification’, for instance in conjoint studies).
  • Rintoul and Puleston test a ‘gamified’ scale with a ‘smooth’ slider in two visual versions: one version used just the slider, the other version included an additional animated feature, a person figure that changes his posture according to the position of the ‘slider’ handle on the scale — the person rises and applauds when the highest level is chosen on one end to that person remaining seated leaning backwards ‘asleep’ when the lowest level is chosen on the other end. The researchers show that the animated version reduces differences in response patterns between countries. The concern is that respondents would be affected by animation, trying to experiment with the figure or ‘please’ it. In this study the ‘applauding person’ did not lead to higher frequency of the top-level; yet, conspicuously, more responses in the US and Australia were attracted by the person ‘asleep’ at the low end of the scale than in the simple version. (Note: The ‘rating’ scale did not show numeric levels.)

To the credit of Rintoul and Puleston, their approach is well-reasoned, moderate in its application of ‘gamified’ elements, and realistic. But as advocates of ‘gamification’ techniques they tend to play down the potential problems with those techniques. Some of their visual instruments seem to alter the type of scale or mix between measurement scales where the properties of those new instruments are not properly understood (e.g., sorting items into Likert-type categories, the “water tap” scale, rating scales using faces and text with no numeric levels). A critical question should be raised with regard to every gamified feature: how relevant is it to the mental construct, behaviour, or decision situation studied (e.g., a scenario, a time constraint instructsing “answer within 2 minutes”); at the very least it should not conflict with the focal construct. There is also concern whether additional items chosen when using pictorial images truly reflect respondents’ preferences or interests or were they chosen just as part of the ‘game’? Utilisation of ‘gaming’ features in an artificial way and of little relevance to the original issue studied can seriously hurt the attainment of the research objectives.

Another study that compared four conditions of design format found just modest support for the application of gamifying elements (4). The formats tested were text-only, decoratively visual (DV) , functionally visual (FV, i.e., interactive visual features), and gamified questionnaire. The researchers, Downes-Le Guin and colleagues, address the argument that sliders and other visual features help to combat the problem of ‘straightlining’ in Likert-type scales by increasing response variation between items (an issue addressed also by Rintoul and Puleston). They found limited support for such improvement in the FV condition and none in the DV condition. The FV and gamified designs showed some benefits in increasing enjoyment and interest but no improvement in data quality. The gamified version adds in fact a kind of back-stage view from a game-video to a question display similar to FV. Even when some gaming features like choice of avatar are added, it still functions not substantially more than as a stage decoration while bearing no relevance to the content of questions. It cannot be too surprising that this design has demonstrated little effectiveness.

If researchers wish to introduce characteristics of games right into the content of questions, they may borrow concepts from the fields of game theory, decision making, particularly through the prism of behavioural economics, and simulations. These areas have aspects that are particularly relevant to marketing (competition) and consumer behaviour which can be used to make questions more interesting, intriguing (e.g., as a puzzle), relating more closely to realistic market situations, and being of greater self-relevance (e.g., by quoting or bidding prices for products). There are also limitations to this line of questioning. Questions posed as game-like problems are usually more difficult to comprehend and answer, and take more time to do so. In addition, problems presented in experiments of behavioural economics often reveal biases in judgement that need to be addressed or accounted for. Therefore, the level of sophistication has to be balanced against the ability and willingness of respondents to “solve” the problem-questions.

This course of “gamified” research should have more reason to look for examples and ideas in the domain of “serious games” (e.g., applied in training and education) than video games for fun and entertainment. There are important areas of marketing research where advanced models and techniques of gamification can indeed contribute such as: (1) studying reactions to scenarios of customer experiences in service delivery, or (2) simulation of shopping trips in stores and malls.

Introducing gamified elements and techniques to marketing survey questionnaires can help to improve respondent behaviour, receiving a more favourable attitude towards the survey and smoothing their progress through the questionnaire. The benefits, however, are not clear-cut and are subject to much debate. The contribution of gamification depends greatly on how it is implemented — not all interpretations are welcome from a research perspective. There is justified concern that an entertainment-driven approach will lead to superficial game-like applications that only distract consumers-respondents from answering questionnaires accurately and sincerely. From playful interactive features to more advanced mechanisms like scenarios and problems with game-logic that also encourage more thinking, it is important to keep the gamified design sensibly relevant and related to the domain of the question.

May you all my readers have a Joyful and Successful New Year 2013

Ron Ventura, Ph.D. (Marketing)


(1) “From Game Design Elements to Gamefulness: Defining Gamification”; Sebastian Deterding, Dan Dixon, Rilla Khaled, & Lennart Nacke, 2011; Conference Proceeding MindTrek ’11 (ACM)

(2)  Ibid. 1

(3) “Beyond Colour and Movement: Measuring the Impact of Dynamic Answer Formats on Respondent Behaviour”; Duncan Rintoul and Jon Puleston, 2012; AMSRS Conference (Australian Marketing and Social Research Society) [See another paper related to this research presented at ESOMAR Asia ’12 Conference). http://iibsor.uow.edu.au/consulting/UOW109801.html (Step down the page of Rintoul to “Downloads”) / See also an article with more examples by Deborah Sleep, “Improving Online Market Research through Gamification”, The Guardian: Media Network blog, posted 15 Aug., 2012  http://www.guardian.co.uk/media-network/media-network-blog/2012/aug/15/online-market-research-gamification

 (4) “Myths and Realities of Respondent Engagement in Online Surveys”; Theo Downes-Le Guin, Reg Baker, Joannne Menchling, & Eric Ruylea, 2012, International Journal of Market Research, 54 (5), pp. 1-21 http://www.marketstrategies.com/user_area/content_media/Online_Survey_Engagement.pdf

Read Full Post »