Touch-screens are becoming the norm of display and interaction on mobile devices, from smartphones to tablets — devices with screen sizes in the range of 4” to 10”. Maximal area of the device’s face is dedicated to the screen, leaving a thin surrounding frame with enough space primarily for the physical ‘On’ button (e.g., awakening the screen, returning to the ‘Home’ display). Most controls for operating a smartphone or tablet and their applications are now virtual, represented as visual icons, symbols and keystrokes on the screen. Users can interact with the device (even for dialing a phone number) by pointing, swiping and similar hand (finger) gestures applied to the screen’s display. It all sounds and feels great, and mostly functions alright, but not all is bright — there is still much room for improvement and better fine-tuning.

The focal devices of this article are smartphones with screens normally between 4” and 5.4” and tablets that carry mostly screens in size of 7” to 10” (extra-size smartphones, also-known-as ‘phablets’, embody a screen larger than 5.4”). They essentially enlarge the real-estate of the screen by doing away with physical controls on the device (buttons and keypads). Operation of the device and interaction with its applications is delegated almost wholly to the touch of virtual controls and other finger-gestures.

This new form of handheld computer-type devices provides a highly advanced class of viewing verbal and pictorial content and interacting with them through manual gestures. Touch-screens were available already in the turn of the century with Personal Digital Assistants (PDAs). The touch-screens of smartphones and tablets are yet empowered in several important aspects: (1) they can be operated with the touch of fingers without need for a pen or stylus; (2) the screens are larger; and (3) the images are in much higher quality. The differences do not end here, if only to mention the communication abilities of the more recent mobile devices. Smartphones in particular can be said to converge a phone and a PDA in a single device, but with some additional capabilities that neither mobile phones nor PDAs have had in earlier times.

The first critical problem to address with touch-screen mobile devices concerns writing. A user is likely to encounter difficulties frequently when writing text with a virtual keyboard — it is rather easy to miss target character keystrokes. The difficulty is not simply in typing text but in getting the words spelled correctly, and overall avoiding character typing errors. The difficulty to produce a text without errors is likely to turn out more acute and agitating with smartphones and the smaller tablets (i.e., 7-8”). It may also cause users to leave spaces in the wrong places, and inversely to concatenate words. Correcting errors can be furthermore annoying when the user cannot find the direction arrows or point his or her finger to stand at the right position of correction; going backspace is not useful if one already moved to another line when the earlier error is noticed or any other correction of text is demanded.

Mobile devices foster writing correspondence texts (e.g., e-mails, chatting, social media updates and comments) even faster than with other modes, specially when users are in motion.  People tend to write correspondence as such more quickly and haphazardly, taking less care to avoid mistakes, and textproofing before sending is usually not in high priority or time-affordable. The result is that producing a well-thought and error-free text message on a touch-screen with a virtual keyboard may be an irritating mission (e.g., either abort message or send it with some errors).

• Writing alphanumeric text with a 12-key physical pad is hardly convenient, and is usually time-consuming. In that sense QWERTY-type keyboards, physical or virtual, are better. There is yet an important difference to notice: The keys on a physical keyboard (e.g.,  Nokia E5 that followed on the original Blackberry phones) can be quite small but they feel like separated bumpers (i.e., giving the user a tactile feedback where the finger rests) whereas a virtual keyboard is completely flat and smooth. The cost of the physical keyboard is of course the smaller screen.

Mistyping is mostly associated with failure to accurately ‘hit’ the intended character keystroke with one’s index finger, and often enough with the thumb (e.g., when in motion and only one hand is free to hold the device and write). That is because virtual character keys tend to be too small for our fingers used for texting (less so on 10” tablet screens). The kind of errors that may result are typing the wrong character, typing the same character inadvertently twice, or  not typing the designated character. Apparently, failing to execute selected actions also occurs with images, such as when having to press virtual buttons or activating icon and text hyperlinks. These controls could be related to the device and its utilities or embedded in websites and imported apps. These issues are well-explained by Steven Hoober in an article in UXMatters (“Common Misconceptions About Touch”, 18 March 2013). Hoober makes an important distinction between seeing clearly text and icon targets and touching them effectively, and he recommends target sizes for them (in measures of points and millimeters).

Hoober refers to an additional sensitive and critical consideration in preventing users from taking accidentally the wrong action: he calls this ‘preventing interference errors’. He clearly suggests to avoid placing controls for actions with opposite consequences too near each other lest trying to touch-press one control could result in adversely activating the other unwanted control. This applies especially to actions associated with catastrophic results or outcomes that are difficult to undo. For instance, he recommends separating sufficiently the locations of controls for Send and Delete actions (Hoober recommends a distance of at least 8mms and preferably 10mms between centres of the controls [the target point of finger contact]).

Touch-screen devices benefit indeed from a larger screen real-estate for image display. But there is nonetheless competition on that real-estate for the content of display, and competition can be quite tough especially on devices with screens smaller than 7” in size. The competition is prominently between images of controls and the content of device utilities, webpages and apps. It applies primarily to the interface of a virtual keyboard that requires a relatively large space (in some cases up to 50% of screen area). However, there could be other controls needed for operating the device and specific utilities, websites or apps (e.g., designers may have to give up on some pictorial imagery in order to allow enough space for action controls like “Add to Basket”).

Focusing on the virtual keyboard: when called-upon to write, it pops-up and hides  other content of the display (e.g., e-mail message, shopping webpage) in the lower part of the screen. It may hide content that the user actually needs to see while proceeding in composing a message or responding to content in a website. The smaller the screen, on one hand a larger part of the underlying display is hidden, on the other hand the keystrokes have to be smaller. Unlike with a physical keyboard, the virtual one can at least be dropped out when not in use and called again when needed for writing. But it can be disturbing if every few moments one has to drop out the keyboard and surface it again to resume writing. With larger screens there should still be enough space for text in the e-mail message editor that one can scroll; with screens 7” or less one may be able to see only up to four lines at a time and even that in small type difficult to read (changing zoom may help but also cause trouble — more below).

Virtual keyboards on mobile devices are split into two or three displays due to space limitations (e.g., Latin letters as for English or German [but with some order variations], numeric figures and symbols, and an extra keyboard for non-Latin alphabets as Hebrew, Arabic, Cyrillic). But in any particular set of keyboard display, some character keys or controls may have to be forsaken for space limitations. As suggested above, it is most annoying when the direction arrows are eliminated (e.g., on a Samsung 7” tablet) because it makes it more difficult to go back and forth across a text while composing and editing it.

Relying on gestures can save space for screen real-estate and help in making interactions fluid and efficient, but working with a touch-screen has limitations. Raluca Budiu of Nielsen & Norman Group (user experience research and usability experts, 19 April 2015) lists some of the main problems that may arise for users: (1) The leading problem concerns typing, and particularly the need to continuously divide attention between the content written and the keypad area; (2) Poor tactile feedback, small keypads and crowded keys make the typing experience more troublesome; (3) The target size of controls or keys has to be considerably larger with touch interface to optimize reaching time and minimise errors compared with a mouse; (4) Since there can be many target areas on a touch-screen (especially of smartphones), it is easy to make accidental touch errors (see Hoober’s ‘interference error’) — some errors can “leave the user disoriented and unsure of what happened”. Budiu notes that respecting the Undo usability heuristic is furthermore important with mobile devices.

References to those main problems could be found in the earlier paragraphs of this article. Two more issues are addressed below:

Scrolling over a touch-screen — Mobile devices do not apply a scrolling bar — the user can scroll by swiping the index finger in a swoosh movement up or down over the touch-screen. The smaller the screen, and if one is in a landscape mode, more scrolling may be needed (shifting left and right is also possible). Trouble may start when the window display is populated by ‘clickable’ tiles or pictures: if the user does not swipe the finger quickly and lightly enough over the image, he or she may activate the underlying link rather than scroll across the window. When that happens, the user may arrive to a different window display, and one has to find the way back. More disturbing, when the content is online and connectivity can be slow on occasion, the user may remain stuck for a long time before being able to return to the desired location of content and resume work.

Zooming and automatic change of size —  Since type on touch-screens of mobile devices can be small and uneasy to read, one can zoom-in to enlarge the display appearance and the text in it. This is usually done by swiping the index and thumb fingers away from each other over the screen (conversely, one can zoom-out to reduce size but see more content by bringing the fingers closer together). But caution: one has to be accurate, and this does not always work so well. One may accidentally “blow” a picture image over the whole screen, for instance. When writing an e-mail message zooming can be helpful as one toggles between writing and reading the composed text. Yet, these devices are smart and sometimes they try to adjust the size for you according to the identified mode of use; sometimes it is appropriate but on occasion it causes trouble and nuisance. In more drastic cases, whilst trying to enlarge the type on a webpage, the system may lock in a loop and continue zooming-in until the user can see nothing coherent and has to start over again.

• Note that the scrolling and size problems were encountered much more frequently on a Samsung tablet, either 7” or 10”, than on an Apple’s 10” iPad .

People may discover at times that although they were sure they could see exactly where their hand should reach and act, it somehow missed the target. That may happen because perception augmented by cognitive conception and processes of location and action are not the same in the human brain. These processes are connected (i.e., they share and pass information between them) but are nonetheless distinct. Visual information flows and is processed in two pathways: (a) perceptual but non-conceptual information is passed through the ventral stream to the temporal lobe where percepts are interpreted into meaningful images of scenes and objects; (b) visuospatial (location) and visuomotor (action) signals are transferred through the dorsal stream to the parietal lobe to guide, for instance, our manual movements. The ventral-temporal (semantic) visual system allows to identify a target for action yet the dorsal-parietal (pragmatic) visual system is responsible in parallel for determining where the target is and how to act upon it. Furthermore, action requires only a subset of information from percepts, including size, shape and orientation of a target object to complete a task, much less than what we perceive and even recognize as seeing. The conceptual identity of the target is mostly not required.

Jacob and Jeannerod (2003), distinguishing between Semantic and Pragmatic vision as cited above, argue that pragmatic vision processed in the parietal lobe is more complex and multi-layered than has been theorised and described in literature on vision. Humans may believe they act on whatever they perceive (as an image) but in fact they usually act on the nonconscious signals that arrive directly to the parietal lobe. Recognising and identifying clearly the target and understanding what to do with it are therefore not enough — the target should be designed in a form that permits (affords) the visuomotor system to perform the action correctly and efficiently. The semantic and pragmatic processes occur simultaneously. In some instances the semantic system may assist the pragmatic system but usually deliberate intervention is not needed. A user should not have to tilt the tablet, for example, while trying to accurately and slowly direct his finger to touch the small backward or forward arrows of the browser on the touch-screen. This is an example of an effortful action users should not be driven to.

Using mobile devices with touch-screens has advantages and can be a gratifying experience. But there is also a lot that can be done to improve that experience, moreover if the aspiration is that consumers will use these devices much more frequently for performing more tasks, and especially that they will use tablets more than desktop and laptop computers in the future. Although the touch-screen mobile devices promote to use fingers, they should support the use of a pen or stylus and perhaps even encourage it with smaller screen devices (for typing and not just for drawing). It is also helpful to enlarge the images of keystrokes, icons and symbols as one approaches to touch any of these controls. These are just hints and there are probably many more ways interaction designers can create to improve mobile users’ experiences, making them more effective and enjoyable.

Ron Ventura, Ph.D. (Marketing)

Reference:

Ways of Seeing: The Scope and Limits of Visual Cognition; Pierre Jacob and Marc Jeannerod, 2003; Oxford University Press.

Additional recommended reading:

Mobile Computing; Jesper Kjeldskov; In Encyclopedia of Human-Computer Interaction (2nd edition, Chapter 9); Interaction Design Foundation.