Unremarkable Augmentation in Busy Practitioners’ Surrounding: Exploring Peripheral Interaction with Teachers and Nurses
Pengcheng An 1,*, Saskia Bakker 2, Miguel Cabral Guerra 3, Jesper van Bentum 2, and Berry Eggen 3
1 School of Design, Southern University of Science and Technology, Shenzhen, China
2 Philips Design, Eindhoven, the Netherlands
3 Department of Industrial Design, Eindhoven University of Technology, Eindhoven, the Netherlands
When practitioners are in the midst of their busy day-to-day work—such as teachers facilitating learners in the classroom or nurses caring for patients in the hospital—they face complex social or material environments and intensive, intertwined activities. Yet unprecedentedly, these practitioners also need to incorporate technologies, as increasingly necessary supports, in their surroundings. A challenge thereby concerns practitioners’ limited attention: current human-technology interfaces usually demand users’ focus of attention, whereas interacting with digital devices is often neither the only nor the central task in practitioners’ ongoing activities. This work explores the design of peripheral interaction to understand how surrounding technologies could leverage the periphery of human attention to tacitly augment practitioners’ daily routines. Via a series of field explorations with teachers at school and nurses in neonatal intensive care units (NICU), we illustrate how peripheral interaction designs could seamlessly enrich the practitioners’ action repertoire (readily available actions) or enhance their reflection-in-action (sensemaking of the unfolding situation) without interfering with their ongoing routines. From these cases, we extract two relevant design properties to inform future practice: i.e., the designed interaction being subsidiary to the main practice and open to practical knowing. Six considerations are provided to help designers to achieve these properties in practice.
Keywords – Calm Technology, Peripheral Interaction, Practical Intelligence, Professional Practice, Routines, Ubiquitous Computing.
Relevance to Design Practice – Via design explorations with real-world professionals, this article surfaces new implications of designing human-technology interfaces for frontstage professionals (teachers, nurses, etc.). Rich design insights are formulated to inform future design practices of innovating technology-enhanced workplaces that could tacitly augment professionals’ repertoire and sensemaking without demanding their focal engagements.
Citation: An, P., Bakker, S., Cabral Guerra, M., van Bentum, J., & Eggen, B. (2023). Unremarkable augmentation in busy practitioners’ surrounding: Exploring peripheral interaction with teachers and nurses. International Journal of Design, 17(1), 71-90. https://doi.org/10.57698/v17i1.05
Received January 18, 2022; Accepted April 2, 2023; Published April 30, 2023.
Copyright: © 2023 An, Bakker, Cabral Guerra, van Bentum, & Eggen. Copyright for this article is retained by the authors, with first publication rights granted to the International Journal of Design. All journal content is open-accessed and allowed to be shared and adapted by the Creative Commons Attribution 4.0 International (CC BY 4.0) License.
*Corresponding Author: anpc@sustech.edu.cn
Pengcheng An is an assistant professor at the School of Design, Southern University of Science and Technology. He is a researcher at the intersection of Design and Human-Computer Interaction (HCI). An received his Ph.D. from the Department of Industrial Design, Eindhoven University of Technology (TU/e), and completed a postdoctoral program and teaching duties in the same department. He also served as a postdoctoral research fellow and sessional lecturer at the School of Computer Science at the University of Waterloo. His research explores the design and implications of human-artifact intelligent systems emerging in everyday contexts, including IoT environments (e.g., classrooms and workplaces) and embodied, multi-modal interface ecologies.
Saskia Bakker is senior data designer at Philips Experience Design, where she specializes in design for human-AI collaboration in the context of medical decision making. Before joining Philips, she was an assistant professor on peripheral interaction design in the Industrial Design Department of TU/e, the Netherlands. She held visiting lecturer positions at the UCL interaction center (London) and at the Human-Centered Ubiquitous Media group at LMU (Munich) and was chair of the steering committee of the ACM TEI conference series.
Miguel Cabral Guerra is a Group Design Manager at Catawiki. Before that, he worked at TU/e as a Hybrid Teacher and at Essense as a service Designer. He studied Industrial Design Engineering at The Hague University of Applied Sciences and completed his Master’s education in TU/e. He is a designer with experience creating user-friendly products, services, and systems. He is specialized in design methodologies, innovation strategies, and concept development with a strong emphasis on user involvement and user research.
Jesper van Bentum is a user-centered designer passionate about technology and solving complex user and societal problems. In 2017, he received his B.Sc. in Industrial Design Engineering from the HAN University of Applied Sciences. In 2020, after a one-year internship at the Department of Family Care Solutions at Philips Research in Eindhoven, he received his M.Sc. in Industrial Design from TU/e. Since April 2020, he has been working as a scientist in the Department of Personal and Preventive Care at Philips Research in Eindhoven. He focuses on investigating new methods to improve user experience using both quantitative and qualitative user research methodologies.
Berry Eggen is a full professor and director of research of the Industrial Design Department at TU/e, where he chairs the Future Everyday research group. Over the years, he has led design research groups in the areas of information ergonomics, multimodal interaction (including lighting and sound), and intelligent interfaces. His current research activities include interaction design research for the healthcare domain, and the exploration of decentralized systems approaches to ambient intelligence. Within the field of multimodal research, his interests include lighting and sound innovations in the home and public spaces and the exploration of light and sound as calm environmental information displays.
Introduction
The functioning of our society prominently relies on the dedication of numerous practitioners, such as teachers, nurses, clinicians, firefighters, production engineers, chefs, waiters/waitresses… to name a few. Practitioners are referred to as the specialists who consistently cope with a set of practical situations that require proficiency in practice (Schön, 1983). Typically, many practitioners’ day-to-day work contains a substantial part that can be called the frontstage practice (Eraut, 1995; Goffman, 1959). For instance, as Figure 1 illustrates, teachers’ frontstage practice is facilitating learners throughout a classroom session, and nurses engage their frontstage when caring for patients during a nursing shift. In frontstage practice, practitioners are typically faced with complex social or material surroundings, in which they need to proficiently respond to dynamic situations and orchestrate multiple threads of activities within limited time frames. These frontstage occasions thereby can be distinguishable from practitioners’ work in backstage: e.g., when teachers grade student work and prepare for future classroom sessions, or when nurses fill in a report about their patients’ status, or other jobs that they often do while sitting behind a desk continuously (as also shown in Figure 1). Similarly, many other practitioners have their respective frontstage practice: e.g., think about the typical practice contexts of clinicians, firefighters, chefs, etc.
Figure 1. Practitioners’ backstage and frontstage occasions, using the examples of nurses and teachers.
Practitioners’ frontstage occasions constitute an increasingly relevant yet highly challenging design context for creating human-technology interaction. On the one hand, unprecedentedly, practitioners are expected to develop new competencies of incorporating technologies, as increasingly necessary supports, in their frontstage routines (Konttila et al., 2019; Mouza, 2019). For instance, utilizing up-to-date digital equipment to enhance the quality and efficiency of classroom instruction has become an essential aim in teachers’ professionalization (Li et al., 2019; Mouza, 2019). And nowadays, nurses need to be trained more than ever to cope with the pervasive presence of medical technology devices, which bring crucial benefits to contemporary healthcare settings (Konttila et al., 2019; Locsin, 2017).
On the other hand, this continuing emergence of technologies, along with the trend of ubiquitous computing (Takayama, 2017; Weiser, 1993), also inevitably escalates the complexity in practitioners’ frontstage contexts. Emerging technologies, originally meant as supports, can unintentionally interfere with practitioners’ already busy routines, imposing excessive mental burden or even burnout on them. For example, despite the well-built digital infrastructures in modern classrooms, the routine use of computer technology for classroom teaching was less than expected (Vega & Robb, 2019). This was partially because using these technologies was experienced by teachers as time-consuming and cognitively demanding (Cuban et al., 2001; Li et al., 2019; Vega & Robb, 2019). For another example, despite the extensive training for coping with technology facilities in hospital settings, nurses were still found to encounter anxiety or burnout caused by an overwhelming quantity of clinical monitoring alarms from various medical devices on a daily basis (Braithwaite, 2008; Varisco et al., 2020).
The above examples reveal both societal relevance and urgency for design research to probe meaningful ways of seamlessly supporting practitioners’ frontstage practice. While a large body of research explored human-computer interaction in the backstage of practitioners (e.g., in office or desktop settings), practitioners’ frontstage scenarios remain relatively underexplored. And this has become the motivation of our design inquiry. A major challenge in designing for the frontstage concerns the way practitioners mobilize their attention: in the contexts resembling the backstage, interacting with the designed interface is often the users’ primary or only task (e.g., an interface of video editing or statistical analysis software). In frontstage, performing the designed interaction is mostly neither the only, nor the central part of practitioners’ ongoing activities. As a busy artifact ecology (Bødker & Klokmose, 2012), the frontstage is a dynamic compound of people (e.g., clients or coworkers), objects (e.g., tools, materials, environments…), and events, which require a practitioner’s attention (or responses) to multiple targets at the same time or in an interleaved manner.
However, most human-computer interfaces are currently designed to engage users in the center of human attention (Weiser & Brown, 1997), i.e., demanding focal attention during use. As illustrated in Figure 2, while interacting with an off-the-shelf digital device (e.g., a laptop computer, a tablet, a mobile phone, or a smartwatch…), a user often needs to remain their visual focus on its graphical user interface (GUI) to interpret information, and perform a series of precise operations, in order to fulfill a particular task. Such focal interactions do not always fit in the frontstage occasions; they could interfere with practitioners’ already complex and intensive routines and impose excessive mental load. This hinders practitioners from seamlessly incorporating supportive technologies in their ongoing activities.
Figure 2. Current off-the-shelf computing devices often require users’ continuous focal attention during use.
To enable the surrounding technologies to more seamlessly augment practitioners’ frontstage routines, this work explores an alternative style of human-computer interaction: i.e., peripheral interaction (Bakker et al., 2015). This burgeoning field has stemmed from the renowned vision of calm technology (Weiser & Brown, 1997), which argues that computing technologies should not only be able to engage users in the center of human attention; instead, certain human-technology interactions should be enabled to take place in the periphery of human attention. This way, technology interfaces could naturally fit in various levels along the human attention continuum (Bakker & Niemantsverdriet, 2016). And users could thereby avoid being overburdened in their daily lives where digital artifacts are increasingly emerging.
While prior design explorations have suggested the value of peripheral interaction in lowering users’ attentional threshold for technology use, this work further surfaces how peripheral interaction could unremarkably augment practitioners’ frontstage practice. We conducted field design explorations with school teachers and nurses at a neonatal intensive care unit (NICU) as they are often used as representative examples of frontstage practitioners (Eraut, 2000; Locsin, 2017). We implemented two types of peripheral interaction designs in their surroundings: peripheral information display (Matthews et al., 2004; Pousman & Stasko, 2006) and peripheral physical interaction (Bakker et al., 2015; Edge & Blackwell, 2009). The design cases presented in this paper were all evaluated with their respective target practitioner groups through field studies. These evaluations have been reported in previous publications (An et al., 2017, 2019; Cabral Guerra et al., 2019; van Bentum et al., 2019). By studying these cases together and connecting them to related theories on professional practice, we articulate two potential opportunities for peripheral interaction designs to tacitly augment practitioners’ frontstage routines (see Figure 3): i.e., enriching their repertoire of action (readily available actions to deliver on the spot), and enhancing their reflection-in-action (intuitive sensemaking of the unfolding situation). To inform future design and research, we generalize across these cases to extract intermediate-level knowledge (Höök & Löwgren, 2012), including a set of interaction design qualities and considerations. By elaborating on these design qualities and considerations, we intend to support future designers who aim to seamlessly integrate emerging technologies as unremarkable augmentation in future practitioners’ surroundings.
Figure 3. The overall framing of how peripheral interaction designs could augment practitioners’ frontstage routines.
Framing: Augmenting Practitioners’ Daily Routines with Peripheral Interaction Design
In this section, we first address related theories on professional practice to understand the essence of practitioners’ frontstage routines and then briefly review prior HCI explorations encompassing peripheral interaction. Based on the theoretical implications, we connect the two major categories of peripheral interaction designs to the key aspects of practitioners’ routines (as illustrated in Figure 3). These connections formulate our overall framing of how peripheral interaction could augment practitioners’ frontstage routines, which our design research cases will later contextualize.
Theoretical Background: Understanding Practitioners’ Daily Routines
Understanding and designing for routines has long been an important topic in human-computer interaction. Much prior research has aimed to articulate the nature of (domestic) everyday routines and thereby gather implications for design (Bakker et al., 2012; Tolmie et al., 2002; Wakkary et al., 2016). For example, Tolmie et al. (2020) famously put that routines are the glue of domestic life, which tacitly connects everyday activities as a coherent whole. However, few works set out to understand routines from the lens of professional practice. In this subsection, we address related theories on professional practice to understand the key aspects of practitioners’ frontstage routines. Namely, we articulate that to practitioners, routines are not only the glue of their frontstage activities but also the carrier of their practical intelligence (Sternberg & Wagner, 1986). Moreover, we elaborate on two key aspects of practitioners’ routines that guarantee their intelligent practice in frontstage occasions: repertoire of action and reflection-in-action (see Figure 3).
Frontstage Routines: The Carrier of Practitioners’ Practical Intelligence
In social sciences, many related studies can be found to examine professional practice that involve frontstage occasions: such as the work of teachers (Brante, 2009), nurses (Storesund & McMurray, 2009), clinicians (Laxmisan et al., 2007), jazz musicians (Peplowski, 1998), mediators (Balachandra et al., 2005)… to name but a few. As commonly reflected in these studies, practitioners’ frontstage practice is often characterized by intensity (i.e., rapid pace or limited time), complexity (i.e., complicated social and material settings), and uncertainty (i.e., dynamic, varying circumstances). Such frontstage situations might be highly demanding or daunting to a novice practitioner. However, a relatively experienced practitioner could survive by relying on their established routines. Established routines make professional practice fluent and skillful in its own context (Moorman & Miner, 1998) and enable practitioners to not only perform repetitive tasks faster, but also improvise more efficiently with non-routine elements in their practice (Eraut, 2000; Schön, 1983). From a situational perspective (Billett, 2001), professional routines can be seen as a set of tacit procedures or rules constituted in practitioners’ interactions with people, tools, or environmental factors in their work context. Adopting the view of tacit knowing (Polanyi, 1962), effective routines embody a practitioner’s skills, competences, or know-how that enable them to intelligently cope with practical situations. In this sense, practitioners’ routines can be seen as the carrier of their practical intelligence (Sternberg & Wagner, 1986), a type of practical comprehension (Polanyi, 1962) that is inherent to their effective, simultaneous doing and thinking in the frontstage, rather than prolonged, analytical cognition in the backstage (as Figure 1 illustrates).
Practitioners can progressively develop routines by practicing consistently in their contexts (Schön, 1983). As a practitioner’s practice becomes increasingly routinized, more particular components in their practice will become tacit (Polanyi, 1962) without requiring their focal deliberations; therefore, his/her performance becomes increasingly spontaneous and fluent. For example, experienced teachers can orchestrate multiple learning activities in parallel during a class session; and experienced nurses can well manage to care for various patients and monitor different medical devices simultaneously. In comparison, their novice counterparts might feel overwhelmed in similar situations. The establishment of professional routines could also be (partially) explained by the cognitive theories regarding skill acquisition (Dayan & Cohen, 2011) or habituation (Gardner et al., 2012; Wood & Rünger, 2016). To better understand how practitioners’ routines serve as the carrier of their practical intelligence, below we address two key aspects of effective frontstage routines which enable practitioners to act intelligently.
Repertoire of Action
Practitioners’ frontstage tasks are not simply repetitions; instead, they also involve a lot of improvisations with the unfolding situation (Schön, 1983). Effective routines do not only benefit practitioners in repetitive performance but also grant them a repertoire of action: i.e., a set of ready actions that they can quickly choose from in response to varying practical situations (Schön, 1983). This way, routines determine practitioners’ skillfulness in improvising with the uncertainty in their daily work (Balachandra et al., 2005; Moorman & Miner, 1998). Practitioners’ repertoires often involve interactions with their familiar tools. For example, a news photographer acts more skillfully with their own equipment than with someone else’s. This implies that repertoire could be developed and manifested between the practitioners and the artifacts in the practice context (Patel, 2008). When tools are successfully integrated into practitioners’ repertoire, they become ready-to-hand (Heidegger, 2005), meaning practitioners do not have to pay focal attention when interacting with them. Instead, practitioners could focus beyond them and focus more on the higher-level objective that they use a tool to achieve (Polanyi, 1962). Many field studies have observed examples, in which professionals such as airport operators (Billett, 2001; Suchman et al., 1999), Xerox technicians (Orr, 2016), or artisans (Cook & Yanow, 1993) achieve their workaday practice relying on their repertoire developed with certain artifacts in the practice context.
Besides tools, practitioners can also tacitly rely on their physical environment as part of their action repertoire. Rich examples can be found in a body of research regarding the theory of distributed cognition (Hollan et al., 2000; Hutchins, 2005). Through studying activities in different practice contexts (e.g., a supermarket, workshop, or kitchen), related research revealed how people tacitly utilize environment as external representations (Hollan et al., 2000; Hutchins, 2005; Risko & Gilbert, 2016) to reduce the time and mental resources needed for accomplishing practical goals. Similarly, Kirsh (1995) reported that in several environments (e.g., an office), practitioners naturally ease their task performance by hiding/highlighting affordances of things in the surrounding (e.g., covering things that are not to be used for now), or creating piles arranged in certain spatial orders. As observed and generalized by Hollan et al. (2000), many types of workflow (routines) embody a coupled, inseparable system between practitioners and their surroundings, in which the surrounding often serves as practitioners’ cognitive ally. To summarize, as a key aspect of professional routines, the repertoire of action enables practitioners to skillfully cope with varying practical situations by tacitly relying on their tools and surrounding. This implies the value of designing technologies to fit in and enrich their repertoire and thereby expand their skill set in the frontstage.
Reflection-in-Action
While delivering actions spontaneously in the frontstage, practitioners also need to rapidly make sense of the latest changes and unique elements occurring in the unfolding situation in order to optimize their performance on the spot (Schön, 1983). Schön (1983) has initially referred to this sense-making process as reflection-in-action, emphasizing its nature of being intuitive and momentary, as well as occurring in action. Reflection-in-action is another key aspect of effective professional routines: although itself also needs to be routinized (via extensive practicing over time), once habituated, it can help practitioners avoid over-routinized performance or rigidity in practice and stay responsive to the practical situations (Schön, 1983; Yanow & Tsoukas, 2009). In other words, it determines practitioners’ responsiveness and reflectiveness in practice. Over time, it can also serve as a source of renewal (Schön, 1983) that enables practitioners’ routines to cumulatively evolve. Therefore, reflection-in-action has been widely considered as a challenging yet important competence to be acquired for many professions which feature simultaneous doing and thinking (Eraut, 2000; Schön, 1987; Yanow & Tsoukas, 2009).
In many respects, reflection-in-action can be distinguished from reflection-on-action (Schön, 1983): i.e., the common approach of reflection that entails a prolonged, analytical thinking process taking place away from action, such as in a backstage occasion (Eraut, 1995). First and foremost, reflection-on-action is aimed for impacts on future episodes of practice, whereas reflection-in-action primarily intends to benefit practitioners’ ongoing performance or their understanding of the situation at hand. For example, during classroom teaching, a teacher may intuitively sense the atmosphere in the classroom to feel whether his/her explanation has been clear to the students, or whether there are individuals who need extra attention. Or, in a baseball game, a pitcher may tacitly deliberate on his/her ongoing performance of pitching and accordingly modify his/her upcoming actions in real time. Another unique feature of reflection-in-action is that it is often somewhat conscious but not highly elaborate (Eraut, 1995; Schön, 1983). Due to the limited attentional resources and short time frame to think about their performance in the frontstage, practitioners need to rely on their intuitive feel for things or tacit know-how (Schön, 1983, 1987), so that their sensemaking would not interrupt their flow of practice. Moreover, similar to the repertoire of action, reflection-in-action could also be mediated by (tangible or intangible) artifacts such as practitioners’ surrounding tools or materials (Schön, 1983; Yanow & Tsoukas, 2009). This implies the design opportunity to use technologies to augment practitioners’ intuitive sensemaking and enhance their reflection-in-action during frontstage performance.
To summarize, established routines are the carrier of practitioners’ practical intelligence, which enables them to cope with the complex, intensive, and highly dynamic situations in the frontstage. Therefore, at the very least, technologies to support frontstage practitioners should be compatible with their routines, not interfering with their existing workflows or overburdening their minds. Moreover, to extend practitioners’ frontstage intelligence, technologies could aim to contribute to the two key elements in their routines: repertoire of action and reflection-in-action. In the next subsection, we connect these implications with HCI design research.
Related Design Research on Peripheral Interaction
The burgeoning field of peripheral interaction (Bakker et al., 2015; Edge & Blackwell, 2009; Hausen, 2014) explores the design of human-computer interaction that can take place in the periphery of users’ attention. This idea originated from the calm technology vision (Weiser & Brown, 1997). The vision is proposed in light of the technological trend that computing devices are becoming increasingly ubiquitous in our daily surroundings. In contrast, each device often means to engage users in the center of their attention. As a risk, more and more of our attentional resources will be occupied in interacting with these devices, and we might be overburdened in our daily lives. To avoid this undesirable situation, Weiser and Brown argue that computing devices should not be designed only to engage users in the center of their attention. Instead, many devices could be enabled also to support users in the periphery of their attention, and shift between the center and the periphery as desired (Weiser & Brown, 1997). This proposition is based on the human capability of utilizing the periphery of attention, commonly manifested in daily life. For example, while working in a room, we could effortlessly notice the change of weather through the change of incoming daylight or via the raindrops drumming on the window without shifting our focus away from the work. In a similar way, computing systems should also leverage the periphery of users’ attention to unobtrusively facilitate their daily tasks without disrupting their main focus. Following this vision, two major types of peripheral interaction designs have been explored: peripheral information display and peripheral physical interaction.
Peripheral Information Display
Many design cases have been focused on presenting information in the periphery of users’ attention. For example, Ishii et al. (1998) built AmbientROOM as a case to illustrate how information can be subtly displayed by ambient light, sound, and object movement in an office setting. Subsequently, a stream of research continued the exploration under various terms, including peripheral display (Matthews et al., 2007), ambient information system (Pousman & Stasko, 2006), awareness systems (Markopoulos & Mackay, 2009), and information decoration (Eggen & Van Mensvoort, 2009). As one of the earliest design instances, Dangling String, presented in the paper by Weiser & Brown (1997), is a network cable driven by a motor; it spins according to the busyness of the network communication within the Ethernet cable, generating motion and sound to subtly provide awareness to people about network activities in the office. For another example, Data Fountain, presented by Eggen and Mensvoort (2009), is a display system using a full-sized fountain; its three water streams represent the current value of the dollar, euro, and yen, unobtrusively communicating this information to people who are interested to know it. Many more studies have explored this direction with various types of media (Colley et al., 2019; Müller et al., 2019) and devices (Reich & Dunne, 2016; Xiao & Benko, 2016). As related to the context of nursing in the intensive care unit, several prior design cases have been presented, which shared similar design motivation as calm technology, including CareTunes (Bogers, 2017), UltiMo (Pinch Studio, 2018), and Doplor (Redert, 2018). In the classroom context, a variety of peripheral information displays have been reported, such as Lantern (Alavi & Dillenbourg, 2012), CawClock (Bakker et al., 2015), and Lernanto (van Alphen & Bakker, 2016), to name but a few.
Peripheral information display conveys low-resolution, unobtrusive messages and ultimately aims for that users absorb its information without consciously performing the act of reading (Weiser, 1999). As illustrated in Figure 3, we argue that this paradigm of display has the potential to support professional practitioners in their process of reflection-in-action. As mentioned earlier, practitioners need to rapidly make sense of the situation at hand during their intensive frontstage routines. Given that their attention is often already occupied by multiple streams of information in the context, a focal display might compete with their existing focus and overburden their minds. If extra supportive information can be conveyed to them without requiring their focal reading, they would be more likely to incorporate this information into their intuitive and momentary sensemaking processes during action. Later in this paper, two design cases will be discussed to demonstrate how peripheral information displays could be designed to enhance practitioners’ reflection-in-action.
Peripheral Physical Interaction
While most early design instances inspired by calm technology have been peripheral information display, more recent explorations such as by (Bakker et al., 2015; Edge & Blackwell, 2009; Olivera et al., 2011) have suggested that not only information perception but also physical interactions (interactions involving bodily movement) with computing artifacts can be carried out in the periphery of users’ attention. Bakker and Niemantsverdriet (2016) described peripheral physical interaction as users’ imprecise yet intentional manipulations of the computing devices, taking place in the intermediate region of users’ attentional continuum: i.e., somewhere between the conscious and subconscious levels. To inspire design with real-world examples, Bakker et al. (2012) investigated the peripheral (bodily) actions that were prevalently performed in our daily lives. For example, when we are reading a newspaper, we can easily reach our coffee mug and take a sip without disrupting our focus on the newspaper.
Designs of peripheral physical interaction could utilize various interaction styles, such as tangible interaction (Bakker et al., 2013; Edge & Blackwell, 2009; Olivera et al., 2011), freehand gesture (Hausen et al., 2013), or lower-limb or full-body interaction (Matthies et al., 2013; Probst, 2016). For instance, Edge and Blackwell (2009) presented a table-top tangible interface to enable office workers to update their task progress by manipulating physical tokens without focally engaging them. Bakker et al. (2013) designed a mobile tangible tool for teachers to control the ambient lamp of each student in the classroom to convey quiet and nonverbal messages to them. Probst (2016) introduced a set of peripheral interfaces for users to perform basic controls over desktop computer applications through simple bodily input (e.g., moving a foot or tilting the chair). Moreover, peripheral physical interaction is suggested to intersect with several concepts in HCI which have similarly concerned interactions below the level of fully focused attention or as users’ secondary tasks: such as microinteraction (Ashbrook, 2010; Wolf et al., 2011), implicit interaction (Serim & Jacucci, 2019), casual interaction (Pohl & Murray-Smith, 2013) or reflexive interaction (Matthies et al., 2019). Compared to interactions relying on focal and precise manipulations, peripheral physical interaction seems to have more potential to become ready-to-hand (Heidegger, 2005) in practitioners’ busy frontstage. Once the operation of supportive technologies could become easy-to-initiate on the spot, these technologies are more likely to be integrated into practitioners’ existing repertoire for coping with various emerging situations in the frontstage. Therefore, we argue that peripheral physical interaction could be designed to meaningfully enrich practitioners’ repertoire of action, which are to be concretized later by our cases studies.
Design Explorations with School Teachers and NICU Nurses
In this section, we present four Research-through-Design (Zimmerman et al., 2007) cases, which have been aimed to understand and facilitate the front-stage routines of two different groups of professionals: school teachers and hospital nurses (at a NICU). We have included the two professions since teachers and nurses have long been studied as exemplar types of practitioners whose day-to-day work embodied many core characteristics in professional practice (Eraut, 2000; Evetts, 2014; Schön, 1983). All design cases underwent a field evaluation with the relevant practitioners, which have been reported in peer-reviewed publications (An et al., 2017, 2019; Cabral Guerra et al., 2019; van Bentum et al., 2019), providing detailed empirical findings about the proposed designs. The four cases have covered both types of peripheral interaction design for each professional context (see Figure 8). As a result, analyzing these design cases together enables us to generalize beyond a single professional context or a particular application domain, and therefore extract intermediate-level design knowledge (Höök & Löwgren, 2012) that can benefit a relatively broad range of design practice. We now review these cases below for contextualizing our earlier framing (Figure 3) and laying the ground for knowledge generalization.
Design Exploration 1: FeetForward
FeetForward (An et al., 2017) is a set of foot-based shortcuts designed to assist teachers while using interactive whiteboards (IWB) during lectures. The system, as shown in Figure 4, consists of a set of pedals that can be customized to perform specific tasks, allowing teachers to easily access and control their computer or IWB with simple foot tapping. This method of interaction was chosen based on classroom observations and co-design workshops with teachers, who reported that their hands were often occupied with other materials and they did not want to use additional devices, such as a remote or wearable interface, which would require extra mental effort to remember to use. The pedals were placed below the interactive whiteboard to minimize disruptions to the flow of teaching. For example, during lecturing, a teacher may need to have two windows displayed on the IWB at the same time. To do so, previously, the teacher would need to resize and reposition each window using the stylus, or walk to his/her computer to use the mouse or keyboard. Either way, the teacher would need to execute a series of precise operations that would require focal engagement and cause a disruption to the ongoing flow of lecturing. Using the pedals as shortcuts by simply foot tapping, a teacher can easily resize and reposition windows on the IWB, or access frequently used online resources, without the need for precise actions or walking to their computer.
Figure 4. FeetForward's prototype and field deployment: (a)(b) it is a foot-based and customizable peripheral interface; (c)(d) we deployed the system in classrooms to support teachers’ use of interactive whiteboards.
An evaluation of FeetForward was conducted over a five-week period in actual secondary school classrooms (An et al., 2017). The self-report behavior automaticity data from teachers suggested that during the period, the system demonstrated the potential to shift technology-related tasks to the periphery of a teacher’s attention. The participating teachers generally felt that using the pedals saved them time and effort, and on a number of occasions, prevented disruptions to their teaching flow (e.g., having to walk to their computer or performing precise sequential operations). During the five-week study, the teachers explored different customized functionalities to establish their preferred way of using FeetFoward. In some cases, the system was found to encourage teachers to use digital resources more frequently and in a more improvisational manner. These results suggest that peripheral physical interaction can support teachers to more easily incorporate pedagogically-relevant technology into their front-stage repertoire, thus extending their digital competency.
Design Exploration 2: ClassBeacons
ClassBeacons (An et al., 2019) is a set of wirelessly connected lamps placed on students’ desks, serving as peripheral (ambient) information display (see Figure 5). These lamps calmly depict how the teacher has been allocating physical proximity and interpersonal interaction to students around the classroom: i.e., classroom proxemics (Martinez-Maldonado et al., 2020). As Figure 5 illustrates, each ambient lamp slowly and subtly changes color from yellow to green as the teacher cumulatively spends time around the lamp, supporting adjacent students. Such spatial information enables teachers to intuitively reflect on their whereabouts, and actively plan their upcoming allocation of time and attention to individual students. The yellow-green spectrum was chosen for its subtlety and neutrality (e.g., less judgmental than colors like red), to make the display unobtrusive and open to context-dependent interpretations. Classroom proxemics is a tacit yet important aspect of teacher competence (Lim et al., 2012; Martinez-Maldonado et al., 2020), which ought to be reflectively managed, to positively influence learners’ behaviors (Gunter et al., 1995; Shores et al., 2017). However, in practice, it is often rather challenging for teachers to consciously manage their unfolding proxemics, simply because they do not have extra time or mental resources to keep track of their whereabouts in the midst of teaching. ClassBeacons is therefore designed to situate the proxemics data in the physical space, as peripheral environmental cues, to offload teachers’ reflective processes.
Figure 5. ClassBeacons' prototype and field deployment: (a) the system supports teachers’ reflection-in-action on how they have divided time and attention over students in the classroom; (b) the display is based on teachers’ real-time positioning data; (c)(d) the system deployed in a classroom; (e) each lamp depicts how long the teacher has been around it cumulatively by subtly changing from yellow to green.
ClassBeacons was evaluated with a group of secondary school teachers in their real-life classroom sessions (An et al., 2019). The results showed how this peripheral information display blended into the teachers’ classroom routines, and triggered their meaningful reflections in action. Namely, according to the teachers, the ambient lamps were experienced as a natural part of the surrounding, and they appreciated being able to receive the information simultaneously when they were observing the students—a crucial and frequent component in their existing routines. Teachers’ encountering with the display were described as quick, effortless and intuitive, which could occur serendipitously, as well as intentionally. In a diverse set of examples, the system benefited their sensemaking on the fly: including helping them to confirm their ongoing proxemic performance, notice the unwanted patterns in their current teaching behaviors, or responsively modify their interactions with students (An et al., 2019). These evidences illustrate how a peripheral information display could seamlessly embed relevant data or information in teachers’ surrounding, and thereby unobtrusively enhance their reflectiveness within front-stage practice.
Design Exploration 3: Beepless Pedal
Beepless Pedal (Cabral Guerra et al., 2019) is a foot switch for nurses to silence (non-actionable) alarms generated by a neonatal monitor, while their hands are occupied by activities inside the neonatal incubator (see Figure 6). Working in a neonatal intensive care unit (NICU), nurses have to encounter a great number of clinical alarms on a daily basis, in which a large proportion consists of non-actionable or false alarms. Yet still they need to manually silence a non-actionable alarm from a neonatal monitor, as a move of confirming that they have perceived the alarm and the situation, otherwise shortly (in 30s), the alarm will be sent to the handheld devices of all nurses so that someone could eventually check and confirm the situation. To silence an alarm, the nurse could press a touchscreen on patient vital sign display, or press a button on the remote control of a neonatal monitor (see Figure 6 a), both located outside the incubator. However, a NICU nurse’s daily routines largely involve manual activities inside incubators, such as changing diapers, feeding, taking blood sample, replacing the patients’ ventilator masks, etc. In these situations, the nurse’s hands are occupied, making it impossible to respond to the alarm without disrupting their task at hand. Instead, they may have to: pause the task and take their hands out of the incubator, press the button for silencing alarms, sanitize their hands, and resume the task inside the incubator. Beepless Padal is hence designed to enable nurses to effortlessly silence an alarm without having to stop their current manual activity, and go through a tedious process of task interruption-resumption (Figure 6).
Figure 6. Beepless Pedal's prototype and deployment in a NICU: (a) the alarm silencing button on the remote control (often) attached to the patient monitor, covered by the additional yellow part came from the Beepless Pedal prototype; (b) & (c): the Beepless Pedal attached to the lower part of the incubator; (d) a nurse working at the incubator with Beepless Pedal next to her foot.
The working prototype of Beepless Pedal was implemented in a hospital for a field test involving several NICU nurses and their neonatal patients (Cabral Guerra et al., 2019). According to the observation, Beepless Pedal indeed was mostly used when the nurses were sitting beside the incubator and already engaged in patient handling. In addition, they also used the design in other situations when their hands were occupied or sanitized. And when they were on the move, or operating the incubator monitor, they would still press the button attached on the monitor to silence an alarm. As experienced by the nurse, Beepless Pedal was useful since it provided a meaningful additional approach for silencing alarms, making it much less demanding on occasions where their primary activity is inside the incubator. As a result, they could have uninterrupted care for the neonatal patients. In other words, Beepless Pedal expanded the nurses’ skill set of efficiently reacting to clinical alarms in different situations, which implies how peripheral physical interaction could enrich nurses’ repertoire and increase their proficiency in the front stage.
Design Study 4: CheckMates
CheckMates (van Bentum et al., 2019) is a peripheral information display intended to ease NICU nurses’ thoughtful and proactive planning in their dynamic task flows (i.e., reflection-in-action). The CheckMates system consists of multiple in situ light-objects (Figure 7). Each light-object calmly depicts the amount of supply left in a medical device around an incubator. This information can support nurses to flexibly decide the timing of refilling or replacing the supply of each medical device, so that this recurrent task could be carried out proactively, with minimal conflict with many other (time-sensitive) tasks in their intensive workflow. As the implementation example in Figure 7 shows, the light-objects are placed about an incubator to indicate: the remaining water of the incubator heating and humidifying unit, the remaining water of the patient ventilator, and the remaining medication within the medical syringe infusion pump. One hour before a device runs out of the supply (estimated upon its current consumption rate), its light-object begins to change continuously from green to blue: the more it turns to blue, the less supply remains. Several minutes before the device’s inherent supply alarm is triggered, the light-object will turn complete blue and start breath-like pulsing.
Figure 7. CheckMates‘ system and deployment in a NICU: (a) a light object is attached to a selected machine, such as a water container of the incubator; (b) the color change spectrum of each light-object; (c) multiple light objects implemented around the incubator (within yellow circles).
Through the in-situ light-objects, CheckMates offers an extra one-hour ambient notification timeframe, in addition to the inherent alarms of these medical devices (e.g., with attention-demanding sounds and red lights triggered 15 min before the supply runs off). This gives nurses more room for planning ahead or improvising in situ, to dynamically cope with supply refilling with other (more time-critical) tasks, and avoid the unwanted overwhelming moments in which multiple alarms may be triggered to call for their immediate reactions. Moreover, CheckMates enables the supply status of medical devices to be more visible from distance, in nurses’ highly nomadic workflow. Although these devices have their existing supply indicators, they require nurses to get close and check focally (e.g., the printed water scale of a water tank in Figure 7a). Plus, NICU patient room is usually darker than a regular room, making the reading of these indicators even more effort-taking.
The CheckMates system was evaluated in a neonatal intensive care unit (NICU) setting (van Bentum et al., 2019). The participating nurses were in their familiar work environment while performing their daily routines. Instead of testing with real neonatal patients in incubators, neonatal patient mannequins were utilized. As experienced by the nurses, the CheckMates system offloaded their tasks of monitoring the supply status of medical devices on the fly. A number of examples revealed how the system could enable their proactive and flexible planning during their ongoing task flow, which is a crucial aspect of their reflection-in-action. And this was appreciated by the nurses, since it would help them better reduce preventable alarms from the medical devices in the surrounding (van Bentum et al., 2019), which could cause nurses’ alarm fatigue (Cvach, 2012) or burnout (Braithwaite, 2008), and hinder the development of their patients (Lai & Bearer, 2008).
Generalization and Discussion: Relevant Interaction Design Properties and Considerations
In this section, we extract intermediate-level knowledge (Höök & Löwgren, 2012), which are often interaction design properties or qualities that are generalized beyond single design artifact (instance), and can inform future design practice in related contexts (Höök & Löwgren, 2012; Stolterman & Wiberg, 2010; Wakkary et al., 2016). The four design cases studied in this paper were evaluated in real-life practice environments by the target practitioner groups, with the empirical results published respectively (An et al., 2017, 2019; Cabral Guerra et al., 2019; van Bentum et al., 2019). The four cases encompassed both types of peripheral interaction (information display and physical interaction), as well as two different, yet representative contexts of frontstage practice: school teaching and hospital nursing (Eraut, 2000; Evetts, 2014; Schön, 1983). Following the approach demonstrated by (Höök et al., 2016; Wakkary et al., 2016), our case studies aimed at distilling essential interaction design properties which commonly manifested across the cases and played a crucial role in enabling the peripheral interaction to blend in and benefit pratitioners’ frontstage routines. Initially, the four cases were subjected to a constant comparative analysis (Glaser, 1965) led by the first author. After that, we have been continually discussing and evolving our generalization over one year, in consultancy with several experienced design researchers, until the final outcome was reached. Based on the empirical data gathered from the field evaluations of our design cases, we now present a detailed and contextualized account of the two properties we have extracted, as well as six actionable considerations that can help designers and design researchers to achieve these properties in practice (illustrated by Figure 8).
Figure 8. An illustration of our generalization from the four design instances (represented by four icons) to the design properties and considerations.
Property 1: Designed Interaction Being Subsidiary to the Main Practice
We formulate the first design property of peripheral interaction for front-stage practice as: the designed interaction being subsidiary to the main practice. Subsidiary means less important than, but related or supplementary to. On the one hand, being subsidiary means the designed peripheral interaction is beneficial but not critical to practitioners’ main activity. Primarily, peripheral interaction should be aimed for making practitioners’ secondary tasks less taxing; so that they can have more cognitive (or motor-sensorial) resources assigned to the essential or critical parts of the current practice. On the other hand, being subsidiary also means that the designed peripheral interaction can be performed relying on practitioners’ subsidiary awareness (Polanyi, 1962), meaning the interaction is treated as a congruent component of the whole ongoing practice rather than a separate task. That being said, the designed peripheral interaction needs to be seamlessly woven into the ongoing main practice without disrupting the flow of preexisting routines or overburdening practitioners’ thoughts. Practically, there are three considerations to help achieve this design property:
- Design consideration 1.1: support tasks that are beneficial but not critical to the main practice.
- Design consideration 1.2: enable momentary, effortless engagements of practitioners.
- Design consideration 1.3: provide support in the proper places, at the proper moments, using proper modalities.
We now discuss these considerations in detail with concrete examples:
Consideration 1.1: Support Tasks that are Beneficial but not Critical to the Main Practice
With this design consideration, designers need first to identify what tasks in the front-stage are suitable to be supported by peripheral interaction. In general, we argue that peripheral interaction is especially valuable for supporting practitioners’ secondary tasks, or sub-tasks, which can benefit their current main practice but should not occupy much attention. Conversely, tasks which are of essential importance and often demand high engagement, cautiousness, or prolonged analytical thinking may not be suitable to be accomplished through peripheral interaction.
Take the two cases from classroom teaching as examples: the essential activities for teachers are pedagogical interactions with students (e.g., lecturing or close supervising). Both designs supported teachers’ tasks which are meaningful but secondary to their essential activities: interacting with technological tools in lecturing (FeetForward) and monitoring teacher-proximity distribution in supervising (ClassBeacons). On the one hand, these tasks are not critical: the performance of these tasks is not the most significant determinant of teachers’ success compared to tasks such as observing or verbally interacting with students. Nonetheless, getting supported in these secondary tasks did meaningfully contribute to teachers’ performance of essential activities: enabling more flexible use of digital resources for lecturing (FeetForward) and supporting teachers’ conscious allocation of time and attention for supervising (ClassBeacons).
Similarly, the two cases from NICU supported nurses’ beneficial-but-not-critical secondary tasks: silencing non-actionable alarms (Beepless Pedal) and proactively managing the timing of refilling medical devices (CheckMates). Although somewhat relevant, these secondary tasks are not as critical as the tasks that directly involve caring for patients (e.g., collecting blood from neonatal patients through heel prick, which requires careful and precise moves) or responding to changes in patients’ vital signs. And time-wise, a delay or temporary suspension of these non-critical tasks is less likely to cause severe consequences when compared with tasks of higher time-criticalness: e.g., feeding the neonatal patient, which has a strict schedule, or coping with patients’ desaturation or bradycardia, which are often prioritized over other tasks. Nonetheless, by supporting those relevant yet non-critical tasks, the designs could indeed benefit nurses’ more critical tasks: preventing disruption to their ongoing operations inside the incubator (Beepless Pedal) and reducing nurses’ juggling moments where supply refilling tasks compete with high-priority or time-critical tasks for nurses’ attention (CheckMates).
This consideration embodied in front-stage contexts echoes the earlier explorations on peripheral interaction in general. Prior research on both information display (Matthews et al., 2007; Pousman & Stasko, 2006) and physical interaction (Bakker et al., 2015) similarly argued that peripheral interaction should be designed for relevant-but-not-critical tasks, and not any human-computer interaction is suitable to be carried out in the periphery of attention. Furthering the earlier research, we contextualize and differentiate this consideration in terms of front-stage practice. As different from generic users or users in domestic or leisure contexts, practitioners already have explicit and urgent high-level goals to pursue (e.g., teaching students or nursing patients), which go beyond simply using the designed interface. Hence, the objective of design should also go beyond merely facilitating users’ performance of the designed interaction per se, or, simply promoting them to do multitasking. Instead, the design objective should be to ecologically benefit their practice as a whole, in which the designed interaction only serves as a congruent particular (Polanyi, 1962).
Consideration 1.2: Enable Momentary, Effortless Engagements of Practitioners
As revealed in the cases, practitioners often had limited time and attention available for using supportive technologies during their busy and intensive workflow. The property of being subsidiary means the designed interaction is able to be performed during practitioners’ main activities, relying on their subsidiary awareness rather than focal awareness (Polanyi, 1962). This can be achieved when the interaction only requires practitioners’ momentary, effortless engagements. Such engagement could be their interpretation of displayed information or execution of a sensorimotor sequence, or both.
The two designs of peripheral information display were both appreciated by their target practitioners for being glanceable and intuitive. In the case of ClassBeacons, the teachers felt that they could process the displayed information in a couple of seconds since it only requires glancing rather than reading (An et al., 2019). And when they were busy interacting with or observing the students, they could somewhat attune to the display without feeling overburdened. Similarly, the NICU nurses already had many information streams to keep track of. They appreciated that CheckMates helped shift some observational tasks from requiring closely reading an indicator to relying on a glance from a distance. Correspondingly, the two designs of peripheral physical interaction (FeetForward and the Beepless Pedal) supported quick and effortless shortcuts to simplify practitioners’ certain sensorimotor sequences into microinteraction (Ashbrook, 2010).
A series of concepts coined by prior research on peripheral interaction could also help achieve momentary, effortless engagements. Regarding information display, the notification level (Matthews et al., 2007; Pousman & Stasko, 2006) should be carefully considered to balance the unobtrusiveness and noticeability of designed peripheral information. Moreover, information capacity (i.e., how many pieces of information to be displayed), and representation fidelity (i.e., how detailed each information piece is represented) (Pousman & Stasko, 2006) are also relevant parameters to be decided in light of the context to balance informativeness and intuitiveness. These trade-offs are especially salient in practitioners’ front-stage. In regard to physical interaction, microinteraction (Ashbrook, 2010; Wolf et al., 2011) proposes using minimalist actions to achieve an interaction in order to minimize its interruption to users. Whacking gesture (Hudson et al., 2010) proposes inexact and inattentive interaction: e.g., whacking the smart phone in your pocket to quickly mute an incoming phone call without taking the phone out. Bakker et al. (2015) argued for easy-to-initiate and easy-to-discard interaction to make the interaction ready-to-hand (Heidegger, 2005). These suggestions would inspire designers to create different solutions to peripheral physical interaction. Additionally, from the perspective of motor skill acquisition (Dayan & Cohen, 2011) or behavior automaticity (Gardner et al., 2012; Wood & Rünger, 2016), shorter and more effortless interactions (e.g., interactions consisting of fewer sensorimotor units or require less accuracy) are also easier to learn and to practice and hence more likely to become a spontaneous component in practitioners’ routines.
Consideration 1.3: Provide Support in the Proper Places, at Proper Moments, Using Proper Modalities
Another implication of being subsidiary is that the designed interaction could seamlessly fit in the existing artifact ecology (Bødker & Klokmose, 2012) of the frontstage practice: i.e., the compound of events, artifacts, and environmental factors (e.g., locations or spatial arrangement of things) in practitioners’ surroundings. To achieve this, designers need to adopt an ecological lens and contextually examine the frontstage practice in which the designed interaction is situated. We argue that such an ecological understanding can help leverage the powerful idea of distribution, by which we mean: appropriately distributing practitioners’ use of supportive technology over the proper locations in their surroundings, to the proper moments of time in their workflows, as well as to the proper modalities of their cognitive or sensorimotor resources.
As revealed in the design cases, situating a piece of information in its own relevant location could offload practitioners’ mental burden of interpreting the information. As indicated by the participating teachers, the spatially distributed display of ClassBeacons was intuitive at a glance: they did not have to think about which piece of information was related to which location of the classroom or which group of students (An et al., 2019). Conversely, if they had used a screen-based visualization (e.g., a Heatmap), they might have to make extra efforts to read the screen, and map between the screen representation and the reality. In the case of CheckMates, the light objects co-located with their associated medical devices, thus requiring minimal effort for nurses to comprehend which medical device needs refill. Therefore, it is important to note that the idea of distribution does not only concern the ubiquity (Weiser, 1993) of computing artifacts, but also their ubiety—the right technology at the right place (Buxton, 2018). Another aspect of designing an appropriate distribution of information is ensuring that practitioners receive the information at the right time. For ClassBeacons, the teachers’ interactions with the display were naturally distributed to times when they were with students or moving around the classroom, eliminating the need for them to actively plan when to access the information display. In the case of CheckMates, the nurses had regular schedules for recording vital data from patient monitors in different rooms. During these moments, it was natural for them to also notice the light objects from the corridor and be aware of when to refill a device (van Bentum et al., 2019). These spatial and temporal properties enabled the two display systems to naturally fit in and augment practitioners’ surroundings, turning the whole environment into a tacit spatial visualization.
The two cases of physical interaction also embodied the idea of distributing practitioners’ technology-related tasks to the right places and moments in time. These systems served as decoupled interfaces (Hausen, 2014) from practitioners’ pre-existing devices. FeetForward allowed teachers to customize shortcuts that were decoupled from their computers, enabling them to perform certain functions more easily in front of the interactive whiteboard while lecturing. And these pedals allowed them to access functionalities that were previously hidden behind their computers and required precise operations and deliberate effort to remember to use. Beepless Pedal provided a decoupled control of the alarm silencing button on the patient monitor, complementing the button when nurses were sitting in the incubator and caring for patients (see Figure 6). As found in the field evaluations, both peripheral physical interaction systems have achieved spatial and temporal ubiety, making the secondary tasks of practitioners less demanding and more convenient to operate. This resonates with the theories of cognitive offloading (Risko & Gilbert, 2016) and distributed cognition (Hollan et al., 2000; Hutchins, 1995), which have used ample real-world examples (e.g., of supermarket employees, chefs, or ship navigators…) to unveil how people ally with their surrounding artifacts and environmental factors to tacitly distribute their cognition in action and simplify their task at hand. However, such processes can often be rather implicit and unremarkable, and hence not that easy to be noticed and leveraged in design practice, unless designers establish intimate comprehension about practitioners’ existing surroundings and workflows.
Beyond spatiality and temporality, we further argue there is another type of distribution that is worth considering in design: the distribution across modalities, i.e., the different sensorial or motor resources of practitioners. According to theories of multitasking (Kahneman, 1973; Salvucci et al., 2009; Wickens, 2002), in order to be performed along with practitioners’ other ongoing tasks, the designed peripheral interaction needs to avoid interfering with those tasks by competing for resources from a same modality. For instance, two tasks that both require the focal visual channel are likely to have resource interference and thus difficult to be performed together. Conversely, if one task requires the focal visual channel and the other requires the ambient visual channel, they are relatively easier to co-occur (Wickens, 2002). This may explain why ClassBeacons naturally blended into the teachers’ classroom observation: while they used focal visual channel to observe the facial expressions or body languages of students, they could meanwhile sense the color of the lamps through the ambient visual channel (An et al., 2019). Some HCI studies also explored distributing peripheral information to the auditory channel: using sonification or ambient soundscape to augment practitioners’ awareness without interrupting their main tasks relying on focal vision, e.g., knitting (Smith et al., 2015) or skiing (Hasegawa et al., 2012)).
In addition to sensory channels, designers can also consider the distribution of modalities across practitioners’ motor resources. For example, the Beepless Pedal distributes the task of silencing alarms to the foot, avoiding disruption to the primary tasks being performed with the hands. FeetForward was found to be meaningful by the teachers because their hands were often occupied by materials such as test tubes, textbooks, or styluses (An et al., 2017); and they were concerned about having to remember to use new handheld or wearable devices in their already busy routines. In both design cases, turning secondary tasks into imprecise, coarse, and simple foot interactions was found to be an effective way to support practitioners and prevent disruption to the main focus and task flow. Such inter-limb distribution is commonly used by practitioners such as drivers, tailors, or potters (Velloso et al., 2015). Taking this as an example, future designs could consider a broader range of inter-limb distributions based on the specific contextual factors. For instance, bi-manual activities such as typing or cutting papers with both hands, are also a common way of distributing motor resources in everyday practice, and have inspired studies on peripheral physical interaction leveraging bi-manual styles (Edge & Blackwell, 2009; Hausen, 2014).
Property 2: Designed Interaction Being Open to Practical Knowing
We formulate the second design property of peripheral interaction for frontstage practice as: the designed interaction being open to practical knowing. Practical knowing (Schön, 1987) refers to practitioners’ personal, experiential and contextual understandings about their own practice, which is established over time, being immersed in real-world practical situations. It can include the repertoire, or tacit know-how (Amin & Roberts, 2008; Polanyi, 1962) that practitioners rely on to perform tasks or interact with surroundings; it can also include the personal interpretive systems (Schön, 1983; Yanow & Tsoukas, 2009) they use to account for their practical situations on the fly (e.g., what are the relevant problems to address right now?) or assess their own performance (e.g., what is the right way of doing it?). Practitioners are the experts specialized in their own practice context. And what is considered to be intelligent practice is highly context-dependent not only across but also within professions (Amin & Roberts, 2008; Billett, 2001; Schön, 1983). Therefore, the designed interaction should try to respect or leverage practitioners’ established practical knowing, in order to extend, rather than challenge their existing competence. Namely, the design should avoid prescribing their actions or giving authoritative, arbitrary assessment of their performance/situation without incorporating their own understandings. Instead, the property of being open to practical knowing is about affording appropriate space for practitioners to decide the use of, or assign meanings to the designed interaction. There are three actionable considerations for designers to achieve this design property:
- Design consideration 2.1: aim for adding meaningful alternatives instead of replacing existing approaches.
- Design consideration 2.2: enable practitioners’ modification and experimentation in practice.
- Design consideration 2.3: encourage practitioners’ own interpretations of the presented information.
We now elaborate on these considerations through concrete examples:
Consideration 2.1: Aim for Adding Meaningful Alternatives Instead of Replacing Existing Approaches
Practitioners’ frontstage repertoire consists of their established approaches of doing things. We argue that designers should consider the new peripheral interactions as meaningful alternatives to expand practitioners’ repertoire instead of replacements to their existing approaches. And those existing approaches should still be made available for them. The reasons are two-fold: on the one hand, aiming for replacing or dramatically changing practitioners’ current ways of doing things could result in new complexity for practitioners to adapt to, and hinder the designed interaction from being successfully incorporated into their practice. On the other hand, although the designed peripheral interactions may provide meaningful support, given the varying situations in real-world practice, they may not always guarantee the most practical or coherent solution for practitioners to accomplish a task. In specific situations, relying on a previous approach could be more practical than relying on the new interaction.
For instance, as reported by the teachers, the foot-based shortcuts of FeetForward served as alternatives to the tasks performed on their computer (An et al., 2017). Oftentimes, using FeetForward was more time-efficient and effortless. However, when the teachers had already been sitting by their desks or using their computers, performing the tasks on the computer became more practical. As they appreciated, FeetForward expanded their options for doing those tasks while they could still use a previous approach when desired. Similarly, Beepless Pedal was a meaningful alternative to the alarm-silencing button on the patient monitor (Figure 7). The pedal was considered most helpful when the nurses’ hands were busy, sanitized, or wearing gloves (Cabral Guerra et al., 2019). However, on some occasions, using the button was more practical, for instance, when interacting with the patient monitor or on the move.
This consideration can also apply to information displays. ClassBeacons afforded an alternative way for teachers to deliver proximity: making conscious decisions based on how they have already distributed their whereabouts over the classroom. Yet it was not meant to replace their existing way: as stated by some teachers, without ClassBeacons, they would deliver proximity mostly according to students’ help requests. This was still their prioritized way when they had ClassBeacons in the classroom. However, in the vast situations where students are not explicitly requesting help, ClassBeacons helped them consciously manage proximity as resources for, rather than a byproduct of, teaching (An et al., 2019). CheckMates supported nurses’ task of filling the water tank of neonatal incubators; this task was previously supported by two things: (1) the water level scales printed on the transparent surface of the water tank (see Figure 7), and (2) the audio and red-light alarm of the incubator system which calls for immediate action when water reaches a critical level. CheckMates was not designed to replace these two things. Instead, it was intended to supplement them and add an alternative approach for nurses to do this task. As reported by the nurses, when they were on the move, it was intuitive for them to notice the CheckMates’ light objects and roughly keep track of the supply status. This helped them to plan ahead in order to find appropriate moments to refill the devices (van Bentum et al., 2019). Alternatively, if they would like to know detailed status of the water supply, they could closely check the scales on the tank. Or, if the nurses had been busy with other more prioritized situations, they could also choose to ignore CheckMates and the water level scales and only react to the auditory alarms.
Consideration 2.2: Enable Practitioners’ Modification and Experimentation in Practice
In the two cases of physical interaction, we have learned that practitioners could have different needs and preferences in terms of using the designed interfaces. Such varying needs and preferences manifest their established practical knowing (e.g., tacit know-how, or repertoire), which is highly personal, experiential and context-dependent. To stay open to the divergent practical knowing developed by practitioners, the designed interface should support practitioners’ modification and experimentation in their own context of practice.
Differing preferences could result from both practitioners’ bodily habits and mental inclinations. For instance, the placement of Beepless Pedal was decided based on the observation that nurses often put their feet on the lower support of the chair when sitting by the incubator (see Figure 6). However, there was one nurse who was used to sitting with her feet on the ground, and she felt that the pedal could be placed a bit closer to her feet (Cabral Guerra et al., 2019). Another example reflects practitioners’ different inclinations in cognitive processes: the FeetForward system used graphical icons to convey the functionality of each pedal. While this was intuitive to most teachers, one teacher indicated that textual cues might work better for him because he generally felt texts more comprehensible than icons (An et al., 2017). Hence, designers could consider how to enable user modifications of both physical and digital aspects of the interface to better fit practitioners’ divergent bodily habits and cognitive inclinations.
While the above examples concern the level of usability (e.g., ergonomics or learnability), the coming example surfaces why practitioners’ modification and experimentation may also need to be supported at the level of functionality. In the FeetForward case, the teachers assigned quite diverse shortcut functions to their pedals, and they had very different opinions about which was the most meaningful function to them (An et al., 2017). This revealed their differing styles of using digital resources for lecturing, shaped by their previous practice and training. For instance, one teacher used images from his textbook a lot, so his favorite shortcut was opening a folder to retrieve textbook images quickly. Another teacher had been using multiple web-based applications in teaching, and her favorite shortcut was switching among opened tabs in the internet browser. The teachers appreciated that FeetForward encouraged them to experiment with the functions to find their own best fit.
More importantly, we have learned that practitioners’ modification of a designed interface does not happen all at once. Instead, it can be a prolonged and speculative process during use, in which practitioners continually experiment with and modify the interface to reveal different possibilities for incorporating it in their context of practice. Such a process is similar to what has been described as design-in-use (Wakkary & Maestri, 2007) in everyday domestic lives. In the case of FeetForward, none of the teachers had their functional customization finalized in the beginning. Instead, many functions they assigned at an earlier stage were replaced later by other functions that were considered more useful after exploration. Towards the end of the five-week deployment, the teachers were still assigned new functions, seemingly suggesting that the teachers had not finished their customization with FeetForward. Such experimentation may never end, since the practice context may vary over time, and practitioners may need to renew their toolkits or skillsets accordingly. As stated by the teachers, if FeetForward were deployed longer, they might adjust its functions to fit new topics or lesson styles planned for future teaching units. Corresponding to this, a recent study also reveals teachers’ need for customizing digital tools over time (Vermette et al., 2019). Taken together, we argue that designers should carefully consider how they could support practitioners’ prolonged, continuous experimentation-in-use.
Consideration 2.3: Encourage Practitioners’ own Interpretations of the Presented Information
This consideration has been elicited from the two cases of information display. As mentioned earlier, practical knowing also includes practitioners’ interpretive systems used for sense-making in the unfolding practice: such as intuitively assessing what is the most relevant problem to address or evaluating their performance on the spot. Such interpretive systems should be incorporated rather than challenged by design. Therefore, we argue that the designed peripheral display should leave proper space in its presented information to encourage practitioners’ own interpretations. There are two reasons for doing so: first, this could valuably leverage practitioners’ on-the-ground knowledge about their own context. Second, this could prevent the designed system giving arbitrary (Sengers & Gaver, 2006) assessment or prescription that does not fit a practical situation.
The case of ClassBeacons could be used to illustrate how complex and context-dependent it could be to interpret a practical situation. First, there is no standard pattern for an ideal distribution of teacher proximity in a class session. What an ideal proximity pattern looks like differs from session to session, depending on many contextual factors: such as the classroom layout, the needs of the individual students, the content or style of the specific lesson, as well as the professional beliefs of the teacher. While many of these factors are inaccessible to the system, most of them are comprehensible to teachers. For this reason, ClassBeacons was designed as a neutral portrayal of proximity data and obviated using negative colors such as red or orange. This avoided the color of the lamps being simply understood as good or bad and encouraged teachers to give in-depth contextualized interpretations (An et al., 2019): e.g., whether they had been delivering proximity in accordance with their expectations or beliefs, or whether there were unwanted patterns in their proximity behaviors that needed to be changed. For instance, depending on their practical knowing, a yellow lamp could be interpreted as correct in a particular situation (e.g., these students have been doing well independently); while in another case, it could mean a lack of proximity (e.g., the teacher has spent too much time in somewhere else and forgot to interact with the students here).
The color codes of CheckMates were also intended to neutrally portray the current situation for nurses rather than providing prescriptions. Hence, a calm and neutral green-blue spectrum was chosen so that the display would not arbitrarily render the urgency or criticalness of the situation. The participating nurses envisaged how they could generate contextualized interpretations in practice (van Bentum et al., 2019). For example, a blue light object could mean that the nurse should refill the device right now, since there is not a more urgent task at hand, and it is feasible to refill the supply before feeding the neonatal. However, in another situation, the nurse could deliberately ignore a blue light object for the moment, since a more prioritized task calls for action: e.g., responding to a clinical alarm from the patient monitor. This way, CheckMates increased the nurses’ situational awareness while leaving proper space for them to interpret the situation and avoided bringing extra stress to their already intensive workflow.
Vision: Extending “Practical Intelligence” via Unremarkable Augmentation in Future Workplaces
Our design exploration took its point of departure in the emerging field of peripheral interaction, which aims to disburden the use of computing systems and unobtrusively blend them into daily routines. Given their already complex, intensive everyday workflow, peripheral interaction seems especially meaningful to frontstage practitioners. Yet according to our design exploration, the value of peripheral interaction is beyond merely disburdening practitioners’ use of technologies. Instead, as surfaced in our design cases, peripheral interaction holds great potential for extending practitioners’ practical intelligence (Sternberg & Wagner, 1986). Practical intelligence is often tacit, yet it can be seen in practitioners’ spontaneous performance of skills, as well as their proficient sense-making in action (Schön, 1983). Our design cases have demonstrated how these two aspects could be respectively facilitated by designing peripheral physical interaction and peripheral information display across the practice contexts of school teachers and NICU nurses. It is implied that in future workplaces, we could deliberately leverage peripheral interactions to unobtrusively augment practitioners’ practical intelligence.
The term intelligence has gained increased attention in nowadays society thanks to the continuing breakthrough in Artificial Intelligence (AI). Many research efforts have been made for machine intelligence to outperform humans in particular tasks (Schmidt, 2017a). In 2017, an AI algorithm AlphaGo Master (Silver et al., 2017), defeated the world’s number one player in Go. In 2019, its derivative AlphaStar defeated a professional player in StarCraft, a game representing much higher uncertainty in decision-making (Arulkumaran et al., 2019). Meanwhile, media and academic surveys (Grace et al., 2018) render an alarming picture of future professions: e.g., experts interviewed by Grace et al. (2018) believed that by a probability of 50%, machines will outperform humans in all tasks in half a century, and take over all human professions in 120 years, such as translator (by 2024), truck driver (2027), retail worker (2031), or surgeon (2053). Therefore, in the near future, emerging technologies may challenge practitioners not only by escalating the complexity of their work—a challenge that has motivated this research—but also by automating their jobs and thus taking over their professions. The latter seems to be an even more severe societal challenge.
Nevertheless, as pinpointed by Schmidt (2017a), instead of worrying that the advancement of technology may take over human professions, we should actively work on better enabling symbiotic systems (Licklider, 1960) between human and computing technology. Through a human-computer interaction approach, such symbiotic systems aim to leverage technology to amplify human cognition and perception (Schmidt, 2017b) and enable human-AI complementarity (Holstein et al., 2019) to extend human intelligence rather than challenge it. In this way, we could enable a preferable future where human practitioners are empowered rather than sidelined.
This vision is not as distant as it may initially appear. Actually, one could argue that in daily lives, our intelligence has already been extended by computing technology: think about the digital tools we use while working at the office. For instance, a search engine or an online encyclopedia could be seen as our external memory to retrieve information precisely when needed. A statistical application frees us from particular or repetitive calculations in analysis, so we can put more cognitive resources on higher-level patterns of the data. According to the extended mind theories (Licklider, 1960) or cognitive offloading (Dror & Harnad, 2008), these examples show how digital tools can extend our intellectual capability. Nonetheless, as mentioned at the beginning of this paper, the above examples from office or desktop settings tend to resemble the backstage occasions of professional practice. Yet, little design research has been focused on extending practitioners’ intelligence embodied in their frontstage practice. In the backstage, practitioners’ tasks are relatively cerebral, i.e., involving largely prolonged analytical thinking. By contrast, in the frontstage, practitioners carry out intensive, simultaneous thinking and doing in a complex social and material surrounding, in which they have to rely on their tacit knowing (Polanyi, 1962) in action, rather than explicit calculation or prolonged analysis out of action (Eraut, 2000; Schön, 1983). As a result, computing interfaces aimed to augment frontstage practice ought to be designed differently.
Our design cases unveiled the various facets such tacit, underlying intelligence could entail in the frontstage of teachers and nurses and how they could be augmented by designs. For instance, teachers’ intelligent practice can be manifested by their effortless and flexible incorporation of digital resources during the moment of lecturing (An et al., 2017). Their intelligence can also be revealed when they can intuitively reflect in action on how well they have been providing interpersonal interaction to each individual learner (An et al., 2019). As for a NICU nurse, we could say that this nurse is acting intelligently when they can proficiently deal with non-actionable alarms while keeping their main activity in an incubator uninterrupted (Cabral Guerra et al., 2019). Moreover, a NICU nurse’s intelligence could be implied when they can responsively, and proactively manage the supplies of surrounding medical devices along with other unfolding caring tasks (van Bentum et al., 2019). The design cases studied in this paper have surfaced how practical intelligence could be incrementally and unremarkably extended by peripheral interactions without overburdening practitioners.
This resonates with Polanyi’s theory about how tools could extend our practical comprehension (Polanyi, 1962). Namely, when a person practices intelligently with their tool, the use of the tool becomes part of their tacit knowing (Polanyi, 1962): i.e., the tool itself can be tacitly relied on, so that the person can better focus on the goal that they use the tool to approach. This is also vividly explained by Heidegger’s metaphor of readiness-to-hand (Heidegger, 2005): when using a hammer, the hammer itself is tacitly relied on, as if it becomes an extension of our hand; and our focus is on the nail that we use the hammer to approach. Similar metaphors have been made by Schmidt using glasses and binoculars (Schmidt, 2017b): before using binoculars, we have to deliberately decide the moments of use and explicitly pick them up to observe our target; differently, glasses extend our perception more tacitly, since we are less attentive to the glasses per se but more focused on the target of observation.
These metaphors surface the same implication: tools that extend our intelligence in practice should become the instrument of attention rather than the focus of attention. In other words, to extend practitioners’ practical intelligence, technology should be designed to be tacitly relied on rather than focally attended to in their frontstage ecology. Only by doing so could the designed technology systems be compatible with practitioners’ economical and dynamic way of mobilizing attention, and thus incorporated into their spontaneous skill performance and intuitive sense-making. While our design inquiry has implied meaningful potentials, more future explorations are still needed to broaden and deepen our knowledge about this uncharted design space.
In a nutshell, facilitating human-technology symbiosis to empower future practitioners is a relevant challenge in our current societal context. We argue that designing peripheral interactions in practitioners’ surroundings can be a promising and unique approach to this challenge for two-fold reasons. First, it specifically extends the form of intelligence embodied in practitioners’ intensive, simultaneous doing and thinking. Second, it reveals that our intelligence does not only concern what is focused on in the center of our attention but also what can be tacitly relied on in the periphery of our attention.
Conclusions
This paper explores how to support practitioners with technologies that can seamlessly fit into their busy frontstage occasions. Nowadays, practitioners are increasingly required to incorporate supportive technologies in their complex and intensive workflows; however, they only have limited time and attentional resources available for interacting with technology devices. Current human-computer interaction (HCI) paradigms often demand users’ continuous focal attention, assuming the designed interaction as the users’ primary or only task during the moment of use. Relying solely on such focal interaction hinders supportive devices from fitting in professional routines, in which interacting with technology devices is often practitioners’ secondary task performed along with other unfolding activities. We thereby have explored a supplementary HCI style in the emerging field of peripheral interaction, which argues for enabling users to interact with computing systems not only in the center, but also in the periphery of their attention. In the paper, we first connected related theories regarding professional routines to the design research in peripheral interaction. Subsequently, we concretized the potential and implication of peripheral interaction in real-world contexts by presenting four design cases aimed for supporting school teachers and NICU nurses. Our design cases have demonstrated how peripheral interaction could unobtrusively support practitioners by enriching their repertoire of action and enhancing their reflection-in-action. Generalizing upon our design cases, we formulated a set of design insights (two properties and six considerations) that could inform future work in leveraging peripheral interaction to support busy practitioners. Finally, we addressed our vision of how peripheral interaction could be used as a meaningful approach to empower future practitioners by seamlessly and incrementally extending their practical intelligence.
Acknowledgments
We would like to express our sincere gratitude to all the school teachers, NICU nurses, students, domain experts, and other stakeholders who have generously volunteered their time and provided invaluable support to our design research projects. Our sincere appreciation goes to our esteemed colleagues and students at Eindhoven University of Technology for their invaluable assistance and perceptive suggestions that have enabled us to continually refine and strengthen our work over the years. This line of research has been skillfully supervised by Dr. Saskia Bakker and Prof. Berry Eggen. Pengcheng An’s ongoing research into classroom technologies has been supported by the Shenzhen Grant 2022081517130800. Lastly, we extend our thanks to the dedicated reviewers and editing team for their diligent work and constructive feedback in helping us enhance this paper.
References
- Alavi, H. S., & Dillenbourg, P. (2012). An ambient awareness tool for supporting supervised collaborative problem solving. IEEE Transactions on Learning Technologies, 5(3), 264-274. https://doi.org/10.1109/TLT.2012.7
- Amin, A., & Roberts, J. (2008). Knowing in action: Beyond communities of practice. Research Policy, 37(2), 353-369. https://doi.org/10.1016/j.respol.2007.11.003
- An, P., Bakker, S., & Eggen, B. (2017). FeetForward: On blending new classroom technologies into secondary school teachers’ routines. In Proceedings of the conference on human-computer interaction (pp. 327-347). Springer. https://doi.org/10.1007/978-3-319-67684-5_20
- An, P., Bakker, S., Ordanovski, S., Taconis, R., Paffen, C. L. E., & Eggen, B. (2019). Unobtrusively enhancing reflection-in-action of teachers through spatially distributed ambient information. In Proceedings of the SIGCHI conference on human factors in computing systems (Article No. 91). ACM. https://doi.org/10.1145/3290605.3300321
- Arulkumaran, K., Cully, A., & Togelius, J. (2019). AlphaStar: An evolutionary computation perspective. In Proceedings of the genetic and evolutionary computation conference companion (pp. 314-315). ACM. https://doi.org/10.1145/3319619.3321894
- Ashbrook, D. L. (2010). Enabling mobile microinteractions [Doctoral dissertation]. Georgia Institute of Technology. https://smartech.gatech.edu/handle/1853/33986
- Bakker, S., Van den Hoven, E., & Eggen, B. (2013). FireFlies: Physical peripheral interaction design for the everyday routine of primary school teachers. In Proceedings of the 7th international conference on tangible, embedded and embodied (pp. 57-64) ACM. https://doi.org/10.1145/2460625.2460634
- Bakker, S., & Niemantsverdriet, K. (2016). The interaction-attention continuum: Considering various levels of human attention in interaction design. International Journal of Design, 10(2), 1-14.
- Bakker, S., van den Hoven, E., & Eggen, B. (2012). Acting by hand: Informing interaction design for the periphery of people’s attention. Interacting with Computers, 24(3), 119-130. https://doi.org/10.1016/j.intcom.2012.04.001
- Bakker, S., van den Hoven, E., & Eggen, B. (2015). Peripheral interaction: Characteristics and considerations. Personal and Ubiquitous Computing, 19(1), 239-254. https://doi.org/10.1007/s00779-014-0775-2
- Balachandra, L., Barrett, F., Bellman, H., Fisher, C., & Susskind, L. (2005). Improvisation and mediation: Balancing acts. Negotiation Journal, 21(4), 425-434. https://doi.org/10.1111/j.1571-9979.2005.00075.x
- Billett, S. (2001). Knowing in practice: Re-conceptualising vocational expertise. Learning and Instruction, 11(6), 431-452. https://doi.org/10.1016/S0959-4752(00)00040-2
- Bogers, K. (2017, Sept.). CareTunes: Music as a nurses’ monitoring tool. Critical Alarms Lab. https://delftdesignlabs.org/projects/caretunes/
- Bødker, S., & Klokmose, C. N. (2012). Dynamics in artifact ecologies. In Proceedings of the 7th Nordic conference on human-computer interaction making sense through design (pp. 448-457). ACM. https://doi.org/10.1145/2399016.2399085
- Braithwaite, M. (2008). Nurse burnout and stress in the NICU. Advances in Neonatal Care, 8(6), 343-347. https://doi.org/10.1097/01.ANC.0000342767.17606.d1
- Brante, G. (2009). Multitasking and synchronous work: Complexities in teacher work. Teaching and Teacher Education, 25(3), 430-436. https://doi.org/10.1016/j.tate.2008.09.015
- Buxton, B. (2018, January 12). Ubiety: On design, place and the importance of manners [Video]. Youtube. https://youtu.be/GEeZFW7PFBI
- Cabral Guerra, M., Kommers, D., Bakker, S., An, P., van Pul, C., & Andriessen, P. (2019). Beepless: Using peripheral interaction to reduce the number and sound of clinical alarms. In Proceedings of the conference on designing interactive systems (pp. 607-620). ACM. https://doi.org/10.1145/3322276.3323696
- Colley, A., Raudanjoki, Ö., Mikkonen, K., & Häkkilä, J. (2019). Plant shadow morphing as a peripheral display. In Proceedings of the 18th international conference on mobile and ubiquitous multimedia (pp. 1-5). ACM. https://doi.org/10.1145/3365610.3368410
- Cook, S. D. N., & Yanow, D. (1993). Culture and organizational learning. Journal of Management Inquiry, 2(4), 373-390. https://doi.org/10.1177/105649269324010
- Cuban, L., Kirkpatrick, H., & Peck, C. (2001). High access and low use of technologies in high school classrooms: Explaining an apparent paradox. American Educational Research Journal, 38(4), 813-834. https://doi.org/10.3102/00028312038004813
- Cvach, M. (2012). Monitor alarm fatigue: An integrative review. Biomedical Instrumentation & Technology, 46(4), 268-277. https://doi.org/10.2345/0899-8205-46.4.268
- Dayan, E., & Cohen, L. G. (2011). Neuroplasticity subserving motor skill learning. Neuron, 72(3), 443-454. https://doi.org/10.1016/J.NEURON.2011.10.008
- Dror, I. E., & Harnad, S. (2008). Offloading cognition onto cognitive technology. In I. E. Dror & S. Harnad (Eds.), Cognition distributed: How cognitive technology extends our minds (pp. 1-23). John Benjamins Publishing. https://doi.org/10.1075/bct.16.02dro
- Edge, D., & Blackwell, A. F. (2009). Peripheral tangible interaction by analytic design. In Proceedings of the 3rd international conference on tangible and embedded interaction (pp. 69-76). ACM. https://doi.org/10.1145/1517664.1517687
- Eggen, B., & Van Mensvoort, K. (2009). Making sense of what is going on ‘around’: Designing environmental awareness information displays. In P. Markopoulos, B. de Ruyter, & W. Mackay (Eds), Awareness systems (pp. 99-124). Springer. https://doi.org/10.1007/978-1-84882-477-5_4
- Eraut, M. (1995). Schon shock: A case for refraining reflection-in-action? Teachers and Teaching, 1(1), 9-22. https://doi.org/10.1080/1354060950010102
- Eraut, M. (2000). Non-formal learning and tacit knowledge in professional work. British Journal of Educational Psychology, 70(1), 113-136. https://doi.org/10.1348/000709900158001
- Evetts, J. (2014). The concept of professionalism: Professional work, professional practice and learning. In S. Billett, C. Harteis, & H. Gruber, (Eds.), International handbook of research in professional and practice-based learning (pp. 29-56). Springer. https://doi.org/10.1007/978-94-017-8902-8_2
- Gardner, B., Abraham, C., Lally, P., & de Bruijn, G.-J. (2012). Towards parsimony in habit measurement: Testing the convergent and predictive validity of an automaticity subscale of the self-report habit index. The International Journal of Behavioral Nutrition and Physical Activity, 9, 102. https://doi.org/10.1186/1479-5868-9-102
- Glaser, B. G. (1965). The constant comparative method of qualitative analysis. Social Problems, 12(4), 436-445. https://doi.org/10.2307/798843
- Goffman, E. (1959). The presentation of self in everyday life. Doubleday.
- Grace, K., Salvatier, J., Dafoe, A., Zhang, B., & Evans, O. (2018). Viewpoint: When will AI exceed human performance? Evidence from AI experts. Journal of Artificial Intelligence Research, 62. https://doi.org/10.1613/jair.1.11222
- Gunter, P. L., Shores, R. E., Jack, S. L., Rasmussen, S. K., & Flowers, J. (1995). On the move using teacher/student proximity to improve students’ behavior. Teaching Exceptional Children, 28(1), 12-14. https://doi.org/10.1177/004005999502800103
- Hasegawa, S., Ishijima, S., Kato, F., Mitake, H., & Sato, M. (2012). Realtime sonification of the center of gravity for skiing. In Proceedings of the 3rd conference on augmented human (Article No. 11). ACM. https://doi.org/10.1145/2160125.2160136
- Hausen, D. (2014). Peripheral interaction. Ludwig-Maximilians-Universität München.
- Hausen, D., Boring, S., & Greenberg, S. (2013). The unadorned desk: Exploiting the physical space around a display as an input canvas. In Proceedings of the conference on human-computer interaction (pp. 140-158). Springer. https://doi.org/10.1007/978-3-642-40483-2_10
- Heidegger, M. (2005). Being and time. Rowman & Littlefield.
- Hollan, J., Hutchins, E., & Kirsh, D. (2000). Distributed cognition: Toward a new foundation for human-computer interaction research. ACM Transactions on Computer-Human Interaction, 7(2), 174-196. https://doi.org/10.1145/353485.353487
- Holstein, K., McLaren, B. M., & Aleven, V. (2019). Co-designing a real-time classroom orchestration tool to support teacher-AI complementarity. Journal of Learning Analytics, 6(2), 27-52. https://doi.org/10.18608/jla.2019.62.3
- Höök, K., Jonsson, M. P., Ståhl, A., & Mercurio, J. (2016). Somaesthetic appreciation design. In Proceedings of the SIGCHI conference on human factors in computing systems (pp. 3131-3142). ACM. https://doi.org/10.1145/2858036.2858583
- Höök, K., & Löwgren, J. (2012). Strong concepts: Intermediate-level knowledge in interaction design research. ACM Transactions on Computer-Human Interaction, 19(3), 1-18. https://doi.org/10.1145/2362364.2362371
- Hudson, S. E., Harrison, C., Harrison, B. L., & LaMarca, A. (2010). Whack gestures: Inexact and inattentive interaction with mobile devices. In Proceedings of the 4th conference on tangible, embedded, and embodied interaction (pp. 109-112). ACM. https://doi.org/10.1145/1709886.1709906
- Hutchins, E. (1995). Cognition in the wild (Vol. 7). MIT Press.
- Hutchins, E. (2005). Distributed cognition. Cognition, Technology & Work, 7(1), 5. https://doi.org/10.1007/s10111-004-0172-0
- Ishii, H., Wisneski, C., Brave, S., Dahley, A., Gorbet, M., Ullmer, B., & Yarin, P. (1998). ambientROOM: Integrating ambient media with architectural space. In Summary of the SIGCHI conference on human factors in computing systems (pp. 173-174). ACM. https://doi.org/10.1145/286498.286652
- Kahneman, D. (1973). Attention and effort. The American Journal of Psychology, 88(2), 339-340. https://doi.org/10.2307/1421603
- Kirsh, D. (1995). The intelligent use of space. Artificial Intelligence, 73(1-2), 31-68. https://doi.org/10.1016/0004-3702(94)00017-U
- Konttila, J., Siira, H., Kyngäs, H., Lahtinen, M., Elo, S., Kääriäinen, M., Kaakinen, P., Oikarinen, A., Yamakawa, M., & Fukui, S. (2019). Healthcare professionals’ competence in digitalisation: A systematic review. Journal of Clinical Nursing, 28(5-6), 745-761. https://doi.org/10.1111/jocn.14710
- Lai, T. T., & Bearer, C. F. (2008). Iatrogenic environmental hazards in the neonatal intensive care unit. Clinics in Perinatology, 35(1), 163-181. https://doi.org/10.1016/J.CLP.2007.11.003
- Laxmisan, A., Hakimzada, F., Sayan, O. R., Green, R. A., Zhang, J., & Patel, V. L. (2007). The multitasking clinician: Decision-making and cognitive demand during and after team handoffs in emergency care. International Journal of Medical Informatics, 76(11-12), 801-811. https://doi.org/10.1016/J.IJMEDINF.2006.09.019
- Li, Y., Garza, V., Keicher, A., & Popov, V. (2019). Predicting high school teacher use of technology: Pedagogical beliefs, technological beliefs and attitudes, and teacher training. Technology, Knowledge and Learning, 24(3), 501-518. https://doi.org/10.1007/s10758-018-9355-2
- Licklider, J. C. R. (1960). Man-computer symbiosis. IRE Transactions on Human Factors in Electronics, HFE-1(1), 4-11. https://doi.org/10.1109/THFE2.1960.4503259
- Lim, F. V., O’Halloran, K. L., & Podlasov, A. (2012). Spatial pedagogy: Mapping meanings in the use of classroom space. Cambridge Journal of Education, 42(2), 235-251. https://doi.org/10.1080/0305764X.2012.676629
- Locsin, R. C. (2017). The co-existence of technology and caring in the theory of technological competency as caring in nursing. The Journal of Medical Investigation, 64(1.2), 160-164. https://doi.org/10.2152/jmi.64.160
- Markopoulos, P., & Mackay, W. (2009). Awareness systems: Advances in theory, methodology and design. Springer. https://doi.org/10.1007/978-1-84882-477-5
- Martinez-Maldonado, R., Schulte, J., Echeverria, V., Gopalan, Y., & Shum, S. B. (2020). Where is the teacher? Digital analytics for classroom proxemics. Journal of Computer Assisted Learning, 36(5), 741-762. https://doi.org/10.1111/jcal.12444
- Matthews, T., Dey, A. K., Mankoff, J., Carter, S., & Rattenbury, T. (2004). A toolkit for managing user attention in peripheral displays. In Proceedings of the 17th annual symposium on user interface software and technology (pp. 247-256). ACM. https://doi.org/10.1145/1029632.1029676
- Matthews, T., Rattenbury, T., & Carter, S. (2007). Defining, designing, and evaluating peripheral displays: An analysis using activity theory. Human-Computer Interaction, 22(1), 221-261. https://dl.acm.org/doi/10.5555/1466595.1466602
- Matthies, D. J. C., Müller, F., Anthes, C., & Kranzlmüller, D. (2013). ShoeSoleSense: Proof of concept for a wearable foot interface for virtual and real environments. In Proceedings of the 19th symposium on virtual reality software and technology (pp. 93-96). ACM. https://doi.org/10.1145/2503713.2503740
- Matthies, D. J. C., Urban, B., Wolf, K., & Schmidt, A. (2019). Reflexive interaction: Extending the concept of peripheral interaction. In Proceedings of the 31st Australian conference on human-computer-interaction (pp. 266-278). ACM. https://doi.org/10.1145/3369457.3369478
- Moorman, C., & Miner, A. S. (1998). Organizational improvisation and organizational memory. Academy of Management Review, 23(4), 698-723. https://doi.org/10.5465/AMR.1998.1255634
- Mouza, C. (2019). Beliefs, models, and practices on fostering teacher learning in technology integration. Contemporary Issues in Technology and Teacher Education, 19(3), 302-304.
- Müller, H., Colley, A., Häkkilä, J., Jensen, W., & Löchtefeld, M. (2019). Using electrochromic displays to display ambient information and notifications. In Proceedings of the international joint conference on pervasive and ubiquitous computing and international symposium on wearable computers (pp. 1075-1078). ACM. https://doi.org/10.1145/3341162.3344844
- Olivera, F., García-Herranz, M., Haya, P. A., & Llinás, P. (2011). Do not disturb: Physical interfaces for parallel peripheral interactions. In Proceedings of the conference on human-computer interaction (pp. 479-486). Springer. https://doi.org/10.1007/978-3-642-23771-3_36
- Orr, J. E. (2016). Talking about machines: An ethnography of a modern job. Cornell University Press.
- Patel, K. (2008). Thinkers in the kitchen: Embodied thinking and learning in practice. Harvard Graduate School of Education.
- Peplowski, K. (1998). The process of improvisation. Organization Science, 9(5), 560-561. https://doi.org/10.1016/j.pragma.2013.09.015
- Pinch Studio. (2018, Feb.). UltiMo: A silent patient monitor. Critical Alarms Lab. https://delftdesignlabs.org/projects/ultimo/
- Pohl, H., & Murray-Smith, R. (2013). Focused and casual interactions: Allowing users to vary their level of engagement. In Proceedings of the SIGCHI conference on human factors in computing systems (pp. 2223-2232). ACM. https://doi.org/10.1145/2470654.2481307
- Polanyi, M. (1962). Tacit knowing: Its bearing on some problems of philosophy. Reviews of Modern Physics, 34(4), 601-616. https://doi.org/10.1103/RevModPhys.34.601
- Pousman, Z., & Stasko, J. (2006). A taxonomy of ambient information systems: Four patterns of design. In Proceedings of the working conference on advanced visual interfaces (pp. 67-74). ACM. https://doi.org/10.1145/1133265.1133277
- Probst, K. (2016). Peripheral interaction in desktop computing: Why it’s worth stepping beyond traditional mouse and keyboard. In S. Bakker, D. Hausen, & T. Selker (Eds.), Peripheral interaction (pp. 183-205). Springer. https://doi.org/10.1007/978-3-319-29523-7_9
- Reich, J., & Dunne, L. E. (2016). Multi-modal wearable ambient display: An investigation of continuous glucose monitoring. In Proceedings of the international symposium on wearable computers (pp. 24-27). ACM. https://doi.org/10.1145/2971763.2971801
- Risko, E. F., & Gilbert, S. J. (2016). Cognitive offloading. Trends in Cognitive Sciences, 20(9), 676-688. https://doi.org/10.1016/j.tics.2016.07.002
- Redert, R. (2018). Doplor: Artful warnings for a quiet intensive care. Retrieved from https://delftdesignlabs.org/projects/doplor/
- Salvucci, D. D., Taatgen, N. A., & Borst, J. P. (2009). Toward a unified theory of the multitasking continuum. In Proceedings of the 27th SIGCHI conference on human factors in computing systems (pp. 1819-1828). ACM. https://doi.org/10.1145/1518701.1518981
- Schmidt, A. (2017a). Augmenting human intellect and amplifying perception and cognition. IEEE Pervasive Computing, 16(1), 6-10. https://doi.org/10.1109/MPRV.2017.8
- Schmidt, A. (2017b). Technologies to amplify the mind. Computer, 50(10), 102-106. https://doi.org/10.1109/MC.2017.3641644
- Schön, D. A. (1983). The reflective practitioner: How professionals think in action. Temple Smith.
- Schön, D. A. (1987). Educating the reflective practitioner: Toward a new design for teaching and learning in the professions. Jossey-Bass.
- Sengers, P., & Gaver, B. (2006). Staying open to interpretation. In Proceedings of the 6th conference on designing interactive systems (pp. 99-108). ACM. https://doi.org/10.1145/1142405.1142422
- Serim, B., & Jacucci, G. (2019). Explicating “implicit interaction”: An examination of the concept and challenges for research. In Proceedings of the SIGCHI conference on human factors in computing systems (Paper No. 417). ACM. https://doi.org/10.1145/3290605.3300647
- Shores, R. E., Gunter, P. L., & Jack, S. L. (2017). Classroom management strategies: Are they setting events for coercion? Behavioral Disorders, 18(2), 92-102 https://doi.org/10.1177/019874299301800207
- Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., Chen, Y., Lillicrap, T., Hui, F., Sifre, L., van den Driessche, G., Graepel, T., & Hassabis, D. (2017). Mastering the game of go without human knowledge. Nature, 550(7676), 354-359. https://doi.org/10.1038/nature24270
- Smith, T., Bowen, S. J., Nissen, B., Hook, J., Verhoeven, A., Bowers, J., Wright, P., & Olivier, P. (2015). Exploring gesture sonification to support reflective craft practice. In Proceedings of the 33rd SIGCHI conference on human factors in computing systems (pp. 67-76). ACM. https://doi.org/10.1145/2702123.2702497
- Sternberg, R. J., & Wagner, R. K. (1986). Practical intelligence: Nature and origins of competence in the everyday world. CUP Archive.
- Stolterman, E., & Wiberg, M. (2010). Concept-driven interaction design research. Human-Computer Interaction, 25(2), 95-118. https://doi.org/10.1080/07370020903586696
- Storesund, A., & McMurray, A. (2009). Quality of practice in an intensive care unit (ICU): A mini-ethnographic case study. Intensive and Critical Care Nursing, 25(3), 120-127. https://doi.org/10.1016/J.ICCN.2009.02.001
- Suchman, L., Blomberg, J., Orr, J. E., & Trigg, R. (1999). Reconstructing technologies as social practice. American Behavioral Scientist, 43(3), 392-408. https://doi.org/10.1177/00027649921955335
- Takayama, L. (2017). The motivations of ubiquitous computing: Revisiting the ideas behind and beyond the prototypes. Personal and Ubiquitous Computing, 21(3), 557-569. https://doi.org/10.1007/s00779-017-1002-8
- Tolmie, P., Pycock, J., Diggins, T., MacLean, A., & Karsenty, A. (2002). Unremarkable computing. In Proceedings of the SIGCHI conference on human factors in computing systems (pp. 399-406). ACM. https://doi.org/10.1145/503447.503448
- Van Alphen, E., & Bakker, S. (2016). Lernanto: Using an ambient display during differentiated instruction. In Extended abstracts of the SIGCHI conference on human factors in computing systems (pp. 2334-2340). ACM. https://doi.org/10.1145/2851581.2892524
- Van Bentum, J., Kommers, D., Bakker, S., Cabral Guerra, M., van Pul, C., & Andriessen, P. (2019). CheckMates, helping nurses plan ahead in the neonatal intensive care unit. Multimodal Technologies and Interaction, 3(1), 14. https://doi.org/10.3390/mti3010014
- Varisco, G., van de Mortel, H., Cabrera-Quiros, L., Atallah, L., Hueske-Kraus, D., Long, X., Cottaar, E. J. E., Zhan, Z., Andriessen, P., & van Pul, C. (2020). Optimisation of clinical workflow and monitor settings safely reduces alarms in the NICU. Acta Paediatrica, 110(4), 1141-1150. https://doi.org/10.1111/apa.15615
- Vega, V., & Robb, M. B. (2019). The common sense census: Inside the 21st-century classroom. Common Sense Media.
- Velloso, E., Schmidt, D., Alexander, J., Gellersen, H., & Bulling, A. (2015). The feet in human-computer interaction. ACM Computing Surveys, 48(2), 1-35. https://doi.org/10.1145/2816455
- Vermette, L., McGrenere, J., Birge, C., Kelly, A., & Chilana, P. K. (2019). Freedom to personalize my digital classroom: Understanding teachers’ practices and motivations. In Proceedings of the SIGCHI conference on human factors in computing systems (Paper No. 318). ACM. https://doi.org/10.1145/3290605.3300548
- Wakkary, R., Desjardins, A., & Hauser, S. (2016). Unselfconscious interaction: A conceptual construct. Interacting with Computers, 28(4), 501-520. https://doi.org/10.1093/iwc/iwv018
- Wakkary, R., & Maestri, L. (2007). The resourcefulness of everyday design. In Proceedings of the 6th conference on creativity & cognition (pp. 163-172). ACM. https://doi.org/10.1145/1254960.1254984
- Weiser, M. (1993). Hot topics-ubiquitous computing. Computer, 26(10), 71-72. https://doi.org/10.1109/2.237456
- Weiser, M. (1999). The computer for the 21st century. ACM SIGMOBILE Mobile Computing and Communications Review, 3(3), 3-11. https://doi.org/10.1145/329124.329126
- Weiser, M., & Brown, J. S. (1997). The coming age of calm technology. In P. J. Denning & R. M. Metcalfe (Eds.), Beyond calculation (pp. 75-85). Springer. https://doi.org/10.1007/978-1-4612-0685-9_6
- Wickens, C. D. (2002). Multiple resources and performance prediction. Theoretical Issues in Ergonomics Science, 3(2), 159-177. https://doi.org/10.1080/14639220210123806
- Wolf, K., Naumann, A., Rohs, M., & Müller, J. (2011). A taxonomy of microinteractions: Defining microgestures based on ergonomic and scenario-dependent requirements. In Proceedings of the conference on human-computer interaction (pp. 559-575). Springer. https://doi.org/10.1007/978-3-642-23774-4_45
- Wood, W., & Rünger, D. (2016). Psychology of habit. Annual Review of Psychology, 67, 289-314. https://doi.org/10.1146/annurev-psych-122414-033417
- Xiao, R., & Benko, H. (2016). Augmenting the field-of-view of head-mounted displays with sparse peripheral displays. In Proceedings of the SIGCHI conference on human factors in computing systems (pp. 1221-1232). ACM. https://doi.org/10.1145/2858036.2858212
- Yanow, D., & Tsoukas, H. (2009). What is reflection-in-action? A phenomenological account. Journal of Management Studies, 46(8), 1339-1364. https://doi.org/10.1111/j.1467-6486.2009.00859.x
- Zimmerman, J., Forlizzi, J., & Evenson, S. (2007). Research through design as a method for interaction design research in HCI. In Proceedings of the SIGCHI conference on human factors in computing systems (pp. 493-502). ACM. https://doi.org/10.1145/1240624.1240704
