Framing Smart Consumer Technology: Mediation, Materiality, and Material for Design
Sumit Pandey
Department of Informatics, University of Oslo, Oslo, Norway
This article develops a conceptual understanding of smart consumer technology for interaction design research, articulated through the technological mediation, materials, and materiality of its concrete artefacts. It contrasts current forms of understanding that are based on functional and utilitarian criteria. Our central argument is that a bottom-up, material centric understanding would allow us to frame smart consumer technology as a material for design, and consequently allow for a clearer critical, reflective, and exploratory engagement with its design space. Applying Mediation theory, we develop a phenomenological analysis of technological mediation using the common artefacts of smart consumer technology while at the same time outlining their material composition and materiality. Complementing mediation theory with research through design, we use this understanding to frame material explorations and to articulate the outcomes and reflections derived from a concrete speculative design prototype. This article highlights how theory and exploratory practice can be engaged together to develop an evolving understanding of smart consumer technology as a material for design.
Keywords – Interaction Design, Smart Consumer Technology, Smart Products, Mediation Theory, Research through Design, Machine Learning.
Relevance to Design Practice – This article presents an understanding of smart consumer technology as a material for design that is developed in terms of its concrete artefacts and highlights how this understanding can be used in, and evolved through, new design explorations while at the same time critically articulating their implications for design.
Citation: Sumit Pandey (2018). Framing smart consumer technology: Mediation, materiality, and material for design. International Journal of Design, 12(1), 37-51.
Received May 15, 2017; Accepted February 07, 2018; Published April 30, 2018.
Copyright: © 2018 Pandey. Copyright for this article is retained by the author, with first publication rights granted to the International Journal of Design. All journal content, except where otherwise noted, is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 2.5 License. By virtue of their appearance in this open-access journal, articles are free to use, with proper attribution, in educational and other non-commercial settings.
*Corresponding Author: sumitp@ifi.uio.no
Sumit Pandey is a design researcher and a PhD fellow in Interaction Design at the University of Oslo where his primary research area is the design of objects, methods, and processes that lead to reflective and more open forms of engagement with emerging smart technologies. In addition to research, he is a practicing designer who has extensive experience consulting for enterprise and consumer start-ups to collaboratively create strategic interaction design frameworks and solutions, across a breadth of delivery platforms such as mobile, web, desktop applications, and interactive TV.
Introduction
Networked, computational, and recently, intelligent materials are increasingly meshed within everyday life as smart technology. While the smart label has been applied to a wide variety of everyday artefacts (Staff, 2017; Kiritsis, 2011), an understanding of smart consumer technology itself has largely remained feature centric. For instance, Maass and Janzen (2007) identify smart technology in terms of three different kinds of adaptability: to situational contexts, to actors that interact with them, and to business constraints. Alternatively, Mühlhäuser (2007) defines smart technology as
[A]n entity (tangible object, software, or service) designed and made for self-organized embedding into different (smart) environments in the course of its lifecycle, providing improved simplicity and openness through improved p2u and p2p interaction by means of context-awareness, semantic self- description, proactive behaviour, multimodal natural interfaces, AI planning, and machine learning (p. 163).
Approaches for designing particular characteristics and artefacts of smart technology, such as those described by (Kuniavsky, 2010; Saizmaa & Kim, 2008; Wilson, Hargreaves, & Hauxwell-Baldwin, 2015), or its implications within particular topical contexts, as described by (Haines, Mitchell, Cooper, & Maguire, 2007; Lee, 2014; Lingel, 2016; Strengers, 2014), have been explored within interaction design research. However, explicit engagement with building an understanding of smart technology itself as a material for design has been limited. An understanding of smart technology as a material for design would allow for clearer articulation, critiques, and reflections on existing interactions with it. Further, considering how smart technology is getting embedded in everyday appliances and artefacts, understanding it as a material for design can allow us to shape it more effectively in combination with other physical materials. Consequently, such an understanding would help frame the design space of smart technology for design practice and exploration. The importance of understanding interactive materials and their materiality has also been emphasised in interaction design research’s recent ‘material turn’ that has explored areas like computational materials and the materiality of technological artefacts in everyday life (Fernaeus & Sundström, 2012; Redström, 2005; Robles & Wiberg, 2010; Vallgårda & Redström, 2007; Wiberg, 2014, 2016; Wiberg et al., 2013).
To understand smart technology as a material for design, we develop an articulation of both its materiality and material composition (as a material in design; Redström, 2005) and how it affects everyday life (as a material that designs; Pierce & Paulos, 2010). With material composition, we refer to what makes up a smart artefact, like its computational and physical materials (Vallgårda & Redström, 2007). On the other hand, materiality refers to how smart artefacts are experienced through their material properties (Wiberg et al., 2013). However, smart technology covers a broad spectrum of technology that is used both within industrial and consumer contexts. For the purposes of this article, we restrict the scope of our discussion to smart technology as it gets featured within everyday lives. For reasons of clarity, we refer to such tangible artefacts situated within a domestic context, as smart consumer technology. This allows us to meaningfully investigate smart technology without getting too general and abstract.
Our research is primarily informed by post-phenomenology and mediation theory since they offer a theoretical understanding and methodology for describing and understanding human-technology relationships. Post-phenomenology argues that technologies in use shape (and in-turn, are shaped through) human experiences and existence, and consequently, human-world relations. This phenomenon is called technological mediation (Ihde, 1990; Verbeek, 2005a). Verbeek (2005b) developed this concept theoretically to formulate mediation theory and a vocabulary of mediation. In it, he suggests that humans both handle and perceive with technology, and is critical of theoretical approaches that attempt to build a universal understanding of technology in terms of the conditions of its possibility (Verbeek, 2005b). In addition, he also opposes a feature or functionality centric understanding of human-technology interactions arguing that such an understanding too easily reduces the role of products to instrumentality (Verbeek, 2015a). Instead he proposes that such an understanding should be developed through an analysis of technological mediation by concrete artefacts in a use context. This artefact centric approach has been called a material turn in the philosophy of technology (Verbeek, 2015b). Interestingly, this is in line with the material turn in interaction design research that has argued for developing non-instrumental accounts of technology as a material for similar reasons (Wiberg, 2014).
As mentioned earlier, currently, smart consumer technology is primarily described in terms of its utilitarian and functional characteristics like context-awareness and proactive behaviour (Mühlhäuser, 2007). From a design standpoint, it offers a set of technical features that can be incorporated within artefacts but does not aid critical and exploratory engagement or understanding of the design space of potential possibilities. Therefore, in this article, we instead attempt to develop a conceptual understanding of smart consumer technology in terms of technological mediation by its concrete artefacts. Verbeek (2005b) links the materials and materiality of technological artefacts to their mediation. He argues that technological artefacts “fulfil their functions as material objects, and by this functioning they shape human actions and experiences” (p. 209). Consequently, we argue that an understanding of mediation would also allow us to reflect on the material composition and materiality of smart consumer technology. Mediation, material composition, and materiality, taken together, would help outline smart consumer technology as a material for design—or something whose materials and mediation can be shaped through design (Verbeek, 2005b).
Moreover, we argue that such an understanding could also help frame a design space for critical and exploratory designerly practices, which, in turn, would help evolve and expand our initial understanding. In fact, the rapidly evolving nature of this technology underlines the importance of an understanding that is evolving and open to critique. To highlight this, we describe the outcome from a material centric and speculative research through design process (Auger, 2013; Wiberg, 2014; Zimmerman, Forlizzi, & Evenson, 2007), which focused on an artefact named Hearsay. We also describe its technological mediation, to discuss how it critically engages with our conceptual understanding and helps expand it.
This work makes contributions to both interaction design theory and practice. First, we develop an initial conceptual understanding of smart consumer technology in terms of its technological mediation, material composition, and materiality. Second, we present how this conceptual understanding can be critically engaged within design practice to develop new design concepts. Lastly, we highlight how theory and exploratory practice can be engaged together to develop an evolving understanding of smart consumer technology as a material for design.
Outline
We begin by outlining our methodology and briefly introducing our theoretical framework—post-phenomenology and mediation theory. Following this, we present an analysis of technological mediation by smart consumer technology in everyday domestic contexts and use this analysis to reflect on its material composition and materiality. Next, we transition to design exploration, structured around a speculative design artefact named Hearsay, and briefly present an analysis of its technological mediation. We conclude by presenting reflections from both our initial analysis and design exploration and their mutually informing nature.
Methodology
Fallman (2008) proposed the interaction design research triangle as a model for interaction design research, framed around the activity areas of design studies, design practice, and design exploration. These activity areas correspond to descriptive-analytical, practice-based, and constructive-exploratory forms of research conduct respectively. Design studies are a highly analytical mode of engagement, intended to describe and further the understanding of people, processes, and/or products within interaction design research. Design studies involve adopting and appropriating theories and practices from other disciplines, like sociology and psychology, and contributing new knowledge to design theory, design methodology, design history, and design philosophy. Design practice consists of hands on, reflexive designerly engagement (Cross, 2001) with real world issues and constraints within non-academic contexts, but with a research question in mind. Finally, design exploration is a synthetic and constructive research conduct that frames alternative realities, comments on on-going societal phenomena and/or highlights possibilities and examples within a design space. Such explorations are driven largely by self-initiated projects formulated around the researcher’s own agenda rather than by expected market or user needs. Fallman also emphasises the importance of controlled movement between the activity areas for generating interesting and novel results.
Our methodological approach is framed by mutually informing movements between design studies and exploration. We begin by using mediation theory and the vocabulary of mediation to build a conceptual understanding of smart consumer technology currently, in terms of the technological mediation of its artefacts in use, its material composition, and materiality. We proceed to expand this understanding by exploring and articulating alternative possibilities for the future, through material exploration and prototyping a new design concept. This allows us to constructively unpack new possibilities and problematics associated with emerging or new constituent materials of smart consumer technology and to generate conceptual narratives that expand our understanding with concrete images of the possible (Folkmann, 2011). Moreover, this helps emphasise the need for a continuously evolving understanding of smart consumer technology. This marks a move from a descriptive to a constructive mode of interaction design research conduct. With constructive design research conduct, we refer to “research that imagines and builds [constructs] new things and describes and explains these constructions” (Koskinen, Zimmerman, Binder, Redström, & Wensveen, 2011, p. 6). Therefore, our methodology incorporates research through design in addition to conceptually working with mediation theory. Research through design is a methodology that uses the “methods, practices and processes (and outcomes) of design practices with the intention of generating new knowledge” (Zimmerman & Forlizzi, 2014, p. 167). While it has its roots in design and artistic research (Frayling, 1994), it has since been adapted and adopted within interaction design research (Basballe & Halskov, 2012; Höök & Löwgren, 2012; Stolterman, 2008) and HCI design (Gaver, 2012; Zimmerman et al., 2007) as well. The use of designerly processes makes it a constructive and analytical research methodology (Gaver, 2012; Koskinen et al., 2011). It generates knowledge that is closely linked with the contextual outcomes and processes of design practice rather than being extensible and verifiable (Gaver, 2012).
In the following section, we briefly introduce our theoretical framework and describe the main concepts that inform our research.
Theoretical Framework
Post-Phenomenology and Mediation Theory
Don Ihde (1976) described phenomenology as “an intense examination on experience in its multifaceted, complex and essential forms” (p. 17) and critically adapted concepts from classical phenomenology in the context of modern technologies. Ihde (1990, 2009), along with other contemporary philosophers of technology (Rosenberger & Verbeek, 2015b; Verbeek, 2005b) refer to this adapted understanding of phenomenology as post-phenomenology. While post-phenomenology retains classical phenomenology’s focus on human experience, it critiques its alienation centric view of technology, opting instead to think in terms of mediation. It conceptually outlines technological intentionality (Ihde, 1990) to emphasise the non-neutrality of technology in mediating human-world relationships. Ihde argued that technologies have intentions of their own that play an important role in shaping people’s relationships with the world based on how they evoke certain kinds of use and how they contextually get used. Similarly, Verbeek suggested that technologies neither merely distance us from the world nor do they exist solely as functional and instrumental objects. Instead he suggested that technologies “are to be understood in terms of the relations human beings have with them, not as entities ‘in themselves’” (Rosenberger & Verbeek, 2015a, p. 19). Rather than determining or being determined, humans and technologies mutually constitute and shape each other through mediated relations that emerge between them (Verbeek, 2015a). Methodologically, the post-phenomenological approach takes actual technological artefacts and developments (rather than philosophical theories) as its point of departure for analysing human-technology relationships (and understanding technology). Rosenberger and Verbeek articulate it as a “philosophy of technology [that] is in a sense a philosophy ‘from’ technology” (p. 10).
Verbeek (2005b) proposed that technological mediation shapes human existence through the translation of actions and involvement with reality and human experience by transforming perceptions and contexts of interpretation. Taken as a whole, technological mediation can describe and help us understand “how [technological] artefacts help shape how humans can be present in the world and how the world can be present for them” (p. 195). He outlines this understanding of mediation in a post-phenomenological theory called mediation theory. While technological mediation allows us to understand technology as a material that designs everyday life through its artefacts, it also enables a reflexive analysis of its material composition and materiality. Verbeek argues that materials, materiality, and mediation are closely linked, in that they mediate through their material presence, on a sensorial level. He suggests that “perceptions and actions always have an aspect of sensorial contact with reality, which is precisely the point of application for mediation by material artefacts” (p. 209). Therefore, mediation theory allows us to make micro-scale analyses of the mediating role of material technological artefacts as a means of building an understanding of technology itself (in terms of the relations and interactions human beings have with them; Rosenberger & Verbeek, 2015a). For instance, a pen mediates a different writing style than a word processor within the same context. Pens allow for free form sketching, scribbling notes and fewer possibilities of editing or copying text. Due to the material properties of a word processor, the style of writing is more prose driven or speech-like (from Verbeek’s (2005b) discussion of Ihde (1990)).
In the following section, we introduce Verbeek’s vocabulary of mediation and briefly highlight how it can be used to describe and understand technology.
A Vocabulary of Technological Mediation
Verbeek (2005b) uses a structure of conjoined transformation of human perception and translation of actions and involvements by technological artefacts to outline a vocabulary for describing technological mediation. He describes a transformation of perception due to technological artefacts amplifying or strengthening some aspects of reality while reducing or weakening others. He argues that discussing the transformation of perception in terms of amplification and reduction allows us to describe and analyse perception in terms of specific forms of access to reality, no richer or poorer than the other. Another aspect of the transformation of perception is its ability to shape and be shaped by the cultural context in which it is situated. Verbeek describes it with the help of the related post-phenomenological concept of multistabilities (Ihde, 1990), which implies that technological artefacts can potentially evoke different forms of use and human relations depending on the individual and cultural context in which they are situated. Through multistable relationships, technologies get embedded and interwoven with culture and consequently transform and get transformed by it.
Verbeek (2005b) adapts the concept of translation from Actor Network Theory (Latour, 1992) in the context of technological mediation to describe the translation of action. He critiques Borgmann’s (1984) device paradigm’s focus on alienation and instead suggests translation be used to understand human engagement and involvement with reality. Verbeek outlines the translation of action as technological artefacts inviting certain kinds of actions, and enabling some forms of involvement and engagement, while inhibiting or discouraging others. He further suggests treating involvement with the artefact, with the product it makes available, and with the reality it mediates, as a “dimension of technological mediation and not as something that technology excludes or renders impossible” (p. 191).
The structure of invitation and inhibition of the translation of action parallels the structure of amplification and reduction of the transformation of perception. However, invitation, inhibition, amplification and reduction are not inherent properties of an artefact but rather a consequence of their role as active and non-neutral mediators of human action and perception. Verbeek (2005a) explains that “when technologies are used, they co-shape human-world relationships: they make possible practices and experiences, and in so doing, they play an active role in the way humans can be present in their world and vice versa” (p. 140).
In the next section, we outline a conceptual analysis of technological mediation by smart consumer technology and then, based on this description, go on to reflectively highlight its material composition and materiality.
Understanding Smart Consumer Technology as a Material for Design
Method
We use phenomenological analysis as a method to develop a conceptual understanding of smart consumer technology as a material for design. Ihde (1986) describes phenomenological analysis as a method of probing for what is genuinely discoverable and potentially there, but not often seen (p.13). He argues that a phenomenological analysis elevates understanding from a literal or direct state to a polymorphic state, one that allows for multiple, potentially unseen, possibilities to be perceived within a phenomenon. In a polymorphic understanding of a phenomena, multiple possibilities co-exist simultaneously, without any kind of hierarchy or one seeming more true/real than others.
Phenomenological analysis allows us to describe everyday experiences with (the concrete artefacts of) smart consumer technology. In addition, it allows us to use these descriptions as an empirical basis for analytically framing a conceptual understanding of technological mediation by smart consumer technology. We use the vocabulary of technological mediation (Verbeek, 2005b) to articulate the analysis. Finally, we use the hermeneutic or interpretative rules discussed by Ihde for developing, constraining and structuring a phenomenological analysis and to avoid subjective judgements. We describe these rules next.
First, Ihde (1986) suggests that a phenomenological analysis should attend to phenomena as and how they show themselves. This means that the analysis should focus on how the phenomenon is actually experienced and perceived without any kind of structure, preconceived bias or knowledge applied to it. Next, a phenomenological analysis should describe and not explain phenomena, avoiding articulations of why a phenomenon occurs but rather working with how it takes place. In our case, this would mean avoiding the technological/functional reasons behind why a phenomenon occurs but rather articulating how smart consumer technologies transform perception and translate actions. Lastly, a phenomenological analysis is developed using the variational method, initially developed by Husserl (1970). The variational method uses descriptions of multiple examples or variations of a phenomenon in order to deconstruct and unpack its essential or invariant features. Ihde describes variations as examples that possibilize phenomena. He describes an experienced phenomenon as consisting of present and absent-within-present aspects and suggests using variations to identify them. Taking the example of a book, he points out that, looking from above, one can see its front cover and thickness. However, the back cover, while visually absent, is still perceptible due to the thickness and weightiness of the book. Further, considering the importance of describing phenomena as and how they show themselves, we limit our focus to the interactions and use of the artefacts rather than how they serve as signs, their meanings or how they are socially or culturally interpreted (Verbeek, 2005b). Our experiences and interactions with multiple artefacts (variations), serve as the basis for developing a polymorphic understanding of smart consumer technology as a material that designs, by mediating human actions and perceptions. Finally, by reflecting on the materials and materiality of the artefacts described, we outline an initial understanding of smart consumer technology as a material in design.
However, we would like to point out that our intent with this discussion is not to get to a comprehensive or exhaustive definition of smart consumer technology but rather to develop an initial bottom-up understanding of its technological mediation, materials, and materiality. In fact, in the remainder of the paper, we use the conceptual understanding which we develop to frame and articulate a design exploration which then helps critique, and reflectively expand this understanding.
Technological Mediation
We start by identifying concrete and diverse examples of smart consumer technology for our analysis. We scope our discussion to the emerging forms of smart technology that, besides being connected, also incorporate forms of automation and intelligence. Popular examples of smart consumer technology include smart-phones, smart TVs and speakers (entertainment), cameras and locks (home security), and health and fitness trackers (fitness and health); (see Brown, 2017; Olick, 2017). This selection of artefacts forms the basis of our phenomenological analysis. An additional factor affecting the choice of artefacts was our own access to them since this analysis is mainly developed through a reflexive description of our own experiences with smart consumer technology.
The Physical (Smart) Artefact
We begin with direct and everyday experiences with smart consumer technology. Primarily, these manifest themselves in everyday experience as tangible or physical (smart) artefacts (or smart artefacts). As smart artefacts, like a camera, activity tracker, TV, or speaker, they offer utilitarian functions which are encapsulated in specific physical forms. However, their physical interface is usually minimal and limited to buttons and/or sockets for power and visual indicators like small lights or displays. The visual indicators present information about the artefact’s state and, in some cases, about the available data. For example, smart cameras and activity trackers have a power socket or a charging dock and indicators that present their powered on and functional state. Some activity trackers also have a small display that shows the time of day, steps walked, and other fitness related data points as well. By presenting a minimal set of controls and data, the smart artefact inhibits extended and active engagement and interaction. However, the visual indicators do amplify awareness of the functional state of the artefact, like its powered-on state.
Additionally, the smart artefact needs to be physically situated and oriented based on expected use, level of accessibility, and visibility. For example, a smart camera needs to be directed towards the physical space that needs to be monitored. Similarly, an activity tracker needs to be positioned at an appropriate location on the body for effective tracking. Their location and orientation invite differing kinds of involvement with their physical form. Since activity trackers are situated very close to the body, their physical form is continuously available. Therefore, it invites physical interactions more often than other smart artefacts like a thermostat or smart camera. For example, after an extended period of use, we may modify the location of the tracker to relieve some discomfort or check the screen/visual indicators on the artefact to get a quantified measure of their activity. Smart cameras or thermostats usually work in the background inviting minimal physical interaction and are not reconfigured unless they stop working. Similarly, smart speakers and TVs involve limited reconfiguration and physical interaction with the physical artefacts themselves. However, minor adjustments notwithstanding, once worn, activity trackers are also designed to be situated in the background and are rarely noticed due to their light weight and sleek physical form.
The Networked (Smart) Artefact
Now, we turn our attention to elements that are absent from direct perception and action (aspects of mediation), in an attempt to elevate our understanding. While often there may be no physical sockets or interface elements that overtly indicate the necessity of a network connection, a smart artefact needs to be configured to join the local domestic or telecom network to be used in a meaningful way. This presents us with a variation that incorporates the absent-within-present networked and remotely available interface as a distinct aspect of technological mediation.
The necessity of a network connection constrains the physical location and orientation of the smart artefact to areas with consistent network availability and reach. For example, a smart camera needs to be situated such that its location is properly covered by the local network so that it does not intermittently disconnect and interrupt the monitored feed. Similarly, an activity tracker needs to be connected to a phone or GSM chip to be able to access the network. However, the minimal physical interface inhibits the possibility of configuring the network connectivity from the artefact itself. Consequently, we configure the artefact’s network connection over a remote interface accessible via a smartphone app or web page. It enables the artefact to connect (and stay connected) using the local network’s settings and credentials. Once connected, the remote interface represents/displays data captured by the smart artefact. This data is usually constructed from one or more specific aspects of everyday life, like the ambient temperature on the thermostat, metrics relating to the body and movement captured by the activity tracker, and people or objects present in the field of view of a camera. Moreover, the represented data is not static in nature but keeps updating continuously. This aligns logically with the fact that smart artefacts are continuously powered on and continuously capture data. The continuously connected data captured by the smart artefact, and the continuous representation of updated data on the remote interface, suggest a perceptible but invisible link between the two. Put differently, the networked (smart) artefact captures/generates and transmits a continuous stream of data once connected, that is represented over remote interfaces. By a stream, we refer to a continuous flow (of data) that is transmitted or received over the network connection. For example, a smart camera streams its captured/tracked video or image while an activity tracker tracks and transmits different data-points about the body, the thermostat senses and streams temperature data and smart speakers and TVs stream content and sometimes represent data from other smart artefacts.
These representations lead to the emergence of diverse connected material forms (or material forms emerging out of a continuous data stream) that mediate how we perceive and experience aspects of everyday life. The smart camera makes both live and recorded camera feeds remotely available over the internet allowing a monitored space to be remotely present on demand across time and space. Similarly, a smart thermostat makes the ambient temperature of a house available to remote apps. Therefore, while each artefact gets activated by being powered on, it depends on continuous data streams to mediate action and perception in different ways. Actions and controls that can manipulate the data, or in some cases reconfigure the artefact, are also presented along with the representations. For instance, an activity tracker allows the user to set thresholds for different kinds of activity goals and to change the nature of activity that was tracked. A smart thermostat allows the user to alter the temperature remotely through the mobile app interface. But even though smart artefacts are continuously powered-on and connected, interaction with them tends to be driven primarily through notifications to us when important/noteworthy events have occurred. For example, thermostats do not require continuous interaction to function and are usually designed to blend into the background of the user’s home. Activity trackers do not need constant monitoring since they can notify us when their battery is low or if we have walked the pre-set number of steps.
The Intelligent (Smart) Artefact
On reflectively considering the nature and experience of the connected material forms made available by networked (smart) artefacts, we are presented with a third variation of smart consumer technology. We note that the representations and notifications presented to users are not the direct/raw captured data stream. Rather they contain interpretations and visualizations that are intended to help users make sense of the data. For example, a smart thermostat co-relates temperature data to power consumption over time, to infer if the power consumption is within an ideal range or not. Here the ideal range is also inferred from historical trends and factors like time of day, weather, season, location and so on. The artefact notifies the user about inferred patterns of interest within the streaming data, usually with recommended actions that they could take. It invites a form of selective and on-demand involvement by amplifying information through notifications and events. For instance, an activity tracker does not require any interaction unless the user has been in a sedentary position for a long period. The tracker then sends a notification to the user either directly or using its remote interfaces with suggestions for taking a break or moving around. A smart camera can notify the user when an unexpected presence is detected in the space being monitored while a thermostat can identify excessive power usage and suggest lowering the temperature.
Along with recommending actions, patterns and meanings inferred from the streaming data, this data also enables the artefact to act autonomously, further reducing notifications and active involvement with it. For example, smart thermostats monitor patterns of use and ambient temperature data to predict our requirements and automatically alter the temperature. Activity trackers infer the nature of our activity by monitoring and sensing data-points about our body like movement and heart-rate, etc. The passive/notification driven nature of interaction with smart artefacts further reduces the perception that they are continuously capturing, streaming, and interpreting the world around them. Lastly, intelligence can also be perceived in the way that artefacts invite different ways of accessing information and the way in which they manipulate connected material forms. For instance, voice-based speakers intelligently recognise voice and understand our intent (to a certain degree) while generating and communicating a response with relevant information through an artificially synthesised voice.
To summarise, through our analysis, we have uncovered three variations of smart consumer technology, based on the ways in which smart consumer technology mediates human perceptions and actions. Within each variant, we have articulated aspects of technological mediation using multiple examples of transformations of perception and translations of action caused by concrete smart consumer technology artefacts. Through these variations, we have attempted to develop a polymorphic understanding of technological mediation by smart consumer technology in everyday domestic settings. At this point we would like to re-emphasise that neither do these variations exist in a hierarchy nor are some of the variations more real/complete than others. Rather, as Ihde (1986) argues, all of these variations co-exist, simultaneously, within our understanding. We summarise our initial analysis in Table 1 and in the following section use this understanding to reflectively unpack the material composition and materiality of smart consumer technology.
Table 1. Summary of technological mediation by smart consumer technology.
| Variations | Translation of actions | Transformation of perception | ||
| Invite | Inhibit | Amplify | Reduce | |
| Physical (smart) artefact |
|
|
|
|
| Networked (smart) artefact |
|
|
|
|
| Intelligent (smart) artefact |
|
|
|
|
Material Composition
The polymorphic understanding of smart consumer technology (see Table 1), suggests that its material composition should also be considered as a hybrid of physically and remotely present materials (Knutsen, 2014). Besides the artefact’s form and interface elements, the smart artefact consists of computational materials (Vallgårda & Redström, 2007), like sensors, actuators, processors, radio transmitters, and local storage, to generate, connect, transmit/receive, and optionally, locally store snapshots of data streams. Further, machine learning algorithms are used to analyse the streaming data to infer patterns of use, meaning, and intent. Machine learning algorithms are also used to infer opportunities for automation, based on patterns of use. However, due to their computationally intensive nature, they are usually run on servers located in remote data-centres. Therefore the remotely located data-centres and networking infrastructures, the data streams, and the algorithms involved, form an integral yet invisible part of the material composition of smart consumer technology. For instance, a smart camera’s networked service analyses and interprets the data stream using machine learning algorithms to infer details such as unexpected presence in the room or activity detected.
The material composition of the physical, networked, and intelligent smart artefact, considered together, mediates perception and actions in two different ways, 1) situated in and directed at a specific aspect of everyday experiences and 2) in the context of general aggregation, representation and control of multiple aspects of everyday experience. Using an analogy from network systems, we refer to them as endpoints and hubs respectively. As endpoints, the artefacts tend to be focused on capturing or tracking a specific aspect of everyday life, like generating activity and movement related data in the case of activity trackers and spatial and visual data in the case of cameras, which are then streamed over a network connection. User interaction primarily happens via their remote interface while the physical artefact is mainly present in the background. Hubs, on the other hand, primarily play representational and control related roles, in that they aggregate endpoints’ interpreted data streams and represent their remote interfaces. They also allow endpoints to be controlled through these interfaces. The connected material form presented by hubs is also extendable in nature, which allows endpoints to make interpreted and streaming data available as visual (in the case of smartphones or TVs) or voice-based apps (in the case of speakers). However, this difference should be understood from the point of view of mediation and material composition and not as a discrete typology of smart artefacts themselves. In fact, due to the generative nature of endpoints and the aggregational nature of hubs, their connected material forms can potentially be complementary in nature. Consequently, an artefact can be designed to be both an endpoint and a hub, as in the case of a smart-watch which can also act as an activity tracker. This highlights how smart artefacts can be designed to be directed at a specific aspect of experience while also playing the aggregational role of a hub. Our intent behind highlighting this difference is to underline the different ways in which the material composition of smart consumer technology manifests itself in everyday life.
Materiality
Based on our analysis of the technological mediation and the material composition of smart consumer technology, we reflectively highlight important aspects of its materiality.
Continuous
Continuous sensing and connectedness is an integral element of the materiality of smart consumer technology. It is through continuous sensing and network streaming (transmission and reception) that smart artefacts deliver most of their functions and, often, parts of their interface. Consequently, the material composition of smart artefacts assumes constant connectivity, sensing, and streaming. Once configured, the artefacts do not require any human interaction to remain connected and to continue streaming. However, a connected smart artefact can get disconnected, either by the user, or unintentionally, due to network or power outages. Once disconnected, the tracked data stream becomes unavailable for transmission and/or for aggregation and control. Therefore, we can observe that the continuous nature of materiality is temporal, and to a certain degree, controllable. We can connect, disconnect, and re-connect smart artefacts, switch them on and off, and control their continuity in some cases by altering the rate at which the artefact captures and streams data (like controlling the frame-rate of a smart camera).
Moreover, smart consumer technology tends to be situated in the background of human experience and perception. Activity trackers are worn on the body while smart thermostats are situated in a central area of the home, but in both cases, their sensors, transceivers, and network services and algorithms work in the background, implicitly enabling the remote monitoring and control of fitness information, and the monitoring and manipulating of ambient temperature respectively. Generally speaking, this is what Ihde (1990) refers to as a present absence of technology or a background relationship between humans and smart consumer technologies. However, in the absence of streaming data and network connectivity we directly experience both the physical and remotely available material elements of smart artefacts and their foundational significance in delivering functions that may otherwise be taken for granted.
Absent Presence
While smart artefacts and their material composition have a present absence in use, the connected material forms they generate have an absent presence instead. The physical artefact usually has minimal or no interface and is controlled, configured, and monitored over a remote interface. This allows an artefact’s material composition to be absent from direct perception and yet be remotely present and controllable by a user. The remotely present materials are also configured in a kind of absent presence. While physically absent, the remotely present materials need to be present and available for the artefact to function. A smart camera’s feed can be viewed remotely from its smartphone app, but it depends on network services to stream and deliver its tracked feed and may stop operating if it gets disconnected. The absently present nature of connected material forms and remotely present materials highlights new ways in which we relate to smart consumer technology and how it mediates our relations with the world. On the one hand, it allows for the possibility of extending our reach and availability along with that of our environment across space and time, but on the other it also results in the creation of black-boxed artefacts which depend on absent (remote) materials. For instance, a smart camera makes our lived environment available to us remotely and gives us a sense of security. However, if the networked services transmitting the data stream become unavailable, it can render the (otherwise functional) physical artefact unusable as well. Moreover, since the remotely present materials are largely unseen and proprietary in nature, they result in a general lack of understanding of smart consumer technology. This may create issues relating to repair and long-term durability, since an artefact may stop functioning if its absent remote materials become unavailable. Moreover, the absently present materiality of the connected material forms may situate the physical artefact as a utilitarian carrier of function—that of creating data streams, limiting its psychological durability (Van Hinte, 1997; Verbeek, 2005b) and engagement.
Represented Interpretations
The materiality of smart consumer technology is representational and contextually interpreted, since it involves the presentation of the data stream from the artefact in very specific ways along with contextual inferences drawn from the data. An activity tracker tracking heart-rate data streams it to its network service which situates it in a health and activity related context. It infers contextual cues about the nature of activity and approximate calorie burn. The data, along with the inferred cues, is represented back to the user using an interface language that is situated in the same health related context. Therefore, the activity tracker app uses language and visual cues from the health and fitness domain to present comparative and time-based views of activity using metrics like ideal and target calorie burn. However, a different service could potentially situate the same data in an entirely different context and consequently infer different meanings out of it as in the case of pplkpr (http://pplkpr.com/), an artistic project that uses heart-rate data to infer and quantify the quality of interactions between people. The fact that different meanings can be deduced from data dependent on the context in which that data is situated highlights the importance, while designing, of exploring potential alternative contexts for the interpretation and representation of data streams, in order to be able to understand their affordances, possibilities and challenges.
Encoded Subjectivities
Reflecting on the inferences which can be drawn from intelligent (smart) artefacts, (and especially the recommendations which they make) we can start to see hints of underlying subjective meaning making mechanisms. For example, consider a smart thermostat that recommends lowering the temperature to a green, environment-friendly zone, or an activity tracker that suggests moving more during the day to be healthier. The algorithms involved in identifying these patterns and making these suggestions are doing so based on general patterns of environmentally friendly energy consumption or healthy amounts of movement during the day. However, phenomenologically speaking, there are no absolutely or universally true representations of reality. Therefore, such interpretations and recommendations are indicative of subjective meaning making that gets encoded within machine learning algorithms. We also observe how some forms of machine learning outcomes can potentially be easier to quantify and interpret than other, more subjective ones. For instance, depending on the cultural context, emotions, expressions and gestures can have very different interpretations and meanings.
The conceptual understanding, developed in this section, can be used to critique smart consumer technology and the current state of interactions with it. However, it can also be used as a basis for exploratory and critical designerly engagement. Such design work would not only be informed by this conceptual understanding but could also expand and evolve it. In the following sections we describe such a design artefact, that combined our conceptual understanding with a material centric speculative research-through-design process. Methodologically, it underlines a movement from a descriptive to a constructive mode of engagement and highlights how these seemingly different modes of engagement can be utilized in concert in order to critically develop an evolving understanding of smart consumer technology.
Expanding Understanding through Design Exploration
Approach
Methodologically, our design exploration is framed as research-through-design, and conjoins a material centric process (Wiberg, 2014) with speculative design (Auger, 2013; Dunne & Raby, 2013). A material centric design research process works through deep and reflective engagement with the materials in design to understand them better and how they might co-shape a design process and its outcomes (Wiberg, 2014). In our design process, we worked back and forth between materials and materiality (Wiberg, 2014), and considered the materials being explored as conversational objects that talk back to the designer (Schon, 1992). Wiberg argues that such an engagement changes our relationship to, and experience of, these materials. This helped develop an understanding of the way that intelligent materials, like machine learning algorithms, are (and can be) used in smart consumer technologies, through hands on exploration. We complemented the material centric process with speculative design, to critically engage with the conceptual understanding of smart consumer technology developed previously. Auger (2013) describes speculative design as a practice-based design methodology that exists free of commercial constraints and uses design artefacts both as the means of enquiry and as points of departure for narratives describing alternate systems and/or worlds. From our perspective, the discursive, experiential, and narrative driven nature of speculative design allows us to explore alternate forms of technological mediation and material composition, and to critically expand our understanding of smart consumer technology as a material that designs. However, in our process and account, we do not address concerns and problematics related to purposefulness, user needs, and requirements along with conducting formal evaluations and user studies. The importance and role of such exploratory research intents has been highlighted in prior interaction design research such as (Arnall, 2014; Auger, 2013; Backlund et al., 2007; Hallnäs & Redström, 2002; Knutsen, 2014; Wiberg, 2014).
This section is structured around a speculative design artefact—Hearsay (Figure 1). We briefly introduce it and present its formal and material composition. We then articulate its technological mediation using a short phenomenological analysis.
Figure 1. Hearsay—a smart voice activated lamp: Material elements and functional states.
Top: Uncovered state; Bottom: Covered state.
Design Exploration: Hearsay
Hearsay is the outcome of a material centric speculative design process that is situated within the domain of a specific kind of smart consumer technology-intelligent voice-based smart artefact, like the Amazon Echo (https://www.amazon.com/Amazon-Echo-Bluetooth-Speaker-with-WiFi-Alexa/dp/B00X4WHP5E) and Google Home (https://madeby.google.com/home/). At its core Hearsay is a smart voice activated lamp that is always in conversation. It generates quirky and whimsical (rather than efficient and functional) responses while incorporating transparency (rather than obscurity) in its aesthetic, interaction, and mediation (Figure 1). We explored alternate, non-utilitarian design possibilities and oppositions (DiSalvo, 2012; Pierce & Paulos, 2014) to mass market products in our design process. Pierce and Paulos (2014) describe this design strategy as one that employs functional oppositions and leads to counter-functional things as outcomes. Conceptually, counter-functional things, are artefacts that figuratively counter some of their own essential functionality. Hearsay is a counterfunctional thing that employs oppositions to highlight alternate possibilities in the design space for smart consumer technology. Additionally, through narratives of its use, we present a brief phenomenological analysis of its technological mediation. The variations outlined in this analysis can be referred to as designed variations (Pierce & Paulos, 2013), since they describe phenomena related to a concretely designed artefact but use conceptual or potential narratives of use as a way of highlighting alternate possibilities within a design space. Ihde (1986) also discusses the use of story devices as a way of uncovering variations within phenomena, suggesting that within the context set by the story, experience takes shape. He argues that stories derive their power of suggestion from familiarity or from elements of ordinary experience. Generally, Hearsay demonstrates how engaging with exploratory research through design can help expand and evolve our initial conceptual understanding.
A Note on Material Exploration
While the entire design process is not the explicit focus of this article (see Pandey & Culén, 2017), reflections from material explorations (Hallnäs, Melin, & Redström, 2002) of machine learning, conducted as a part of the process are important in order to understand Hearsay’s designed interactions. In our explorations, we used open source software rather than cloud-based machine learning services to better understand the internal mechanics and complexity of machine learning algorithms. Within the context of smart consumer technologies, machine learning usually takes the form of supervised learning. Supervised learning algorithms learn from labelled data-sets that consist of pre-selected inputs and their true (or correct) output labels. Multiple supervised machine learning algorithms were explored, both with audio and visual data like speech, image capture and live video. Two specific examples are, neuraltalk (Karpathy, 2016) to generate textual interpretations of images and video streams, and the seq2seq conversation modelling algorithm (Sutskever, Vinyals, & Le, 2014), used to generate textual responses to snippets of speech.
The interpreted outcome from all the explorations was highly generative in nature. For instance, new sentences were generated during image interpretation using neuraltalk that were not present in the training data-set. This highlights the fact that the algorithm does not merely remember the image descriptions from the training data but infers the relationship between the description and the objects in an image. Second, rather than remembering specific outputs correlating to the inputs in the training data-set, the algorithms infer relationships at a granular level, like the relationship between the words in a sentence (to be able to construct new and meaningful sentences). This allows algorithms to reasonably adapt to a wide variety of new inputs that share some patterns of similarity with the training data set. For example, seq2seq was able to create new responses to completely new and arbitrary input dialogs. Adaptability meshes well with the continuous nature of smart consumer technology discussed in the previous section, where the input data-stream, being situated in everyday life, may be quite unpredictable and varied.
In the following section, we describe Hearsay’s formal, functional, and material composition.
Formal, Functional, and Material Composition
Hearsay’s physical form draws inspiration from the playful yet striking aesthetic of Italian radical design (Malpass, 2017), specifically that of the Memphis-Milano design group (http://memphis-milano.org) and Studio Alchimia (http://www.alchimiamilano.it/). The removable cover (lampshade) is translucent and shows a faint outline of an evocative physical form inside. The evocative form contrasts with the minimal cover, giving Hearsay a layered aesthetic. Functionally, Hearsay is a lamp, which can be switched on and off using voice commands. If the removable cover of the lamp is kept on (covered state), the audible responses are muted (but are still generated and saved) and the interactions are limited to controlling the lamp. Removing the cover, un-mutes the artefact and reveals the evocative form (uncovered state). The form is used to highlight the artefact’s material composition, like the speaker, microphone, network connection, and a transcript of all the conversations and responses (captured both while muted and unmuted). In the uncovered state, the light from the lamp is also dimmed to create a soft and intimate environment for conversations. Hearsay connects to the internet using a pre-configured wireless router that needs to be attached to the user’s modem via an Ethernet cable. Hearsay automatically connects to the internet once the router is attached. Once connected, Hearsay is always listening and responds as soon as it detects audible and discernible voices. This is in opposition to most voice-based interfaces, that get activated using a hotword [a particular keyword like Alexa (https://developer.amazon.com/alexa), OK Google (https://madeby.google.com/home/), etc.].
From a material standpoint, Hearsay generates a data stream through a microphone that is activated whenever audible voices are detected. The microphone is connected to a portable computer that streams the captured audio to a speech to text network service, which transcribes and streams the text back to the artefact. A response to the transcribed text is generated locally on the computer using a seq2seq machine learning algorithm (Sutskever et al., 2014) trained on a dataset consisting of movie subtitles (Danescu-Niculescu-Mizil & Lee, 2011). The program also infers the user’s intent related to switching the lamp on or off. However, machine learning is not just used for utilitarian purposes, like interpreting commands and playing confirmations as responses. Rather, it enables on-going, continuous and, being trained on movie subtitles, at times surprising and intriguing conversations (see Figure 2). It incorporates generativity and adaptability to allow for always-on interactivity and open-ended (and non-utilitarian) conversations. Additionally, Hearsay is designed to be experienced like an endpoint, directed at conversations and voices in its vicinity. However, it does not have a remote interface, like a smartphone app, and its interface elements, like volume and display control knobs, are located on the artefact itself. It is designed to work only with voice detected from within a close proximity, making it a self-contained or standalone counter-functional smart artefact from an interaction design perspective.
Figure 2. Hearsay in possible scenarios/narratives of use. Top: examples of real responses from the artefact.
(Click on the figure to see a larger version.)
Technological Mediation by Hearsay
The phenomenological analysis of Hearsay’s technological mediation is developed by describing snapshots of narratives of use highlighted in Figure 2. The narrative is based on real responses from the artefact in use. Throughout our analysis, we allude to the different functional states presented in the narrative.
Hearsay is situated at a particular location within the house based on expected use and the network coverage afforded by its dedicated router unit. As a physical (smart) artefact, it invites the user to choose its operational state (uncovered or covered). By leaving the lampshade on, the artefact presents itself as a smart lamp that invites the user to use voice-based commands to switch it on and off (functional intelligence). However, Hearsay does not depend on a hotword to begin listening. Hence, it listens continuously and generates responses anytime it captures audible and discernible voices. Even so, since Hearsay is muted, and the internal display is not visible in the covered state, the perception of generated responses is reduced. Therefore, as such, the covered state positions the artefact in a background relationship with the user, while being continuously connected and implicitly involved in everyday life. It streams and implicitly responds to captured bits of conversation in its vicinity, although these are not made explicit to the user (implicit participation). However, since Hearsay is a lamp, the lampshade needs to be periodically removed, like during initial configuration (when the bulb is fitted in) or for maintenance (like changing the bulb). Through these mundane actions, the otherwise hidden evocative physical form and material elements are introduced to the user. Due to the vibrant colours of this form and the light from the display seeping through the translucent lampshade, the artefact amplifies the form’s presence and invites exploration. The minimalistic covered and the evocative uncovered state of the artefact create a contrasting aesthetic that invites curiosity and exploration (see Figure 1).
The uncovered state physically amplifies the connected material form that Hearsay makes available—the captured conversations and their generated responses. These are displayed in a small screen, on the artefact. Captured conversations are not deleted, and so the user can explore them using a physical control on the artefact. This state physically presents Hearsay as a networked and intelligent (smart) artefact. The playful and often surprising nature of the generated responses invite greater engagement and exploration of the conversations from the past. During exploration, the user may also serendipitously discover implicitly captured conversations from the past (serendipitous surprise and implicit participation). The uncovered state also creates opportunities for the artefact to implicitly capture conversations and audibly play the generated response along with presenting it on the screen (serendipitous surprise). Hearsay inhibits being remotely controlled since the controls are only accessible on it. This reduces the absently present perception of its connected material form. The artefact invites physically situated and present interactions and involvement with its material composition and its connected material form, by requiring the user to be in the close vicinity. Engagement with the artefact and its material composition can also invite the user to keep it in the uncovered state along with positioning it in a much more accessible and closer location, like their desk (explicit participation).
Critical Reflections
Hearsay encapsulates the polymorphic variations of smart consumer technology in its physical form, aesthetics and interactions. Hence, it presents an opposition to existing forms of smart consumer technology physically (through its form and aesthetic), conceptually (through narratives of technological mediation), and functionally (through layered interactions). While it presents a minimal aesthetic in its covered state, comparable to existing smart consumer technology, the uncovered state presents a uniquely vibrant form that separates it from the background. Further, the form also communicates the artefact’s networked, computational and intelligent material composition, rather than rendering it invisible/inaccessible. It highlights the implicit involvement of smart consumer technology in everyday life, while also making gaps in machine learning and interpretation more transparent by presenting unintentional and erroneous transcriptions and responses. The material centric exploration leads to reflections relating to the generative and adaptable nature of machine learning algorithms. Because it can generate responses for a wide variety of conversation snippets, it allowed us to define a continuous listening and response based interaction for Hearsay. A rule-based system, on the other hand, could only respond to pre-configured commands and would throw an error message in other cases. Moreover, it would generate predictable responses, thereby reducing the serendipitous nature of the interaction. The nature of the responses generated through the use of a non-utilitarian movie subtitle dataset, helped frame the surprising and casual nature of the interaction. While the conceptual speculations affected the choice of algorithms explored, they in turn were also informed by the material exploration. The generative nature of machine learning implied that we could not predict Hearsay’s responses. Therefore, we needed to implement the algorithms and experience the responses generated ourselves, to be able to speculate on the narratives of possible use. Hearsay is an example of framing smart consumer technology as a material for design while also engaging critically with its design space, leading to outcomes that question dominant design paradigms, like invisibility and absent presence (in this case). The design exploration also shows new possibilities of interaction and technological mediation through a combination of conceptually and materially grounded design processes.
Discussion
In this section, we present reflections from the activity areas of design studies and design explorations while also highlighting the movements between them. However, rather than being comprehensive or conclusive, the discussion is intended to be inspirational and suggestive for future interaction design research.
Design Studies: A Conceptual Understanding
Starting from experiences with concrete artefacts allowed us to conceptualise smart consumer technology in terms of its technological mediation, material composition, and materiality, rather than criteria related to its utilitarian functions. Even though smart consumer technology consists of diverse artefacts, we have shown that it is possible to conceptually understand it in terms of concrete examples of technological mediation. From a design perspective, we think that a utilitarian understanding reduces the role of technological artefacts to just their instrumentality. Consequently, we see designed interactions with smart consumer technology mostly being directed towards automation and efficiency, with limited attention to the wider possibilities and problematics that might arise as can be seen in a BBC news story about cayla dolls (“German parents told to destroy cayla dolls”, 2017); Finley, 2016; Kastrenakes, 2016)]. We instead argue that a conceptual understanding, developed in terms of technological mediation, is useful within interaction design research for a nuanced and polymorphic articulation of the multistable nature of smart consumer technology (as physical, networked, and intelligent artefacts). First, it allows designers to question existing design paradigms, and explore and envision alternate possibilities of interaction, experience, and use. An example of a dominant interaction design paradigm can be seen in most smart artefacts not having manual controls and depending on smartphone applications for control and configuration. While these artefacts do incorporate the functional criteria for smart technology, such criteria do not help us to critically examine the continued use of prevalent design paradigms. Next, we think that an absence of critiques or exploration of alternatives can potentially reinforce existing interaction paradigms and make them seem fundamental to the smartness of a consumer technology. For instance, smartphone applications may seem fundamental to making a technological artefact smart. However, oppositions, in terms of material compositions and interactions, as are found in Hearsay, can be explored. Lastly, it also allows designers to think about the materiality and mediation of smart consumer technology in conjunction with other materials to shape better composites in terms of their aesthetics, form, and interaction (Redström, 2005; Vallgårda & Redström, 2007).
Design Exploration: Critical and Constructive Engagement
Hearsay’s material centric speculative design process critically engaged smart consumer technology as a material for design. The material exploration expanded on the understanding of smart consumer technology as a material in design. The speculative and conceptual exploration, critically engaged with it as a material that designs. Together, they helped frame the design concept in terms of its material composition, aesthetic, interactions, and narratives of technological mediation. Its material composition and layered interaction, (covered and uncovered state), scaffolds a polymorphic understanding of it as a physical, networked, and intelligent (smart) artefact. However, rather than emerging from an analytical reflection on absently present interfaces of smart consumer technology, these variations can be experienced in interactions with the physical artefact itself. Moreover, while we recognise that multistability is a natural outcome of technological artefacts in use, we think the layered nature of interaction and material composition scaffolds a polymorphic understanding through the emergence of multiple stabilities (see Figure 2). The layered and contrasting nature of its physical form and interaction is intended to invite exploration and physical engagement while the lack of a remote interface is intended to inhibit absently present interaction and involvement. Moreover, it amplifies the physical presence of the artefact in everyday life by requiring interactions to take place from within a close vicinity. Enabling interactions solely through the physical controls is also meant to involve the user in the artefact’s function while also inviting exploration and amplifying an understanding of the extent of the artefact’s implicit and explicit participation in everyday life. Therefore, from a mediation centric perspective, Hearsay presents an opposition to the background, and absently present nature of human-smart consumer technology relations, by being an example of an artefact that asks for involvement with itself (Verbeek, 2005b). We think that the absently present nature of the materiality of smart consumer technology has wider implications for other areas of consideration such as privacy, product durability, and obfuscated understanding of its functionality. As with Hearsay, an understanding of smart consumer technology as a material for design allows for the exploration of design concepts that explicitly consider these issues and presents alternate forms of technological mediation and material composition.
Movements: Framing Speculations—Critically Evolving Understanding
Movements from the conceptual understanding of smart consumer technology to design exploration can help frame designers’ intent and encourage thought which converges on areas for material and conceptual exploration. We have also argued that understanding smart consumer technology as a material for design, helps in framing clearer critical, reflective, and exploratory engagements with its design space. For instance, during Hearsay’s design process, our thinking converged on designing oppositions to the absently present/background nature of smart consumer technology, through physically situated, discovery-centric, and layered interactions. Additionally, we argue that our conceptual understanding also allows for more meaningful readings (Bardzell, Bardzell, & Stolterman, 2014) of design proposals—in terms of their technological mediation, material composition, and materiality. For example, we describe narratives of use for Hearsay, and use them to speculate on the potential forms of technological mediation in a domestic context.
At the same time, our work also underlines the importance of movements from design exploration to design studies as a way of critically engaging with, expanding, and evolving the existing understanding of smart consumer technology. For instance, describing Hearsay’s material composition, interactions, and mediation in terms of oppositions to our conceptual understanding, highlights possibilities different from the status quo. Moreover, movements from material centric exploration can be formulated as reflexive insights about the materiality of the emergent materials of smart consumer technology. We argue, considering the evolving nature of the material composition of smart consumer technology, such movements are crucial for a continuously expanding understanding of this technology.
Conclusion
“To the things themselves!” Verbeek (2005b) re-framed Husserl’s (1970) famous quote while calling for a material turn in phenomenology. In a similar vein, our work echoes the significance of focusing on the (existing and future) artefacts of smart consumer technology in interaction design research. Artefacts form the centrepiece of both our conceptual understanding and designerly exploration along with the mutually informing movements between them. We have highlighted how designerly engagement can lead to outcomes that can evolve and expand the current state of our understanding. We see a lot of potential in future design research in this space that poses different questions, explores diverse forms of material composition, interaction, and technological mediation, and continues challenging and evolving our understanding of smart consumer technology.
Acknowledgments
I would like to thank my supervisors, the reviewers, and the editors of this special issue for providing crucial and constructive criticism on earlier drafts of the manuscript and Jorun Børsting and Michelle Cheung for their help with early prototype development. This project was partially financed by the EU Creative Europe project The People’s Smart Sculpture, under grant number EC-EACEA 2014-2330.
References
- Arnall, T. (2014). Exploring “immaterials”: Mediating design’s invisible materials. International Journal of Design, 8(2), 101-117.
- Auger, J. (2013). Speculative design: Crafting the speculation. Digital Creativity, 24(1), 11-35.
- Backlund, S., Gyllenswärd, M., Gustafsson, A., Ilstedt Hjelm, S., Mazé, R., & Redström, J. 2007). Static! The aesthetics of energy in everyday things. Retrieved from http://www.redstrom.se/johan/papers/wonderstatic.pdf
- Bardzell, J., Bardzell, S., & Stolterman, E. (2014). Reading critical designs: Supporting reasoned interpretations of critical design. In Proceedings of the Conference on Human Factors in Computing Systems (pp. 1951-1960). New York, NY: ACM.
- Basballe, D. A., & Halskov, K. (2012). Dynamics of research through design. In Proceedings of the Conference on Designing Interactive Systems Conference (pp. 58-67). New York, NY: ACM.
- Borgmann, A. (1984). Technology and the character of contemporary life: A philosophical inquiry. Chicago, IL: University of Chicago Press.
- Brown, P. (2017). 20 billion connected internet of things devices in 2017, IHS markit says. Retrieved April 10, 2017, from http://electronics360.globalspec.com/article/8032/20-billion-connected-internet-of-things-devices-in-2017-ihs-markit-says
- Cross, N. (2001). Designerly ways of knowing: Design discipline versus design science. Design Issues, 17(3), 49-55.
- Danescu-Niculescu-Mizil, C., & Lee, L. (2011). Chameleons in imagined conversations: A new approach to understanding coordination of linguistic style in dialogs. Retrieved from http://www.cs.cornell.edu/~cristian/papers/chameleons.pdf
- DiSalvo, C. (2012). Adversarial design. Cambridge, MA: MIT Press.
- Dunne, A., & Raby, F. (2013). Speculative everything: Design, fiction, and social dreaming (1st edition). Cambridge, MA: MIT Press.
- Fallman, D. (2008). The interaction design research triangle of design practice, design studies, and design exploration. Design Issues, 24(3), 4-18.
- Fernaeus, Y., & Sundström, P. (2012). The material move how materials matter in interaction design research. In Proceedings of the Conference on Designing Interactive Systems (pp. 486-495). New York, NY: ACM.
- Finley, K. (2016). Nest’s hub shutdown proves you’re crazy to buy into the internet of things. Retrieved October 22, 2017, from https://www.wired.com/2016/04/nests-hub-shutdown-proves-youre-crazy-buy-internet-things/
- Folkmann, M. N. (2011). Spaces of possibility: The imaginary in design. The Design Journal, 14(3), 263-281.
- Frayling, C. (1994). Research in art and design. Retrieved from http://researchonline.rca.ac.uk/384/3/frayling_research_in_art_and_design_1993.pdf
- Gaver, W. (2012). What should we expect from research through design? In Proceedings of the Conference on Human Factors in Computing Systems (pp. 937-946). New York, NY: ACM.
- German parents told to destroy cayla dolls over hacking fears. (2017, February 17). BBC News. Retrieved from http://www.bbc.com/news/world-europe-39002142
- Haines, V., Mitchell, V., Cooper, C., & Maguire, M. (2007). Probing user values in the home environment within a technology driven smart home project. Personal and Ubiquitous Computing, 11(5), 349-359.
- Hallnäs, L., Melin, L., & Redström, J. (2002). Textile displays: Using textiles to investigate computational technology as design material. In Proceedings of the 2nd Nordic Conference on Human-Computer Interaction (pp. 157-166). New York, NY: ACM.
- Hallnäs, L., & Redström, J. (2002). From use to presence: On the expressions and aesthetics of everyday computational things. ACM Transactions on Computer-Human Interaction, 9(2), 106-124.
- Höök, K., & Löwgren, J. (2012). Strong concepts: Intermediate-level knowledge in interaction design research. ACM Transactions on Computer-Human Interaction, 19(3), 23:1-23:18.
- Husserl, E. (1970). Logical investigations (J. N. Findlay, Trans.). New York, NY: Humanities Press.
- Ihde, D. (1976). Listening and voice: A phenomenology of sound. Athens, OH: Ohio University Press.
- Ihde, D. (1986). Experimental phenomenology: An introduction. Albany, NY: SUNY Press.
- Ihde, D. (1990). Technology and the lifeworld: From garden to earth. Bloomington, IN: Indiana University Press.
- Ihde, D. (2009). Postphenomenology and technoscience: The Peking university lectures. Albany, NY: State University of New York Press.
- Karpathy, A. (2016, September 2). NeuralTalk2 [Web log post]. Retrieved from https://github.com/karpathy/neuraltalk2
- Kastrenakes, J. (2016, March 31). This very expensive juicer can connect to Wi-Fi but can’t actually juice your produce. Retrieved from https://www.theverge.com/2016/3/31/11337444/juicero-wifi-connected-smart-juicer-is-ridiculous
- Kiritsis, D. (2011). Closed-loop PLM for intelligent products in the era of the Internet of things. Computer-Aided Design, 43(5), 479-501.
- Knutsen, J. (2014). Uprooting products of the networked city. International Journal of Design, 8(1), 127-142.
- Koskinen, I., Zimmerman, J., Binder, T., Redström, J., & Wensveen, S. (2011). Design research through practice: From the lab, field, and showroom. Amsterdam, The Netherlands: Elsevier.
- Kuniavsky, M. (2010). Smart things: Ubiquitous computing user experience design. Amsterdam, The Netherlands: Elsevier.
- Latour, B. (1992). Where are the missing masses? The sociology of a few mundane artifacts. In Shaping Technology/Building Society, ed. W. E. Bijker and J. Law. Cambridge, MA: MIT Press.
- Lee, C. (2014). Adoption of smart technology among older adults: Challenges and issues. Public Policy & Aging Report, 24(1), 14-17.
- Lingel, J. (2016). The poetics of socio-technical space: Evaluating the internet of things through craft. In Proceedings of the Conference on Human Factors in Computing Systems (pp. 815-826). New York, NY: ACM.
- Maass, W., & Janzen, S. (2007). Dynamic product interfaces: A key element for ambient shopping environments. Retrieved from http://aisel.aisnet.org/cgi/viewcontent.cgi?article=1033&context=bled2007
- Malpass, M. (2017). Critical design in context: History, theory, and practices. New York, NY: Bloomsbury Academic.
- Mühlhäuser, M. (2007). Smart products: An introduction. In M. Mühlhäuser, A. Ferscha, & E. Aitenbichler (Eds.), Constructing ambient intelligence (pp. 158-164). Berlin, Germany: Springer.
- Olick, D. (2017, January 4). Why 2017 will finally be the year of the smart home: Consumers figure it out [CNBC website]. Retrieved from https://www.cnbc.com/2017/01/04/why-2017-will-finally-be-the-year-of-the-smart-home-consumers-figure-it-out.html
- Pandey, S., & Culén, A. L. (2017). Hearsay: Speculative exploration of intelligent voice based interfaces. Retrieved from http://www.nordes.org/nordes2017/assets/pictorials/nordes17f-sub1008-cam-i26_PANDEY_v2.pdf
- Pierce, J., & Paulos, E. (2010). Materializing energy. In Proceedings of the 8th Conference on Designing Interactive Systems (pp. 113-122). New York, NY: ACM.
- Pierce, J., & Paulos, E. (2013). Electric materialities and interactive technology. In Proceedings of the Conference on Human Factors in Computing Systems (pp. 119-128). New York, NY: ACM.
- Pierce, J., & Paulos, E. (2014). Counterfunctional things: Exploring possibilities in designing digital limitations. In Proceedings of the Conference on Designing Interactive Systems (pp. 375-384). New York, NY: ACM.
- Redström, J. (2005). On technology as material in design. Design Philosophy Papers, 3(2), 39-54.
- Robles, E., & Wiberg, M. (2010). Texturing the “material turn” in interaction design. In Proceedings of the 4th International Conference on Tangible, Embedded, and Embodied Interaction (pp. 137-144). New York, NY: ACM.
- Rosenberger, R., & Verbeek, P.-P. (2015a). A field guide to postphenomenology. In R. Rosenberger, & P-P. Verbeek (Eds.), Postphenomenological investigations: Essays on human-technology relations (pp. 9-41). Lanham, Md: Lexington Books.
- Rosenberger, R., & Verbeek, P.-P. (2015b). Postphenomenological investigations: Essays on human-technology relations. Lanham, MD: Lexington Books.
- Saizmaa, T., & Kim, H. C. (2008). A holistic understanding of HCI perspectives on smart home. Retrieved from http://ieeexplore.ieee.org/document/4624118/
- Schon, D. A. (1992). Designing as reflective conversation with the materials of a design situation. Research in Engineering Design, 3(3), 131-147.
- Staff, V. (2017). The best smart home gadgets of CES 2017. Retrieved from https://www.theverge.com/2017/1/5/14179438/ces-2017-smart-home-gadgets-iot-internet-of-things-news
- Stolterman, E. (2008). The nature of design practice and implications for interaction design research. International Journal of Design, 2(1), 55-65.
- Strengers, Y. (2014). Smart energy in everyday life: Are you designing for resource man? Interactions, 21(4), 24-31.
- Sutskever, I., Vinyals, O., & Le, Q. V. (2014). Sequence to sequence learning with neural networks. Retrieved from http://arxiv.org/abs/1409.3215
- Vallgårda, A., & Redström, J. (2007). Computational composites. In Proceedings of the Conference on Human Factors in Computing Systems (pp. 513-522). New York, NY: ACM.
- Van Hinte, E. (1997). Eternally yours: Visions on product endurance. Rotterdam, The Netherlands: 010 Publishers.
- Verbeek, P. -P. (2005a). Artifacts and Attachment: A Post-Script Philosophy of Mediation. In J. A. Harbers & H. Harbers (Eds.), Inside the politics of technology (pp. 125-146). Amsterdam, The Netherlands: Amsterdam University Press.
- Verbeek, P. -P. (2005b). What things do: Philosophical reflections on technology, agency, and design. University Park, PA: Pennsylvania State University Press.
- Verbeek, P. -P. (2015a). Beyond interaction: A short introduction to mediation theory. Interactions, 22(3), 26-31.
- Verbeek, P. -P. (2015b). Toward a theory of technological mediation. In J. K. B. O. Friis & R. P. Crease (Eds.), Technoscience and postphenomenology: The manhattan papers (p. 189) Lanham, MD: Lexington Books.
- Wiberg, M. (2014). Methodology for materiality: Interaction design research through a material lens. Personal and Ubiquitous Computing, 18(3), 625-636.
- Wiberg, M. (2016). Interaction, new materials & computing – Beyond the disappearing computer, towards material interactions. Materials & Design, 90(15), 1200-1206.
- Wiberg, M., Ishii, H., Dourish, P., Vallgårda, A., Kerridge, T., Sundström, P., … Rolston, M. (2013). Materiality matters – Experience materials. Interactions, 20(2), 54-57.
- Wilson, C., Hargreaves, T., & Hauxwell-Baldwin, R. (2015). Smart homes and their users: A systematic analysis and key challenges. Personal and Ubiquitous Computing, 19(2), 463-476.
- Zimmerman, J., & Forlizzi, J. (2014). Research through design in HCI. In J. S. Olson & W. A. Kellogg (Eds.), Ways of knowing in HCI (pp. 167-189). New York, NY: Springer.
- Zimmerman, J., Forlizzi, J., & Evenson, S. (2007). Research through design as a method for interaction design research in HCI. In Proceedings of the Conference on Human Factors in Computing Systems (pp. 493-502). New York, NY: ACM.
