Computational Compositions:
Aesthetics, Materials, and Interaction Design
1 Department of Informatics & Media, Uppsala University, Sweden
2 Media, Culture, and Communication, New York University, USA
This paper advocates re-focusing interaction design towards aesthetics and compositions. Inspired by current movements within human-computer interaction, product design, materials science, and architecture, we argue that a central problematic concerned with reconciling the digital and physical has always shaped the field. Rather than bridging atoms and bits, an approach that systematically keeps the digital and physical apart, we advocate thinking of design as composition. Doing so will require articulating an aesthetic vocabulary that formally relates diverse materials whether digital or physical. Texture, a word that applies across materials, substrates, and scales, is advanced as a candidate term within our approach. We provide a model for thinking about texture as a relation and then ground the model through a design project, Icehotel X, rendered in ice and digital displays. Texture guided the quality of composition achieved in the final design executed by a team of diverse experts. This approach opens new avenues for the analysis and practice of interaction design.
Keywords – Aesthetics, Atoms and Bits, Computational Composition, Computational Materiality, Icehotel, Interaction Design, Materials, Texture.
Relevance to Design Practice – With the marked shift towards computational materiality, researchers and practitioners require a supporting vocabulary for crafting and analyzing digital- physical compositions. We articulate a formal term, texture, that exemplifies the kind of relational language needed to advance the aesthetics of interaction in an increasingly computational world.
Citation: Wiberg, M., & Robles, E. (2010). Computational compositions: Aesthetics, materials, and interaction design. International Journal of Design, 4(2), 65-76.
Received March 20, 2010; Accepted July 20, 2010; Published August 31, 2010.
Copyright: © 2010 Wiberg and Robles. Copyright for this article is retained by the authors, with first publication rights granted to the International Journal of Design. All journal content, except where otherwise noted, is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 2.5 License. By virtue of their appearance in this open-access journal, articles are free to use, with proper attribution, in educational and other non-commercial settings.
*Corresponding Author: mikael.wiberg@im.uu.se
One of the things our grandchildren will find quaintest about us is that we distinguish the digital from the real, the virtual from the real. In the future, that will become literally impossible. – William Gibson
The real voyage of discovery consists not in seeking new landscapes, but in seeing with new eyes. – Marcel Proust
Introduction
As computation progressively takes place outside of the box, design increasingly occurs at the boundary between the digital and physical. Or, as Hiroshi Ishii explains: “At another seashore between the land of atoms and the sea of bits, we are now facing the challenge of reconciling our dual citizenships in the physical and digital worlds” (Ishii, 2008).
Reconciling a dual citizenship, however, presumes that the worlds of atoms and bits are fundamentally apart. From this vantage, the job of the designer has been to craft the relation between categorically distinct matters. The cascade of approaches resulting from this assumption can be historically traced. Graphical user interfaces use metaphors to cast a relationship between everyday life and information architecture. Ubiquitous computing disguises sensors and processors inside everyday objects so that they contribute function, but show no form. The hybrid quality of tangible devices, which explicitly connect atoms with bits, brings this presumption into even sharper focus. By treating atoms and bits as categorically distinct, however, these approaches proliferate precisely the aesthetic problem they intend to address.
Rather than presume the need to bridge atoms and bits, we advocate shifting interaction design towards valuation of compositions. Inspired by a current moving through a multitude of design disciplines including materials science, product design, interface design and architecture, this paper places aesthetics at the center of a movement towards new computational forms (see Robles & Wiberg, 2010). Our first step in presenting this approach will be to critique the ontological distinction between atoms and bits.
Drawing on aesthetics of interaction research (Dalsgaard & Hansen, 2008; Graves, Hallnäs, & Jacob, 2008; Löwgren & Stolterman, 2004; Redström, 2008; Rullo, 2008; Wright, Wallace & McCarty, 2008), we advocate a vocabulary that helps organize design relationships to achieve a range of effects. Computational compositions require an aesthetic language to guide and analyze new designs. This language should speak effectively at the intersection of social and material worlds and across a range of digital and physical substrates. Our approach differs in that we base our aesthetics within material properties and formal relations. We believe this avoids the fallacy of making a categorical distinction between atoms and bits. The result is a mode of organizing relations that enables an address of compositions across disciplines and at a variety of scales.
To demonstrate our approach we employ the term texture. Texture is useful because it is familiar, relational, and robust. It makes no distinctions between atoms and bits, but does address the relations across materials, surfaces, substrates, and scales. In other words, it exemplifies the use of a formal term to guide aesthetics in design. We apply texture to the design case of Icehotel X. An architecture rendered in frozen water and digital displays, the project radically foregrounds material relations. The case illustrates the process of designing with the aim of creating a composition of unified texture. We then discuss how this approach, which need not distinguish between atoms and bits, might expand work within interaction design.
Atoms and Bits: Categorical Concerns for Interaction Design
The change from atoms to bits is irrevocable and unstoppable.
– Nicholas Negroponte
Computing destabilizes the way we think of atoms. Everyday objects are increasingly innervated by processors, sensors, and information networks. Cultural goods that once seemed forged to particular mediums have given way to digital instantiations that never seem to take form at all. If “the change from atoms to bits is irrevocable and unstoppable,” then the central challenge for contemporary design is to make “being digital” just a little less hard (Negroponte, 1995). Human-computer interaction is the field that takes as its mission the easing of being digital. By organizing the interface between computing and everything else, it is primarily concerned with theorizing and representing the relations between atoms and bits. As such, aesthetics are central to the task. From structuring information architectures through metaphor, to embedding sensors invisibly within a physical device or crafting new digital-physical hybrids, materiality is always in question. By scrutinizing the dominant aesthetics guiding the expressions of materiality and computing in three different moments – graphical user interfaces, ubiquitous computing, and tangible interaction – we argue that interaction design systematically reproduces the very distinction between atoms and bits that it intends to overcome.
Metaphor: The Aesthetics of Graphical User Interfaces
By the late 1980s, graphical user interfaces (GUIs) dominated the genre of computer displays. With the advent of the Apple Lisa (1984), personal computing became a process of navigating surface representations with little evident relation to the underlying architecture of the machine. Like programming languages and command line interfaces, GUIs were part of the broad project to democratize computing through abstraction. By layering symbolic systems on top of machine architecture, the intention was to scaffold operability. Ease of use was tantamount to freedom from engaging with underlying levels of detail. The chief task of the designer, then, was to create a symbolic system that effectively mediated human-computer interactions. Within the paradigm of GUIs, metaphor became the predominant strategy for organizing relations between surface and structure, digital and physical, human and machine.
Though GUI designers radically re-imagined the relations between structure and representation, they conservatively borrowed logic from the non-computational world to determine form. In an extreme case of extended analogy, computers became desktops while machine states, memory, and processes gave way to complex visual systems of files, folders, and office applications. Despite the malleability of bits, metaphorically relating them to atoms seemed the clearest grounds on which GUIs ought to proceed. Thus, the chief task for interaction design was to search for a semiotic substitute for being close to the machine.
Underlying this privileging of metaphors was the belief that they effectively interfaced between psychology and symbol systems (see Blackwell, 2006 for an account of the contemporary role metaphors play in interaction design). Substantiated by scholarship like that of rhetorician George Lakoff and linguistic anthropologist Mark Johnson (1981), this perspective argues that metaphors both reveal and fundamentally structure mental concepts. Thus, by presenting users with visual metaphors of already familiar non-computational systems, interaction designers could literally ease use. Good design was a correspondence between users’ pre-existing models of the physical and social world and software design. As interaction design professionalized, metaphor became further reified through metrics for system assessment (Nielsen, 1994).
Metaphor was by no means the only possible aesthetic strategy for the GUI (see Manovich, 2007 on early aesthetics for computer interaction). For example, Douglas Englebart’s landmark 1968 presentation (Englebart, 1968) of the On-line System (NLS) demonstrated human-computer interaction in a completely different visual logic. NLS was conceived of as an instrument for “augmenting human intellect,” not as a machine designed for ease of use. “Console workers” could compute through a system of “hierarchical view controls” in which the form and manipulability of data could be specified. Users could generate text-based lists, from which terms on the list might allow the user to descend into a deeper layer of data organization or to jump to a different graphical view. View controls could be authored by the user and subsequently chained into variable sequences. Expressing a more cybernetic orientation, the GUI of the NLS was an emergent, complex form.
Shaping GUI aesthetics through metaphors was also not without strong critique. Pushing against the “sacred cow” of interaction design, personal computing and object-oriented programming pioneer Alan Kay advocated instead the paradigm of “user illusion” (Kay, 1990). Metaphor, he charged, was a “compelling snare,” a “siren song” that promised to “describe a correspondence between what the users see on screen and how they should think about what they are manipulating” (Kay, 1990, p. 199). A fool’s errand, the search for an iconic system homologous with its physical counterpart missed the “magic” of computing. Screen as paper is only useful when computational functions exceed the properties of a page. Copy and paste, delete, undo and edit break the metaphorical correspondence. Similarly, digital desktops are only interesting because they don’t clutter, and because they afford multiplicities of view. Metaphor breaks precisely where the action is. Nevertheless, the predominate aesthetics of GUIs helped bring about a deconstruction of computational form. With new systems of representation standing in, it became possible to ask why the machine needed to be visible at all.
Disappearance: The Aesthetics of Ubiquitous Computing
The most profound technologies are those that disappear. They weave themselves into the fabric of everyday life until they are indistinguishable from it. – Mark Weiser
If GUIs decoupled architecture from representation, ubiquitous computing completely ejected computation from its “box”. Emerging in the early 1990s from Xerox Palo Alto Research Center (Xerox PARC), ubicomp promised to resolve the problem of relation between atoms and bits by abandoning the semiotic system of the GUI in favor of a world in which the computer literally disappeared.
Ubicomp was a radical critique of personal computing. Its pioneering visionary Mark Weiser argued that figures like Alan Kay fundamentally misunderstood their operation as democratizing. The “complex jargon,” reliance on metaphor, and misguided form of personal computing condemned computation to live in an “electronic shell.” Aligning “silicone-based information technology” with a specific device effectively prevented computing from integrating with everyday life. At best, the widespread adoption of personal computing offered a future populated by millions of “single boxes” (Weiser, 1991, 94).
Early ubicomp projects expressed their commitment to seamless integration by literally embedding computation within the physical world. Computationally enhanced post-it notes, paper, and bulletin boards, or “tabs,” “pads” and “boards” demonstrated how offices might be populated with hundreds of computers “invisible in fact as well as in metaphor” (for technical details on early ubicomp projects see Want et al., 1995; Weiser & Brown, 1996). These real desks and real pieces of paper were “antidote[s] to windows” that abandoned representation to bring computation directly into the physical world (Weiser, 1991, p. 99).
Ironically, this vision of “the computer for the 21st century” was as conservative as it was radical. Trading metaphors for literal objects only preserves the categorical distinction between atoms and bits. Moreover, like the GUI paradigm, ubicomp privileges the non-computational world. Its forms, organization, and processes are regarded as natural and authentic. Thus, the chief task of interaction design is to preserve the status quo by ensuring that bits “disappear” inside the atoms of everyday life. At best, seamlessly integrated computing affords calm, context-aware processing anywhere at any time. At worst, the aesthetic of disappearance obscures the integrity of the computational as a material that might contribute unique properties that help imagine new forms. The emergence of a third paradigm, tangible interaction, proposes yet another response to this tension.
The Material Turn: The Aesthetics of Tangible Interaction
Converging lines of research indicate that a “material turn” is taking place within interaction design (Robles & Wiberg, 2010). From the creation of computational materials and graspable interfaces to the development of rapid prototyping processes, a range of design projects are re-conceiving the relationship between atoms and bits. These trends provide an opening for the emergence of a new computational aesthetic in which atoms and bits may play more equitable roles.
Within interaction design the material turn manifests under the rubric of tangibility. Explicated by Ishii & Ullmer (1997), tangible interactions bridge the digital and physical by creating computational systems, or “bits”, that can be grasped (Fitzmaurice, Ishii, & Buxton, 1995), manipulated, accessed, and programmed (Klemmer & Landay, 2009). This emphasis on tangibility has helped inspire the re-examination of computation in more material terms (Brownell, 2006, 2008; Landin, 2005; Löwgren & Stolterman, 2004; Vallgårda & Redström, 2007; Vallgårda & Sokoler, 2009). Complemented by growing interest in rapid prototyping processes, computers are increasingly being treated as just another material operating “on the same level as paper, cardboard, and other materials found in design shops” (Bdeir, 2009, p. 397). Similarly, physical materials are being re-imagined as substrates invested with computational properties. From everyday craft materials like felt (Reiger, 2007) and clay (Reed, 2009) to new smart materials like flexinol (a shape-memory alloy), MEFiT (a plastic film that converts mechanical force into electricity), thermochromic ink, and electric plaid (programmable, hand-woven textiles capable of changing color), the very canvas of interaction design is transforming (Matério, 2005).
This move towards informated materials is also occurring at the architectural scale. Structures like louvers now exhibit “soft mechanics,” a move away from hard structural joints and towards dynamic forms (Coelho & Maes, 2009). Projects like Urban Pixels (Seitinger & Perry, 2009), BIX Matrix, an electronic “skin” covering the Kunshaus Graz art hall in Austria, the SPOTS low-res digital façade at Potzdamer Platz in Berlin, or Light BRIX, a physical wall of virtual bricks, blur distinctions between interface and façade (Bullivant, 2005, 2006, 2007). Even the Eiffel Tower has been refinished in dynamic, blinking lights.
Materiality also appears in the social sciences, where analysts are increasingly interested in the physical aspects of information work. Pushing back on socio-technical perspectives that treat epistemologies and ontologies as distinct (Hirschheim & Klein, 1989), a range of researchers argue that even networked social processes fundamentally intertwine with the material world. Thus, they argue that shifting focus towards physical infrastructures will provide a more accurate understanding of information work (Graves-Brown, 2000; Lazzari, 2005; Miller, 2005). Captured under terms like “sociomateriality” (Orlikowski & Scott, 2008; Suchman, 2007), onto-epistemology (Barad, 2007), and material-semiotics (Haraway, 1991), materiality signifies, for the social sciences, the inseparably “recursive relationship between people and things” (Lazzari, 2005, p. 127).
Taken together, these approaches demonstrate the emergence of a new paradigm in interaction design. Following previous paradigms, tangible computing takes up the question of relation between atoms and bits. Also like previous paradigms, it proposes to reconcile the distinction and, in so doing, reproduces the categories that frustrate the development of a stable aesthetic advance. The paradigm does, however, depart from previous approaches by avoiding the hierarchical privileging of the non-computational over the informational world. Hybrids, like tangible bits, place the digital and physical in more commensurate relation but they do nothing to resolve the ontological consequence inherent in the categories themselves.
Addressing this ontological distinction and its codetermining epistemological consequence requires a language or vocabulary (Krippendorf, 2005; Lowgren & Stolterman, 2007) that exceeds descriptions of “assemblages” (Orlikowski & Scott, 2008) and instead organizes constructive techniques for design. There is not yet a stable aesthetic mode for designing compositions through the paradigms of interaction design. By disregarding the categories of atoms and bits in favor of a more intimate concern for properties, structures, surfaces and expression, we can speak of the ways in which materials come into relation. These relations allow a more specific conversation to emerge (at a variety of scales) about the forms of computational composition. One such relation is texture.
Texture – A Point of Departure from Atoms and Bits
By moving away from the long-standing preoccupation with distinctions between atoms and bits and towards the articulation of formal relations at a variety of scales we believe we can develop the language for aesthetic discourse within interaction design. As an initial foray, we’ve used the notion of texture to illustrate the possibilities of speaking of compositions rather than of categories. Texture is a formal relation that refers to the feel, appearance, or consistency of a surface or substance. It can be expanded to address elements in combination and to discuss qualities of works in a variety of fields. Texture is also part of the technical language of materials science (Randle & Engler, 2000), and applies to areas as diverse as geology, architecture, cuisine, and cosmology (as a type of theoretical topological defect in the structure of spacetime). It is employed more symbolically to describe “feel” in fields like literature, music, and art. Here, we use texture to address formal relations between surfaces and structures (see Figure 1).
Figure 1. Texture relations in a composition.
We chose texture as a point of departure because it neither depends upon nor entails any categorical distinction between atoms and bits (Robles & Wiberg, 2010). The term applies to feel, but it also addresses appearance, or the way underlying infrastructure is communicated to an observer; it operates phenomenologically. Thus, it may be used to describe forms that depend on physical manifestation, information representation, or both. Moreover, the term functions at a variety of scales and serves well for interlocutors interested in sliding between “hard” (e.g. material science and architecture) and “soft” (e.g. focused on affect and poetics) sciences that often meet within the domain of design.
To understand how a formal relation like texture might be applied to a composition, consider the two substances, A and B (Figure 1). Each has structuring properties, and each communicates a texture (represented by the lines above squares A and B). The communicated texture stands in some relation to the structuring properties (arrows 1 and 2). Texture can be manipulated or redesigned by transforming the structure, expression, or relation between surface and substrate.
Now consider designing a composition from A and B. Achieving some quality of wholeness requires crafting relations between A and B. Texture, in this case, involves a complex of substrate relations (D), appearances (C), the relative distance between the composing elements (3), and the overall relation between structure and surface (E).
We can critically evaluate the texture of a composition and, in so doing, can ask what kinds of textures we generate in a design moment of computational compositions. New forms can be analyzed independent of their ontological status and, as a result, can be discussed aesthetically. If the texture of the new form stands in elegant relation to the composing elements, we might call this good design. If the texture emerges from the mere juxtaposition of elements, or perhaps inadequate relation between expressing and structuring properties – like laminate plastic surfaces imitating wood or information architectures mimicking desktops – we might call this kitsch. From this vantage, the central task for design is not to reconcile atoms and bits but rather to expand the repertoire of relations between architecture and appearance (i.e. to expand the distances of arrow 1 and 2) such that new compositions might become available for interaction.
From Formal Relations to Computational Compositions
If terms like texture are to be useful they must address relations irrespective of digital or physical status, and they should provide a means for generating new ways of thinking about design. Drawing on design theorists like Nelson and Stolterman (2003), we now generate a larger conceptual framework in which the address of formal aesthetic relations enables computational forms to be considered compositions.
Compositions belong to the realm of aesthetic choice; they can be evaluated with regard to the connections crafted between elements within the composition. These connections may occur vertically (between expression and structuring properties) or horizontally (across multiple substrates). Connections give a design its particular aesthetics, what theorists Nelson and Stolterman might call “presenting features,” or “the qualities that inform the senses most directly” (Nelson & Stolterman, 2003, p.219). Compositions need not involve full expression of all connections. Rather, connections may allow some subset of properties to present while others merely scaffold a particular appearance. Vertical connections in a computational composition may be particularly dynamic in that presenting features might shift based on switches between multiple relations of expressing and structuring properties. Dynamic connections naturally involve complexity.
Complexity addresses emergent and dynamic properties of connection. As Gestalt psychology tells us, the human brain interprets complex and multiple patterns from simple visual elements working in composition. In other words, connections scale via complexity and, thus, specific formal relations come to assemble emergent aspects of appearance. Complexity brings structure, presentation, and human perception into relation.
Speaking of connections that scale via complexity should not be interpreted as advocacy for a kind of generalized holistic emergence made possible by thinking relationally rather than categorically (for a critique of the ontology of relation see Harrison, 2007). Instead, connections and complexity allow designers to make and evaluate choices at multiple scales. The art of manipulating work at these scales to achieve a unifying effect is the work of composition. According to Nelson and Stolterman, “Composing is an activity where judgements are made using aesthetic principles like balance and symmetry, about relationships between details and the whole” (Nelson & Stolterman, 2003, p.209). Though composition provides a way of thinking about integrating elements – whether digital or physical – into meaningful wholes, it also risks the logical fallacy of presuming that a whole expresses all of the properties possessed by its various parts. This need not be the case. Composition works outside the scope of strictly linear relations; the character, or texture of a whole may be emergent, arising from the dynamic interplay between connections.
Composition not only provides a way of designing and assessing particular connections, it also enables aesthetic judgments to form about the overall work. At times, compositions may prove displeasing because features are in competition with each other or with the goal of an overall design. A work may fail to achieve the desired aesthetic form because connections – whether vertical or horizontal – are in competition. Alternatively, a composition might be regarded as a compromise; a work in which not all features come to full expression. For example, compromise may mean using one material to stand in for another, or it might mean that a material foregoes part or all of its properties or functions within a larger composition. Finally, compromise can characterize an entire composition that lacks proper materials, or is unable to unify elements into a coherent or meaningful whole. Thus, compositions can be addressed in terms of overall coherence or in the way in which they present or perform their form. Coherence signifies those compositions that function not only technologically, but also aesthetically.
Aesthetics, then, is a stance through which legitimate compositions are distinguished from kitsch. For some communities, these aesthetic distinctions express greater reliance on truth to materials than others. For example, in architectural terms, to be true is to be honest, to avoid “hiding the buildings’ real structure and creating an illusion of another sort of architecture” (Farrelly, 2009). Within interaction design, “true” compositions have been those in which “presenting features – the qualities that inform the senses most directly” are revealed in “ultimate appearance” (Nelson & Stolterman, 2003, p.219). These judgments are, of course, highly dependent on social, cultural and historical conditions. Nevertheless, they are centrally within the domain of aesthetic discourse. Design, the process of bringing things into relation, is the proper site for formalizing these considerations. The next section applies composition to a concrete design task. Through the merger of two different materials – one physical, one digital – at the architectural scale, we demonstrate how unified texture is achieved.
Design Case: Icehotel X
Two hundred kilometers north of the Arctic Circle in a remote landscape dominated by the darkness of long winters lies Sweden’s most popular tourist destination, the Icehotel. For the past twenty years, the frozen edifice has served as a nexus for convening international teams of artists, designers, engineers, and architects. Each year, the team transforms the mundane substances of the tundra—snow and ice—into remarkable contemporary designs. The result is a spectacular architecture more than 5500 m² (59,209 ft2) filled with ice suites, ice rooms, ice galleries, an ice church, and an ice bar. Every setting is composed almost exclusively in a single material – frozen water. Any material additions are strongly considered, debated, and meticulously designed. Thus, this Arctic architecture provides a strong example for thinking through computational composition precisely because the questions of materiality and unified form lie at the heart of its design (Robles & Wiberg, 2010).
In 2008 and 2009, we were part of a multi-profession collaborative design team charged with communicating the experience of the Icehotel to a broader audience. The remote geographic location of the Icehotel required rather creative initiatives. It was decided that perhaps one of the best ways to give a sense of the site was to erect a full-scale frozen environment within a city center 2000 kilometers south of the Arctic. The resulting project, Icehotel X, served as our inspiration and test-bed for aesthetic formulations of composition within interaction design.
Icehotel X, Copenhagen
Located in Copenhagen, Icehotel X was designed to express “the spirit of the Icehotel” (Wiberg, 2009). Its predominate architectural element was to be the same glacial ice used in the original building. Valued for its purity and clarity, the ice would be preserved within a refrigerated environment approximately 100 m² (1076 ft²), thus allowing visitors to experience full immersion. Despite the integrity of the building material, Icehotel X did not effectively communicate the desired experience. The transplanted material enabled replication, but not composition. In order to achieve greater effect, we decided to integrate digital elements into the architectural design.
Figure 2. The exterior (left) and interior (right) of the Icehotel. The basic building material is frozen water.
We believed that coupling the transparent qualities of ice with animated luminosity via large displays would better afford the desired experience. Working with display technologies presents a certain aesthetic risk. As contemporary devices, they offer the temptation of full computational potential and thus of highlighting their high-resolution, high-speed, or networking attributes. Our feeling was that maximizing these expressions placed the computational in competition with the ice architecture in a variety of ways. A screen formed through more cinematic conventions would command inordinate attention, literally pulling visitors away from their current surroundings and towards a remote locale (Barthes, 1975; Spigel, 2001). Moreover, a large display with clear images produces spectators whose ideal vantage is all the way across the room, not in close connection with the ice or each other. Finally, large screens risk transforming spaces into technology showrooms rather than immersive experiences. Dominant forms of contemporary display received some consideration early in the project (see Figure 3, left). Ultimately, they were rejected.
Figure 3. An early vision of Icehotel X (left) versus the final installation in Copenhagen (right).
Three things were crucial for the Icehotel X installation: that it appear as a unified architectonic installation, that it serve as an extension of the Icehotel, and that it have integrity as a unit in itself. Given the range of disciplines represented in the collaboration – from sculptors and architects to lighting and food designers – and the emphasis on crafting a sense of wholeness, thinking through a shared language of composition seemed the best approach for all to take. Through the lens of achieving a coherent texture, we were able to simultaneously integrate properties of digital elements into the space while avoiding the creation of a high-tech demo room or kitschy clutter in which bits competed with atoms. The final design for Icehotel X contained ice walls, ornamentations, sculptures, and two 8 meter long, 2.4 meter high walls composed of low-resolution LED pixels (see Figure 3, right). Guided by an interest in achieving a texture, or feeling for the north, our composition organized connections at a variety of scales. We made choices about both structuring and expressing features across our candidate materials. In the design, atoms and bits are not reconciled; rather, their connections scaffold each other as part of an overall design.
We first concentrated on the form of the digital display. We wanted to take advantage of both its dynamic properties and its luminosity without creating the kind of focal point for representation generally associated with a screen. We wanted to take care that the presenting features of the display did not detract from the unity of the overall composition or compete with the material integrity of the Arctic ice. To break the connection between luminosity and resolution (and thus competition for viewer attention), we crafted an extremely low-res display. Made of bulbs mounted in a grid and spaced 5 cm apart, the resulting resolution (160 x 48 pixels) insured that the presenting features of the display were technically incapable of creating competing representations of the North.
With a luminous, low-resolution, dynamic material now available, we could build our solution at an architectural scale. Doing so, however, would bring large, heat-emitting, electronic surfaces into contact with pristine blocks of frozen glacial water: mutually assured destruction. Just as ice might freeze the displays or melt into dangerous moisture for the circuits, the lights could easily thaw the ice, ruining the design. Our solution was to organize the relation by introducing a third material, 4 mm near-opaque plastic film. By insulating the display grid surface we prevented temperature loss and fluid exchange. Simultaneously, the opacity of the film competed with the discrete structure of our raw pixels, smoothing their appearance into more continuous display. The resulting form established connections between digital content and its representation in the architectural space.
We created an animated sequence in order to take advantage of the dynamic optical properties of our architecture. In an iterative design process carried out with engineers, light designers, and film editors, we arrived at agreement about the ideal texture of the space. Through a series of abstract representations (Figure 5) of rushing water, snowstorms, bonfires, and the northern lights (typical elements of a visit to the far North) the final composition communicated to urban inhabitants the feel of a remote geography while never replacing the real experience of travel.
We worked against tendencies towards material competition by conscientiously organizing structural properties, surfaces, and representations. Just as composition organized dynamic qualities of light and space but not properties of networks or images, the slow-moving animated sequences determined the presentation of underlying features. In this piece of work, the viewer is not a cinematic spectator positioned across the room. Instead, they might be close enough to feel the ice or even able to ignore the composition altogether in favor of the social interaction taking place within the room. The resulting installation was more a dynamic architecture of light than an isolated piece of equipment (Figure 3, right; Figure 4; Figure 5).
Figure 4. Vertical connection – low-resolution graphics and low-resolution picture (left).
Horizontal connection – material alignment of ice and pixels into an integrated wall (right).
Figure 5. Animation frames from the Icehotel X installation.
Once this new form was designed, we began literally sculpting Icehotel X. Bringing the materials into horizontal connection required millimeter by millimeter adjustments. Ultimately, the two were in such tight relation that light traveled through both; ice became part of the optical medium of digital display (Figure 4, right).
Complementing the digital display and extending its representational capacities, artists and lighting designers executed ice block carvings and sculptures. “Ice screens” depicting thematic images of fish, reindeer, and northern landscapes became a medium akin to the pixel walls. Even ice becomes a different material when looked at through a computational lens. Through this process we upgraded the perceived status of the digital technology to a design material. Digital displays became architectural elements used to help define this space as to create a balanced composition. Taken together, this complex of strategies blurred the distinction between medium and representation, substrate and surface.
Ultimately, texture allowed experts in multiple design professions to have sustained conversation about the presenting features of a final form. Moreover, it allowed each expert to think about how to organize qualities of light, transparency, dynamism, or representation within their particular medium so that they built upon the contributions of others. There were no atoms, bits, or bridges. Instead, supported by a language of formal aesthetic relations, a successful design collaboration took place across interaction designers, lighting designers, engineers, sculptors, artists, and architects. In the process, the close care for the composition allowed the overall project to become invested with meaning. Icehotel X opened to the public in April 2009.
Conclusion: The Aesthetics of Computational Compositions
Every useful thing is a whole composed of many properties; it can therefore be useful in various ways. The discovery of these ways and hence of the manifold uses of things is the work of history.
- Karl Marx
Thus far, this paper has focused on aesthetics by addressing the problematic of atoms and bits. Looking back at a successive series of paradigms in the field – GUIs, ubicomp, and tangible computing – we have argued that the presumption of categorical distinction between atoms and bits underlies the field and frustrates its aesthetic ambitions. We have argued that moving forward requires articulating an approach to design that is concerned with formal relations rather than the ontological status of materials. In so doing, we might open new design spaces, bridge multiple fields, and derive new modes of composition in which computation and interaction play crucial roles.
We demonstrated our interest in exploring a relational vocabulary by reclaiming the notion of texture. Texture is suited to design conversations taking place in a variety of fields ranging from interaction design, architecture, and materials science to art, literature, and music. The term supports a regard for digital-physical compositions by addressing relations at the level of structure and surface, substrate, overall appearance, and feel. It also speaks independently from the ontology of atoms and bits.
It is by no means, however, the only formal term that speaks this way. Light, temperature, color, and gravity are candidate concepts for further elaboration as part of a discourse about computational compositions. Taken together, these aesthetic terms point to the under-theorizing of concepts like material, scale, datum, rhythm, and transformation within the field of interaction design. Ideally, evaluating designs through these formal relations can help highlight the limits of any technological approach. We might be better able to see what cannot be achieved through the contemporary interface.
Nevertheless, the implications of this subtle but fundamental shift for the aesthetics of interaction design can be hard to see. We have attempted, in part, to make visible the implications by utilizing Icehotel X as a case for our approach. Another technique might be to draw the contemporary moment into analogy with other historical moments in which technical processes and materials required aesthetic reconfiguration (for a masterful exegesis on the formation of Modern aesthetics capable of reconciling materials like iron and concrete into architectural practice, see Giedon, 1941; on the ubiquity of print as a resource for imagining the future of computing see Weiser, 1991).
For example, we might draw the analogy between computation and optics, or materiality and glass. Glass was the essential material of an “optical revolution” (Crary, 1990; Friedberg, 2006; Jay, 1988). Though known since antiquity, it was not until the 17th century that the material became widely utilized in its modern form. Like computation, this material formulation of optics moved through a stage of scientific instrumentation before it moved out of its box. Shaped into finely ground lenses and polished mirrors, it was central to the production of scientific knowledge. It powered telescopes and microscopes and literally transformed the scale of human vision.
Like computational forms at the moment of ubiquitous computing, a moment arrived when it became technically possible to move from the scale of instruments to contexts. Recast as windows and mirrors, glass entered the architectural register. This move towards architectural ubiquity relied on more than strictly technical developments. Thus far, we have said little about the relationship between the material and social, a crucial consideration for design theories of interaction. We would like to suggest that material transformations are always fundamentally bound up in social and historical processes. Thus, engineering new forms is engineering new interactions.
In the case of optics and glass, an interest in symbolically binding engineering feats with national power made the re-formulation as much material as symbolic. During Louis XIV’s absolute reign, the French became dominant players in the production of highest quality glass, with the construction of Versailles furnishing a steady market for innovations. Here, materiality, aesthetics, and statecraft intertwined (Mukerji, 1997), composing new forms, spaces, and subjects. The height of symbolic composition was the Galerie des Glaces, or Hall of Mirrors. A luminous passage 240 feet long, the Galerie is bounded on one side by seventeen mirror-clad arches and on the other side by seventeen arcades spanned by windows. The space spectacularly extends the possibilities of organizing luminosity, transparency, and dematerialization. In the process, the very optical invention that produced scientific knowledge was reformulated and scaled. The presenting features of the resulting composition asked subjects to re-imagine relationships between interior and exterior, power and vision, transparency, spectacle, and trade. Ultimately, the re-formation of a material carried epistemological, social, philosophical and moral consequences (Colomina, 1996; Friedberg, 1994;2006; Giedon, 1941).
Glass and computers have more in common than an elemental relationship between sand and silicone. Revisiting the historical organization of an optical material draws our attention to the symbolic at work in the material. It reminds us that material and technological configurations are socially organized. More importantly, it tells us how central aesthetics are to shaping discourses for design (see Hayles, 2005; Thrift, 2003, 2004, on the fantasy of computational categories determining design directions). Stories about aesthetic shifts in the management of glass might be told in a number of historical moments: the Crystal Palace of the 1851 Great Exhibition which was emblematic of industrialization and the ambitions of colonial empire; Paul Scheerbarts’ socialist utopian vision on the moral powers of glass architecture and transparency (Scheerbart, 2001); or the codetermination of computing and glass in the formulation of the curtain wall (Martin, 2003). Material histories of computation will also show how fundamental aesthetic choices are to larger social visions, even though we may not yet see in what manner.
A focus on relational language and computational materiality opens new aesthetic directions. Some of the strongest tangible projects point towards a new mode of composition, but they lack the formal vocabulary for properly articulating the significance of their aesthetic contribution. For example, Qi and Buechley’s (2010) recent pop-up book project demonstrates this emerging compositional sensibility. The project literally crafts a representation of flowers in which mechanical movements drive digital sensors and electronic circuits. More importantly, the very technique of assemblage requires the application of conductive paints in a fashion akin to Japanese brush art. The precise length of the stroke determines the electronic function, thus marrying circuit design, brush painting, and paper craft. This merger of representation and mechanism is reminiscent of the development of collage as an artistic technique.
Collage, credited dually to Georges Braque and Pablo Picasso, enlarged the reach of painting by demonstrating the power of composition to integrate new materials into aesthetic wholes. Picasso’s Composition with Fruit, Guitar, and Glass (1912) (Figure 6) embedded fragments of newspaper directly in the work, not as yet another substrate to draw on top of, but as part of the very image. Paper presented connections at the levels of both structure and surface. The strength of composition helped innovate techniques within established artistic traditions. So too, it is possible to think about a suggested range of directions for developing a relationship between the organization of process and product within the contemporary moment. Like the pop-up book, these works do not bridge categorically different materials; rather, they simply make visible new aesthetic directions.
Figure 6. Compotier avec fruits, violon et verre, Pablo Picasso (1912).
We would like to make one final remark about the long-term implications of thinking compositionally. Perhaps the computational properties we seek were already part of the pre-computational world. Consider water displays. Taking advantage of already present structuring features of surface tension and gravity, these systems render images in falling water drops. Even infosthetic versions like the Bitall installation, in which synchronized magnetic valves control water drops, allowing users to translate digital images from the web into tangible, if momentary images, are not technologically impressive inventions. Yet, it is likely that water could not be conceived through the features of discrete drops in flows unless viewed in a computational moment. From that perspective, a mundane, ubiquitous substance as simple as water provides exactly what we need. Sometimes, the best way to abandon the quaint distinction between the digital and the real is not to build new bridges, but to see with new eyes.
Acknowledgments
The authors would like to recognize the funding organizations that made this collaboration possible. Our thanks to the European Union (Objective 2) project xID 2.0 – Interaction Design in Extreme Environments, and to the Kempe Foundation. Our thanks as well to project partners at Icehotel, Philips, Creative Camp, and to Jung von Matt.
References
- Baljko, M., & Tenhaaf, N. (2008). The aesthetics of emergence: Co-constructed interactions. Transactions on Computer-Human Interaction, 15(3), 1-27.
- Barad, K. (2007). Meeting the universe halfway: Quantum physics and the entanglement of matter and meaning. Durham, NC: Duke University Press.
- Barthes, R. (1975). En sortant du cinéma. Communications, 23(1), 104-107.
- Bdeir, A. (2009). Electronics as material: LittleBits. In Proceedings of the 3rd International Conference on Tangible and Embedded Interaction (pp. 397-400). New York: ACM Press.
- Blackwell, A. (2006). The reification of metaphor as a design tool. Transactions on Computer-Human Interaction, 13(4), 490-530.
- Brownell, B. (2006). Transmaterial: A catalog of materials that redefine our physical environment. Princeton, NJ: Princeton Architectural Press.
- Brownell, B. (2008). Transmaterial 2: A catalog of materials that redefine our physical environment. Princeton, NJ: Princeton Architectural Press.
- Bullivant, L. (2005). 4dspace: Interactive architecture, architectural design. New York: Wiley-Academic.
- Bullivant, L. (2006). Responsive environments: Architecture, art and design. New York: V&A Contemporary Publications.
- Bullivant, L. (2007). 4dsocial: Interactive design environments, architectural design. New York: Wiley Academic.
- Chen, X., Shao, F., Barnes, C., Childs, T., & Henson, B. (2009). Exploring relationships between touch perception and surface physical properties. International Journal of Design, 3(2), 67-77.
- Coelho, M., & Maes, P. (2009). Shutters: A permeable surface for environmental control and communication. In Proceedings of the 3rd International Conference on Tangible and Embedded Interaction (pp. 13-18). New York: ACM Press.
- Colomina, B. (1996). Privacy and publicity: Modern architecture as mass media. Cambridge, MA: MIT Press.
- Crary, J. (1990). Techniques of the observer: On vision and modernity in the nineteenthcentury. Cambridge, MA: MIT Press.
- Dalsgaard, P., & Koefoed Hansen, L. (2008). Performing perception—Staging aesthetics of interaction. Transactions on Computer-Human Interaction, 15(3), 1-33.
- Desmet, P. M. A., & Hekkert, P. (2007). Framework of product experience. International Journal of Design, 1(1), 57-66.
- Engelbart, D. C. (1968). Doug Engelbart 1968 demo. Retrieved December 5, 2009, from http://sloan.stanford.edu/MouseSite/1968Demo.html.
- Farrelly, L. (2009). Basics architecture: Construction + materiality. Lausanne, Switzerland: AVA Publishing.
- Fitzmaurice, G., Ishii, H., & Buxton, B. (1995). Bricks: Laying the foundations for graspable user interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 442-449). New York: ACM Press.
- Friedberg, A. (1994). Window shopping: Cinema and the postmodern. Berkeley, CA: University of California Press.
- Friedberg, A. (2006). The virtual window: From Alberti to Microsoft. Cambridge, MA: MIT Press.
- Giedon, S. (1941). Space, time, and architecture: The growth of a new tradition. Cambridge, MA: Harvard University Press.
- Graves-Brown, P. M. (2000). Matter, materiality and modern culture. London: Routledge.
- Graves, P. M., Hallnäs, L., & Jacob, R. (2008). Introduction to special issue on the aesthetics of interaction. Transactions on Computer-Human Interaction, 15(4), 14.
- Graves, P. M., Iversen, O., Krogh, P., & Ludvigsen, M. (2004). Aesthetic interaction: A pragmatist’s aesthetics of interactive systems. In Proceedings of the 5th Conference on Designing Interactive Systems: Processes, Practices, Methods, and Techniques (pp. 269-276). New York: ACM.
- Haraway, D. (1991). A cyborg manifesto: Science, technology, and socialist-feminism in the late twentieth century. In D. Haraway (Ed.), Simians, cyborgs, and women: The reinvention of nature (pp. 183-201). New York: Routledge.
- Harrison, C., & Hudson, S. (2009). Texture displays: A passive approach to tactile presentation. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 2261-2294). New York: ACM Press.
- Harrison, P. (2007). “How shall I say it...?” Relating the nonrelational. Environment and Planning A, 39(3), 590-608.
- Hayles, K. (2005). Computing the human. Theory, Culture, & Society, 22(1), 131-151.
- Hirschheim, R., & Klein, H. (1989). Four paradigms of information systems development. Communications of the ACM, 32(10), 1199-1216.
- Ishii, H. (2008). Tangible bits: Beyond pixels. In Proceedings of the 2nd International Conference on Tangible and Embedded Interaction (pp. xv-xxv). New York: ACM Press.
- Ishii, H., & Ullmer, B. (1997). Tangible bits: Towards seamless integration of interfaces between people, atoms, and bits. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 234-241). New York: ACM Press.
- Jay, M. (1988). Scopic regimes of modernity. In H. Foster (Ed.), Vision and visuality: Discussions in contemporary culture (pp. 3-23). New York: New Press.
- Kay, A. (1990). User interface: A personal view. In B. Laurel (Ed.), The art of human-computer interface design (pp. 31-61). Cambridge, MA: Addison-Wesley.
- Kesteren, I., Stappers, P., & Bruijn, S. (2007). Materials in products selection: Tools for including user interaction in materials selection. International Journal of Design, 1(3), 41-55.
- Klemmer, S., & Landay, J. (2009). Toolkit support for integrating physical and digital interactions. Human-Computer Interaction, 24(3), 315-366.
- Krippendorff, K. (2005). The semantic turn: A new foundation for design. Boca Raton, FL: CRC Press.
- Lakoff, G., & Johnson, M. (1981). Metaphors we live by. Chicago: University of Chicago Press.
- Landin, H. (2005). Fragile and magical: Materiality of computational technology as design material. In Proceedings of the 4th Decennial Conference on Critical computing: Between Sense and Sensibility (pp. 117-120). New York: ACM.
- Lazzari, M. (2005). The texture of things: Objects, people, and landscape in northwest Argentina. In L. Meskell (Ed.), Archeologies of materiality (pp. 126-161). London: Blackwell.
- Löwgren, J., & Stolterman, E. (2004). Design av informationsteknik: Materialet utan egenskaper. Lund, Sweden: Studentlitteratur.
- Löwgren, J., & Stolterman, E. (2007). Thoughtful interaction design: A design perspective on information technology. Cambridge, MA: MIT Press.
- Manovich, L. (2007). Alan Kay’s universal media machine. Northern Lights, 5(1), 39-56.
- Martin, R. (2003). The organizational complex: Architecture, media, and corporate space. Cambridge, MA: MIT Press.
- Matério (2005). Material world 2: Innovative materials for architecture and design [Vol. 2]. Basel, Switzerland: Birkhäuser.
- Miller, D. (2005). Materiality. Durham, NC: Duke University Press.
- Mukerji, C. (1997). Territorial ambitions and the gardens of Versailles. Cambridge, UK: Cambridge University Press.
- Negroponte, N. (1995). Being digital. New York: Vintage Books.
- Nelson, H., & Stolterman, E. (2003). The design way: Intentional change in an unpredictable world. Englewood Cliffs, NJ: Educational Technology Publications.
- Nielsen, J. (1994). Usability engineering. San Diego: Academic Press.
- Orlikowski, W., & Scott, S. (2008). Sociomateriality: Challenging the separation of technology, work and organization. The Academy of Management Annals, 2(1), 433-474.
- Qi, J., & Buechley, L. (2010). Electronic popables: Exploring paper-based computing through an interactive pop-up book. In Proceedings of the 4th Conference on Tangible and Embedded Interaction (pp. 121-128). New York: ACM Press.
- Randle, V., & Engler, O. (2000). Introduction to texture analysis: Macrotexture, microtexture, and orientation mapping. Singapore: Gordon and Breach Science Publishers.
- Redström, J. (2008). Tangled interaction: On the expressiveness of tangible user interfaces. Transactions on Computer-Human Interaction, 15(4), 37-53.
- Reed, M. (2009). Prototyping digital clay as an active material. In Proceedings of the 3rd Conference on Tangible and Embedded Interaction (pp. 339-342). New York: ACM Press.
- Reiger, H. (2007). Giving materials a voice. In Proceedings of the 1st Conference on Tangible and Embedded Interaction (pp. 125-126). New York: ACM Press.
- Robles, E., & Wiberg, M. (2010). Texturing the ”material turn” in interaction design. In Proceedings of the 4th International Conference on Tangible and Embodied Interaction (pp. 137-144). New York: ACM Press.
- Rullo, A. (2008). The soft qualities of interaction. Transactions on Computer-Human Interaction, 15(4), 55-79.
- Scheerbart, P. (2001). The gray cloth: Paul Scheerbart’s novel on glass architecture. Cambridge, MA: MIT Press. (Original work published 1914)
- Seitinger, S., Perry, D., & Mitchell, W. (2009). Urban pixels: Painting the city with light. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 839-848). New York: ACM Press.
- Spigel, L. (2001). Welcome to the dreamhouse: Popular media and postwar suburbs. Durham, NC: Duke University Press.
- Suchman, L. A. (2007). Human–machine reconfigurations: Plans and situated actions. Cambridge, UK: Cambridge University Press.
- Thrift, N. (2003). Closer to the machine? Intelligent environments, new forms of possession and the rise of the supertoy. Cultural geographies, 10(4), 389-407.
- Thrift, N. (2004). Electric animals: New models of everyday life. Cultural Studies, 18(2/3), 461-482.
- Vallgårda, A., & Redstöm, J. (2007). Computational composites. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 513-522). New York: ACM Press.
- Vallgårda, A., & Sokoler, T. A. (2009). Material focus: Exploring properties of computational composites. In Extended Abstracts of the SIGCHI Conference on Human Factors in Computing Systems (pp. 4147-4152). New York: ACM Press.
- Want, R., Schilit, B., Adams, N., Gold, R., Petersen, K., Goldberg, D., Ellis, J., & Weiser, M. (1995). The PARCtab ubiquitous computing experiment. IEEE Personal Communications, 2(6), 28-43.
- Weiser, M. (1991). The computer for the 21st century. Scientific American, 265(3), 94-101.
- Weiser, M., & Brown, J. S. (1996). Designing calm technology. In T. Rutowski (Ed.), The ”100” show: The 18th annual of the American center for design (pp. 159-163). New York: Watson-Guptill Publications.
- Wiberg, M. (2009). Designing interactive architecture – Lessons learned from a multi-professional approach to the design of an ambient computing environment. International Journal of Ambient Computing and Intelligence, 1(3), 1-18.
- Wright, P., Wallace, J., & McCarthy, J. (2008). Aesthetics and experience-centered design. Transactions on Computer-Human Interaction, 15(4), 1-21.
