Home > Vol 3, No 3 (2009) > Rocchesso

Designing Continuous Sonic Interaction

Davide Rocchesso 1,*, Pietro Polotti 2,3, and Stefano Delle Monache1

1 IUAV University of Venice, Venice, Italy; 2 University of Verona, Verona, Italy; 3 Conservatory of Music G. Tartini, Trieste, Italy

Continuous interaction and multisensory feedback present tremendous challenges to designers who are mostly educated along the lines of visual thinking and discrete interactions. For this investigation, a method of research through pedagogical examples, called basic design and developed in some post-Bauhaus design schools, has been adopted to investigate various interaction primitives mediated by continuous sound feedback. Four basic design exercises, prototypical for different kinds of interaction with kitchen tools, are conducted. The exercises take on different design issues, which are tackled using a variety of strategies. These include the use of a classroom setting, shared observation, and self-reflection. The exercises emphasize auditory perception in interaction, where continuous sonic feedback is realized through parametric control of sound synthesis algorithms. In sum, the present work aims to exploit the value of basic design approach through explorations of sonic features in continuous interaction.

Keywords – Basic Interaction Design, Sound, Continuous Interaction.

Relevance to Design Practice – The design of the sonic qualities of interactive objects is hampered by the difficulty of developing, testing and sharing sounds in action. Experiences inspired by the basic design pedagogy and by experimental phenomenology prove useful in developing a shared understanding and design solutions for some interaction primitives affected by sonic feedback.

Citation: Rocchesso, D., Polotti, P., & Delle Monache, S. (2009). Designing continuous sonic interaction. International Journal of Design, 3(3), 13-25.

Received June 23, 2009; Accepted November 11, 2009; Published December 28, 2009.

Copyright: © 2009 Rocchesso, Polotti, and Delle Monache. Copyright for this article is retained by the authors, with first publication rights granted to the International Journal of Design. All journal content, except where otherwise noted, is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 2.5 License. By virtue of their appearance in this open-access journal, articles are free to use, with proper attribution, in educational and other non-commercial settings.

*Corresponding Author: roc@iuav.it

Davide Rocchesso received a Ph.D. from the University of Padova, Italy in 1996. Between 1998 and 2006, he was with the Computer Science Department at the University of Verona, Italy, as an Assistant and Associate Professor. Since 2006, he has been with the Department of Art and Industrial Design of the IUAV University of Venice as Associate Professor. He has been the coordinator of EU project Sob (the Sounding Object) and local coordinator of the EU project CLOSED (Closing the Loop Of Sound Evaluation and Design) and of the Coordination Action S2S2 (Sound-to-Sense; Sense-to-Sound). He is currently chairing the COST Action IC-0601 SID (Sonic Interaction Design). He has authored or co-authored over one hundred publications in scientific journals, books, and conferences. His main research interests are sound modeling for interaction design, sound synthesis by physical modeling, and design and evaluation of interactions.

Pietro Polotti is part of the Vision, Image Processing and Sound group of the University of Verona and teaches Electronic Music at the Conservatory of Trieste in Italy. He also collaborates with the University IUAV of Venice within the Interaction research group. Previously, he was employed by the Politecnico di Milano as part of the IST TAI-CHI European project. He obtained a Ph.D. in Communication Systems from EPFL, Switzerland in 2002 with a thesis on a new method for sound synthesis based on wavelet transforms, called fractal additive synthesis. Over the last several years, his interests moved towards sonic interaction design.

Stefano Delle Monache graduated in law at Università Cattolica del Sacro Cuore of Milano, and in electronic music and technologies of sound at Conservatorio di Musica G. Verdi of Como. In 2007/2008, he received a research grant at University of Verona in the scope of the EU-founded CLOSED (Closing the Loop of Sound Evaluation and Design) project. He is currently a PhD candidate at IUAV where his research on product sounds is mainly related to the application of design methods and practices, including sketching, to a context of interaction design where auditory feedback is of central importance.

Introduction

This investigation focuses on design practices that make use of embodied, continuous interaction, with auditory display as the main channel for user feedback. The repertoire of audio sensors, actuators, processing and synthesis techniques that are widely accessible nowadays make sonic interaction design a privileged framework for experimental design of continuous interaction in embodied interfaces (Polotti & Rocchesso, 2008; Rocchesso & Serafin, 2009). In his most recent book—The Design of Future Things, Norman (2007) illustrates the urgency of “naturalistic, continual feedback” by means of some auditory examples, such as the sound of the boiling water in the kettle (as an alarm) or the sound of a motor drill (as a continuous monitor). His advice is followed in this present research by addressing situations where the user is continuously involved in object manipulation, and where human-object interaction may be altered by specific design choices in continuous feedback. Of particular interest here are the expressiveness and aesthetics that may emerge from elementary acts of use of everyday computational things. To explore this, this investigation considers a number of perspectives. As observed by Hallnäs and Redström (2002b), “A proper aesthetics of computational things concerns time gestalt,” and there are “many similarities with musical expression.” Taking this observation very seriously, interactions were designed for this investigation that made use of prior work in sound and music computing.

In order to develop abstract conceptualizations of interactions for analysis, critiques, and nurturing of visions, Löwgren (2007a) exploited common design practices and pedagogies. Among practitioners and students, a design discourse develops around shared ideas and experiences that give rise to a continuous debate. This work also exemplifies extensions to classic pedagogical and research practices such as basic design and experimental phenomenology, although direct experimentation of alternatives in a constrained problem-based setting is emphasized.

The reference environment for this investigation is the kitchen, where some interaction primitives such as screwing the top piece of a coffee maker, chopping a carrot, or pouring water from a carafe are all enhanced by expressive augmentation through sound. Four basic design exercises are examined, each having a well-defined interaction objective and clear constraints. Through doing so, three different ways to proceed from problem formulation to exploration of solutions are illustrated. These include the involvement of students and the sharing of experiences; introspection and inter-observation within a design team; and installations in public exhibitions and collection of reactions. Despite the diversity of phenomena investigated here, these exercises demonstrate how one or two physics-based sound models can fit the requirements for each object and provide a rich and expressive sonic feedback. This means that, with continuous interaction, the main sound design problem is not that of finding which sound is appropriate for a given gesture. Instead, the main problem is that of finding a sensible fitting of the interaction primitives with the dynamical properties of some sound models, in terms of specific perceptual effects.

Background

Embodied Interfaces, Continuous Interaction, and Sound

The exercises reported in this article consider the role of sound in continuous interactions as they may occur in the manipulation of objects, either directly or through interfaces. In human-computer interaction, the virtues of direct manipulation have been widely appreciated and exploited (Shneiderman, 1983). Direct manipulation’s kernel principles include the persistent representation of the object of interest; the inclusion of physical actions; and rapid incremental reversible operations with immediate feedback. In traditional WIMP/GUI interfaces, however, the principles of direct manipulation are applied to a world where the key components of interaction are largely in the digital domain, as there is physical separation between actions (mouse movements) and feedback (displayed output). An attempt to move interaction into the physical realm led to Tangible User Interfaces (Ishii & Ullmer, 1997), which rely on direct manipulation of tokens having some representational capabilities. The fact that users are manipulating tokens eliminates a level of indirection, since part of the feedback is indeed where the action is. However, a hermeneutic level is often introduced, due to the nature of tokens as representations. This is what embodied interfaces (Dourish, 2001) tend to avoid by reducing mediations at a minimum, in the spirit of phenomenological thinking.

In contrast to an embodied interface, a disembodied interface (as most existing machine interfaces use) tends to give a schizoid combination of perception and action. For the auditory world, this was well understood and extensively described by Murray Schafer (Schafer, 1994), who also coined the term schizophonia to indicate the separation from sound sources induced by recording and broadcasting means. The danger of schizophonia was felt much earlier by the composer Belá Bartók, who wrote in 1937 (Bartók, 1992) that “[...] the less foreign bodies are interposing themselves between the human body and the vibrating body or, the longest the time during which the human body controls the vibration is, the more the created musical sound will be immediate and, so to speak, human” (p. 289). In the context of musical instrument design, the enactive approach (Essl & O’Modhrain, 2006) is concerned about how perceptually-guided action defines the experienced playability of an instrument. Generalizing to non-musical, everyday situations, enaction can be seen as continuous embodied interaction.

Before the industrial revolution, most of human actions in the world were essentially continuous. Opening a door meant grabbing the handle, turning it, and pushing it so that the door swings on its hinges. These continuous actions are far more prone to expressive manipulation than simple button triggering, as found for example in modern elevator doors. Indeed, as studies in musical acoustics show (Juslin, Friberg, & Bresin, 2002), dynamic expressiveness can be induced through on/off switches only by temporal fluctuations of repeated triggering patterns. Being inherently expressive, continuous actions and gestures are supposed to be more “natural” than triggers. Naturalness here means that control is left to the human manipulator rather than transferred to some machinery – or as Bartók would observe, “less foreign bodies are interposing” (p.289). According to the tightness of sensory feedback to the door handle, control can be more or less direct or physical. To illustrate further, sailing using the tiller is in a sense more engaging than using the wheel to control the rudder. In the latter there is a decoupling that allows application of smaller forces, but less feeling about the forces acting on the boat.

When sustained feedback is elicited by triggers, the person acting on the trigger has the impression of autonomous life, and experiences a sense of causality. For example, in computer jargon, exceptions are thrown, programs are launched, and so on. These are expressions similar to those used by the subjects of Michotte’s (1963) famous experiments to describe causality in motion patterns. In interfaces, causality induced by automatization and triggers can be easily fooled by simultaneous extraneous feedback. In other words, one may easily get a fictitious sense of causality, as feedback comes from confirmation for a complete gesture, and not from continuous monitoring of the gestural progress. Conversely, enactive and embodied interfaces are based on a closed loop based on motor skills, where control is exerted via continuous and simultaneous perception and action. Therefore, in such interfaces, experienced causality tends to correspond more closely to physical causality. This elicits aesthetic experiences that are now recognized as crucial for the evaluation of an interactive artifact. Interfaces that have such a “tightly connected loop between [...] action and response” (Löwgren, 2007b, p. 87), enabled by continuous feedback, have been called pliable.

Approaches in Design

When designing objects for embodied multisensory continuous interaction as done in this investigation’s four design exercises, the controllable dimensions are many. There are countless possibilities for augmenting objects, and sensors and actuators should be considered together as they will certainly interfere in the action–perception loop. How can this complexity be tackled? One possibility is to think in terms of basic phenomena constructively. So, one should consider fundamental interaction gestalts, the basic, immediate and inherently meaningful actions such as pushing, pulling, shaking (Svanaes, 2000), that are exploited in “natural” interactions. Interaction gestalts naturally resonate with the concept of movement primitives, as they are considered in the literature of motor sciences as building blocks for complex motor skills (Schaal, Ijspeert, & Billard, 2003). Interaction gestalts may also result from abstraction of actual interactions (Hallnäs & Redström, 2002b). In a more holistic sense, some authors (Lim, Stolterman, Jung, & Donaldson, 2007; Löwgren & Stolterman, 2004) describe interaction gestalt as the bridge connecting user experience and interactive artifact. Still, it is agreed that these gestalts are not properties of objects but are rather emerging properties of user-object interaction unfolding in time. In the context of musical instrument design, Essl and O’Modhrain (2006) proposed the hypothesis of weak sensorimotor integration that is a grouping of actions according to some shared physical behavior that can be abstracted from the specific physical objects.

In post-Bauhaus design schools (Ulm, Yale, Chicago), prominent designers and educators such as Tomás Maldonado, Josef Albers, and László Moholy-Nagy organized their classes around themes by proposing exercises with well-defined objectives and constraints (Anceschi, 2006; Lupton & Phillips, 2008). Students acquired their basic compositional skills by searching for solutions to the problems, and by sharing and discussing the results. The elements to work with could be raw materials, ready-made artifacts, scientific facts, or algorithmic procedures. With this practice of basic design, research on the design fundamentals was advanced while teaching in a very effective way if one considers the complexity of the themes. To illustrate, Albers’ (2006) color exercises demonstrate a synthesis of centuries of scientific research on color perception through a set of compelling examples. Further, extensions of the basic design method to multisensory communication were introduced at the New Bauhaus in Chicago and are now being practiced in several design schools (Riccò, 2006).

In contemporary interactive contexts, the raw materials that designers have to work with and the phenomena that they wish to influence are complex and multifaceted. However, several contemporary theorists and educators (Anceschi, 2006; Findeli, 2001) claim that basic design, although in a renovated form that we may call basic interaction design, is still a valuable method. In particular, a method of inquiry may proceed by analyzing actions, extracting interaction gestalts, and designing exercises around a specific interaction gestalt (Franinovic, 2008, 2009). Exercises are typically tackled by constructively sharing observations, similar to what experimental phenomenologists usually do to come up with explanations for phenomena that are complex and difficult to reduce to collections of separable and measurable mechanisms (Bozzi, 1978; Sinico, 2008).

Everyday Objects and Gestures

To exemplify the basic interaction design method, this investigation starts with an analysis of actions in the kitchen (Visell, Franinovic, & Hug, 2007) and considers the actions involved in preparing a cup of coffee with the moka pot, an espresso coffee maker. Of note is the interaction primitive of screwing the parts of the moka together, which is abstracted as a case of dynamic connection and used as the center of a design exercise. Another interaction primitive examined is that of cyclic motions typically performed in the kitchen, like cutting vegetables. The nature of cyclic motions and their difference from discrete ones is discussed, as is the continuous feedback that may accompany them. Finally, two exploratory exercises aimed at developing an augmented sense of resistance when performing certain continuous actions, such as pouring and stirring, are presented. These four proposed exercises are indicative of a methodology being followed in work on continuous multisensory interaction.

In the exercises described in this article, objects (e.g., a moka pot) and materials (e.g., carrots) are used that have not been explicitly designed for our research or pedagogical purposes. Even though basic interactions are abstracted at the analysis stage, their diverse realizations make purposeful use of found objects. This tradition of objet trouvé is deeply rooted in twentieth-century art and is applicable here as well. Besides being often more widely accessible and cheaper than custom-made prototype objects and materials (Franinovic, 2009), using these readymade objects highlights the oscillatory balance between function and expression that should characterize design practices, especially in interactive contexts (Hallnäs & Redström, 2002a). This oscillatory attitude allows an understanding of the expressive features of objects in use, while at the same time elicits new uses, or misuses, of objects. Again, in this respect music may provide many examples, ranging from the bow (from hunting tool to string exciter) to the turntable (from reproduction of music to generation, via scratching, of music) (Delle Monache, Polotti, Papetti, & Rocchesso, 2008).

Exercises in Sonic Interaction Design

This section introduces and discusses three general design themes: connection between objects, cyclically repeated actions, and contradictory feedback. One specific design exercise for each of the first two themes is proposed and some experimental results are illustrated and discussed. For the third theme, two different exercises are conceived. One of these exercises involves the manipulation of transformational aspects of interaction, and the other considers the manipulation of structural properties of the objects involved in the interaction. The exercises can be considered as preliminary to the actual design process, and their purpose is to develop an experiential awareness in the designer. At this preliminary stage, a basic design attitude naturally merges with sketching techniques (Buxton, 2007), and objective evaluation is more effectively replaced by shared observation and discussion.

It is important to observe that, in basic sonic interaction design, sound synthesis models and algorithms are to be considered among the raw materials to work with. They play the same role that colored paper sheets played in Albers’ exercises (Albers, 2006). In the following descriptions, the materials are not described in detail, but it is worth noticing that a common palette of sound models based on physics-based sound synthesis is used (Delle Monache et al., 2009). In particular, we make use of the sonic manifestations of simulations of friction phenomena (Avanzini, Serafin, & Rocchesso, 2005) and impact phenomena (Avanzini & Rocchesso, 2004).

Theme I: Continuous Coupling for Connections

An important topic in product design is that of connections, or how to create complex objects by joining simpler elements together. Different connections are characterized by different temporal dynamics. The act of joining and splitting parts unfolds in time, and the temporal exploration of the continuum between “attached” and “detached” can be represented pictorially via traces, as shown in Figure 1.

Figure 1. Temporal dynamics of connections. Representation from Klöckl (2008).

The vertical axis represents the degree of tightness in a connection. Although being a continuum, this tightness dimension can be categorically perceived as a small number of discrete states. For example, torque measurement systems (see for instance Ingersoll-Rand products, http://www.ingersollrand.pl/) discretize the state of a screw connection into three stages: (i) torque too low, (ii) OK, (iii) torque too high. A low torque may compromise the effectiveness of the connection. Excess torque may make the disconnection of the two parts difficult. For such measurement systems, audio-visual signals have been designed to inform the user symbolically about the state, using a color visual code (yellow, green, red) and an auditory code based on counting (one, two, or three beeps).

The Moka and the Three Stages of Coupling

An example of critical connection is found in the moka coffee maker, a piece of Italian design from the 1930s now in widespread use worldwide. In order to prepare a cup of coffee, one has to fill the boiling chamber with water, fill the filter with ground coffee, and connect the three parts (boiling chamber, filter, and top container) by means of a screw connection. Instruction manuals for moka pot brewing systems (e.g. Sweet Maria’s, n.d.) use much space to describe how to screw the top and bottom together, the proper hold of the pot, motions used, and the required arm strength. How to detect deviations from proper tightness is also discussed, with an example given of being able to see steam escaping from the point of connection. One of the reasons for such expressive descriptions is that the gesture of closing the moka is continuous, although it can be discretized into the three stages of tightness of screw connections.

It seems possible, then, to apply some pressure or torque sensor to a moka and provide the same kind of discrete audio-visual feedback as it is given by torque measurement tools. One concern, though, is that this will likely reduce much of the fascination and emotional engagement that people experience when exerting continuous manipulations of well-balanced mechanisms (Buxton, 2005). Given these motivations, the following basic design exercise is proposed:

One difficulty to overcome is that audio-visual symbolic signals, as the ones found in torque measurement systems, would result in a disembodied feedback because they enforce the discretization of gestures that are naturally continuous. Also, they rely on symbolic coding rather than on enactive experience. This is forbidden by the constraints of the exercise, which is the nature of basic design exercises. In basic design exercises, constraints are usually supported by a rationale, although a different set of constraints may make perfect sense in a different exercise.

If choosing between visual and auditory feedback for the moka exercise, audition is certainly to be preferred because, in temporal patterns and tasks, it affords higher resolution and it tends to prevail over possible non-coherent cues coming from other senses (Handel, 2006). Taking into account this observation, the design space is further constrained leaving only sonic interactive feedback as being able to satisfy the requirements of Exercise 1. Still, the possibilities are countless, and one may exploit metaphors, analogies, or other design tricks to narrow the design space further. To do this here, prior art of sound and music computing is exploited. Exercise 1 is approached by imposing a correspondence between two seemingly unrelated acts: screwing the pieces of a moka together and running a bow across a violin string. Although these actions belong to vastly different contexts, they are both prone to dry friction and the resulting dynamics are inherently related.

The Violin and the Three Qualities of Playing

Stringed musical instruments played with a bow, such as the violin, offer an interesting example to orient the approach to this design exercise. Starting with the requirement of three discrete stages, violins indeed exhibit three main kinds of sonic quality, which can be explored by varying the three principal control variables: bow force onto the string, bow velocity, and bow position along the string. The oscillatory regime corresponding to “good tone” production is called Helmholtz motion. A graphical tool used to display at a glance the different oscillatory behaviors of a bowed string is the Schelleng diagram (Schelleng, 1973), schematically drawn in Figure 2 for constant bow velocity. On the sides of “normal” tone production, two other regions are displayed. Given a bow position and velocity, weaker forces produce a “surface sound” eliciting higher modes. Stronger forces, on the other hand, produce aperiodic, “raucous” sounds.

Figure 2. The Schelleng diagram, in the force–position plane, for constant bow velocity.

Similarly, the region of “playability” can be highlighted on the force–velocity plane for constant bow position. Different models of bow-string interaction can be analyzed in terms of the oscillatory regimes they produce, and Schelleng diagrams can be produced experimentally (Serafin, 2004). The wider the Helmoltz region results, the more easily playable the friction model is. A model that displays a wide playability region, while at the same time being so versatile as to reproduce a large variety of oscillatory regimes, is the elasto-plastic friction model (Avanzini et al., 2005). This model was chosen for use in Exercise 1.

By exploring the space of control parameters, we can gradually step into different oscillatory regimes, and listeners are immediately capable to catch the quality of the physical actions associated with such sound manifestations. Namely, three distinct areas of effort (too low, ok, too high) can be readily associated with the three main tone-quality areas described by the Schelleng diagram. As such, the requirements of our sound-design exercise can be met.

Sonifying the Moka

During a doctoral workshop at the University Iuav of Venice, Exercise 1 was tackled by using preparatory materials (a disassembled moka, sensors and actuators, sound synthesis software modules) to put together convincing demonstrations that were experienced individually and discussed in groups. This is the kind of research through pedagogy, or pedagogy through research, that basic design practices recommend. A force sensor was applied between the filter and gasket of a moka, and a force signal was used to control the elasto-plastic friction model that is part of the SDT toolkit (Delle Monache et al., 2009). Although one may map the sensed force directly to the force control parameter of the elasto-plastic model, clearer and more compelling transitions are obtained by co-varying other parameters of friction, similar to what a violinist actually does when performing a musical gesture (Demoucron, 2008). Moreover, as a general criterion for the sonification of everyday objects, it is preferable to make sparse use of short sound pulses rather than modulating a sustained tone. The latter would be anti-ecological and detrimental to the sustainability of a soundscape. Therefore, short friction bursts were generated by a pseudo-random process while the moka components were being coupled. The resulting sound dynamically changes its timbral quality as the coupling gets progressively tighter. More precisely, the timbre moves gradually from a sound of glass harmonica for loose coupling, to assuming a rubber quality for the tight stage. When the coupling gets into a “too tight” state, the sound resembles that of a squeaking hinge. Being as this investigation focuses on basic design methods rather than on reproducible scientific experiments, a detailed report of the sound synthesis algorithm and parameters is not presented here.

Discussion

The sonified moka was experienced directly by ten people who attended the workshop. The unanimous impression was that sound enhanced the screw connection and made the task more engaging. Feedback about the degree of closure was reported to be very natural. The experience of acting directly on the moka was judged to be very different from simply listening to the produced sound. The auditory feedback blended with the kinesthetic perception coming from exerting manual force on the moka components. Indeed, some people started to play with the moka and tried to push the boundaries of meaningful interaction, up to the point that, at the end of session, the force sensor was broken due to excessive pressure. This highlights a problem that is common to any object that affords expressive interaction, which is that it elicits unexpected or excessive manipulation.

When considering sound augmentation of everyday artifacts, the problem of localization of the speaker is quite serious. The question is whether to attach a speaker to the object or not, although physical constraints may limit choice here. Using an embedded loudspeaker usually enhances the unity of experience, even though the ventriloquist effect (Handel, 2006) provides a certain degree of perceptual coherence even for speakers displaced from the artifact. Both options were tried during the workshop. When an actuator was directly attached to the moka, everybody agreed that the sound truly became part of the object, not only because of its apparent point of emission, but also for the acoustic properties of the aluminum chambers of the moka, which affected sound production.

In this paper, the results of many hours of direct experimentation, shared appreciation, and discussion were condensed. In basic design, as well as in experimental phenomenology, the process of knowledge acquisition is distilled in a few self-speaking demonstrations (see, e.g. Albers, 2006; Sinico, 2008). In interaction design, the dissemination of interactive experiences is problematic, but video examples can often replace first-hand experience effectively. A video documentation of the moka realization and of all the basic design exercises presented in this article is available online at http://www.soundobject.org/BasicSID/.

Theme II: Multisensoriality and Expressiveness in Rhythmic Tasks

Since the 1990s, some researchers have been considering continuous gestures in graphical user interfaces, like crossing targets or traversing hierarchical cascading menus, and trying to model them as limits of sequences of discrete gestures. The steering law was derived and applied in this framework (Accot & Zhai, 1997). However, when the task requires cyclic movements, experiments show that these movements cannot be simply treated as a concatenation of discrete gestures, as their rhythmic nature allows better exploitation of the physical properties of the neuro-motor system. Actually, Fitts’ law has been reported to break down in rhythmic tasks, where performance turns out to be supported by much higher indexes of performance (Smits-Engelsman, Van Galen, & Duysens, 2002).

Some other researchers have looked at what is in between the starting position and the final target in goal-directed movements, and have discovered that there are kinematic patterns (Bootsma, Fernandez, & Mottet, 2004). Specifically, when difficulty is raised in a reciprocal aiming task, velocity profiles take an asymmetric bell shape, with an attack shorter than the release. So, there is much more behind pointing tasks. The continuous support of discrete events contains much information that we are likely to exploit in everyday activities. In particular, the expressive content of our actions is largely mediated by these continuous motions, even when they are aimed at discrete events, such as cutting a carrot evenly into pieces.

In music performance, preparatory and ancillary continuous movements are fundamental to convey the intended expressive character to a sequence of notes or drum strokes (Dahl, 2005). When attending the performance of a drummer, one integrates visual and auditory information. While the former is supported by continuous profiles of an array of kinematic variables, the latter comes in discrete bursts of impact noise. The visual continuous stream and the auditory burst-based stream get integrated and synchronized according to different conditions. For example, the tempo of a rhythm and the naturalness of visually-perceived movements are known to influence the perceived synchrony (Arrighi, Alais, & Burr, 2006). Also, the weight given to different sensorial channels may depend on the degree of expertise. Expert drummers tend to trust auditory feedback more than visual feedback when evaluating audio-visual synchrony of drum gestures (Petrini et al., 2009). These observations strongly support the use of auditory display to provide feedback in cyclic tasks, and to afford the development of a specific virtuosity. A skilled person will be capable of matching speed with accuracy, and consistency with adaptability, with a minimum of attention and effort (Dahl, 2005). Virtuosity transforms the fatigue of repeating some gestures thousands of times into a joy of performing (Sennet, 2008).

As early as 1991, Gaver, Smith, and O’Shea (1991) simulated a bottling plant and demonstrated that auditory display can support rhythmic activities in collaborative environments. More recently, Lantz and Murray-Smith (2004) published a study on rhythmic interaction with mobile devices. The authors used a model based on Dynamic Movement Primitives to recognize and classify a number of rhythmic gestures of people interacting with mobile devices. The sonic feedback was intended to give continuous information about the quality of performance of a specific gesture. Jylhä and Erkut (2009) investigated the cyclic and rhythmic dimensions of clapping and the different hand clap types as means to sonically interact with computer applications. The dynamic relation between conscious tempo alteration and unconscious entrainment confirmed the power of rhythmic actions as effective means to establish continuous negotiations with the computer.

Rhythmic Cutting in the Kitchen

In the context of human-computer interaction, the action of rhythmic cutting has already been recognized as a source of useful information for achieving a sort of computer-assisted cooking (Kranz et al., 2007). A basic design exercise can be conceived to exploit the effectiveness of a well-defined feedback in such cyclic activities.

In several traditions of food preparation, cutting carrots into pieces of various shapes is considered a sort of performative activity, and sometimes even a key to meditation (Snow, n.d.). Chopping techniques have a long tradition in both eastern and western cultures. Prehension and control of movements, which means the application of a minimum force and release while stressing on a beat, are considered to be fundamental aspects that open the cook’s experience to concentration. This is condensed in an ancient Chinese piece of cooking advice which offers that a good cook must learn first to cleave a grain of boiled rice (Sennett, 2008).

To further support the importance of these motions, anthropologists have observed how deeply psychological states can be affected by shifts in tempo and intensity of cyclic gestures, as in the case of the drumming of shamans (Hodgkinson, 1996). These phenomena occur for enactive involvement in the action, and not by mere exposition.

Sensorized kitchen tools may enrich the enactive experience of food preparation, to the point that cutting precise rondelles, as an example, becomes an act more easily achievable and enjoyable. For that purpose, the synchronization and balance between senses can be manipulated. The feedback strategy is crucial in determining how effectively synchronization is achieved. Synchronization is performed through anticipation (Aschersleben, 2002), and different rhythmic feedbacks can change how the user internally uses the anticipatory error that supports the interaction loop. This leads to the next design exercise.

Exercise 2 was tackled by the authors as a self-reflective basic design exercise. In other words, it built upon the shared experiences and discussions within the team of investigators. Self-reflection, or introspection, is the key tool of experimental phenomenology, or descriptive experimental psychology, dating back to the method of understanding by demonstration advocated by Franz Brentano in the 19th Century (Vicario, 1993). Basic design and experimental psychology are, in this respect, both using the practice of shared observation as the only possible way of assessing the properties of objects. The fact that this sharing may include students, naïve subjects, or just the investigators themselves, is of minor importance for the drawn conclusions, even though a wider panel may increase the robustness of results. Bozzi (1978) proposed the inter-observational method, where an experiment is performed by jointly exposing a small group of subjects to the stimuli. Since the members of the group have to agree on a report, problems of outliers and degree of expertise are largely reduced. At the same time, joint observation and discussion contribute to make the description of facts more stable and rich. In the design practice, it is clearly more convenient to let the team of designers wear the hat of subjects and provide an inter-observational report. Even though experimental phenomenologists recommend the direct participation of the experimenter without a privileged position with respect to the subjects, a potential bias is recognized in reducing the group of subjects to the team of designers. However, a justification for this convenient choice may be obviously found in the difference in the objectives between experimental psychology and design.

The experimental setup (see Figure 3) was made of three integrated sensing modules, namely a beat follower of the sounds of the blade hitting the chopping board, a tracker of vertical knife acceleration, and a color tracker of the portion of the carrot being cut. The sensed data were fused in order to provide a continuous sound feedback for the longitudinal progression and a rhythmic sound feedback for the cyclic action, thus supporting the coordination between the two hands during the action.

Figure 3. Cutting a carrot.

By means of a contact microphone applied to the chopping board, impact sounds generated by cutting actions were detected and treated as triggers of adaptive beats aimed at providing the user with a feedback about the regularity of her/his action. The beat tracker was based on the mean of the elapsed time between successive cuts, computed over a few (four to six) gestures.

Also, in order to estimate the size of the vegetable to be cut, a camera was placed parallel to the chopping board and attached to one of its edges. By color-tracking the orange color of the carrot (green for the zucchini), and by estimating the size of the orange blob, it was possible to detect the progress of the action, as displayed by the size of the integer part of the carrot. The mass value that decreased from a certain maximum to zero was used to control the configuration of the already mentioned friction sound model, providing a dynamic timbral transformation. The knife was equipped with a 3-axes wireless accelerometer. While cutting, the vertical acceleration was sensed as a direct continuous manifestation of user gestures.

Each of the sensing modules had a specific role in the control of sonic feedback. Three kinds of rhythmic feedback were programmed and tested for a comparative evaluation:

These different feedback strategies are explained as follows.

Beat with adaptive tempo and synchronous rhythm. The tracked mean tempo drove a metronome in milliseconds, and the user strived to be synchronous with the metronome-driven sound feedback. The user set the tempo by cutting the first rondelles and, afterwards, interacted with the computer-generated adaptive beat. The feedback beat had an inertial time for changing its value in case of a persistent change in rhythm of the user’s action. Additionally, small and/or isolated deviations in periodicity of the cutting gesture did not influence the rate of the feedback beat, thus steering the user to recover a regular pace.

Beat with fixed tempo and upbeat rhythm. The reference tempo of the sound response was fixed at 75BPM (800 ms inter-beat interval). Furthermore, the sound feedback occurred on the upbeat with respect to the onbeat defined by the impact of the blade on the chopping board. Therefore, the temporal offset of such periodic feedback for a regime of regular cutting at 75BPM was 400 ms. Rhythmical deviations from such reference target were subtracted from the offset, as if the metronome were spurring the player to get back in time. In other words, both in case of acceleration and deceleration of the cutting pace, the delay time of the sound feedback got shorter and the user was impelled by the closer upbeat to recover the reference tempo.

Beat with adaptive tempo and upbeat rhythm. The regular iteration of the gesture was driven by an upbeat sound response, which was delayed with respect to the impact of the knife on the cutting board by an amount of time corresponding to half the inter-beat interval. Unlike mode B, here the sound feedback depended on the user pace, similarly to what happened in mode A.

For all three cases, the impact of the knife on the cutting board was also sonically augmented, and a wide palette of physics-based impact sounds were considered. In particular, we tested a number of cross combinations of different virtual materials (glass, wood, metal, hybrids) and size in order to make a clear distinction between the impact sound and the feedback sound.

In addition, a continuous sound feedback was used to inform about the nuances of the cyclic motion by mapping the sensed acceleration data of the knife to the pressure control parameter of a physics-based friction sound model. This sound reinforced the rhythmic feedback as well as provided an auditory indication of progress of the whole action, which was obtained by timbral variation of the friction sound. By raising the pitch and by making the sound sharper, the sonic feedback conveyed the feeling of the reducing mass of the vegetables. In detail, the sound corresponding to the entire carrot or zucchini was low-pitched and spectrally richer, and it became timbrically slimmer and higher in pitch when the carrot or the zucchini mass diminished.

Discussion

The three proposed modes were tested with two different kinds of vegetables, carrots and zucchinis. It was evident that the difference in hardness between the two vegetables induced different gestures for accurate rhythmic cutting. In the case of zucchini, the user preferred to lift the whole knife, so that cutting came mainly from the weight of knife and forearm. In the case of the carrots, cutting was more effective if the tip of the knife was kept on the board, thus providing a pivot for the up-down movement of the knife itself. A number of experiments were freely performed and audio-visually recorded. Using the reported impressions of the users/designers, and an analysis of the video recordings, it was possible to qualitatively evaluate the three different modes.

Beat with adaptive tempo and synchronous rhythm. In mode A, the cyclic motion appeared to be fragmented in two stages, corresponding to dropping and lifting the blade. The performance seemed to chase the sound feedback, with fast accelerations in the final part of each of the two cycles, as if each sound feedback acted as a new start. Abrupt reactions and a small overall gesture width were noticeable. As an effect, the blade remained on the board for a shorter time, as compared to the other two modes. Maintaining a perfect synchronism in mode A seemed to be more unnatural and, thus, less successful when compared to modes B and C.

Beat with fixed tempo and upbeat rhythm. Mode B seemed to require the attitude and skills of a player performing music with a metronome. The nuance of the gesture was quite regular during cutting and holding, while it accelerated subtly in the final part of the lift. The descent/ascent motion was quick, as if waiting for the sound feedback and then chasing the next cut, albeit not as rushed as in mode A. In fact, the overall width of the gesture was larger than in mode A, even if smaller than in mode C.

Beat with adaptive tempo and upbeat rhythm. Mode C afforded maintaining a regular pace while not being strictly fixed to a reference tempo as in mode B. The fact that in mode C the sound feedback came after each stroke, and before the next one, made it an apparently better listening reference point, since it could be pre-attentively related to the sonified impacts of the knife on the board. Somehow, mode C recalled the “tick-tack” of a clock and this was possibly the reason of its effectiveness; a binary subdivision of time that could be very easily internalized by the user. From the video analysis of the cutting action, it appeared evident how in mode C the gesture was wider and more fluid than in the other two modes. Furthermore, the sound feedback tempo adapting to any persistent change of rhythm in the cutting action made the action more flexible and allowed the cutter to act in a more relaxed way with respect to mode B. For example, a slower beat at the beginning of the action as well as at the end, when the blade got nearer to the fingers, was often appropriate.

The continuous feedback due to the knife movement spurred contradictory observations. Someone judged it as a useful and consistent reinforcement of the rhythmic feedback, while others described it as potentially disturbing. Nevertheless, the timbre change providing information about the action progress received a unanimously positive evaluation. The “shrinking” sound feedback seemed to induce the users to withdraw their fingers from the last bit of carrot well in time not to have a finger cut.

Even though the focus here was on chopping, the sonically augmented knife and chopping board may be used for other purposes, such as smashing garlic. If the goal is that of precise chopping, it was noticed that the hardness of the vegetables influenced how a gesture is performed and, in turn, that this affects the sonic feedback. However, the evaluation of the different feedback strategies holds across these different kinds of chopping. Even so, it may well be the case that with a different objective, these observations and conclusions would not apply to other actions such as smashing.

Theme III: Contradictory Feedback in Interactions

The third proposed theme was extensively explored for the realization of an interactive installation based on everyday objects and actions, a sonic dining table called the Gamelunch (Delle Monache, Polotti, Papetti, & Rocchesso, 2007; Polotti, Delle Monache, Papetti, & Rocchesso, 2008; Delle Monache et al., 2008). The components of the installation were an ordinary table, some cutlery, and other dining items (see Figure 4). The actions were those of preparing and consuming food and drinks, like pouring, cutting, mixing, and so on. This exercise also adopted a basic design approach for the analysis of the actions and the definition of their artificial auditory feedbacks. One of the investigation strategies in the Gamelunch was to design sound feedbacks in a contradictory way, in order to let the user experience unexpected sonic responses while interacting with the ordinary objects of the sonic table.

Figure 4. The Gamelunch: a sonified interactive dining table.

When continuous feedback is coupled to continuous movements, the coordination of different sensory channels is known to affect the action (Lagarde & Kelso, 2006). For example, audition may convey a better sense of velocity or acceleration as compared to vision or haptics, thus eliciting smoother trajectories in a target-reaching task (Rath & Rocchesso, 2005). Little research has been done to investigate the emotional impact of continuous sounds on continuous gestures, even though it is recognized, in the area of product sound quality, that evaluating the feel of an object in use is different from rating different sounds presented over the headphones (Spence & Zampini, 2006). In the area of emotional design, it has been observed that users interpret displays as emotionally expressive even if there is no explicit representation of emotion (Böhner, De Paula, Dourish, & Sengers, 2005). So, it becomes possible to “distort” a display to manipulate its emotional impact. In the context of (non-interactive) pure observations, the effects of sound incongruity on product expression were investigated by Ludden and Schifferstein (2007), who found that surprise is a relevant and controllable design parameter.

Playing with the relationship between different sensory channels gives the opportunity to exploit the power of contradiction in multisensory interaction. A possible strategy is in fact to distort one (or some aspects of one) of the displays, while preserving the natural characteristics of the other ones. Empirically, it was observed that by properly designing a sonic feedback, one can facilitate or resist an action. Producing a sonic feedback, that is unexpected with respect to one or more aspects of a specific interactive context, has some effects on the performed actions. By means of what we could consider a manipulation of both cognitive expectations and emotional impact of the feedback, one is able to stress the relevance of a specific portion of the information conveyed by the distorted channel. Experimental realizations of this approach were tested as part of the Gamelunch installation.

For the purpose of having a stable and portable configuration, a table was enhanced by embedding a number of sensors, acquisition boards, loudspeakers and cables in the tabletop. Magnetic, light, and force sensors were invisible to the user and their purpose was to make the objects laying on the table sensitive. These objects included a couple of bowls and two dishes. Other graspable objects, including a knife, a fork, a couple of bottles, a tray, and a decanter, were made sensitive by means of 3-axes wireless accelerometers. Embedding the loudspeakers in the table improved localization of the sound sources and enhanced the embodied qualities of the objects.

As a guiding principle, it was hoped to preserve some physical consistency between sound and movement to produce a situation that when the action was more intense, the sound feedback should be energetically consistent in terms of loudness or increased number of events. In other words, the feeling of causal continuous control of the action should be maintained. This was considered to be fundamental in order to preserve the veridicality of the sonic feedback and to avoid the risk of arbitrariness that otherwise would occur. Under this constraint of energetic consistency, two main kinds of contradiction have been explored, one related with the temporal development of the actions, and the other deeply connected with the nature of materials that were manipulated. These two modalities correspond to what Warren and Verbrugge (1984) called transformational and structural invariants, respectively. The following exercises were developed within the Gamelunch environment, which was showcased, in successive stages of development, in a number of public exhibitions. Behavioral and verbalized reactions of users were collected and helped bring about a convergence of ideas toward design solutions for the proposed problems.

Manipulating Transformational Invariants

The manipulation of the transformational aspects of actions was addressed in the action of pouring a liquid from a decanter. In terms of basic design, the problem of the “resisting” decanter can be formulated as:

The previously mentioned solid-friction sound model (Avanzini et al., 2005) was used to sonify the inclination of the decanter, giving the feeling of a resisting force that contradicted the feeling of the decanter becoming lighter as liquid was poured from it. The same principle was also used for sonifying manipulations of a fork. In this case, a squeaking sound was generated when the fork was lifted. A feeling of heaviness in the action, and particularly of resistance in the elbow was unanimously perceived by the numerous users who tried to pour some water from the decanter or to eat something with the sonified fork. In addition, two opposing types of reactions were registered among users of the decanter and the fork: some users interrupted their action as if recalling they were on diet, while some other users were stimulated to emphasize the gestures. Both responses demonstrate the performative potential of these objects.

Manipulating Structural Invariants

The second focus of this third theme consisted in contradicting something not directly related to the action, like the material of the objects. Along these lines, the following exercise investigated the action of stirring a soup:

The researchers collectively came to an agreement that the sonic feedback used was similar to the sound of a tool mixing floating stones, which was in contradiction with the visual image of the soup in the dish. An effect of resistance to movement due to the acoustic manifestation of solid obstacles in the soup was perceived.

In general, contradictory feedback may be exploited in contexts where some actions have potentially serious consequences. It is well known that warning messages tend to be ignored in routine situations (Raskin, 2000). It may well be the case that some additional resistance introduced via multisensory emotional stimulation in presence of critical gestures induces higher attention, as in the case of the decanter and the fork. This is an important issue for future design of objects with a high degree of user-autonomy, such as cars (Norman, 2007). In support of this, the sustained experience of effort has proven to be an effective design ingredient to prevent modal errors in interfaces (Raskin, 2000).

Facilitatory Contradiction in Sonic Feedback

These exercises with contradictory feedback can also be applied in other areas by subverting their objectives. Instead of aiming at contrasting an action, one could pursue the opposite effect of facilitating it by means of the same sound design strategies based on contradiction. For example, when the sound of a knife is that of a fast and smoothly sliding blade, the action of cutting will be perceived as easier and smoother. In terms of basic design, an exercise with a swift-cutting knife could be formulated in the same way as Exercise 3 by simple inversion of the requirements of the objective, especially since both exercises focus on the action, or rather on the manipulation of transformational invariants. A swift counterpart of the solid soup of Exercise 4 can also be obtained by playing with the material identity (structural invariants). For example, a liquid sound feedback in the action of dressing a salad can alter one’s perception of the material in the bowl and make people feel the action of tossing the salad is lighter than it actually is. In the Gamelunch implementation of this specific case, it was noticed how another kind of distortion of the action could occur, in the sense that people were induced to mix the salad by means of a circular movement as if it were liquid. A similar effect was obtained with a fork by arbitrarily changing the sound produced by the action of piercing a bit of food. A gummy sound could give the sensation of an easy and soft action, even if using the fork to pierce a hard bit of solid caramel stuffed with nuts.

Even though rigorous experimentation has not yet been conducted, it is remarkable to note how most of the people who experienced the Gamelunch agreed on the effectiveness of the distortion in terms of acquiring consciousness of the role of sonic feedback in everyday actions. The principle of contradiction seems to be an extremely effective way to bring the attention of people on sounds produced by their actions and to make them conscious of the importance of the information they receive through their auditory channel, often underexploited or even neglected.

Other Interaction Primitives

In the early 1960s at the Ulm design school, Tomás Maldonado proposed the Antiprimadonna (Anceschi, 2006), an exercise with the objective of training designers to produce non-hierarchical visual patterns. The idea was that, after extensive practice with this task, the designer would be better in letting hierarchies emerge in a controlled way.

Similarly, the exercises on contradictory feedback can be considered as training toward better awareness in the manipulation of structural and transformational invariants in sonic interactive systems. In case of structural invariants, the goal could be to exploit the sonic identity of materials for supporting continuous actions by providing the feeling of the effect of the action on the state of the involved materials/objects. An example could be the case of liquids or powders that produce strong physical or chemical effects in little quantities. The sound should help to manipulate these materials by sonically representing the properties of their substances. In the Gamelunch exhibit, two bottles, one for oil and one for vinegar, were enhanced with corresponding sounds. The quantity of the poured liquid and the effect of the substance were represented by the sound activated when the bottle was tilted. Considering the bottle of vinegar, given the condition that the loudness of the sound was a hint about the quantity poured, and, could one gauge how sour the salad would be simply by listening? It appears that given a good sound design, one could actually “taste” with their ears.

Transformational properties can also be exploited for controlling interactive behaviors. A possible approach consists in anticipating the development of an action, and providing sonic cues about the consequences of certain gestures. For example, we sonified a tray with some glasses on top of it. If the tray were to be slightly inclined, a dripping sound was emitted to portend what would happen if the tray would be tilted further.

Several further interesting explorations are possible with the Gamelunch framework. For example, it would be interesting to see how users engage with each other around the sonic dining table. Some informal observations made at public exhibitions indicate that interesting orchestrations of interactions sometimes emerged, which might be of great interest to researchers. The work could also be extended by going deeper into the analysis of single interaction primitives, to make sense of how and why sonic feedback affects the action. This may be possible by analyzing the traces of the sensed signals, as has been done before (Rath & Rocchesso, 2005).

Conclusions

The examples and problems proposed and discussed here can be considered as exercises in basic sonic interaction design. One exercise on continuous feedback for mechanical connections was developed in a classroom setting, thus demonstrating the pedagogical essence of basic design. Another exercise, on cyclic continuous actions, was developed by the designers themselves, and enabled them to produce and compare different interactive sketches aiming at guiding and facilitating cyclic actions. Other exercises were shown as part of the development of a complex installation, the Gamelunch, where different gestural primitives were investigated in terms of the effect produced by the application of a contradictory sonic feedback to the actions themselves. Kitchen scenarios were chosen for the exercises, only because they could be instrumental to experimenting with continuous multisensory interaction under well-defined constraints. There are many other environments and tools that may benefit from carefully-designed multisensory feedback, in terms of accuracy, safety, affection and engagement. In this sense, this work can be seen as a reference for a more general line of investigation into sonic interaction design.

Acknowledgments

This research was supported by the project CLOSED (http://closed.ircam.fr/) and by the Action COST- IC0601 on Sonic Interaction Design (http://www.cost-sid.org/). We wish to thank Stefano Papetti for software development and for many insights. Thanks to Carlo Bassetti for describing torque measurement to us, and to Kristian Klöckl for his systematic investigation of connections.

References

1. Accot, J., & Zhai, S. (1997). Beyond Fitts’ law: Models for trajectory-based HCI tasks. In Proceedings of the SIGCHI Conference on Human factors in Computing Systems (pp. 295-302). New York: ACM.
2. Albers, J. (2006). Interaction of color.New Haven, CT: Yale University Press. (Original work published 1963)
3. Anceschi, G. (2006). Basic design, fondamenta del design [Basic design, foundations of design]. In G., Anceschi, M. Botta, & M. A. Garito (Eds.), L’ambiente dell’apprendimento – Web design e processi cognitivi [Learning environment – Web design and cognitive processes] (pp. 57-67). Milano, Italy: McGraw Hill.
4. Arrighi, R., Alais, D., & Burr, D. (2006). Perceptual synchrony of audio-visual streams for natural and artificial motion sequences. Journal of Vision, 6(3), 260-268.
5. Aschersleben, G. (2002). Temporal control of movements in sensorimotor synchronization. Brain and Cognition, 48(1), 66-79.
6. Avanzini, F., & Rocchesso, D. (2004). Physical modeling of impacts: Theory and experiments on contact time and spectral centroid. In Proceedings of the Conference on Sound and Music Computing (pp. 287-293). Retrieved May 15, 2009, from http://recherche.ircam.fr/equipes/repmus/SMC04/scm04actes/P46.pdf
7. Avanzini, F., Serafin, S., & Rocchesso, D. (2005). Interactive simulation of rigid body interaction with friction-induced sound generation. IEEE Transactions on Speech and Audio Processing, 13(5), 1073-1081.
8. Bartók, B. (1992). Mechanical music. In B. Suchoff (Ed.), Belá Bartók – Essays (pp. 289-300). Lincoln, NE: University of Nebraska Press. (Original work published 1937)
9. Böhner, K., De Paula, R., Dourish, P., & Sengers, P. (2005). Affect: From information to interaction. In Proceedings of the 4th Decennial Conference on Critical Computing (pp. 59-68) New York: ACM.
10. Bootsma, R., Fernandez, L., & Mottet, D. (2004). Behind Fitts’ law: Kinematic patterns in goal-directed movements. International Journal of Human-Computer Studies, 61(6), 811-821.
11. Bozzi, P. (1978). L’interosservazione come metodo per la fenomenologia sperimentale [Inter-observation as a method for experimental phenomenology]. Giornale Italiano di Psicologia [Italian Journal of Psychology] 5, 229-239.
12. Buxton, B. (2007). Sketching user experiences: Getting the design right and the right design. San Francisco: Morgan Kaufmann.
13. Buxton, W. (Winter 2005). Experience design vs. interface design. Rotman Magazine, 47-49.
14. Dahl, S. (2005). On the beat: Human movement and timing in the production and perception of music. Unpublished doctoral thesis, KTH School of Computer Science and Communication, Stockholm, Sweden. Retrieved November 12, 2009, from http://kth.diva-portal.org/smash/record.jsf?pid=diva2:11296
15. Delle Monache, S., Polotti, P., Papetti, S., & Rocchesso, D. (2007). Gamelunch: A physics-based sonic dining table. In Proceedings of the International Computer Music Conference (pp. 41-44). San Francisco: Computer Music Association.
16. Delle Monache, S., Polotti, P., Papetti, S., & Rocchesso, D. (2008). Sonically augmented found objects. In Proceedings of the Conference on New Interfaces for Musical Expression (pp. 154-157), Genova, Italy.
17. Delle Monache, S., Devallez, D., Drioli, C., Fontana, F., Papetti, S., Polotti, P., & Rocchesso, D. (2009). Sound design toolkit – Users’ guide. In Closing the loop of sound evaluation and design (Deliverable 2.3, pp. 1-62). Verona, Italy: University of Verona. Retrieved November 12, 2009, from http://closed.ircam.fr/uploads/media/CLOSED_D2.3.pdf
18. Demoucron, M. (2008). On the control of virtual violins – physical modelling and control of bowed string instruments. Unpublished doctoral dissertation, Kungliga Tekniska högskolan, Stockholm, Sweden. Retrieved November 12, 2009, from http://urn.kb.se/resolve?urn=urn:nbn:se:kth:diva-9424
19. Dourish, P. (2001). Where the action is: The foundations of embodied interaction. Cambridge, MA: MIT Press.
20. Essl, G., & O’Modhrain, S. (2006). An enactive approach to the design of new tangible musical instruments. Organised Sound, 11(3), 285-296.
21. Findeli, A. (2001). Rethinking design education for the 21st century: Theoretical, methodological, and ethical discussion. Design Issues, 17(1), 5-17.
22. Franinovic, K. (2008). Basic interaction design for sonic artefacts in everyday contexts. In Focused - Current design research projects and methods (pp. 95-112). Bern, Switzerland: Swiss Design Network.
23. Franinovic, K. (2009). Toward basic interaction design. Elisava Temes de Disseny Journal. Retrieved November 12, 2009, from http://tdd.elisava.net/coleccion/25/franinovic-en
24. Gaver, W. W., Smith, R. B., & O’Shea, T. (1991). Effective sounds in complex systems: The arkola simulation. In Proceedings of the SIGCHI Conference on Human factors in Computing Systems (pp. 85-90). New York: ACM.
25. Hallnäs, L., & Redström, J. (2002a). Abstract information appliances: methodological exercises in conceptual design of computational things. In Proceedings of the 4th Conference on Designing Interactive Systems (pp. 105-116). New York: ACM.
26. Hallnäs, L., & Redström, J. (2002b). From use to presence: On the expressions and aesthetics of everyday computational things. ACM Transactions on Computer-Human Interaction, 9(2), 106-124.
27. Handel, S. (2006). Perceptual coherence – Hearing and seeing. New York: Oxford University Press.
28. Hodgkinson, T. (1996). Siberian shamanism and improvised music. Contemporary Music Review, 14(2), 59-66.
29. Ishii, H., & Ullmer, B. (1997). Tangible bits: Towards seamless interfaces between people, bits and atoms. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 234-241). New York: ACM.
30. Juslin, P., Friberg, A., & Bresin, R. (2002). Toward a computational model of expression in performance: The GERM model. Musicae Scientiae, Special Issue 2001/2002, 63-122.
31. Jylhä, A., & Erkut, C. (2009). A hand clap interface for sonic interaction with the computer. In Proceedings of the 27th International Conference on Human Factors in Computing Systems (pp. 3175-3180). New York: ACM.
32. Klöckl, K. (2008). Changing connections. The role of connections in products between traditional and new technologies. Unpublished doctoral dissertation, IUAV University of Venice, Venice, Italy. Retrieved November 12, 2009, from http://rice.iuav.it/103/
33. Kranz, M., Schmidt, A., Maldonado, A., Rusu, R. B., Beetz, M., Hörnler, B., & Rigoll, G. (2007). Context-aware kitchen utilities. In Proceedings of the 1st International Conference on Tangible and Embedded Interaction (pp. 213-214). New York: ACM.
34. Lagarde, J., & Kelso, J. (2006). Binding of movement, sound and touch: Multimodal coordination dynamics. Experimental Brain Research, 173(4), 673-688.
35. Lantz, V., & Murray-Smith, R. (2004). Rhythmic interaction with a mobile device. In Proceedings of the 3rd Nordic Conference on Human-Computer Interaction (pp. 97-100). New York: ACM.
36. Lim, Y. K., Stolterman, E., Jung, H., & Donaldson, J. (2007). Interaction gestalt and the design of aesthetic interactions. In Proceedings of the 2007 Conference on Designing Pleasurable Products and Interfaces (pp. 239-254). New York: ACM.
37. Löwgren, J. (2007a). Inspirational patterns for embodied interaction. Journal of Knowledge, Technology and Policy, 20(3), 165-177.
38. Löwgren, J. (2007b). Pliability as an experiential quality: Exploring the aesthetics of interaction design. Artifact, 1(2), 85-95.
39. Löwgren, J., & Stolterman, E. (2004). Thoughtful interaction design: A design perspective on information technology. New York: MIT Press.
40. Ludden, G., & Schifferstein, H. (2007). Effects of visual–auditory incongruity on product expression and surprise. International Journal of Design, 1(3), 29-39.
41. Lupton, E., & Phillips, J. C. (2008). Graphic design: The new basics. New York: Princeton Architectural Press.
42. Michotte, A. (1963). The perception of causality. New York: Basic Books.
43. Norman, D. A. (2007). The design of future things. New York: Basic Books.
44. Petrini, K., Dahl, S., Rocchesso, D., Haakon Waadeland, C., Avanzini, F., Puce, A., & Pollick, F. E. (2009). Multisensory integration of drumming actions: Musical expertise affects perceived audiovisual asynchrony. Experimental Brain Research, 198(2-3), 339-352.
45. Polotti, P., Delle Monache, S., Papetti, S., & Rocchesso, D. (2008). Gamelunch: Forging a dining experience through sound. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Extended abstracts, pp. 2281-2286). New York: ACM.
46. Polotti, P., & Rocchesso, D. (Eds.). (2008). Sound to sense - Sense to sound: A state of the art in sound and music computing. Berlin: Logos.
47. Raskin, J. (2000). The human interface: New directions for designing interactive systems. Reading, MA: Addison-Wesley.
48. Rath, M., & Rocchesso, D. (2005). Continuous sonic feedback from a rolling ball. IEEE MultiMedia, 12(2), 60-69.
49. Riccò, D. (2006). Basic design sinestesico. Fondamenti teorici [Synesthetic basic design. Theoretical foundations]. In G. Anceschi, M. Botta, & M. A. Garito (Eds.), L’ambiente dell’apprendimento – Web design e processi cognitivi [Learning environment – Web design and cognitive processes] (pp. 97-110). Milano, Italy: McGraw Hill.
50. Rocchesso, D., & Serafin, S., (2009). Sonic interaction design [Editorial for the special issue on Sonic Interaction Design], International Journal of Human-Computer Studies, 67(11), 905-906.
51. Schaal, S., Ijspeert, A., & Billard, A. (2003). Computational approaches to motor learning by imitation. Philosophical Transaction B, 358(1431), 537-547.
52. Schafer, M. (1994). Soundscape - Our sonic environment and the tuning of the world. Rochester, VT: Destiny Books.
53. Schelleng, J. C. (1973). The bowed string and the player. The Journal of the Acoustical Society of America, 53(1), 26-41.
54. Sennett, P. (2008). The craftsman. New Haven, CT: Yale University Press.
55. Serafin, S. (2004). The sound of friction: Real-time models, playability and musical applications. Unpublished doctoral dissertation, Stanford University, Palo Alto, CA. Retrieved November 12, 2009, from http://imi.aau.dk/~sts/serafinthesis.pdf
56. Shneiderman, B. (1983). Direct manipulation: A step beyond programming languages. IEEE Computer, 16(8), 57-69.
57. Sinico, M. (2008). Demonstration in experimental phenomenology. Theory & Psychology, 18(6), 853-863.
58. Smits-Engelsman, B., Van Galen, G., & Duysens, J. (2002). The breakdown of Fitts’ law in rapid, reciprocal aiming movements. Experimental Brain Research, 145(2), 222-230.
59. Snow, K. (n.d.). From keys to the open gate. Retrieved November 12, 2009, from http://www.snowlight.com/keys.html
60. Spence, C., & Zampini, M. (2006). Auditory contributions to multisensory product perception. Acta Acustica united with Acustica, 92(17), 1009-1025.
61. Svanaes, D. (2000). Understanding interactivity: Steps to a phenomenology of human-computer interaction. Unpublished doctoral dissertation, Norwegian University of Science and Technology, Trondheim, Norway. Retrieved November 12, 2009 from http://www.idi.ntnu.no/~dags/interactivity.pdf
62. Sweet Maria’s home coffee roasting (n.d.). Retrieved November 12, 2009, from http://www.sweetmarias.com/
63. Vicario, G. B. (1993). On experimental phenomenology. In S. C. Masin (Ed.), Foundations of perceptual theory (pp. 197-219). Amsterdam: Elsevier Science.
64. Visell, Y., Franinovic, K., & Hug, D. (2007). Sound product design research: Case studies, participatory design, scenarios, and product concepts. In Closing the loop of sound evaluation and design (Deliverable 3.1, pp.14-20). Zürich, Switzerland: Hochschule für Gestaltung und Kunst. Retrieved November 12, 2009, from http://closed.ircam.fr/uploads/media/closed_deliverable_3.1_public.pdf
65. Warren, W. H., & Verbrugge, R. R. (1984). Auditory perception of breaking and bouncing events: A case study in ecological acoustics. Journal of Experimental Psychology, 10(5), 704-712.

		International Journal of Design is a peer-reviewed, open-access journal. All journal contents are allowed to be shared and adapted in accordance under the Creative Commons Attribution 4.0 International (CC BY 4.0) License.