Download the pdf
Being interviewed together with Caroline Hummels at TEI 13 about our panel discussion
This investigation explores relations between 1) a theory of human cognition, called Embodied Cognition, 2) the design of interactive systems and 3) the practice of ‘creative group meetings’ (of which the so-called ‘brainstorm’ is perhaps the best-known example). The investigation is one of Research-through-Design (Overbeeke et al., 2006). This means that, together with students and external stakeholders, I designed two interactive prototypes. Both systems contain a ‘mix’ of both physical and digital forms. Both are designed to be tools in creative meeting sessions, or brainstorms. The tools are meant to form a natural, element in the physical meeting space. The function of these devices is to support the formation of shared insight: that is, the tools should support the process by which participants together, during the activity, get a better grip on the design challenge that they are faced with. Over a series of iterations I reflected on the design process and outcome, and investigated how users interacted with the prototypes.
In creative meetings, participants do not always have a clear understanding of their creative challenge right from the start. Part of the problem is that each participant may understand the challenge, as it is initially introduced by the problem-owner, differently. In general, many creative challenges are complex, ill-defined problems to begin with (aka ‘wicked problems’). Especially when multiple stakeholders are involved, each with their own interpretation of what the challenge is ‘really’ all about, a better insight into ‘what the problem really is’ needs some time to evolve. In practice, shared insight into the creative challenge co-evolves with the team’s practical activities towards addressing the challenge. In current practice, people use all kinds of physical tools to develop this shared insight, such as: sticky-notes, whiteboard and markers, sketching paper, prototyping materials, photographs brought from home, and so on. The question is whether we can augment these physical tools using interactive technology in a meaningful way, without disrupting the creative flow, or the natural, improvised and flexible ways by which people currently interact with each other and their physical artifacts.
A theory that might be helpful for designing such integrated technological tools, as part of the creative space, might be Embodied Cognition. This is a theory about the basic way in which people are able to think and act. It claims that what we call cognition is fundamentally dependent on our ongoing embodied activity. Bluntly speaking: no active body, no thoughts. The theory argues against the classic idea that thinking is something that happens purely ‘internal’ to us. It rejects the idea that the mind is the ‘software’ running on the ‘hardware’ of the brain. Thinking, instead, emerges in action, out of continuous embodied interactions between brain, our body and the way our body is ‘situated’ in a physical- and social context. According to Embodied Cognition, cognition is best seen as a dynamic coupling (Clark, 1997; Dourish, 2001), or a process of coordination (Suchman, 2007; Clancey, 1997), or, as phenomenologists call it, as getting ‘grip’, through skilled action (Dreyfus, 2002; Merleau-Ponty, 1963). Artifacts, such as the sticky-note in a brainstorm session, play an important part in the embodied cognitive process. As Kirsh (2010) states, physical artifacts are ‘things to think with’. They help people to ‘offload’ thinking to the environment, to coordinate their own activities with those of others, and to create and hold active online couplings in the continuous feedback loop between action and perception.
One reason Embodied Cognition may be useful is because the field of interactive systems design shows a growing trend towards trying to integrate physical form and digital process. Looking at it from the perspective of Industrial Design, this entails adding interactive behaviors to physical products, using sensors, actuators, and the like. From the perspective of computer science, it means creating so-called ‘tangible’ interfaces, where various physical objects can be used to control digital information, a follow-up on the familiar ‘graphical’ interface. A related trend is that of ‘contextual’ interfaces that depend crucially on available cues in the local environment. These developments go under such headings as ubiquitous computing, tangible interaction, wearable computing, augmented reality, and so on. The bulk of this design work has yet to find its way to the commercial markets, but the prospect is that in the near future this will certainly happen. The current popularity of interacting via mobile devices, such as the smartphone or tablet, with Apps making use of GPS location, the accelerometer, and so on, signals a development that moves away from the classical ‘desktop interface’. One may say it moves interaction with digital processes ‘back into the real world’, mixing it seamlessly with physical objects, environments, and social contexts.
Through designing and researching two concrete interactive prototypes I explored the following research question:
RQ 1. How may we design interactive systems in support of embodied cognition?
The main objective of my investigation has been to reframe our conceptualization of interactive systems design, such that designers may start to think in new ways about what it is they are trying to do, based on Embodied theory. At the same time, the practical attempt of the theory to design also brings insight the theory itself, which defines my second research question:
RQ 2. How does (the practical attempt at) designing interactive systems supporting shared insight in creative meetings, inform the theory of embodied cognition?
In a series of design iterations I undertook the following activities: 1) detailed observational studies of naturally occurring human practices (either with our without our prototypes); 2) participatory workshops involving potential users from several creative companies and organizations, executed at the site-of-practice, including situated interviews and ‘acting-out’ design concepts; 3) design explorations and prototyping, and reflecting on these in reference to the theoretical framework; 4) detailed observations of human activity in response to an experimental manipulation, using two variations on a prototype as conditions, and 5) general theoretical reflections. Together, these research activities enabled me to answer my research questions.
NOOT, in its final form, consists of a system of tangible clips with which one can create time-markings in a continuous audio-recording of the creative session. The tangible clips can be placed everywhere in the physical space, e.g. on the wall, on the table, on sketches or annotations, or on a mock-up or prototype. With a physical horn one can activate audio-playback that allows one to listen to the part of the conversation that was going on when the clip was first activated. In this way NOOT provides small segments of ‘history’ of the conversation, attached to meaningful physical items in the space, which can be fed back into the current conversation. In the final reflection I offered that NOOT couples individual moments of reflection-in-action to the overall group conversation, thereby supporting the formation of shared insight.
FLOOR-IT enables people to create digital photographs of any of the sketches or written texts (or other visual elements) created during the session. The series of personal snap-shots form a ‘trace’, reflecting one’s evolving line-of-thought. Each person’s personal ‘trace’ is physically projected as a circle of digital images around the body, on the floor. On that floor, which is quite large (six projectors were used to create the canvas), small groups of people engage in a creative conversation, while their traces are publicly visible as projected around their bodies. The traces form a conversational ‘scaffold’, to which people can point and refer during the talk. Furthermore, by using foot gestures, images may copied from one trace to the other, and they may be combined to form new clusters, new traces, which stay fixed on the floor. The emerging, overall trace on the floor represents the growing ‘shared insight’ of the team as a whole which continues to support the ongoing interactions of the creative team. In a user study, comparing FLOOR-IT with a variation that projected the pictures on a shared wall, it was discovered that such traces function to help people position themselves socially in relation to others. Referring to ones personal trace during ongoing talk helps not so much to share factual information rather than that it serves to present yourself as a valuable partner in the activity, and to invite others to do so as well.
Embodied Cognition is a broad field of inquiry, with roots in quite disparate research traditions. Based on my reflections on the design iterations, I was able to discern four variations of the theory that each have their own particular consequences for design. I call these the 1) distributed representation and computation perspective, 2) the socially situated practice perspective and 3) the sensorimotor & enactment perspective.
The distributed representation and computation perspective is perhaps most easily understood by those familiar with computational principles and it has proven to be a useful and relevant set of principles for interaction designers. Yet it actually hinders interaction designers in getting to the heart of the notion of embodiment. Instead, based on my design investigations I offer that the prime ingredients needed for understanding how my prototypes support shared insight are 1) the sensorimotor aspect of cognition (how insight emerges from real-time coupling of perception and action) and 2) the social situatedness of cognition (how cognition is socially coordinated between people). Moreover, sensorimotor coupling and social situatedness are strongly integrated in one unified embodied activity (Goodwin, 2000). In particular, the studies revealed how people would create expressive traces in the environment. Expressive traces, e.g. a physical sticky-note, a NOOT clip, or a trace in FLOOR-IT, are both the outcome of people’s earlier actions, as well guiding further action. That is, traces become part of people’s sensorimotor couplings. At the same time, expressive traces are also social artifacts, created in and for a social context, publicly available and socially accountable. They function to coordinate people’s social positioning in the physical space. Expressive traces form the linking pins between social interaction and sensorimotor coupling, thereby supporting the emergence of shared insight.
I offer a number of pitfalls and opportunities for designers that want to ground design in embodied cognition theory. I start with the claim that the classic interface concepts, which rely on information processing metaphors, are best explained with a ‘Distributed Representation and Computation’ version of embodied cognition. This goes also for many of the so-called ‘tangible media systems’, where tangible objects essentially ‘encode’ digital information in physical form. However useful, they stand in the way of designing for a more fundamental form of Embodied Cognition, as they too easily draw us back into ‘Cartesian thought’.
Based on Socially Situated Practices and Sensorimotor Coupling and Enactment, I propose a more fundamental form of Embodied Cognition Design. Embodied Cognition Design brings forth interactive systems that transform our ways of perceiving, our possibilities for acting, our ways of interacting socially with others, and it helps us to create endurable ‘expressive traces’ in the environment. In any concrete product proposal, all of these aspects will be part of the unified experience of the user. Through Embodied Cognition Design we may search for completely new roles for digital computing technology in human practices. This means going beyond the classical, Cartesian functions of storing, processing and presenting representational data. One consequence of this vision is that the ‘function’ of an artifact can no longer be predefined before one starts designing the ‘interface’: in Embodied Cognition Design, concrete interactions between the user and the system bring forth, or ‘enact’ the meaning that the system has for the user. This means one has to design the interactive behavior and ‘what the system is for’, both at the same time, with no a priori distinction between the digital- and the physical aspect, nor even the embedding context. One may for example design in iterative fashion, building series of functioning prototypes, which can be tried out such as to stay in close contact with the user and his context of practice throughout the entire project.
TRACES 1 (aka Floor-it)
Van Dijk, J. and Van der Lugt, R. (2013) Scaffolds for shared understanding. AI EDAM, Special Issue on Design Communication, 27, 107–117.
Van Dijk, J. (2009) Cognition is not what it used to be. Reconsidering usability from an embodied embedded cognition perspective. Human Technology, Volume 5, Number 1, May 2009, pp. 29-46
Van Dijk, J. Kerkhofs, R. van Rooij, I., and Haselager, W.F.G. (2008) Can there be such a thing as embodied embedded cognitive neuroscience? Theory & Psychology, Vol 18(3): 297-316
Haselager, W.F.G., van Dijk, J., and van Rooij, I. (2008) A lazy brain? Embodied Embedded Cognition and Cognitive Neuroscience. In: Paco Calvo and Antoni Gomila (Eds): Handbook of Cognitive Science. An embodied approach.
Van Dijk, J., Van der Lugt, R., & Hummels, C.C.M. (2014) Beyond Distributed Representation: Embodied Cognition Design Supporting Socio-Sensorimotor Couplings. Proceedings of TEI’14, x-x, München. New York: ACM.
Van Dijk, J., Mousette, C., Kuenen, S. & Hummels, C.C.M. (2013) Radical clashes: what Tangible interaction is made of. Proceedings of TEI13, feb 2013, Barcelona.
Van Dijk, J. Roest, J. Van der, Lugt, R. & Overbeeke, C.J. (2011) NOOT: A tool for sharing moments of reflection during creative meetings. C&C’11, November 3–6, 2011, Atlanta, Georgia, USA.
Van Dijk, J. & Vos, G.W.(2011) Traces in Creative Spaces. C&C’11, November 3–6, 2011, Atlanta, Georgia, USA.
Van Dijk, J. Van der Lugt, R. & Overbeeke, C.J. (2011) Marking the moment: Coupling NOOT to the situated practice of creative sessions. Work-in-Progress Workshop at TEI’11, Portugal.
Van Dijk, J. & Brouwer, C.E. (2011) Making sense of brainstorms: some NOOTs to reflect on. Participatory Innovation Conference (PINC11), Sonderborg, Danmark.
Brouwer, C.E. & Van Dijk, J. (2011) Brainstorming: Talk and the representation of ideas and insights Participatory Innovation Conference (PINC11), Sonderborg, Danmark.
Van Dijk, J. Van der Roest, J. & Messager, E. (2011) NOOT: Tangible tags supporting reflection in brainstorms. Demo Presented at CHI-Sparks, june, Arnhem, Netherlands
Vos, G.W. & Van Dijk, J. (2011) Traces in Creative Spaces. Demo Presented at CHI-Sparks, june, Arnhem, Netherlands
Van Dijk, J., van der Lugt, R. & Overbeeke, C.J. (2009) Let’s take this conversation outside: Supporting embodied embedded memory. in: Conf. Proceedings of DPPI’09, October 13-16, Compiegne, France.
Sluijs, J.M. & Dijk, J. van (2009) Noot: a tangible interaction tool supporting memory in creative practices. Proc of CHI-NL 2009, Leiden, Netherlands. June, 11, 113-116.
Van der Linden, M., Van Dijk, J., & van Turennout, M. (2003). Context-dependent activity in auditory cortex during speech perception. Poster presented at the 9th Annual Meeting of The Organization for Human Brain Mapping, New York City, USA.
Van Dijk J. & Bleeker, G.J. (2001) Bookreview: A universe of consciousness. How matter becomes imagination. J. of Cogn Sys Research, 2, 241-244.
Workshops (invited & organized)
Van Dijk, J., Mousette, C., Kuenen, S. & Hummels, C.C.M. (2013) Radical clashes: what Tangible interaction is made of. Panel discussion at TEI’13. youtube link
Van Dijk, J. & Frens, J.W. Being there, doing it: the challenge of embodied cognition for design. Workshop at Creativity and Cognition ‘11, November 3–6, 2011, Atlanta, Georgia, USA.
Van Dijk, J. (2009) “Interaction Design: Exploring the role of the body and the environment” Workshop at International ICT days, KAHO Gent, November 12.
Van Dijk J. (2009) “The other body” Workshop at Co-Design Live event, HU, December 11. www.hu-kennisconferentie.nl
Van Dijk, J. (2011) Mijn intieme relatie met het briefje op de koelkast. Bijdrage voor Weblog Intieme Technologie van Ratheneau Instituut. http://intiemetechnologie.wordpress.com
Van Dijk, J. (2008) De ongrijpbare eindgebruiker. MORF, #8, (juni) Tijdschrift voor vormgeving (popular).
Van Dijk, J. (2005) Verschillende artikelen op kennislink over cognitie en de hersenen.
Reviewer for conferences
Tangible, Embodied and Embedded Interaction 2014 (Associate Chair)
Tangible, Embodied and Embedded Interaction 2013.
Tangible, Embodied and Embedded Interaction 2012.
Creativity And Cognition ‘11
Participatory Innovation ‘11
2013 Nominated for best paper award TEI’13 (tba february 2014) Link
2011 Honorable mention, Emma Candy award Best Contr. to Creative Communication (NOOT: a tool for sharing moments of reflection during creative meetings, CC’11, Atlanta). - Link
2000. First prize annual essay contest University Nijmegen. Nijmegen University Press. Van Dijk, J. (2000) Er waart een spook door de wetenschap van de geest. Brief aan mijn vader.
Invited talks (selection)
Dijk, J. (2013) Embodied Cognition for Interactive Systems Design. Invited talk at dept. of Communication, University of Southern Denmark, November 2013, Kolding, Denmark. youtube link
Dijk, J. (2013) Embodied Cognition Design. Invited talk at SPIRE, University of Southern Denmark, May 15, Sønderborg, Denmark.
Dijk, J. (2012) Tracing embodied interactions. Presentation at Design Hot Pot. Waag Society. June 25, Amsterdam.
Dijk, J. (2012) Being there: Doing it: Design for the embodiment of cognition. Guest lecture at Delft University. Dept of Architecture, PROTOSPACE lab, 23 april, Delft, Netherlands.
Dijk, J. (2012) Being There: Doing it: Design for Embodied Cognition. Guest Lecture at University of Utrecht, dept of Computer Science, 23 feb, Utrecht, Netherlands.
Van Dijk, J. (2010) Creative Learning by Augmented Scaffolding Systems. Talk for jaarvergadering KIVINIRIA, Hogeschool Utrecht, January, 2010.
Van Dijk, J. & Haselager, W.F.G. (2010) Ethical and philosophical considerations of AmbI. Discussion lead at Symposium Ambient Intelligence, Artificial Intelligence department, may 28, Nijmegen.
Van Dijk, J. (2009) “At the table with your Body in a Situation: Reporting on a Pitching Innovation Game”. Talk at MTNL2009, December 14 Link
Haselager, W.F.G. & van Dijk, J. (2009) “It’s the dance, stupid!” Workshop on Embodied Embedded Cognition at 3TU Winterschool Human Technology Interaction', Drenthe, 28 januari.
Van Dijk, J. (2009) “Everyday life, cognition, and participatory design.” Talk at the conference dinner of Flirting with the Future SIDR conference, Eindhoven University of Technology, Eindhoven, 16 April.
Blankestijn, D. & Van Dijk, J. “Human Technology.” Talk at European Week 2009, Eindhoven University of Technology, Eindhoven, May 25.
Van Dijk, J. (2008) Talk at Beleving van de beleving @ hogeschool utrecht 20 nov 2008 Material Design/ tijdschrift Product http://www.productmagazine.nl/nieuws/artikel/609_De_beleving_van_beleving
(The web pages may show error messages, which can be ignored, and for which my sincere apologies; the software will be updated ASAP).