Contents

The Augmented Subject

Publication data

Introduction

Augmented Reality

The X-Subject

The Interface

References

Zoltán DRAGON

University of Szeged, Hungary

The Augmented Subject

Technological Interfaces of Subjectivity and Geography

This work is licensed under Creative Commons 3.0

You are free to share - to copy, to distribute and transmit the work under the following conditions:

- You must attribute the work in the manner specified by the author or licensor (but not in any way that suggests that they endorse you or your use of the work).

- You may not use this work for commercial purposes.

- You may not alter, transform, or build upon this work.

In William Gibson's Spook Country, Hollis Henry, the protagonist of Gibson’s latest trilogy of novels, is invited to a spooky installation in the very first chapter: with a virtual reality headset device she can revisit the death of the actor, River Phoenix, on the spot (Gibson 2008, 9). The on-location virtual artifact is part of the trend once known as "locative art" - today it is not really an art form, we simply use a more general term, augmented reality instead.

Today we are – more and more without the clumsy goggles and devices that accompany them – like Hollis: not sure of what we see, how we see, and of the flimsy boundary between virtual and real, online and offline; and a bit like the body that is there and not there at the very same time, mapped and unmapped in an uncanny cartographic way.

I will talk about the specific problem the emplacement and the positioning of the subject presents in today’s increasingly augmented everyday realities. My particular interest lies in the technological aspect of the augmentation of space, of geography that results in what Nanna Verhoeff calls “performative cartography” that questions not only our modern understanding of – mostly urban – spaces, but also reveals a problematic aspect of subjectivity. If the space that the subject finds itself in is augmented, it is inevitable to pose the question what this augmentation brings for the subject. I will identify a common element, a digital-material glue that helps us understand how the subject is mapped onto the very cartography it pretends to master: the interface. While theorists have recently paid much attention to both the question of the subject and of space in the digital world separately, I would like to refocus the discussion on what connects the two, and claim that it is the technological aspect of this connection that defines the augmentation of space and subject alike.

Augmented Reality

In general terms, augmented reality can be thought of as a digital layer on our reality comprised of geo-registered data retrieved and visualized through the screen of a mobile interface. As such, augmented reality, through contemporary mobile screen technology, not only takes place, but it also makes space (Verhoeff 2012, 134): the screen of the interface reconfigures the geographical space around us and creates a data-filled vision of it in front of us, thus making us capable of not only understanding our present situation (the deictic “You are here” type of positioning of traditional cartographic logic), but also of navigating this space. In other words, augmented reality fills the space with visualized data connected to geo-registered points, thus augmenting (but not completely overlaying or overtaking) our perception of reality.

Let me illustrate my point with a somewhat unique application that might even raise questions about the medial definition of the cinema, as well as show the basic principles on which augmented reality operates. The Augmented Reality Cinema application takes geo-registered data of well-known film sequences and allows the user to activate the digital layer in their reality on-location. This way it is not the subject who enters the diegetic realm, but the diegesis enters the subject’s reality, inviting the subject to the location of the particular shots, outside the institution of the cinema (or rather, the user-subject navigates the filmic sequence to its location).

http://youtu.be/R6c1STmvNJc

I could have chosen other applications as well, of course, as this is an evolving technology, but I think the diegetic space of the filmic sequence can more spectacularly integrate with the representation of our reality than any other data-driven apps.

What is pivotal in augmented reality is an upgraded cartography. Traditionally, cartography is concerned with making and reading maps, in two-dimensional meaning-making attempts, separating the observer and the observed (Verhoeff 2012, 139). Today, however, with the introduction of interactive, digital mapping tools featured by mobile and essentially networked media technologies, the function and the working mechanism of mapping have fundamentally changed. New mapping practices blur the boundary between observer and observed, and put the user in the role of navigator with the use of Global Positioning System (GPS), Geographic Information Systems (GIS), geo-browsers (like Google Earth), map-hacking, geo-tagging, and allowing the creation of mash-ups. According to Verhoeff, this not only leads to new representations of space, but also provides “tools to modify maps” and “to fill maps with different kinds of knowledge” (139).

It is thus a new way of putting oneself onto the map: it does not depend on the cartographer, but rather on how connected and networked one is, since contemporary digital cartography allows the user to manipulate and navigate the datalayers comprising the ever evolving map, thus allows for acts of creating and recreating space in a process of re-locating oneself.

According to Verhoeff, this means that we have moved from representational cartography to what she coins as performative cartography. Performative cartography is “a procedural form of simultaneous making and reading space by exploring a hybrid space of atoms and bits, both the physical and the virtual, through interaction between on- and off-screen navigable space” (137). This practice brings together three domains central to the visual regime of navigation: screen, space, and mobility. Performative cartography emerges “during movement, as a particular form of interactive navigation: fixed maps do not dictate the itinerary, but rather maps and views evolve and emerge along the way” (145).

Verhoeff compares the process of performative cartography to bricolage, that is, a type of “creation by recombining ready-made bits and pieces” in the way Claude Lévi-Strauss or Gerard Gennette talk about it (146), but I think this is not the case here. Performative cartography and digital mash-ups are examples of what Lev Manovich calls deep remix, which is a truly hybrid media format (Manovich 2008). The difference between a bricolage or remix and the deep remix is that while in a remix the combined elements preserve their logic of representation (an image is an image even if it is transposed to text or music), in a deep remix the logic of representation is brought to a common platform: that of the programmable language of software.

Let me refer to a mash-up example to illustrate the hybridity of new media that is also at play in performative cartography. Canadian indie band Arcade Fire teamed up with Chris Milk to create an intriguing Google Experiment project called The Wilderness Downtown, which takes the band’s track, We Used to Wait, and turns it into a hybrid mash-up of video footage, geo-imagery and Google’s map application, most notably its street-view option. The user navigates to the website the project is located at and enters the name of his or her home town – the algorithm then searches for the coordinates and assembles a personalized show. In a curious act of digital flânerie, the user gets multiple points of view shots of video and street-view imagery stitched together, adjusted to the movement of the faceless protagonist of the clip, using several, dynamically changing browser windows as a multiplication of screens within the screen.

http://thewildernessdowntown.com

A nostalgic gesture indeed, the clip in fact utilizes the spatial memory of the user-subject, connecting the psychological factors with an algorithmic rendition of space. It dynamically, in a procedural manner, repositions the subject through the interface of the mash-up to recreate the once visited spatial coordinates that allow the subject not only the recollection of psychic, but also of bodily engagement within the given spatio-temporal setting. The subject’s spatial memory thus comes to be aligned (and overseen) by digital and performative cartography – in André Nusselder’s reading of Lacan, a fantasy screen navigated through the interface.

The X-Subject

Steve Pile and Nigel Thrift define subjectivity as “rooted in the spatial home of the body … united and orchestrated to a greater or lesser extent by narrative, and as registered through a whole series of senses” (Pile and Thrift 1996, 11). Locating the “spatial home of the body” and registering through senses recalls the Freudian definition of the ego, as “first and foremost a bodily ego,” that is “not merely a surface entity, but is itself the projection of a surface” (Freud 1961, 27). Jacques Lacan takes his cue from the projective quality of the surface and suggests that the three-dimensional space in which the subject resides and functions is not a given, we are not born into space: it is the result of identification in the field of the Other that prepares the subject to comprehend and navigate space (Adams 2003, 148). Lacan’s idea is instructive here: from his famous mirror stage idea on, he emphasizes the presence of a screen, a projective surface on which various scenarios of identification can play out.

André Nusselder, in Interface Fantasy: A Lacanian Cyborg Ontology, argues that the subject in the age of information technology lives on the screen (Nusselder 2009, 2), which echoes among many critical views, Nicholas Mirzoeff’s claim that “modern life takes place onscreen” (Mirzoeff 1999, 1), but he enlarges on this statement in a different way. For Nusselder, the computer screen is essentially a psychological space, the screen of fantasy, which distorts “the ideal relationship” of the user-subject and the sign (Nusselder 2009, 3). “Therefore,” he says, “the relation of user and sign is … cybernetic” (3).

While I agree with the claim that the introduction and use of digital computing devices distort the way the user-subject and the signs of representation engage, I think the subject does not “live on the screen.” As I will discuss it later, the cybernetic relation of user-subject and sign can work only if we stay close to the Lacanian graph of visual regime: the screen needs to be seen as a digital interface through which the relation of the user and the sign can play out. In other words, life does not take place on the screen but rather through it. It is an important distinction, as I do not think the subject has ultimately become completely virtual – though I will keep arguing for it being ultimately augmented.

To think about subjectivity in the age of Web 2.0, we need to consider that the subject is co-present in blunt reality as much as it can be online. It means that the digital aspect of subjectivity needs to be taken into consideration as well in defining the subject of the 21st century. What it entails is an intricate process of tracing one’s avatar – not only the visually composed set of images (let alone the blue Na’vis of James Cameron’s groundbreaking film) that we use as funny (or not so funny) identifications of ourselves on social networking sites or personal websites, but also the invisible digital footprints that we inevitably leave behind when browsing the World Wide Web.

This online subjectivity layer is in fact more than what we may think it is: we know what sites we have visited, what videos we have just watched on YouTube or whom we befriended on Facebook – but the internet knows a lot more about us. When you log into Amazon.com, the algorithm will instantly recommend you a wide range of items: how did it know that I am interested in science fiction? Well, it tracks my online route, but it uses not only the data I leave behind when I navigate away from its pages, but collects the digital footprints I may take afterwards as well. It knows where I come from, it knows what I do there, and it knows exactly whither I leave. A lot of companies out there do nothing but trace my footsteps and compile a virtual map of my online travels. My online avatar thus is an ever growing dataset composed of my navigation, my browsing habits, which then “pictures me” so that I can be identified when entering the site of commerce.

To see who follows your invisible digital footprints, one handy tool to refer to is the Collision plugin for Mozilla’s Firefox browser: it gives you an interactive infographic representation of those sites that you connect with without actually clicking on their hyperlinks by accident or on purpose. To illustrate how intricate this graph can turn, see my 5-minutes long browsing on regular sites: I start out to read my emails, check my friends on Twitter and Facebook, navigate to a couple of Hungarian news portals, watch a YouTube video, and the like – and see how more bubbles pop up adjacent to the particular sites: sites I have never ever visited, sites that follow me nonetheless. When talking about avatars or online representations of subjectivity, this is what should be meant: the intentioned plus the unintentional pieces of navigation that one performs when browsing. (Note that as I am talking, more and more companies announce their DNT strategies, such as Twitter or even Microsoft with the introduction of its new Internet Explorer browser: they allow the user to opt out or even to block 3rd party sites and services tracking their online browsing.)

In line with Nusselder’s discussion of Lacanian psychoanalysis in the digital age, one could assert that the unintentioned part of the dataset that builds my avatar is basically the digital unconscious that symptomatically returns in the most unexpected corners of the World Wide Web. He suggests that “information technologies … seem to design or create a second, parallel world” (an idea that corresponds to the main hypothesis of virtual domains like Second Life), and we can communicate, interact with this world in a way that user-data relation becomes transparent (Nusselder 2009, 8), an idea that could easily be linked to what Jay David Bolter and Richard Grusin calls “immediacy” (Bolter and Grusin 2000).

Recently, Beth Coleman investigated how our perception of reality is changing by the minute and how our online persona is getting more and more emphasis in how we see ourselves. In Hello Avatar: Rise of the Networked Generation, she elucidates the difference between offline and online dimensions that is usually at the core of discussion of contemporary digital identity, and contests the use of such a binary opposition. Instead, she points out how the two have really become one inseparable experience and goes on to offer the term “x-reality” (can be read either as “ex-“ or “cross-reality”) that is the quasi-virtual realm intertwining the two domains of experience.

While Coleman improves on the subject-object relationship in the digital age compared to many of the contemporary considerations of the topic by adding the programming, technological layer into the equation at some point (even attributing some kind of disruptive aspect to technological agency), I think that by putting the ultimate emphasis on the agency of the user-subject in interaction, the immediacy of the interface becomes a ghostly non-presence. Moreover, I also question the disruptive aspect of technology in augmented reality – especially in arguments that aim at connecting the on- and offline domains. I agree with both Nusselder and Coleman that transparency for the interface is pivotal, but I consider it problematic to locate agency on either the side of representation or that of the subject of representation.

As I have outlined above, it is through the interface that space is made: it is the performative cartography that uses geo-registered data, user interaction and interface technology that creates spatial representation. However, it is only through the programmable interface that the user can locate and constitute itself as subject who then is able to partake of the process of creating space. It is as if the Lacanian big Other got manifested in the mobile screen technology of the interface: the user-subject identifies itself through the device that helps it locate itself – a gesture that also makes space available to navigate in.

The Interface

In his discussion of the position of the Lacanian Real in the cyperspace, Slavoj Žižek argues that "[w]hat defines the properly ‘human dimension’ is the presence of a screen, a frame, through which we communicate with the ‘suprasensible’ virtual universe to be found nowhere in reality" (Žižek 1997, 60). In this model he refers to the diagram Lacan uses to define the difference between the look and the gaze in The Four Fundamental Concept of Psycho-analysis, where the subject of representation is ultimately deprived of all of its imaginary agency. Interestingly, Lacan also denies this power of the gaze, i.e., the position of the big Other, thus challenging the thereto hierarchically organized relationship between the subject and object of representation (Lacan 1998, 67-78). The structure Lacan proposes and Žižek adapts to a cyberspace context is thus very similar to the situation I am describing.

The only difference that Žižek does not address is indicative, however. Both Lacan and Žižek presuppose the structuring role of representation as such: for instance, a painting designates the position of its spectator (think of perspectival organization) that leaves him or her with the feeling of mastery over the visual field. However, the smallest detail or point of disorientation (such as the strange blot in Hans Holbein’s infamous The Ambassadors) annihilates this impression and reveals the “other side” of representation, the gaze, the collision with which is the flimsy structural pane of the screen. The screen is then a mediating instance necessary to keep the balance of the look and the gaze, a pivotal part of the structure of representation, of the visual regime in which the subject comes to be, nonetheless, itself part of the scheme of representation.

In augmented reality situations, however, the screen of the interface (today in quite a material format) pre-presents rather than behaves as part of the representation. I am referring to the technological necessity Verhoeff calls attention to: in digital cartography space is continuously recreated by the user-subject through the interface. While, as we have seen in the example of comparing the practice of performative cartography to bricolage, Verhoeff does not emphasize the role of the interface as much as I think it should be done, it is clear that this pre-presentational turn could not operate without the fundamental presence of the interface, of the mobile screen.

Moreover, we need to acknowledge the agency of the interface as well – as taken over from either the agency of the visual regime of representation or from the subject. William Uricchio has recently announced the algorithmic turn in visual culture (Uricchio 2011) in his analysis of Microsoft’s Photosynth app. Photosynth basically takes user-generated, geo-registered imagery, finds the matches among the myriad of photos and stitches them together in such a way that it does not become a simple 3D panoramic representation of space, but rather a fluid, navigable one. One example is the collaboration with CNN’s broadcasting Barack Obama’s inauguration, when thousands of images were uploaded by professionals and amateurs alike, while Photosynth’s algorithm was busy at work to create a dynamic, navigable experience of the Washingtonian space.

http://bit.ly/bR4EC

While I think the algorithmic turn came a bit earlier (think of Animoto’s service, for instance, that matches color, editing and even music to the “feeling” of the photos you upload and thus creates a video montage), this turn clearly shows how the agency is now ultimately on the side of the interface and technology. It even forces its own digital, non-linear logic on us: in Photosynth there is no fixed point of view, set frame of representation, as it works with multiple authorities, fluid structuring of imagery based solely on data.

The interface, however, is not innocent, and while strives to be transparent, it is not so successful. In Lev Manovich’s definition, the interface “acts as a code that carries cultural messages in a variety of media” (Manovich 2001, 64). The interface is the mode of displaying, receiving, and transmitting information, an electronic surface that allows users interactive engagement with any form of content: the screen, the software and hardware used, and the special logic they entail. However, “far from being a transparent window into the data inside a computer,” Manovich argues, “the interface brings with it strong messages of its own” (65). It means that the logic with which an interface operates is already coded in the transmission of data.

As the technological insertion between the user-subject and space, the interface enables performative cartography. As Verhoeff explains,

The basic principle of screen-based navigation is that we see how we move, while how we move enables vision. This mutually constructive, discursive relationship between seeing and moving is a new principle in real-time, digital cartography. It is the movement that establishes the map; reading space requires navigation, rather than the other way around. Digital maps make use of the logic of tagging, plotting, stitching as forms of interaction. (Verhoeff 2012, 153)

Tagging is labeling objects and locations with metadata (this is what the organizers of the conference did with the map we were given: they augmented our reality, re-mapping a Szeged for us). Plotting is the practice of tagging objects in space, and inserting tagged objects in the map (like live-streamed videos, or photographs you could see in Photosynth, plotting a whole inauguration ceremony in a special way). Finally, stitching is when these practices are integrated into a navigable whole (153).

Stitching is like a 4D model of filmic suture – not the standard suture procedure criticized so severely, but the short-circuited mode as discussed by Žižek who argues that when shot and counter-shot (the basic constituents of the suture operation on a formal level) appear within the same shot, subject and object of the look become co-present in the same virtual, impossible space, creating what he calls, the interface (Žižek 2001, 40). To refer back to the Augmented Reality Cinema app, I could say that what it presents us with is the suture operation translated into the interfaced, short-circuited mode of proto-filmic and post-production sequences in which scenario, curiously enough, the film sequence comes before, precedes, pre-presents the real space for the subject: stitches it into a new, augmented type of diegetic realm.

It is precisely this aspect, the tagging, plotting, and finally, the stitching of the user-subject within performative cartography that is missing from Verhoeff’s discussion: it is as if – contrary what she states about the interactive nature of such a process – the subject could distance itself from the scene it is navigating, exactly like the Cartesian subject of representational cartography. The point, however, is that by this process, the subject itself comes to be: tagged in space, plotted in the map, and stitched into its physical and augmented environment. It is the digital version of the Lacanian scenario of identification through the field of the Other as an establishing gesture of making space: here, however, the augmentative act of the interface takes the place of the field of the Other.

To refer back to Manovich’s definition of the interface, and to augment Verhoeff’s performative cartography with its help, one can detect a crucial change in how our perception of our spatial setting is structured: in Lacanese, the visual field (including the spatial coordinates) is born out of the split between the gaze and the look, with the intrusion of the screen. However, in augmented reality, and especially with regard to performative cartography, it is the screen that gives birth to both subject and space. This is, I would say, the techno-logic of digital representation in spatial terms.

The consequences of this cannot be overemphasized: the interface augments reality and the subject alike, thus taking the ideological power of representation with it, for it is no longer the map that acts as the locus of knowledge, of politics, of power, of instruction, i.e., it is no longer the representation that designates the place of object and subject, but the pre-presentational interface that – as Nusselder showed – distorts the relation between user and sign.

What all this amounts to finally, is the need to reconsider how we think about subjectivity in the age of digitally augmented reality. As our space comes to be augmented more and more by the day, inevitably, the subject – caught up in the flux of augmenting reality – cannot but be part of this interactive gesture, and augment itself by re-locating itself, renewing its self-recognition through the digital interface that instead of plotting our narratives in space, invite us to the procedure of continuous tagging and mapping. With the use of contemporary augmenting scenarios, therefore, the augmented subject is born through the interface.

References

- Bolter, Jay David and Richard Grusin. 2000. Remediation. Understanding New Media. Cambridge, MA: MIT Press.

- Coleman, Beth. 2012. Hello Avatar. Rise of the Networked Generation. Cambridge, MA: MIT Press. Kindle ebook.

- Gibson, William. 2008. Spook Country. New York: Berkley.

- Lacan, Jacques. 1998. The Four Fundamental Concepts of Psycho-analysis. London: Vintage.

- Manovich, Lev. 2001. The Language of New Media. Cambridge, MA: MIT Press.

- -----. 2008. Software Takes Command. Available: http://lab.softwarestudies.com/2008/11/softbook.html Kindle ebook.

- Mirzoeff, Nicholas. 1999. An Introduction to Visual Culture. London: Routledge.

- Nusselder, André. 2009. Interface Fantasy. A Lacanian Cyborg Ontology. Cambridge, MA: MIT Press.

- Uricchio, William. 2011. “The Algorithmic Turn: Photosynth, Augmented Reality and the Changing Implications of the Image,” Visual Studies, Vol. 26., No. 1., March 2011., 25-35.

- Verhoeff, Nanna. 2012. Mobile Screens. The Visual Regime of Navigation. Amsterdam: Amsterdam UP.

- Žižek, Slavoj. 1997. The Plague of Fantasies. London: Verso.

- -----. 2001. The Fright of Real Tears. Krzysztof Kieślowski Between Theory and Post-theory. London: BFI.