Blended Reality: superstructing reality, superstructing selves
8 likes4,727 views
The document outlines the work of the Institute for the Future, particularly its Technology Horizons program, which focuses on understanding technological innovations and their societal impacts over the next decade. It discusses the concept of 'blended reality' where physical and digital worlds are increasingly integrated, leading to transformed identities and social interactions through emerging technologies. Key trends highlighted include the democratization of media production, new communication modes via social media, and the evolving perception of individual identity in a connected world.
![BLENDED REALITY superstructing superstructing
I N T R O D U C T I O N
We are creating a new kind of reality, one in which physical and digital environments, media, and
interactions are woven together throughout our daily lives. In this world, the virtual and the physi-
cal are seamlessly integrated. Cyberspace is not a destination; rather, it is a layer tightly integrated
into the world around us.
Technology enables this transformation but, as is always the case, when we invent new technologies, they
in turn re-invent us. In the realm of blended reality, the technologies and tools that we are creating change
a fundamental part of our existence: the lenses through which we view and interact with the world. We are
beginning to see and feel the world through new sets of eyes and ears—things that were previously invisible
become visible, and we see the familiar in a new way.
Almost ten years ago, we wrote about the sensory transformation we’re about to undergo as technologies
move off the desktop and into the physical environment. We also pointed out that sensory transformations
inevitably lead to major social and cultural transformations because they shape the nature of what we experi-
ence and how we make sense of our surroundings [Cybernomads SR-829]. Blended reality is the manifesta-
tion of these changes. It is a type of sensory transformation that will change people’s lives, their senses of
self and others, and their views of the world around them. In this report we analyze key directions of this
metamorphosis.
In conducting this research, we held ethnographic interviews with people who are pioneering new ways of
living in the blended reality world (see Interview Subjects on page 3). Their perspectives and experiences are
signals of emerging needs, abilities, practices, and routines. To identify key themes and forecasts of the future
social landscape and its implications for organizations and individuals, we analyzed the data and brought
it together with our research on emerging technologies, identity, and sensory transformations in order to
provide a set of forecasts outlining directions of change in the next 10 years. We also conducted research on
the key foundational technologies that are driving the Blended Reality transformation (see Foundation Piece
on page 41).
The following are some key shifts we want to bring to your attention:
Emergence of new genres of personal performances as a key communication modality.
The Web 2.0 generation of social media tools—Twitter, YouTube, Facebook, etc.—will increasingly
change the nature of our interactions with each other, from communications to public perfor-
mances.
Increased ability to participate in each other’s life streams.
As we continuously stream digital bits of our lives—what we are doing, how we are feeling, what we
are thinking, who we are meeting with—we will increasingly come in and out of each other’s lives
on a scale previously unknown.
Growth of a mediated oral culture.
Spurred by the availability and adoption of cheap and portable video tools, the web will continue
its metamorphosis from a primarily text-based vehicle to a multimedia one. Audio and video will
rival the written word as the predominant communication forms for coming generations.
1](https://image.slidesharecdn.com/sr-1221blendedreality-100816120523-phpapp01/85/Blended-Reality-superstructing-reality-superstructing-selves-5-320.jpg)
![BLENDED REALITY superstructing superstructing
annaone and loki play SF0, a San Francisco-based collaborative Kati London is a game designer at Area/Code and co-creator of
production game whose tasks have largely been created by the Botanicalls, a sensor system that allows plants to Twitter their water-
game’s players. (www.sf0.org) ing needs. Kati is interested in sustainability art and games that help
people better relate to the non-human objects in the world. (http://
Stella Artois* is a 16-year-old girl living in Modesto, CA. Home- www.katilondon.com)
schooled her entire life by way of online educational courses and
materials, she has managed, with the help of her mom, her own Jeff McClure is an early adopter of Yelp. Jeff always consults Yelp
education. By age 14, she had completed her required high school when he’s about to check out a new place, whether in San Francisco
curriculum and moved onto online college courses. She has recently or elsewhere. He contributes reviews to the site, and has attended
begun attending classes on campus. Stella manages multiple online several meet-ups. In addition to Yelp, he’s used Flickr and LiveJournal
identities and is involved in a variety of online and non-overlapping for years and has made a number of real-world friends through both
communities, from art to music to her local, real-world community. sites. (http://troymccluresf.livejournal.com)
Gordon Bell is a Microsoft engineer, telepresence pioneer, and Jerry Michalski is an independent consultant under the company
human guinea pig for MyLifeBits, described in a 2007 New Yorker name Sociate. He uses brain software that allows him to capture,
article as, “a personal transaction processing database for every- organize, and retrieve every piece of information/knowledge he
thing.” (http://research.microsoft.com/en-us/um/people/gbell/) comes across. He’s been using his “brain” for 10 years now and it is a
remarkable repository of knowledge for him. He’s also a social media
Nick Bilton works in the R&D lab at the New York Times and is part power user. (http://www.sociate.com)
of a NYC-based hacker collective, NYC Resistor. Nick is a power user
of Botanicalls and is constantly thinking about ways that he can Laura O’Donnell* is a woman in her mid 20s who has managed
tweak, hack, or customize technology to make his interaction with Type 1 Diabetes since her teens using a variety of hardware, soft-
the non-human world a little smoother—or just more fun. (http:// ware, online and social tools. She uses SugarStats, an online start-up
nickbilton.com) service, to visualize this data and track patterns. She is also active on
LiveJournal and Twitter, where she seeks and gives advice, support
Jane DS is a lifecaster on Justin.TV who broadcasts her day-to-day and commiseration with other diabetics, and maintains social net-
life over the Internet. (http://www.justin.tv/jane_ds) works around crafts and art. She also maintains a personal website
Sean Goe* is a prominent blogger and frequent user of a variety of where “all the pieces of [her] on the internet” are collected with links.
social media. Sean Goe has over 20,000 people following his Twitter Skip Rizzo is a doctor at University of Southern California who is at
feed. He operates his own web hosting company and is a fixture in the forefront of virtual reality exposure therapy, a program that uses
the Bay Area tech community. audio, visual, smell, and vibration elements to simulate traumatic
Jess Hemerly is a Research Manager at IFTF and a long-time user of events for, among others, soldiers with PTSD. This is “triggering psy-
social media, from blogs and networking sites to message boards chophysiological experiences that produce high levels of emotional
and IRC. (http://jesshemerly.blogspot.com) engagement.” (http://vrpsych.ict.usc.edu/people/rizzo.html)
Ian Kizu-Blair is one of SF0’s co-creators and states that one of the Nigel Simpson is a developer at Sun Microsystems working on
creators’ motives was to use a digital platform and online interaction the Wonderland platform, a toolkit for developing virtual worlds.
to get people away from their screens and playing in the real world. He works remotely from an island off Seattle, and relies heavily
Ian and his co-creators, Sean Mahan and Sam Lavigne, have also on mpk20, a virtual office space developed by Sun Microsystems,
been commissioned to conceive and develop client-specific games. for interacting with his team members. The Sun office complex in
(http://sf0.org) Menlo Park has 19 buildings, and mpk20 symbolizes the 20th build-
ing. Nigel is in an interesting position as he is both a developer and
Lance Koenders is Senior Vice President of the Digital Home Group user of virtual worlds. He helped develop the mpk20 virtual world
at Intel Corporation. He started exploring Second Life for work and he uses the mpk20 virtual world to hold virtual meetings with
research. During his time as an active Second Life user, he bought his his team members, most of whom work on the east coast. (http://
own island and opened a music venue on it to attract people. Now research.sun.com/projects/mc/mpk20.html)
less active in Second Life, Lance has turned over management of his
virtual world property to a group of assistants. (http://www.linkedin. Keiko Takamura is a singer/songwriter who began exploring
com/in/lancekoenders) Second Life on a whim and has since become a popular Second Life
performer. She was featured on an episode of MTV’s documentary
Clarke Lethin is a recently retired Marine Colonel who has spent the show True Life. (http://www.keikotakamura.com)
last 10 years with One Marine Expeditionary Force of over 45,000
Marines and Sailors at Camp Pendleton, California. From 2005-2008 * Name has been changed at the interviewee’s request
Col. Lethin was Chief of Staff and was heavily involved in develop-
ing the Infantry Immersive Trainer, a state of the art mixed reality
combat environment designed to preview the combat experience
in Iraq.
3](https://image.slidesharecdn.com/sr-1221blendedreality-100816120523-phpapp01/85/Blended-Reality-superstructing-reality-superstructing-selves-7-320.jpg)
![1 | I AM THE STAR OF MY OWN SHOW
Lifecasting involves a continuous broadcast of one’s life using a variety of tools like laptop, video, mobile
phone cameras, and Internet sites such as Justin.TV, Yahoo!, and Qik, to open up their lives to the public. In this
sense, many don’t think of these as performances but rather as sharing their day-to-day
lives with the outside world.
Jane DS is a young woman who lifecasts on Justin.TV.4 Her channel has hundreds of
regular viewers who not only watch her life but also interact with each other and with
Jane through a chat box on her lifestreaming home page. They exchange comments
about what Jane is doing, provide feedback, and ask questions in real time. Jane leaves the
camera on all the time, and viewers continue to chat as Jane sleeps.
A screenshot of Jane
DS’ lifecasting page on
Justin TV. Even without cameras or video equipment, people are increasingly engaging in performances that use text
as the medium. Twitter streams, Facebook pages, and blogs are all forms of public performances meant for
public consumption. People use these social media tools to provide periodic or sometimes frequent updates
on different aspects of their lives—where they are, what they are reading, who they are meeting with, what
they are thinking, etc. Audiences achieve an ambient intimacy with the performances they follow by getting a
sense of the patterns in people’s lives.
Such performances are text-based, although even textual performances are becoming increasingly multime-
dia. vBlogging or mBlogging are emerging blog forms where bloggers use videos and photos to documents
parts of their lives. A Flickr stream containing a series of photos with titles, descriptions, tags, and comments
is another form of a public diary. Unlike the diaries and journals kept under mattresses or locked in drawers,
today’s digital diaries are highly public, unless a user chooses to control access to the information by limiting
it to a defined group of friends, colleagues, or acquaintances.
An important part of the transition to personal production is that individuals and members of social net-
works are now filling the resources and roles that were previously required to produce a show.
From Hired Editors to Self-Editing. Any produced show used to require expert editorial help. Today, how-
ever, we are becoming our own editors, and one of the most important editorial questions we are constantly
asking is whether to post or not to post something. This is becoming a major filter for how we think about
what we are doing, seeing, how we are feeling. We are constantly asking ourselves: is this something I can
post? Where should I post this? Is this a Twitter or a blog post? How should I post it in order to
“You don’t want to post too much make it interesting to my audience? Should this information stay or do I need to delete it?
From Agents to Mods. Who replaces all those producers, managers, stagehands, and army of
or too little. You don’t want to be
other paid staff in the era of personal production? For many, these roles are played by volun-
un-followed [on Twitter]. ” teers and people in the social networks of the “stars.” Jane DS and others we interviewed talked
— Jerry Michalski
about the importance of mods or moderators. She describes moderators as, “people that I
trusted … they were different from just regular viewers.” Popular blogs and online sites such
as BoingBoing rely on moderators to keep the show going by maintaining the site’s etiquette
by deleting incendiary comments or banning troublemakers.5 These mods receive no special training, nor are
they necessarily paid for their work. Many simply emerge out of the audience as active and positive contribu-
tors and then are entrusted with special moderator duties and privileges as a reward for their reputations.
6](https://image.slidesharecdn.com/sr-1221blendedreality-100816120523-phpapp01/85/Blended-Reality-superstructing-reality-superstructing-selves-10-320.jpg)
![BLENDED REALITY superstructing superstructing
From Hecklers to Griefers and Trolls. What moderators are often protecting the show (or site) from is a
new genre of evil-doers that often go by the name of griefers or trolls. They are to today’s digital perfor-
mance spaces what hecklers were in previous eras. These are people who raid your site, hack
it, damage it, or otherwise interfere with the performance through their disregard for a show
or site’s stated or understood guidelines. In Jane’s case, the griefers stole her lifestream and “They were coming back constantly,
posted it to a completely separate site. But these acts of sabotage are also performances in
standing up for me, and they
their own right, another way for attention-seekers to say, “Look at me!” Because their behav-
ior is so disruptive, they almost always get what they want. A popular saying has emerged showed me that they were people
regarding these troublemakers—“Don’t feed the trolls”—meaning if you don’t reward their that weren’t here to cause trouble.
attention-seeking behavior by getting angry, they’ll find another show to heckle.
They basically just clean up my
From Marketers and Promoters to Street Teams. Forget professional promoters. Today’s
promoters are the people in your network who promote your ideas or shows for you by simply chat room, protect me from people
referencing your work and posting a hyperlink. We also see organizations using the Internet to that are trying to cause trouble.
organize real-world promotional activities. Future Builders is a group of teenagers who want to
There’s a lot of times when I don’t
make the world a better place.6 They set out to accomplish this mission by organizing benefit
concerts for young people. The money raised goes to various causes—anti-hunger causes, pay attention to the chat room,
battered women’s shelters, HIV/AIDS organizations. You can sign up on their MySpace page to
especially when I’m just lifecasting.
become a member of their street team and distribute t-shirts, help promote performances, and
help advertise various causes.7 In exchange, street team members get special mention on the So they watch it for me. “
site and even the occasional complimentary ticket to a charity performance. — Jane DS
The idea that “life is theatre” is hardly new. We engage in ongoing performances in our everyday life just by
carrying out our routines. The people around us serve as audiences while our clothes and physical surround-
ings serve as the costumes and sets in which our personal stories and the stories of our identities unfold. We
explored this theme almost ten years ago at the IFTF research exchange on Identity [SR-663B]. At that time
we wrote: “We create our identity through reference points. We know who we are in reference to others …
Identity is a conception of self that we create based on various reference points in our life.”
All of the new audiences we create and that acquire a constant, ambient presence in our lives—our Facebook
friends, Twitter followers, blog readers, lifecast fans, and more—provide new reference points for creating
and shaping our identities, our senses of self. Not only are we creating multiple new reference points and
feedback loops through which we view ourselves, but we are gaining a new level of awareness about their
presence in our lives.
As we do so, we gain a sense of self that is highly complex, malleable, and playful. We gain a new level of
awareness of how to play to different audiences and how to shape our performance streams to make our-
selves interesting and entertaining. We have to be lively and engaging in order to maintain our audiences. At
the same time, we are able to develop multiple channels for playing out and experimenting with different Ten years ago, the
identities if we wish to do so. Outlook Project (now
the Ten-Year Forecast)
Psychologist Paul Bloom points out in his recent article in the November 2008 issue of The Atlantic that “many held a research exchange
researchers now believe, to varying degrees, that each of us is a community of competing selves, with the on personal identity
7](https://image.slidesharecdn.com/sr-1221blendedreality-100816120523-phpapp01/85/Blended-Reality-superstructing-reality-superstructing-selves-11-320.jpg)
![BLENDED REALITY superstructing superstructing
Virtual reality exposure therapy is a very promising, leading edge application of virtual reality environments
to treat the traumatic effects of things like military combat, abuse, or assault. Obviously we can never undo
what has already happened, but it turns out that we can alter our brain’s reaction to memories of the past and
dramatically reduce the ongoing effects of past trauma. For those with post-traumatic stress
disorder—a diagnosis now thought to apply far beyond military combatants—blended reality “Memory is a constructive process.
therapies are helping to slowly ease the psychological and physical impact of the memories as
[Our therapeutic process has
well as increase tolerance for conditions that would historically reactivate stress.
to] leave room for patients to
At the University of Southern California’s Institute for Creative Technologies, Dr. Richard “Skip”
Rizzo runs the Virtual Reality (VR) Assessment and Treatment program for Combat-Related insert their own memories and
PTSD. Dr. Rizzo uses heavily immersive applications including sounds, vibrations, and custom-
ized VR environments to recreate a soldier’s personal traumatic experiences. The immersion
experiences. ”
enhances patients’ imaginations, allowing them to retrieve memories that are often buried or — Skip Rizzo
intolerable without support. As they move through the program, patients report their anxiety
rating in terms of “subjective units of discomfort” and doctors compare it with their heart rate. In clinical
terms, Rizzo describes this as “graded and repeated imaginal reliving of the traumatic event … in a low-threat
context where the patient can begin to therapeutically process trauma-relevant emotions as well as de-con-
dition the learning cycle of the disorder via a habituation/extinction process.” Patients learn new responses
to feared stimuli so that the sound of a car door slamming down the street, for example, doesn’t retrigger the
trauma of urban warfare for an Iraqi vet.
Those who have the greatest need to improve their current lives are pioneering a new approach to experi-
ences, and showing the rest of us what might be possible. There are myriad situations to which we’ll apply pre-
view and redo. Organizations will use these tools to train new employees, while large corporate partners could
produce collaborative mass simulations (think Virtual Financial Crisis). Individuals will improve their confidence
in real-world social interactions by practicing the ones that make them nervous beforehand.
experior (verb) Latin
Furthermore, mixed physical and virtual environments allow people to move beyond their primary
psychophysiological responses and move to their higher mental functions more quickly. This results in better to test, put to the test,
performance in chaotic situations, less stress, and increased safety for everyone involved. These environments ascertain; to try, test,
will be used for business and public service.
experience, prove.
These new capabilities constitute a new approach to the nature of an experience. Conventionally thought of
as something that happens to us once, at a specific time and place, an experience is becoming more like an
ongoing experiment. Experiments are tests, they are repeatable, and they are reproducible. Preview and redo
bring us closer to being able to do all of that—with real life.
In the future we may see the emergence of a kind of “reality GPS,” where the experience of reality is a terrain to
be navigated based on previously drawn maps, real-time data, and post-experience revisions. In such a world
we could reduce risk, increase performance, and be more psychologically prepared for challenges. But with
everything so premeditated and rehearsed, would we also reduce spontaneity and randomness?
23](https://image.slidesharecdn.com/sr-1221blendedreality-100816120523-phpapp01/85/Blended-Reality-superstructing-reality-superstructing-selves-27-320.jpg)
![BLENDED REALITY superstructing superstructing
Haptic Interfaces will emerge as new ways of interacting with the digital world.
Within five years, a significant percentage of our digital interactions with will be through tangible or gestural
means.
The Nintendo Wii utilizes gesturing while iPhone users pinch and poke their touch screen displays. Samsung
Electronics and LG Electronics are racing to popularize their new “haptic” phones that gently vibrate in vari-
ous patterns depending on their settings.
Groups like the Tangible Media Lab are prototyping a whole range of new interfaces like the Audiopad, a
matrix of antenna elements which track the positions of electronically tagged objects on a tabletop surface.
Large format 3D displays are already in development at labs like CalIT2 in San Diego.
As our modern computing infrastructures evolve, fixed and wireless infrastructures will
become ubiquitous.
There are several new important technologies to watch here:
WiMax has a number of carriers deploying its networks worldwide. It’s capable of delivering 70
megabits per second to devices within 30 miles (50km radius). Intel has announced that they’ll soon
be offering WiMax enabled processors. [In the U.S. Intel, Sprint, and Google have announced they
are working together to deliver WiMax services across the U.S. starting this year.]
LTE (Long Term Evolution), which is envisioned as a competitor for WiMax, will offer over 100Mbps
of throughput. Verizon and AT&T have competing formats currently (CDMA and GSM respectively).
LTE will start appearing in the U.S. in 2010 with mass coverage by 2012.
Femtocell, according to Wikipedia, “is a small cellular base station, typically designed for use in
residential or small business environments. It connects to the service provider’s network via broad-
band (such as DSL or cable); current designs typically support 2 to 5 mobile phones in a residential
setting extending service coverage indoors, especially where access would otherwise be limited or
unavailable.” According to Technology Review: 37
Similar in concept to the Wi-Fi routers that many people use to blanket their homes with
wireless Internet access, these little boxes instead provide a network for carrying the
voice and high-speed data services of mobile phones. They’re designed to give band-
width-hungry cell-phone subscribers the strongest possible connections at home.
Wireless HDMI (High-Definition Multimedia Interface) is a connectivity protocol that delivers uncompressed
HD digital video and audio signals to a TV or projector. Early generation chipsets are available from compa-
nies like Tzero, using WiMedia, an Ultra Wide Band (UWB) system.
Over the next 15 to 20 years, computational resources will also become widely available and abundant in
power. Companies like AMD and Intel have already started building and selling chips with multiple proces-
sor cores and have announced multiple plans for 8, 80, and even 800 Pentium equivalent processors on one
chip. As processors become redundant on a chip, processing net useful power may increase at rates beyond
43](https://image.slidesharecdn.com/sr-1221blendedreality-100816120523-phpapp01/85/Blended-Reality-superstructing-reality-superstructing-selves-47-320.jpg)
![FOUNDATION PIECE
Gordon Moore’s forecast 38 of a doubling of power every 18 months. Prices for computation will continue to
drop correspondingly. Anant Agarwal, CEO of startup Tilera whose company currently ships a 64-core embed-
ded processor forecasts a corollary of Moore’s law:
Today, companies like Amazon, Google, Yahoo, Microsoft, IBM, Sun, Hewlett Packard, and others are all offering
utility cloud computing, which links networks of computers into grids and clusters of cloud computing ser-
vices. Over time, supercomputer capabilities like pattern recognition, datamining, simulations and modeling,
and high-resolution interaction will be widely used for mass-market web services and new applications in
media, gaming, and ubiquitous computing.
Over the next ten years, our mobile devices will be able to precisely calculate our position in three dimen-
sions. Personal devices, like iPhones, already include a robust toolbox of sensors that can tell applications,
with some precision, whether we are using GPS and their wireless sources. These devices can even calculate
motion using built in accelerometers. Software like Fireagle allows us to fuse information from these sensors,
and to selectively decide how to share information about our locations with other people and services.
Research labs and startups are prototyping finer grained positioning using photographic techniques that
recognize patterns in image and can compare and match those patterns with the real world captured in a
mobile camera. Earthmine, a company in Berkeley, California, is already building point-cloud databases of 3D
models of cities [where each pixel can be positioned within 20cm of the exact location.]
In the future, we’ll utilize sensors by embedding them in our environments as well as wearing them. A new
projection says that by 2012 more than 3.4 million senior citizens in the U.S. will be using networked sensor
applications to monitor and improve their health.
We are beginning to see some software tools and technologies that enable the creation of startling new
high-resolution, real-world virtual worlds that might serve as compelling frameworks for blended realities.
These are a few of the new software ecologies beginning to emerge.
Forterra is building a toolkit, called OLIVE, a virtual world modeled on the real world, for enterprise
simulations.
Both Google and Microsoft are building 3D versions of the real world. The Google version is a
grassroots effort where ordinary people create models using a free tool called SketchUp. Microsoft
is building digital 3D models by flying aircraft with special radars called Lidar that capture shapes as
well as imagery.
Sun Microsystesm engineers are building a digital replica of a building in Menlo Park, California
using a toolkit Sun is building called Wonderland.
44](https://image.slidesharecdn.com/sr-1221blendedreality-100816120523-phpapp01/85/Blended-Reality-superstructing-reality-superstructing-selves-48-320.jpg)
Blended Reality: superstructing reality, superstructing selves
- 1. superstructing superstructing www.iftf.org Institute for the Future Technology Horizons Program 124 University Avenue, 2nd Floor February, 2009 | SR-1221 Palo Alto, CA 94301
- 2. About the … The Technology Horizons Program combines a deep understanding of technology and societal forces to identify and evaluate discontinuities and innovations in the next three to ten years. Our approach to technology forecasting is unique—we put people at the center of our forecasts. Understanding humans as consumers, workers, householders, and community members allows IFTF to help companies look beyond technical feasibility to identify the value in new technologies, forecast adoption and diffusion patterns, and discover new market opportunities and threats. The Institute for the Future is an independent, nonprofit strategic research group with over 40 years of forecasting experience. The core of our work is identifying emerging trends and discon- tinuities that will transform global society and the global marketplace. We provide our members with insights into business strategy, design process, innovation, and social dilemmas. Our research generates the foresight needed to create insights that lead to action. Our research spans a broad territory of deeply transformative trends, from health and health care to technology, the work- place, and human identity. The Institute for the Future is located in Palo Alto, California. © 2009 Institute for the Future. All rights reserved. All brands and trademarks are the property of their respective owners.
- 3. BLENDED REALITY superstructing superstructing INTRODUCTION 1 Chapter 1 5 Chapter 2 9 Chapter 3 13 Chapter 4 17 Chapter 5 21 Chapter 6 25 Chapter 7 29 Chapter 8 33 FORECASTS 37 FOUNDATION PIECE 41 END NOTES 47
- 5. BLENDED REALITY superstructing superstructing I N T R O D U C T I O N We are creating a new kind of reality, one in which physical and digital environments, media, and interactions are woven together throughout our daily lives. In this world, the virtual and the physi- cal are seamlessly integrated. Cyberspace is not a destination; rather, it is a layer tightly integrated into the world around us. Technology enables this transformation but, as is always the case, when we invent new technologies, they in turn re-invent us. In the realm of blended reality, the technologies and tools that we are creating change a fundamental part of our existence: the lenses through which we view and interact with the world. We are beginning to see and feel the world through new sets of eyes and ears—things that were previously invisible become visible, and we see the familiar in a new way. Almost ten years ago, we wrote about the sensory transformation we’re about to undergo as technologies move off the desktop and into the physical environment. We also pointed out that sensory transformations inevitably lead to major social and cultural transformations because they shape the nature of what we experi- ence and how we make sense of our surroundings [Cybernomads SR-829]. Blended reality is the manifesta- tion of these changes. It is a type of sensory transformation that will change people’s lives, their senses of self and others, and their views of the world around them. In this report we analyze key directions of this metamorphosis. In conducting this research, we held ethnographic interviews with people who are pioneering new ways of living in the blended reality world (see Interview Subjects on page 3). Their perspectives and experiences are signals of emerging needs, abilities, practices, and routines. To identify key themes and forecasts of the future social landscape and its implications for organizations and individuals, we analyzed the data and brought it together with our research on emerging technologies, identity, and sensory transformations in order to provide a set of forecasts outlining directions of change in the next 10 years. We also conducted research on the key foundational technologies that are driving the Blended Reality transformation (see Foundation Piece on page 41). The following are some key shifts we want to bring to your attention: Emergence of new genres of personal performances as a key communication modality. The Web 2.0 generation of social media tools—Twitter, YouTube, Facebook, etc.—will increasingly change the nature of our interactions with each other, from communications to public perfor- mances. Increased ability to participate in each other’s life streams. As we continuously stream digital bits of our lives—what we are doing, how we are feeling, what we are thinking, who we are meeting with—we will increasingly come in and out of each other’s lives on a scale previously unknown. Growth of a mediated oral culture. Spurred by the availability and adoption of cheap and portable video tools, the web will continue its metamorphosis from a primarily text-based vehicle to a multimedia one. Audio and video will rival the written word as the predominant communication forms for coming generations. 1
- 6. INTRODUCTION Blurring of boundaries between “us” and “that.” As computing, sensing, and imaging capabilities migrate from traditional communications and technology tools into everyday objects and environments around us, inanimate things will acquire identity and presence in our lives. Expanded powers to create highly personalized and compelling worlds. Using lightweight digital technologies, we will increasingly create our own worlds and structures that fit our individual needs and contexts. In many cases, the worlds we create will be more emo- tionally charged, more appealing, more satisfying—i.e. more perfect for us than traditional purely physical worlds. Increased awareness of the individual as a network. As our networks become highly portable and we have access to members of our network not on an episodic but on a continuous basis, we will think of ourselves as not one but as one of many and will act and interact accordingly. Arrival of digital immortality. We are continuously documenting and archiving our life streams—from photos and videos to vari- ous types of health and identity data. In the process, we create digital mirrors of our lives that can be aggregated, searched, mined, and used as inputs to new types of simulations. Growing potential for New Taylorism. Frederick Taylor revolutionized the assembly line by taking careful measurement of many parts of the production process, minimizing waste, and increasing productivity. As more behavioral data is collected, the temptation to use such data to improve productivity or to optimize processes in areas far removed from assembly line like knowledge work, education, and learning, will grow. Emergence of new platforms for identity creation and experimentation. Continuous feedback loops with our multiple audiences—our Twitter followers, Facebook friends, blog readers—will re-shape our sense of self. We will increasingly see identity as malleable and something to experiment and play with. Most importantly, we are blending ourselves into one global intelligence, one global body, one global brain. As Jerry Michalski of Sociate said in one of our interviews for this project, “For the first time, divisions imposed on us by geography, class, political, economic, transportation, and many other systems created over many centuries, are being dismantled as we enter each other’s life streams across these boundaries.” Therein lies the greatest hope for the future as for the first time we have an opportunity to connect to each other in new and infinitely more intricate ways. This new reality will offer its share of challenges, includ- ing an existence that is more transparent and complicated than ever before. In the process, we will be “superstructing”—creating structures that go beyond the basic forms and processes with which we are familiar—our economy, our society, and ourselves. All of our institutions, processes, practices, and, indeed, senses of selves were created and functional in the world based on certain boundaries and divisions. We are just in the beginning of superstructing all these aspects in our blended reality lives, where old divisions no longer hold and we acquire new lenses for seeing ourselves and the world around us. 2
- 7. BLENDED REALITY superstructing superstructing annaone and loki play SF0, a San Francisco-based collaborative Kati London is a game designer at Area/Code and co-creator of production game whose tasks have largely been created by the Botanicalls, a sensor system that allows plants to Twitter their water- game’s players. (www.sf0.org) ing needs. Kati is interested in sustainability art and games that help people better relate to the non-human objects in the world. (http:// Stella Artois* is a 16-year-old girl living in Modesto, CA. Home- www.katilondon.com) schooled her entire life by way of online educational courses and materials, she has managed, with the help of her mom, her own Jeff McClure is an early adopter of Yelp. Jeff always consults Yelp education. By age 14, she had completed her required high school when he’s about to check out a new place, whether in San Francisco curriculum and moved onto online college courses. She has recently or elsewhere. He contributes reviews to the site, and has attended begun attending classes on campus. Stella manages multiple online several meet-ups. In addition to Yelp, he’s used Flickr and LiveJournal identities and is involved in a variety of online and non-overlapping for years and has made a number of real-world friends through both communities, from art to music to her local, real-world community. sites. (http://troymccluresf.livejournal.com) Gordon Bell is a Microsoft engineer, telepresence pioneer, and Jerry Michalski is an independent consultant under the company human guinea pig for MyLifeBits, described in a 2007 New Yorker name Sociate. He uses brain software that allows him to capture, article as, “a personal transaction processing database for every- organize, and retrieve every piece of information/knowledge he thing.” (http://research.microsoft.com/en-us/um/people/gbell/) comes across. He’s been using his “brain” for 10 years now and it is a remarkable repository of knowledge for him. He’s also a social media Nick Bilton works in the R&D lab at the New York Times and is part power user. (http://www.sociate.com) of a NYC-based hacker collective, NYC Resistor. Nick is a power user of Botanicalls and is constantly thinking about ways that he can Laura O’Donnell* is a woman in her mid 20s who has managed tweak, hack, or customize technology to make his interaction with Type 1 Diabetes since her teens using a variety of hardware, soft- the non-human world a little smoother—or just more fun. (http:// ware, online and social tools. She uses SugarStats, an online start-up nickbilton.com) service, to visualize this data and track patterns. She is also active on LiveJournal and Twitter, where she seeks and gives advice, support Jane DS is a lifecaster on Justin.TV who broadcasts her day-to-day and commiseration with other diabetics, and maintains social net- life over the Internet. (http://www.justin.tv/jane_ds) works around crafts and art. She also maintains a personal website Sean Goe* is a prominent blogger and frequent user of a variety of where “all the pieces of [her] on the internet” are collected with links. social media. Sean Goe has over 20,000 people following his Twitter Skip Rizzo is a doctor at University of Southern California who is at feed. He operates his own web hosting company and is a fixture in the forefront of virtual reality exposure therapy, a program that uses the Bay Area tech community. audio, visual, smell, and vibration elements to simulate traumatic Jess Hemerly is a Research Manager at IFTF and a long-time user of events for, among others, soldiers with PTSD. This is “triggering psy- social media, from blogs and networking sites to message boards chophysiological experiences that produce high levels of emotional and IRC. (http://jesshemerly.blogspot.com) engagement.” (http://vrpsych.ict.usc.edu/people/rizzo.html) Ian Kizu-Blair is one of SF0’s co-creators and states that one of the Nigel Simpson is a developer at Sun Microsystems working on creators’ motives was to use a digital platform and online interaction the Wonderland platform, a toolkit for developing virtual worlds. to get people away from their screens and playing in the real world. He works remotely from an island off Seattle, and relies heavily Ian and his co-creators, Sean Mahan and Sam Lavigne, have also on mpk20, a virtual office space developed by Sun Microsystems, been commissioned to conceive and develop client-specific games. for interacting with his team members. The Sun office complex in (http://sf0.org) Menlo Park has 19 buildings, and mpk20 symbolizes the 20th build- ing. Nigel is in an interesting position as he is both a developer and Lance Koenders is Senior Vice President of the Digital Home Group user of virtual worlds. He helped develop the mpk20 virtual world at Intel Corporation. He started exploring Second Life for work and he uses the mpk20 virtual world to hold virtual meetings with research. During his time as an active Second Life user, he bought his his team members, most of whom work on the east coast. (http:// own island and opened a music venue on it to attract people. Now research.sun.com/projects/mc/mpk20.html) less active in Second Life, Lance has turned over management of his virtual world property to a group of assistants. (http://www.linkedin. Keiko Takamura is a singer/songwriter who began exploring com/in/lancekoenders) Second Life on a whim and has since become a popular Second Life performer. She was featured on an episode of MTV’s documentary Clarke Lethin is a recently retired Marine Colonel who has spent the show True Life. (http://www.keikotakamura.com) last 10 years with One Marine Expeditionary Force of over 45,000 Marines and Sailors at Camp Pendleton, California. From 2005-2008 * Name has been changed at the interviewee’s request Col. Lethin was Chief of Staff and was heavily involved in develop- ing the Infantry Immersive Trainer, a state of the art mixed reality combat environment designed to preview the combat experience in Iraq. 3
- 8. 1 | I AM THE STAR OF MY OWN SHOW 4
- 9. BLENDED REALITY superstructing superstructing C H A P T E R 1 Forty years ago, Andy Warhol uttered the now-famous phrase: “In the future everyone will be world famous for 15 minutes.” The future is here, but our world fame is no longer measured in minutes. In the words of artist and designer Nick Philip, “everyone will be world famous for 15 megabytes.” Each one of us is becoming a one-person media channel—the directors and stars of our own shows. Technology enables this transformation. In the not so distant past it took an army of people to produce a show. You needed expensive equipment, expensive actors, skilled camera operators, producers, stagehands, agents, promoters, and multitudes of others. Even today, heavily produced TV shows still require a lot of people, money, and equipment. However, this is no longer the only game in town. As the cost of video pro- duction rapidly decreases a whole array of accessible and affordable lightweight, non-professional tools are emerging. These include inexpensive video cameras, such as Flip Mino1, which sells for about $180 USD; cell phones with built-in cameras and ever-improving video capabilities; laptops with cameras and microphones; and web-based programs such as Qik that enable video streaming via cell phones.2 These tools are getting into the hands of people without professional training who are using them to create their own shows. Thus, what was exclusively the province of experts and production studios is now open to everyone, enabling an evolution of a whole new repertoire of personal performance genres. The following are some emerging personal performance genres we uncovered in our ethnographic interviews. The Flip Mino ranges from $180 for the base model to $230 for HD. The Internet has become a public performance space for a variety of creative types (and some not-so- creative types). The tools and platforms performers use include everything from Second Life to YouTube and YahooLive! channels.3 More complex online performances often require advance planning, preparation, coordination, and rehearsal. Keiko, a singer/songwriter we interviewed, started to explore Second Life while living in Japan. She began to experiment with performing in Second Life, gradually amassing a fan base. Despite paralyzing stage fright, she has since started to carry over some of her success in Second Life to coffee shops and venues around the Bay Area. Her performances in Second Life require scheduling, setting up equipment and merchandise, notifying audiences, and managing audience feedback. Another one of our interviewees owns a venue in Second Life, where artists like Keiko can perform. Venues are mostly meant to draw people to build and buy real estate on a venue owner’s island, but they serve the secondary purpose of giving artists exposure within the virtual world. In some cases, Keiko and a group of Second Life musicians merge online and physical venues, with live audiences assembled in a physical location viewing an online performance, and vice versa. 5
- 10. 1 | I AM THE STAR OF MY OWN SHOW Lifecasting involves a continuous broadcast of one’s life using a variety of tools like laptop, video, mobile phone cameras, and Internet sites such as Justin.TV, Yahoo!, and Qik, to open up their lives to the public. In this sense, many don’t think of these as performances but rather as sharing their day-to-day lives with the outside world. Jane DS is a young woman who lifecasts on Justin.TV.4 Her channel has hundreds of regular viewers who not only watch her life but also interact with each other and with Jane through a chat box on her lifestreaming home page. They exchange comments about what Jane is doing, provide feedback, and ask questions in real time. Jane leaves the camera on all the time, and viewers continue to chat as Jane sleeps. A screenshot of Jane DS’ lifecasting page on Justin TV. Even without cameras or video equipment, people are increasingly engaging in performances that use text as the medium. Twitter streams, Facebook pages, and blogs are all forms of public performances meant for public consumption. People use these social media tools to provide periodic or sometimes frequent updates on different aspects of their lives—where they are, what they are reading, who they are meeting with, what they are thinking, etc. Audiences achieve an ambient intimacy with the performances they follow by getting a sense of the patterns in people’s lives. Such performances are text-based, although even textual performances are becoming increasingly multime- dia. vBlogging or mBlogging are emerging blog forms where bloggers use videos and photos to documents parts of their lives. A Flickr stream containing a series of photos with titles, descriptions, tags, and comments is another form of a public diary. Unlike the diaries and journals kept under mattresses or locked in drawers, today’s digital diaries are highly public, unless a user chooses to control access to the information by limiting it to a defined group of friends, colleagues, or acquaintances. An important part of the transition to personal production is that individuals and members of social net- works are now filling the resources and roles that were previously required to produce a show. From Hired Editors to Self-Editing. Any produced show used to require expert editorial help. Today, how- ever, we are becoming our own editors, and one of the most important editorial questions we are constantly asking is whether to post or not to post something. This is becoming a major filter for how we think about what we are doing, seeing, how we are feeling. We are constantly asking ourselves: is this something I can post? Where should I post this? Is this a Twitter or a blog post? How should I post it in order to “You don’t want to post too much make it interesting to my audience? Should this information stay or do I need to delete it? From Agents to Mods. Who replaces all those producers, managers, stagehands, and army of or too little. You don’t want to be other paid staff in the era of personal production? For many, these roles are played by volun- un-followed [on Twitter]. ” teers and people in the social networks of the “stars.” Jane DS and others we interviewed talked — Jerry Michalski about the importance of mods or moderators. She describes moderators as, “people that I trusted … they were different from just regular viewers.” Popular blogs and online sites such as BoingBoing rely on moderators to keep the show going by maintaining the site’s etiquette by deleting incendiary comments or banning troublemakers.5 These mods receive no special training, nor are they necessarily paid for their work. Many simply emerge out of the audience as active and positive contribu- tors and then are entrusted with special moderator duties and privileges as a reward for their reputations. 6
- 11. BLENDED REALITY superstructing superstructing From Hecklers to Griefers and Trolls. What moderators are often protecting the show (or site) from is a new genre of evil-doers that often go by the name of griefers or trolls. They are to today’s digital perfor- mance spaces what hecklers were in previous eras. These are people who raid your site, hack it, damage it, or otherwise interfere with the performance through their disregard for a show or site’s stated or understood guidelines. In Jane’s case, the griefers stole her lifestream and “They were coming back constantly, posted it to a completely separate site. But these acts of sabotage are also performances in standing up for me, and they their own right, another way for attention-seekers to say, “Look at me!” Because their behav- ior is so disruptive, they almost always get what they want. A popular saying has emerged showed me that they were people regarding these troublemakers—“Don’t feed the trolls”—meaning if you don’t reward their that weren’t here to cause trouble. attention-seeking behavior by getting angry, they’ll find another show to heckle. They basically just clean up my From Marketers and Promoters to Street Teams. Forget professional promoters. Today’s promoters are the people in your network who promote your ideas or shows for you by simply chat room, protect me from people referencing your work and posting a hyperlink. We also see organizations using the Internet to that are trying to cause trouble. organize real-world promotional activities. Future Builders is a group of teenagers who want to There’s a lot of times when I don’t make the world a better place.6 They set out to accomplish this mission by organizing benefit concerts for young people. The money raised goes to various causes—anti-hunger causes, pay attention to the chat room, battered women’s shelters, HIV/AIDS organizations. You can sign up on their MySpace page to especially when I’m just lifecasting. become a member of their street team and distribute t-shirts, help promote performances, and help advertise various causes.7 In exchange, street team members get special mention on the So they watch it for me. “ site and even the occasional complimentary ticket to a charity performance. — Jane DS The idea that “life is theatre” is hardly new. We engage in ongoing performances in our everyday life just by carrying out our routines. The people around us serve as audiences while our clothes and physical surround- ings serve as the costumes and sets in which our personal stories and the stories of our identities unfold. We explored this theme almost ten years ago at the IFTF research exchange on Identity [SR-663B]. At that time we wrote: “We create our identity through reference points. We know who we are in reference to others … Identity is a conception of self that we create based on various reference points in our life.” All of the new audiences we create and that acquire a constant, ambient presence in our lives—our Facebook friends, Twitter followers, blog readers, lifecast fans, and more—provide new reference points for creating and shaping our identities, our senses of self. Not only are we creating multiple new reference points and feedback loops through which we view ourselves, but we are gaining a new level of awareness about their presence in our lives. As we do so, we gain a sense of self that is highly complex, malleable, and playful. We gain a new level of awareness of how to play to different audiences and how to shape our performance streams to make our- selves interesting and entertaining. We have to be lively and engaging in order to maintain our audiences. At the same time, we are able to develop multiple channels for playing out and experimenting with different Ten years ago, the identities if we wish to do so. Outlook Project (now the Ten-Year Forecast) Psychologist Paul Bloom points out in his recent article in the November 2008 issue of The Atlantic that “many held a research exchange researchers now believe, to varying degrees, that each of us is a community of competing selves, with the on personal identity 7
- 12. 1 | I AM THE STAR OF MY OWN SHOW happiness of one often causing the misery of another.”8 The self that gets up in the morning, determined to eat only healthy food all day differs from the self that raids the refrigerator in search of ice cream in the evening. According to Bloom, “Within each brain, different selves are continually popping in and out of exis- tence. They have different desires, and they fight for control—bargaining, deceiving, and plotting against one another.” Our transformation into one-person media channels and conscious solo performers will enhance this multi-faceted view of ourselves. Such fundamental transformation in how we play out our identity will inevitably impact how we view others, how we interact with each other, and how we think about personal responsibility and personal obligations. 8
- 13. BLENDED REALITY superstructing superstructing C H A P T E R 2 Social networking tools enable us to build and extend our networks. They not only allow us to become active audience members, but also force us to see ourselves as part of a larger collective of lives and minds. The ability to tap into other people’s digital life streams via Twitter updates, Face- book news feeds, lifecasts, or personal blogs creates a constant peripheral awareness of one’s larger community, and a new sense of virtual intimacy or connections to people who are merely acquain- tances in our offline lives. Just knowing people are out there, either on a chat list or posting messages to Twitter, creates a sense of ambient companionship. We plug into people’s thoughts, knowledge, actions, experiences, interests, concerns, affiliations etc., without expending much effort or imposing on the other people. This ability to tap into the knowledge of the larger community makes these streams valuable sources of information. They are also lead- ing to the creation of new social norms and roles. The phenomenon of seeing ourselves as a part of a larger network and as a body of digital lifestreams opens up new genres of relationships and new interaction paradigms. The terms “knowing someone” and “my friend” have very different meanings in this era of rampant social networking. The way we define relationships is changing drastically. A brief conversation with someone at a party could likely result in them adding you as a “friend” in an online social networking space, and in the case of Facebook, it’s a public way of announc- ing to their network that they “know” you. New social norms and etiquettes are emerging for keeping in touch with “friends” and friends alike. One way to reach out online is Slide, Inc.’s Superpoke application for Facebook. “Poking” someone on Facebook is a gentle way of saying “hi,” but Superpoke is playful and often Superpoke is an application silly. Superpoking someone means selecting a friend from your network and choosing an for Facebook that allows action, whether it’s throwing a virtual farm animal or random object at your friend, or hugging, kissing, or users to playfully “poke” karate chopping them. The more friends you Superpoke, the more action options you earn. During election their friends online. season, Superpoke offered politically oriented actions where users could “throw” the Presidential and Vice Presidential candidates at one another. IFTF researchers, for example, found their Facebook inboxes full of airborne Sarah Palins. 9
- 14. 2 | ME OF MANY Microblogging site Twitter has its own social vocabulary. Users don’t friend each other; they “follow” one another’s streams of brief messages. Your Twitter popularity can be measured in the number of followers you have, whether they are acquaintances, strangers, or friends. The posted messages, called “tweets,” range from serious news to hyperlinks to banal messages like, “I’m eating toast.” Users can also create ad hoc group discussions by using a hash tag. At the recent Technology Horizons fall exchange, Twitter users at the conference tagged tweets with “#blended,” which could then be followed through the Twitter search site, search.twitter.com. New Twitter relationships are often built through these hash tags, as they are considered an area of common interest. Twitter has recently seen exponential growth. TechCrunch reports that as of Twitter.com invites users to March 2008, the site had over 1 million total users and 200,000 active users each week. “Join the conversation” via The total number of messages posted was approximately 3 million per day.9 140-character microblogs. Social networking and other web tools allow individuals to become part of large networks. For example, it is not uncommon for teenagers to have hundreds of “friends” on social networking websites, many of whom they don’t know or have never met in real life. These tools allow us to have access to people’s thoughts, ideas, and experiences—whatever they choose to make available. This is creating a new kind of vir- “I don’t know 13,000 people. I don’t tual intimacy, an expanded peripheral awareness of and access to multiple levels of relation- know anyone who knows 13,000 ships and networks. people. I don’t know anyone who Older web tools also establish this sense of intimacy. For example, the use of Instant Mes- saging service not only establishes presence, but it provides a sense of virtual companion- knows 5,000 people. So I don’t ship. Users can see who is online, and get some sense of connection. Unless a user makes his know how many people I know at or her stream visible only to selected contacts, any Twitter user can follow any other Twitter user’s stream. A user can view someone else’s post, or tweet, on the stream of people he’s this point because — does meet- following and decide whether to reply or simply to take no action, absorb the information, ing a person once for a minute and move onto the next post. But the ability to view the thoughts and random musings of someone you don’t know very well creates an immediate sense of intimacy with that per- mean you know them or not? son. The Obama campaign harnessed this power of virtual intimacy during the recent 2008 I think those things have to be presidential elections. Anyone could follow the BarackObama stream on Twitter, getting his defined. ” updates along with those of other friends and acquaintances they follow.10 Psychologically, this made Obama feel more like someone they “knew” than just a distant politician. — Sean Goe As physical locations, like offices, become less important to work and how work is done, microblogging and social networking are beginning to replace the social environment provided by an office full of coworkers. In a world where more workers are opting for flexible locations be it home or a café, online streams serve as virtual water coolers, providing a sense of camaraderie and practically negating the need for face-to-face social interaction. For example, Yammer, a Twitter-like microblogging service, focuses on business, and only individuals with the same corporate domain in their email addresses can access the company network.11 Yammer was unveiled 10
- 15. BLENDED REALITY superstructing superstructing earlier this year at the TechCrunch50, an annual conference founded by the blog TechCrunch, where 50 start-ups are invited to launch their idea or product in front of fellow entrepreneurs, media, and Silicon Valley venture capitalists. Yammer took the top prize.12 Unlike Twitter, where users provide updates on what they are doing, Yammer asks the question, “What are you working on?” Yammer also has a Blackberry and iPhone client that makes the data accessible on the go. You can keep track of what colleagues are doing even when you’re away from your office, home, or coffee shop. “Twitter’s definitely like the virtual water cooler for me. I don’t work at a job so I don’t see people all day. ” Managing online identity is not as simple as creating and maintaining one multi-purpose — Sean Goe MySpace or Facebook profile. In many cases, users maintain multiple spaces online for the different aspects of their lives, and each of those aspects relates to a different community. The media is also helping us play with our identities, and amplify them. A direct result is that users have to spend more time online controlling their different streams and identities. For example, when we interviewed Stella Artois, a 16-year old who has was home schooled until college, she explained that she has ten different MySpace profiles, and each profile corresponds to a different aspect of her life. She has one MySpace page to stay in touch with her friends from school; another page to stay in touch with fans of a British band she likes; and a different MySpace page all about an Ameri- can band with whom she works. She is aware at all times that these communities exist, but has chosen to keep them separate them from each other by creating a profile for each. Being virtually plugged into these networks allows users to tap into the collective intelligence of the community. These virtual streams are sources of information that we can leverage for research and often get quick answers from. The advantage is that it can all be done without imposing on virtual friends. Twitter has turned into a great resource for knowledge and expertise since anyone can search for any kind of information, updates, advice, and opinions using the Twitter search engine. Following the November 2008 terror attacks in Mumbai, India, Twitter users who tagged tweets with “#Mumbai” traded information at a rate of 50-100 posts a minute. The Twitter stream Mumbai, which aggregated tweets tagged with #Mumbai, became one of the most important resources for netizens following the situation on the ground as well as the global mainstream media reports. As the Associated Press’s Sam Dolnick wrote on December 1, “The The Mumbai Twitter stream lightning-quick updates of the attacks that killed 174 people read like a sketchy but urgent blow-by-blow was one of the first places account of the siege, providing further evidence of a sea change in how people gather their information in to find news after the November 2008 terrorist an increasingly Internet-savvy world.”13 attacks. 11
- 16. 2 | ME OF MANY Simply through casual connections, we’re building vast information networks that give us access to on- demand knowledge that comes not from encyclopedias, but from other people. We can throw a question out into the social ether through Twitter or another social site and get a variety of responses that are tailored to our particular question because the answers come from other people, friends or “friends.” As Google CEO Eric Schmidt pointed out: “The goal is to enable Google users to be able to ask questions such as ‘What shall I do tomorrow?’ and ‘What job shall I take?’”14 By engaging in these communities and networks, we also become sources of information ourselves, offer- ing knowledge or expertise where it applies to us. This participation not only helps raise awareness of our large networks, but also lets us feel active and useful, even if we’re only answering a request for advice on local restaurants. We contribute to the larger knowledge community just by responding to questions. We will tap into our extended online networks to make many personal day-to-day decisions. Before we Networking Intelligence set out to give a talk or present a business proposition, we’ll test out our ideas on our extended networks. was a key theme in our 2007 Ten-Year Forecast and This simple interaction lends itself to rich collaboration opportunities without being cumbersome. What is it will continue to play out most important is that we see and think of ourselves as one of many. Being able to tap into the resources of in the coming years. a personalized community in order to obtain information, knowledge, or resources is more important than personally having this information, or knowing all the answers. This is where the notion of “networking intel- ligence,” introduced in our Ten-Year Forecast, comes from (2007 Ten-Year Forecast Perspectives, SR-1064). “I really like the sense of presence. I have Google Talk set so that it shows up in a window and I like being able to look over and see who’s around. It gives me a feeling of companionship. And these are mostly really good friends. ” — Jerry Michalski 12
- 17. BLENDED REALITY superstructing superstructing C H A P T E R 3 As digital storage increases, people are creating and preserving digital records of every aspect of their lives—how they work, play, socialize, what they think, and more. These digital imprints define who we are, and shape our present and the future. People retain records of even the smallest things, from emails containing shopping lists to random 140-character-or-less thoughts expressed on Twit- ter. This kind of archiving has the potential to grant us digital immortality, where whole pictures of our lives and identities can be gleaned from the comprehensive archives of personal digital records. Digital archiving enables us to extend experience beyond one single moment and provides us tools to easily revisit the experience. A desire to capture an experience creates tension between living the experience as it happens and needing to record it. At the same time, technology will make it easier to passively capture and record data, allowing us to extend our physical life into digital infinity. The following are some of the key practices that people are engaging in as they create digital mirrors of their lives. Digital life archives enable people to have instant access to their recorded thoughts, mementos, social networks, etc. It’s like having all of your life at your fingertips—noth- ing gets lost, regardless of how mundane it may seem. Portable mobile devices make this digital life accessible at any time. It is possible to have access to your data not just when you are sitting in front of your computer, but even when you are on the move. For the past ten years, Jerry Michalski has been using a software called Brain to record A snapshot of some of his thoughts, ideas, and other interesting pieces of information. To date, Jerry has created 94,000 Stephen Wolfram’s nodes or thoughts in his “brain.” He collects the daily minutiae of his life so extensively that he owns the digital scrapbook. most comprehensive personal archive of its kind in the world. Stephen Wolfram, author of New Kind of Science and creator of computational software program Mathemat- ica, set up his archiving experiment online. Wolfram built “A Scrapbook: The Life and Times of Stephen Wol- fram,” an online scrapbook that spans his life since 1959. He has digitized letters, scratch paper, photographs, and other appropriate artifacts and made it all publicly available on the Internet.15 His site is well designed by public standards, but also tells a very personal story. 13
- 18. 3 | DIGITAL IMMORALITY Other companies are developing more lightweight tools to help individuals create and preserve their digital memories. Memiary, a recently launched blogging tool, helps you create and preserve a digital memory of each day in your life.16 Memiary’s founders have positioned it is a “weightless pocket diary.” Users can record up to five memories per day, and the site is limited to personal use only. In addition to the individual’s quest to preserve data, the Library of Congress seeks to collect, archive, and save everything from geo-spatial data to visual media through the National Digital Information Infra- structure & Preservation Program.17 The program also provides instructional informa- tion for individuals on how to preserve their digital memories. Dematerialization is a practice of substituting digital mementos for physical things. It reduces clutter in your life. It reduces your consumption of physical things. It reduces your carbon footprint. If you do it right, it makes your things easier to find. One of the benefits of dematerialization is that it gives you more space in your home for things that have to remain in their physical form, such as furniture and kitchen appliances. And if you are someone who interacts with some kind of device on a daily basis, say a phone or a television or a computer, you could actually have more engagement with the things that have been digitized than you did when they were stuck in a cardboard box in the basement. Microsoft’s data model for A few people we interviewed are beginning to use digital archiving as a way to MyLifeBits, a project that de-clutter their lives while preserving memories of important physical things in their lives. For the time takes a lifetime of memories being, digital archiving and dematerializing consists of creating a digital artifact—usually a photo—of the and digitizes them. original physical artifact. Much of our home clutter is paper-based: our files, old bills, insurance contracts, old letters, family recipes, taxes, and disintegrating photo albums. All of these can be digitized with a scanner. Microsoft engineer and telepresence pioneer, Gordon Bell has been the test subject for a comprehensive archiving project called MyLifeBits, a project that makes Vannevar Bush’s theoretical memex— an ultimate storage and research navigation device that Bush wrote about in the 1940s—a “I keep feeding information in there, reality. From the MyLifeBits website: “Gordon Bell has captured a lifetime’s worth of articles, and that actually works well in books, cards, CDs, letters, memos, papers, photos, pictures, presentations, home movies, vid- eotaped lectures, and voice recordings and stored them digitally. He is now paperless, and is meetings because as I learn things, beginning to capture phone calls, IM transcripts, television, and radio.”18 and as I think of things, I can bring Bell uses software developed by fellow Microsoft researchers Jim Gemmell and Richard things up to contribute to the Lueder. He is essentially building a digital replica of his life experiences. Treating himself as a guinea pig for a future in which many of us will have access to a lot more computing power, meeting quickly because they’re connectivity, and online storage, Bell and his team have been trying to figure out which kinds there. The brain acts as my long- of data make sense to hold onto (for instance, he decided that digital copies of every single term memory in many ways. I can voice call weren’t very useful to him), and what can be done with those data streams once they exist. After over ten years of the project, Bell has about 100 gigabytes of personal data just put things away in there. ” that he figures he really needs. — Jerry Michalski Bell, who admits to being a “rabid cyberist,” has also dematerialized most of his physical 14
- 19. BLENDED REALITY superstructing superstructing memorabilia, the t-shirts, baseball hats, coffee mugs, the commemorative plaques and plates. They are out of his closets, off his shelves, into his computer, and onto his screen saver. He doesn’t have to deal with them in their physical forms ever again and nor will anyone else. In the next ten years, 3D render- ing will be another way of digitizing the physical. Digital life data allows individuals to easily revisit old memories. Instead of storing boxes of photographs in a dark closet, pulling up a collection of digital images or scanned photographs requires a few mouse clicks. But a fear of losing personal data drives today’s more comprehensive archiving practices. Bytes are easily— and accidentally—erased, and with them we lose records of precious memories. People deal with this fear in very different ways. Some people aggregate data through websites like FriendFeed and Facebook; others amass libraries of hard drives, creating and storing backups of backups of backups. People are storing their data on external hard-drives and creating additional back-ups. For instance, Jerry Michalski creates a back-up of all his data, and he told us jokingly that when Friendfeed is an online his house was burgled the thieves stole the back-up so he learned to hide the back-up drive. Many upload service that aggregates their data in their email accounts like Gmail, Yahoo!, and Hotmail as they provide vast storage, and most believe streams from Facebook, that the data is safe and accessible in the cloud. Regardless, the world of digital archiving continues to be one Flickr, Google, Twitter, of personal experimentation. and more. Ever since sociologist Richard Ling observed that mobile phone conversations often take prece- dence over physical interactions, researchers have been interested in understanding the effects of technology on interactions in physical space. In the course of our research we’ve found that “If I had photos, my printed photos, documenting an experience can be a filter on how the documenter experiences the moment. We’ve put them in a shoebox, I would all been at a school play or concert where parents fail to applaud for their own children because they never look at them. This way, I’m are so busy capturing the moment. In this case, the camera is a filter that mediates the experience. The need to document is further amplified since many documenters who post their archives on the forced to look at stuff. ” Internet engage with their audiences or followers. Bloggers and Twitter users experience increasing — Gordon Bell pressure to provide interesting and timely updates to their followers. The need for constant engage- ment creates a tension between living the experience and recording and sharing it. Even the simple act of sending a tweet about a something pulls the Twitter user out of the moment and into a sepa- rate act of documenting the moment. But they also must take the time to consider the right phrasing for their audience, thus changing their relationship to the experience of being whereever they are. For example, they aren’t watching the concert at that moment; they are engaged in their own performance, think- ing about how to best translate their experience to an audience that extends far beyond where they are. 15
- 20. 3 | DIGITAL IMMORALITY Each one of us will create a tremendous amount of digital data about our lives. In the process, we will develop new ways to curate, archive, and orga- nize this abundant data. When your most important things are all digital, your computer hard drive becomes even more precious. More of the daily routine is spent managing and maintaining digital possessions, updating and backing up computers. Fear of losing personal data drives today’s more comprehensive archiving practices. People deal with this fear in very different ways. Some aggregate data through websites like FriendFeed and Facebook; others amass libraries of hard drives, creating and storing backups of backups of backups. But with much of this data going into spaces on the Internet provided by companies, what happens when those companies cease to exist? In many cases, those Facebook users have records are lost. created a “Delete Me” group to petition for Facebook On the other hand, what happens when a user wants to delete their presence from a site entirely and the page deletions. site doesn’t allow it? Facebook currently does not allow users to permanently delete profiles. In protest, one Facebook user deleted all of the information from his profile and changed his username to “Delete Me.” We will need to reconcile the will of the user—and the user’s expectations of data storage or data deletion—with the terms or, worse, fate of companies who provide archival services. More importantly, our digital trails will be searchable and accessible, making it possible for us and for others to look back on the collection of experiences that make up our lives. But we are also in danger of obsessive documentation interfering with our enjoyment of the actual moment. Or are we? It is possible that for some people, the act of documentation makes the moment more enjoyable and even more engag- ing simply because they are hyper-focused on the experience in order to properly capture and “I do think about the wider audience. share it. For example, photographers, both amateur and professional, spend much of their life It is like when they had the Olympic behind the lens, striving to snap those perfect images that tell the story of an experience. torch protest. I was walking back We will develop new and adaptive ways of data mining that will allow us to make sense of our digital trails, and we will need services and devices through which to capture, share, and store home and ran into it accidentally data. Companies can create value by providing tools for documenters and archivists to make and started documenting it, and sense of the data, turning digital trails into a source of revenue. Additionally, some people might then I’m posting on Twitter. I got prioritize digital over physical items. Where you once might have gotten a t-shirt, plaque, or hat, now you get an icon, photo, or other digital object. Where you once might have wanted a new immediate feedback, and even got car maybe you drive less and spend more on the equivalent of virtual mobility. The acquisition people from China who were thank- of virtual goods becomes not a way to retreat from the real world, as critics often claim, but a way to make the physical world more enjoyable. Manufacturers will have to consider which of ing me for it because they wanted their goods can be digitized and which cannot, as well as what part they will play in the creation to see coverage. ” and distribution of new digital goods. — Sean Goe 16
- 21. BLENDED REALITY superstructing superstructing C H A P T E R 4 Let’s admit it, the world is an imperfect place. People don’t always do what we want them to do, they don’t always think or act like we do, trains don’t always run on time, schools don’t work on our schedules, and lessons are rarely structured so that you can learn at your own speed. In fact, much of our existence can be summed up as trying to adapt to someone else’s rules and structures or trying to find ways to get around those rules and structures. But what if you could create a more perfect world, a world that works for you and your friends, a world in which you set the rules and you determine structures, timetables, and rules of engagement? Online environments enable many to do just that. How much they tweak the world varies—some create perfect mirrors of “real” lives in digital environments, while others develop new personas and completely new worlds. Degrees to which people hack realities represent a continuum ranging from mirroring to total reinvention. The following are some of the key practices we engage in and challenges we encounter as we increasingly create worlds on our terms. The first order of business in creating a world that suits you is to create the “me” that you want. Some, like Keiko Takamura, take great care to make their avatars look as much as like them as possible. When Keiko gets a haircut, her avatar gets a haircut. She even looks for clothes in Second Life that match what she wears in real life. Her digital avatar is as literal a representation of her physical image as she can achieve. Stella Artois, on the other hand, works to disguise her age among her online classmates. Although she is highly motivated and has achieved a lot at her young age, she remains self-conscious about her age. Stella believes that if she revealed her age, people would Keiko Takamura’s avatar not take her seriously. She doesn’t lie; she just withholds information that might negatively effect how edu- closely mirrors her cational peers perceive her. Even each of her various MySpace profiles represent a different Stella, each one physical appearance— presenting an isolated and carefully crafted piece of her personality. when she gets a haircut, so does her avatar. 17
- 22. 4 | WORLD ON MY TERMS The second part of creating a world on your own terms is to establish routines, structures, and processes that work for you. Existing educational structures, for example, are set up to serve “I didn’t want these people that masses of students and often operate with sets of rules and processes that are not ideally could be 10 years older or 20 years suited to every student’s individual needs. Stella has avoided dealing with such mass struc- tures and sets of rules largely due to the fact that she has been homeschooled her whole life, older knowing my age—then if I relying on online classes and resources to learn at her own pace. She never had to wake up said anything they’ll not respect early to go to class, never had to fight for a seat on a school bus, never had to deal with the me. You don’t want to tell anything cliques and adolescent battles that made many of us miserable. She could study when she wanted and at her own pace without the distractions and limitations of traditional spaces for because you’re younger. Even on education. Working together, she and her family have created a highly personalized learning campus no one knows my real age world for Stella. ... I want them to respect me for Similarly, Jane, our Justin.TV lifecaster, has used lifecasting tools to craft a world in which she is the star, built around her ordinary life. From being an average teenager living at home, who I am, not judge me on my age. going to school, and working a part-time job, she is the star of her own show, with many sup- I’m the youngest one there. ” porters and resources helping her maintain it. Inside of Justin.TV, Jane’s ordinary life becomes a source of fascination for viewers and fans. — Stella Artois While many of our experiences are blending virtual/digital and physical realities, the most interesting sur- prises and new practices evolve at transition points between the two. Think about meeting someone you’ve been friends with on Facebook or in Second Life and realizing that this person is exactly the same as—or quite different from—how he portrays himself in the “other reality.” Or imagine being a champion snowboarder in a video game, but a klutz on the slopes in real life. In both cases, this territory of transition—where real life expectations and impressions meet online identity and skill, and vice versa—offers quite a few surprises to those navigating the terrain. In many ways it is a pioneer territory. In some cases, the gap between online and physical reality appears to be vast. People are “I could also go at the speed I often surprised to stumble upon the gap and try to find ways and tools to overcome it. Dozens of your Facebook friends and “friends” will sign up online to come to a party you wanted to, and if I wanted to do a organized at a real life restaurant, but when you get there, none of your “friends” actually harder math book or English book, show up. As it turns out, obligations and friendships on Facebook do not carry the same level of commitment as your other relationships. And conversely, you may acquire a much deeper well I could. I don’t have to be stuck and more multifaceted knowledge of people you don’t know just by following their Twitter in whatever grade was my age. ” streams or Facebook feeds. These transitions and intersections between physical and digital are where we find ourselves finding unexpected surprises, learning new ways of navigating — Stella Artois relationships, and interactions. Most of the institutions and rules we have created as a society emerged before digital media and digital inter- actions began to permeate our lives. As we make the switch to a pervasively digital society, we will see wide- 18
- 23. BLENDED REALITY superstructing superstructing scale reinvention of the rules and institutions developed in analog times. People who have already created alternative worlds often see mainstream societies’ rules as disorienting and constricting. So it is for Stella, who finds the experience of going to a “regular” college somewhat strange because it was so different from the educational world she designed. People who today are “outside the norm,” like Stella, will move toward the mainstream as more individuals and groups find ways to customize all facets of their worlds. The ability to create worlds on our terms also makes such worlds very compelling and in some ways, addic- tive. Michael Highland, a video game designer, writes about creating virtual experiences in video games that are more emotionally charged and more satisfying than “real” ones: There was one winter I played a ton of SSX (a snowboarding video game). I got to a point where I was actually really excited to go out and experience real snowboarding again, I even made a CD mix of songs from the game so I could listen to the same music when I got out on the slopes. When I finally did put board to snow I was sorely (pardon the pun) disappointed. The harshness of reality (the icy slope, the cold, my lack of skill) hit me full force. The experi- ence of playing the game was more emotionally fulfilling than actually being out on the real mountain. In the virtual world I was a snowboarding god, in the real world a hopeless amateur. I remember going home that day and returning to my PS2, and the “more perfect” virtual world it supplied me.19 For Michael, the experience of whipping down the slopes in a video game is more emotionally charged and satisfying than the same experience in the physical world. We may have to confront the fact that physi- cal, unmediated reality is simply not as highly engaging or highly personalized as what we experience online or in digital representations of physical occurrences. We are, in fact, creating online worlds that are “better than real” in terms of the quality of experience they enable. As we are able to create “better than real” worlds, we will need to start enhancing the quality of our physical experi- “You see it not only in avatar-based ences in order for them to remain compelling. meta-worlds, or chat rooms, whatever you want to call them, but you see it even with, you know, online dating and stuff. You know, you read a profile, you email back and forth with people, you create a construct of what that person might be like, and then you make a decision whether you want to actually find out how much the reality matches your construction of that person.” — Keiko Takamura 19
- 25. BLENDED REALITY superstructing superstructing C H A P T E R 5 Virtual environments have enabled a small group of early users to have an unprecedented amount of intervention into their own experiences, both before and after the experiences happen. People use virtual or mixed-reality environments to preview or rehearse tasks ranging from performing live in front of an audience to performing life-threatening surgeries. Blended reality “redos” allow us to relive specific, often difficult, experiences or environments which eventually lessens the negative psychological and physical trace effects of those difficult experiences and environments. Today, there are two distinct paths for the development of reality preview and redo pioneered by those with a great need to change the nature of their experience or help other people do so. The first path consists of the most advanced immersive environments in existence today. Created in the lab and used for high-stakes institutional practices such as military combat, risky medical procedures, or global collaboration in large enterprises, these immersive virtual environments are most often mediated by trained experts and experi- enced with a team of people. The second path is more lightweight, built and controlled by the user, and used for managing psychological and physical effects that are seen as barriers to individual achievement or happi- ness, such as shyness. In the coming years, the rest of us will be able to use the same tools to preview and redo a much larger range of experiences. What follow are some examples of where and how preview and redo provide useful training for real life. “We’re trying to train the mind to filter out the unimportant and deal Medical professionals and first response teams are among those using blended reality environments with the important ... Our intent is to train people in teamwork, decision making, and action under stressful conditions. These programs aim to immerse the participants in physical/digital training drills to create a sense of déjà vu when to be able to inoculate the Marines participants experience the real thing. For example, when new emergency room staff encounters so they’re less inclined to be a dying patient, they know what to do right away because, in a sense, they’ve already experienced it. They’ve practiced dealing with specific, tricky medical situations on virtual patients and have to a casualty.” some extent already processed the emotions that can cloud judgment and take a toll on physical — Colonel Clarke Lethin performance. Outside of medial applications, large organizations like IBM are letting employees loose to practice datacenter management in virtual datacenters—without endangering real life bits and bytes. 21
- 26. 5 | PREVIEW AND REDO But the most powerful state-of-art reality previews are taking place in the military in efforts to reduce the risk of battlefield casualties and trauma. As Chief of Staff for One Marine Expeditionary Force at Camp Pendleton, California, Marine Colonel Clarke Lethin has overseen development of the Infantry Immersive Trainer, a “mixed reality combat environment” designed to preview the combat experience in Iraq. Colonel Lethin and his team have recreated the physical scenes of an Iraqi village in an old warehouse, complete with sounds, smells, sights, 2D interactive avatars, and Iraqi actors who play the part of civilians caught in the midst of battle. A platoon goes through the village simulation repeatedly, learning more each time about how their bodies and minds might respond to an incoming RPG or an injured civilian. Mistakes that would be deadly in real life can be made with little consequence, but those mistakes now become learned lessons when The U. S. Military has been heading into real-life combat. using previews and redos to reduce battlefield casualties and PTSD among soldiers. The anonymity of the Internet has always been one of its most powerful dimensions. Anonymity in blended reality is unfolding in increasingly comprehensive virtual environments where audio and video streams from the real world are woven into the experience. Such environments are helping some people practice and learn how to do things—have dinner with others, share art, have an informal chat with a colleague—which have been unavailable to them under real world conditions. The stakes are high for these individuals: they are rehearsing human connection. Keiko Takamura was recently diagnosed with post-traumatic stress disorder (PTSD) and suffers from such severe stage fright that it has prevented her from doing the one thing she most wants to do: express herself through music and share it with others. Despite this, over the past few years Keiko has become a popular performer in the virtual world of Second Life. She plays at least three evenings a week, gets “I’m an avatar who wants to be a paid in Linden Dollars (the currency of Second Life, good for in-world products and services), and has developed a fan base. At home with her guitar, she sits in front of her computer, mic, rock star.” and mixer, and logs into the stream provided by her host, usually an in-world venue owner. — Keiko Takamura She logs her avatar into the virtual venue and starts singing on a virtual stage. Nearby is her virtual tip jar and merchandise table where fans can buy virtual t-shirts for their avatars, video clips, and MP3s. At the counter there may be free coffee and donuts, and a laptop that takes you to Keiko’s MySpace page. Playing in Second Life has allowed Keiko to slowly overcome her debilitating stage fright. But what she really wants to do is turn these experiences into successful performance in the real world. Sometimes she bridges the two worlds, performing live in San Francisco with a group of local Second Life musicians. The live shows are broadcast into the Second Life venue with the musicians’ avatars recreating the real life performance in the virtual world. Playing in Second Life has allowed her to practice overcoming her stage fright, and to learn how to sell her music and interacting with fans and other musicians, some of who she considers mentors. 22
- 27. BLENDED REALITY superstructing superstructing Virtual reality exposure therapy is a very promising, leading edge application of virtual reality environments to treat the traumatic effects of things like military combat, abuse, or assault. Obviously we can never undo what has already happened, but it turns out that we can alter our brain’s reaction to memories of the past and dramatically reduce the ongoing effects of past trauma. For those with post-traumatic stress disorder—a diagnosis now thought to apply far beyond military combatants—blended reality “Memory is a constructive process. therapies are helping to slowly ease the psychological and physical impact of the memories as [Our therapeutic process has well as increase tolerance for conditions that would historically reactivate stress. to] leave room for patients to At the University of Southern California’s Institute for Creative Technologies, Dr. Richard “Skip” Rizzo runs the Virtual Reality (VR) Assessment and Treatment program for Combat-Related insert their own memories and PTSD. Dr. Rizzo uses heavily immersive applications including sounds, vibrations, and custom- ized VR environments to recreate a soldier’s personal traumatic experiences. The immersion experiences. ” enhances patients’ imaginations, allowing them to retrieve memories that are often buried or — Skip Rizzo intolerable without support. As they move through the program, patients report their anxiety rating in terms of “subjective units of discomfort” and doctors compare it with their heart rate. In clinical terms, Rizzo describes this as “graded and repeated imaginal reliving of the traumatic event … in a low-threat context where the patient can begin to therapeutically process trauma-relevant emotions as well as de-con- dition the learning cycle of the disorder via a habituation/extinction process.” Patients learn new responses to feared stimuli so that the sound of a car door slamming down the street, for example, doesn’t retrigger the trauma of urban warfare for an Iraqi vet. Those who have the greatest need to improve their current lives are pioneering a new approach to experi- ences, and showing the rest of us what might be possible. There are myriad situations to which we’ll apply pre- view and redo. Organizations will use these tools to train new employees, while large corporate partners could produce collaborative mass simulations (think Virtual Financial Crisis). Individuals will improve their confidence in real-world social interactions by practicing the ones that make them nervous beforehand. experior (verb) Latin Furthermore, mixed physical and virtual environments allow people to move beyond their primary psychophysiological responses and move to their higher mental functions more quickly. This results in better to test, put to the test, performance in chaotic situations, less stress, and increased safety for everyone involved. These environments ascertain; to try, test, will be used for business and public service. experience, prove. These new capabilities constitute a new approach to the nature of an experience. Conventionally thought of as something that happens to us once, at a specific time and place, an experience is becoming more like an ongoing experiment. Experiments are tests, they are repeatable, and they are reproducible. Preview and redo bring us closer to being able to do all of that—with real life. In the future we may see the emergence of a kind of “reality GPS,” where the experience of reality is a terrain to be navigated based on previously drawn maps, real-time data, and post-experience revisions. In such a world we could reduce risk, increase performance, and be more psychologically prepared for challenges. But with everything so premeditated and rehearsed, would we also reduce spontaneity and randomness? 23
- 29. BLENDED REALITY superstructing superstructing C H A P T E R 6 Sensors, programming tools, and geo-locative technology are enabling living and non-living things around us to communicate through the common human protocols we use to communicate with each other. A growing group of hackers, developers, and artists are building products and systems that allow end users to give objects personalities by assigning them voices and accents, or by tailoring a list of messages the object can send to fit a desired personality. Animals, plants, and things are blogging, using Twitter, and even mapping their movements via Google Earth and Google Maps. Some early terms have emerged to describe these networked things, both living and non- living: blogjects (things that blog), tweetjects (things that tweet), and the broader concept of the “Internet of things”—the idea that things around us are connected to the Internet, and are a part of the same social web we have come to know. While some of the current applications seem novel, many have practical implications for real-world interac- tion. Within this experimental world of communicative and present objects, we’ve identified a few key emerg- ing practices. The objects and non-human living things in the world around us have a lot to say about our world but no way to communicate … until now. Sensors and GPS connected to blogging and microblog- “There’s a lot of really relevent ging applications are allowing us to engage with things in our environment by giving them access things happening all around to our own social tools. But since not everything can be easily networked (yet), experimenters us that are easy to dislocate have to figure out what to start with. The first step is to decide which non-human phenomena and processes we want to better understand. Then we have to decide which objects would best help us yourself from, especially on the do that—which things we want to tell their stories—and engage them, whether by equipping them with sensors and translating that data into blog posts or using Google Maps API to mashup GPS natural scale.” data with a map. — Kati London In 2006, Beatrice da Costa’s Pigeon Blog project involved four separate releases of 16 pigeons equipped with GPS-enabled air pollution monitors.20 The sensors transmitted the carbon monoxide data through an online blogging/mapping tool that subsequently appeared on the web. The information gathered by each bird can be viewed by clicking on that bird’s name. The pigeons blogged and mapped the information as they flew. Another animal mapping project comes from Eugene Popatov at Bryn Athyn Col- lege in Pennsylvania. Using mail-to-map forwarding, GPS data transmitted from William the deer’s collar maps 25
- 30. 6 | SENTIENT WORLD his location every five minutes on a Google Map..21 In both of these cases, the only human intervention is the initial arming of the animals with collars (or in the case of the deer, finding new subjects when the batteries in the GPS collar die). The animals are essentially blogging and mapping on their own. While only a handful of DIY-types, technologists, and researchers are experimenting with the Internet of things today, one of the underlying principles in this blended playground Beatrice da Costa’s Pigeon is that everything can be a node on the network. If you find a way to connect something, it can be a node on Blog followed 16 pigeons the network, whether “it” is a dog, wastebasket, doorway, toilet, or tree. Many things are already networked and tracked the air pollution they came across in flight. with GPS but that data is highly technical and not nearly as friendly to view as it would be through other communication protocols. The Red Funnel Ferries, which run between the Isle of Wight and the mainland United Kingdom, have sent GPS data for a long time. But only recently did Andy Stanford-Clark enable them to publish their locations through Google Maps API that anyone can access online. The ferries also tweet when they arrive or depart.22 This makes it much easier for the average rider to follow the real-time informa- tion provided directly from the ferries instead of relying only on a schedule or a more technical maritime map. Just as things online communicate with us, we can also enable things to communicate with each other. Botanicalls began as an experiment in the ITTP lounge at New York University.23 Researchers placed sensors in 13 plants in the lounge, connected them to a network, and enabled them to call the phone in the lounge when they needed water or light. The plants queried each other first to compare conditions. A plant in the dark could ping other plants in the room to see if they were also in the dark. That information allowed the plant to determine whether it was dark in the room or if it was just in a dark spot and needed “One of the things that we talk to be moved close to a window. The plant would then call the lounge phone and let the about is this idea of everything answerer know its needs. On what may be a more practical scale, smart home technology is putting our appliances on the grid, allowing them to communicate with each other and with being a node on the network. So the grid, to determine when to go into power saving mode to conserve as much energy as why cant my plant let me know possible—and save us money in the process. when it needs to be watered? And everyone’s talked about this for a long time: why doesn’t my Once engaged, and once the common language is established, it’s possible to give objects in our environment personalities that will make them more like friends than things. By tapping refrigerator tell me when my milk is into the world of scripts and programming, we can translate data into more common human bad and things like that? ” protocols and push that information through the digital communications channels we use. Individual voices speaking our language provide a fun experience, thus increasing engage- — Nick Bilton ment with the things around us. For examples, Botanicalls turned their lounge experiment into a consumer kit that, using a combination of 26
- 31. BLENDED REALITY superstructing superstructing soil sensors and Arduino, allows plants to express their watering needs via Twitter. Users can “We got really interested in the idea customize the standard messages included with the system, giving them personalities that of talking objects in general but make them more fun to hear from. Another example from the botanical world is Midori-san (Mr. Green), a blogging plant created by KAYAC Co., Ltd.24 Midori-san lives in Donburi Café then thinking about sustainability in Kamakura, Japan. According to the blog, Midori-san is particularly fond of making plant- and what’s worthwhile to have themed puns that, unfortunately for us, don’t translate well to English. There’s also a widget that can be embedded on a blog or a Facebook profile that allows you to click and turn on conversation with. We thought it the fluorescent light next to Midori-san, and Midori-san will thank you for the photosynthesis would be really interesting if you boost. could take as much data and as much scientific hard info about living things around us, and give A key motivation for the early experiments in this area of blended reality involves exploring sustainability. As humans, we’re limited in gathering information just because we can’t go it access to the more common where other things can, or things we try to gather information about don’t want us around. human protocols. ” But animals and living things in our environment can be harnessed to gather the kinds of information we can’t. What better way to understand our world than to empower the things — Kati London around us to tell us about the world from their point of view? Using these things that already exist in our living world can help us better understand the living world itself. Botanicalls recently ran a sustainability art project called “A Tour Led by the Plants of Manhattan.” 25 Each tree had its own personality and users could call the provided tour number, select the menu item that cor- responds to the number of the location where they’re standing, and hear about the trees and history of that location told by voices meant to represent the botanical profiles of the trees. For instance, a more demand- ing plant had a bit of an attitude compared to an older tree with a calm persona. Some of the plants argued, others flirted, but all of them engaged the participants with a green thing in the middle of a concrete jungle. Andy Stanford-Clark, who lives on the Isle of Wight, has networked his entire house and enabled it to send messages about what’s happening—electric usage, water consumption, phones ringing, etc.—to Twitter.26 With increased awareness comes a feeling of greater responsibility. Wouldn’t you rather have your house tell you that you’re using way too much electricity than wait for the bill from the power company? When something speaks a language different from that in which we are naturally fluent, we are less likely to pay attention to it, or often simply don’t have the ability to communicate with it. Enabling the living and non-living things around us to communicate on our terms and in our language allows us to better relate to non-human processes and phenomena by provid- The Botanicalls sensor ing a framework for reciprocal communication. We can much more easily understand information from a liv- allows plants to ing or non-living thing provided its tweets show up on our Twitter home page or its blog comes through our communicate with their owners to express RSS reader. Even GPS data is easier to comprehend when it’s presented in the form of a familiar Google Map. their needs—you can Giving objects presence makes them active participants in the conversation instead of passive standers-by. even give each plant a personality. 27
- 32. 6 | SENTIENT WORLD And by adding in a touch of personality, the line between “that” and “us” blurs even further. Filtering data through human protocols makes it more accessible so that technical data becomes more easily parsed—and more fun. When something is fun, it tends to be more engaging, and when we are engaged, we are more open to explore and understand something in which we might otherwise have no interest. As more things come online, we’ll not only have people vying for our attention, we’ll have to deal with things, too. With so much information coming at us, will a networked, sentient world make life easier or lead to cognitive over- load? How will we filter and manage information so that needy objects don’t overwhelm us? 28
- 33. BLENDED REALITY superstructing superstructing C H A P T E R 7 Like pioneers who discover and settle new lands, early adopters take pride in forging new digital frontiers. The first users are essential to setting the tone, whether it’s in a social networking space, an online discussion forum, an open source project, or even a cultural community within a virtual world. Early settlers also build infrastructure, and thrive in the chaos of semi-lawlessness as they develop their online codes of conduct. These pioneer users are also often beta testers upon whose feedback developers and community managers rely to push the direction of the product, particu- larly when it comes to social networking applications and open source, user-contributed projects. While most early adopters tend to accept that the digital space will eventually change as the number of new users increases, many of them adopt an “I was here first!” attitude regarding the web, becoming defensive and intolerant of newcom- ers (n00bs) who violate the stated and understood rules of the settlement. But their claims to ownership can be justified by the following practices in which they engage to establish the settlement. Flickr establishes its rules of conduct in its terms of Major sites like Flickr and Yelp have rules for conduct that are stated in their terms of use. But when a digital use, but other sites rely on community first appears, this is often not the case. Early users or developers of an online community may early users to establish a adopt a sort of charter for behavior, often unspoken but well known to the early adopters. Most of the time more informal code people expect that users behave as they would in real life, and generally the rule is simply to be nice. In the of ethics. case of a community like the Second Life music community, courtesies are often constructed based on the way a similar community functions in real life. The Second Life music community operates a lot like the urban coffee house music scene, where users mentor, admire, and support each other. The rules can be as simple as searching a forum for an existing thread before posting a new one on a topic that may have already been discussed. On the other hand, trolls and griefers exemplify bad behavior, posting incendiary comments and responses that violate the established behavioral standards. Online communities, spaces, and lifestreamers like those on Justin.TV frequently select moderators who monitor user behavior and 29
- 34. 7 | DIGITAL PIONEERS in extreme cases ban users who refuse to play nice. Second Life landowners are allowed to develop covenants for their pieces of land and evict anyone who violates those covenants. Every community on Flickr has its own moderator who controls the rules and monitors submissions based on those rules. “Especially because there’s such a Often, early users of a social media sites will establish customs and later, when that online communication between the guys space becomes the property of another company, resent the new terms of use the company that are creating it and running it, puts in place. This was the case when Yahoo! purchased Flickr, and suddenly things were cen- sored and banned that the company deemed inappropriate but that users had considered the game is constantly evolving acceptable and part of the site’s identity. The non-negotiable new rules led the site to censor and so is my interaction with it and the photostreams of many who posted nude photographs—the old art versus pornography debate—without any notice. how it interacts with the rest of ” my life. — anna one Beta testers and participants in open source projects often take a sense of pride and owner- ship in the project which they’ve helped to bug test or to which they’ve contributed some code. Early users have often communicated directly with platform and application developers to help them better understand what users want and need. Bug testers of open source media player project Songbird have been instrumental in the product’s development between releases. After the company’s recent 1.0 release, long-time followers of the project immediately noticed that the face of the project had changed: a new dock icon looks nothing like the black bird that had been the mascot of the company since its inception. Many users turned to GetSat- isfaction, a Web 2.0 site for social user feedback and customer service, to express disappointment and share their own ideas for icons that honored the original bird.27 GeoJSON, a format for encoding geographical structures, was developed entirely by its user “The rules of the covenants allow community and is protected under a Creative Commons Attribution license.28 The users com- me to take his land back with no municate through a GeoJSON subscriber list, posting threads and asking each other questions. Currently 20 projects have adopted the community-developed format. SF0, a collaborative payment of any kind. He’s just gone production game created by Ian Kizu-Blair, Sean Mahan, and Sam Lavigne, relies on users to because he broke the rules. I was not only test out various iterations of the platform, but also to create new game tasks in order to grow the game.29 Without a sense of ownership on behalf of the players, there would be no nice about it; I gave him back the game. amount he paid for them and said, ” ‘Sorry, this isn’t going to work’ — Lance Koenders With the world turning online for entertainment, there’s a natural push to leverage these communities for commercial gain. Yet, natives see marketing attempts in their spaces as disin- genuous and disrespectful. Commercial invasions will usually be met with disdain from the tightly knit group of individuals who established the practice in the first place. According to Keiko Takamura, major musicians doing shows in Second Life are seen as disrespectful to Second Life’s community of regular users who perform in the virtual world. Bono and 50 Cent’s performances came across as apathetic to the real community of musicians within the world. Nobody wants to see their communities sell out, not even in digital spaces. Meanwhile, early users of Yelp helped to make the site a useful resource, posting honest reviews of local busi- nesses. The variety of perspectives made the reviews well rounded, unlike the single opinion of a newspaper 30
- 35. BLENDED REALITY superstructing superstructing columnist. But according to Yelper Jeff McClure, the influx of people who use the site as a forum to whine about their lives degrades the quality of the reviews, and thus the usefulness of the “They don’t really care about the site. community. They don’t really care In some cases, a community will develop such tightly knit bonds that, once newcomers over- about us, they’re just here to throw whelm the space or the space is shut down, a core group early adopters and respected later a one off party and never see it adopters will migrate somewhere new. This was the case with Joshua Davis’ Dreamless.org, which, when closed, led many users to move to Mark O’Sullivan’s o8. Once o8 devolved, many again. ” users moved over to YayHooray!, created by the guys behind Threadless, and still up and oper- — Keiko Takamura ating today.30 Finally, however, an influx of new users whose signal to noise ratio was much more noise than substance led the longtime users to migrate once more and two main spinoff forums now exist, one mostly for international users, and one that houses the predominately U.S. branch of the migrant YayHooray! community. Mark O’Sullivan’s lightweight Vanilla forum software has made it easy for anyone with a server and a little bit of know-how to set up a private, invite-only Internet discussion forum.31 Fitting into a community online is not much different from fitting into a community in the real world. You have to learn the rules, the customs, and the etiquette before you can truly be accepted. Violating those terms, unspoken or not, can cause embarrassment and huge setbacks in gaining respect and acceptance. Meanwhile, communities themselves have to find a balance between protectionism and acceptance. Nobody wants to see their community destroyed by trolls, griefers, or people who just don’t know how to behave. But at the same time, a community cannot survive if it can’t grow a little, so there has to be some willingness to let new people in. As companies try to leverage the social tools that are becoming second nature to many of us, they must remember that they, too, have to respect the community. Spammers will always be spammers. But savvy marketing and promotions can work as long as they don’t violate the culture of the community. The people who comprise that community worked hard to make it what it is, and don’t respond well to a marketing ploy that disregards the standards they’ve set. Furthermore, in terms of acquisitions, a company acquiring another company that already has a loyal community has to be careful to not alienate the exist- SF0 depends on its users ing community. After all, the value isn’t just in the product; it’s in the pioneers who settled there, used it, and to both play the game and provide feedback on made it something of note. the site. 31
- 37. BLENDED REALITY superstructing superstructing C H A P T E R 8 We depend on various types of software tools for information, memory storage, retrieval, and even survival. The data we create as part of our everyday processes of thinking and being manifest in changes in our behaviors, our interactions, and our cognition. Essentially, our technology contrib- utes to real changes in our bodies and our minds. Increasingly, these interactions with software and devices are not only personal, but also embedded in rich social connections and communities. How will this new visibility change our behaviors? In what new ways will we be persuaded— or manipulated? When we say “change,” or even “dependency,” we are not referring to the sudden sharp dependency of a pacemaker or other life-saving implant intervening in the body’s functions. What is interesting here are the slow and incremental changes of feedback cycles over time. Years of self-tracking, simulation, and minute interventions result in a healthier, more stable, and long-lived body. Minute changes in virtual experiences can alter our physical bodies and real-life behaviors, as well as our interactions with others. Central to all of these are iterative cycles, feeding back on one another: as we are exposed to the same patterns of data gather- ing and representation, our bodies and minds respond and change, for better or worse. The last several years have seen an explosion of self-tracking on web-based and mobile platforms: measuring everything from detailed data normally collected in clinical settings (CureTogether.com, for example) to blasé adventurism (Bedpost.com). Independent of the Sites like the Quantified tracking sites themselves, the phenomenon has attracted online communities of interest like the Quantified Self, created by Kevin Kelly Self, convened by Kevin Kelly and Gary Wolf, and FlowingData convened by Nathan Yau. While it is garnering and Gary Wolf, serve to some mainstream media attention, (see for example a recent Wall Street Journal article “The New Examined aggregate personal data while also providing a Life”) perhaps more interesting is the movement to re-integrate these user-led measurements into the community for its users. medical system in a useful way.32 Project Health Design, supported by the Robert Wood Johnson Founda- tion, posits a genre of medical record called “observations of daily living” (ODL). As part of the broader movement towards electronic personal health records, this project seeks to harness the impulses of self- tracking to work in medical contexts while empowering people to collect data about themselves as a form of engagement with their own health.33 33
- 38. 8 | OUR BODIES This is perhaps nowhere more advanced than in the rich communities that have emerged to support the management of chronic illnesses. Laura O’Donnell, one of our intervie- wees, stumbled into this world of self-observation while managing Type 1 Diabetes, a disease notorious for generating numbers. At first she just logged them on paper or in Excel and showed them to her doctors. But slowly, she started trying to find patterns on her own, and then began sharing her findings on her blog and various life-streams. Her audiences there were not always receptive, however, and there were limits to what she could do on her own. She started using a service, like those that are proliferating in the self-tracking world: Sugarstats.com, which would take her logged data and graph, model, and project it in ways that are useful for both Laura and her doctors. Sugarstats collects data for users with diabetes Part of the fascination driving self-tracking projects is the increasing accessibility of the models in which and allows them to track the data are now represented. The ease of visualizing the data to start with allows more complex functions data and create graphs in a and interactions: pattern recognition, projections, and extrapolations. These communities of self-tracking, way that is useful for both patients and doctors. self-modeling, and collective interpretation are creating the seeds of modeled systems from the bottom up, somewhat like the personal health simulations computed from the top down that we have been forecasting for years. Indeed, these too are starting to emerge. Entelos is a company creating personal health simulations both for pharmaceutical trials and self-health services. They use sophisticated mathematical models of each of the body’s core systems: the endocrine system, the metabolic system, and so forth, to show the interrelated responses of the body to differing treatments and interventions—to customize, even personalize, the simula- tions.34 They are thus able to show how different courses of behavior changes, pharmaceutical, and surgical interventions would affect a given person as they age, rather than an abstract average. This identification with the model changes the way that patients react to their future choices. We see this even more clearly with a different kind of model—the avatar. “And that’s where I started to really actually investigate the patterns. There are a number of experiments run by the Virtual Human Interaction Lab (VHIL) at And then, eventually, I was like: Stanford University examining the relationship of avatars’ movements and appearance to “this is what I need to do.” And this our physical bodies and our social contexts, among others. In some cases, just seeing the representation of our bodies in virtual reality changes the way we act, measurably. Seeing our is the thing that if somebody asks avatar’s bodies change—moving differently, growing thinner, growing fatter, looking more me what the best thing to do, like, or less like other people—has effects, possibly very persistent effects, on the way we act in the flesh.35 Likewise, Dr. Walter Greenleaf from the Greenleaf Institute in Palo Alto, is incubat- managing diabetes and doing it ing applications for therapeutic interventions from behavioral change to disability solutions well, I definitely had a moment based on the very premises of motivation and neuroplasticity that the VHIL is uncovering.36 when I realized that is what I Motion capture allows the visual experience of identifying with an avatar to spill into muscle memory, into embodied experience. Jeremy Bailenson’s lab has just started experimenting would say. ” with motion capture technology to allow two users to share virtual space for close motion — Laura O’Donnell mimicry, for example, to each Tai-Chi. 34
- 39. BLENDED REALITY superstructing superstructing One striking thing about many of these examples is the way they are embedded in social con- texts: the data we collect feeds into our identities and our interactions with others as well as “It’s one thing to share it with our relationship to our own bodies and minds. Sharing data, and the opportunities for collec- your health care provider, but tive sensemaking that such sharing creates, is an emerging feature of more and more services, and a user-desired feature when it is not present. Communities serve to reify the identity of an it’s another to share it with your illness or communicate a transformation, like for example, changing gender. Micha Cardenas, friends, whoever they may be. It a researcher and artist at University of California, San Diego and part of the Calit2 collabora- would first of all be other diabet- tion, is about to embark on a performance using motion capture technology in Second Life immersion in parallel with hormonal gender transition. One way she hopes this will feed back ics, because they have the same somatically is by teaching muscle memory for various kinds of gendered gestures, including experience with the blood sugar walking. Although not especially committed to Second Life as a community, Micha chose it as the platform for her performance precisely because of the social context: pushing the bound- numbers as me. But then, if anyone aries of bodily experimentation needed to be grounded in interactions with other people else is says, “Oh, you know, I work in who could react to the performance in its own medium. the health care domain, or I never thought about this, and I want to know more, this would be one way More data is being collected about our selves, our bodies, our health, and it is being collected in situ, rather than in centralized clinical contexts. More aspects of our lives are being consid- to get information. ” ered data in this renaissance of “personal informatics,” while at the same time the data that we — Laura O’Donnell collect is getting better, more accurate, and more granular. Especially as the long awaited bits of genetic and genomic information enter the mix, contentions around who owns, controls, and has access to this data will intensify. As medicine grapples with patient’s rights, doctor’s responsibilities, and HIPPA, users and startups are pushing ahead to define the kind of data that they would, not only, like to own, but also what they are willing to share and with whom. This data is being modeled in increasingly accessible, visual, and meaningful ways that strike chords with our identities: from charts and graphs to avatars, reflecting our senses of individuality and our selves in uniquely motivating ways. These models, from avatars like Micha’s to graphs like Laura’s, are being built, displayed, and interpreted in increasingly social contexts. As new psychological research shows, this digitally mediated sociality has profound psychological effects. While we currently see individuals taking control over their own self-representations, the danger still lurks that intensely psychologically loaded processes will be appropriated in the name of commerce and productivity. 35
- 41. BLENDED REALITY superstructing superstructing F O R E C A S T S We bring together our technology road map with themes identified in our ethnographic research as well as signals from other sources to develop forecasts or foresights of big transformative shifts in our society, organizations, and the larger environment. These are not predictions, i.e. pointed forecasts of when something will happen and what will be the market share of this particular product or service; rather, these can be seen as waves that will be unfolding over the coming decade and beyond. It is essential for organizations to understand these shifts or waves in order to both prepare for them and, in many cases, help shape them. Here are the key shifts we want to bring to your attention: Transition from communications to performances. We have come to view many of the existing media tools and technologies—instant messaging, e-mail, texting, mobile phones—as a means of exchanging information and communicating with each other. The new generation of social media tools—Twitter, YouTube, Facebook—are increas- ingly changing the nature of our interactions with each other from communications to public per- formances. At the core of this transformation is the individual awareness of and constant presence of an external audience that provides instant feedback loops for our performances. It is important to understand this transformation because issues like privacy—which have increasingly become of concern of organizations and individuals—take on a new dimension if we conceive of what we are doing as performances rather than communications. In such performances, being interesting, being able to attract fans/followers, often trumps the privacy concerns. Rising access to each other’s life streams. Our Twitter streams, Facebook pages, lifecasts, Flickr streams, and more, are enabling continuous digital streaming of bits of our lives—what we are doing, how we are feeling, what we are thinking, who we meet with, etc. These streams are open to our network and, increasingly, to the public. Thus we are able to come in and out of each other’s streams and do so on a scale unknown before. If you follow 2000 people on Twitter, every time you open Twitter or one of its plug-ins, you come into contact with the lifestreams of those 2000 people with virtually no effort. Think about how long it would have taken you to acquire this information before. And think of how this will transform what you pay attention to, the breadth of what you know/see about your social network, near and far. We will have access to the lifestreams of those we know or hardly know at all; in fact, “knowing” acquires a whole new meaning as you may know more details about a person’s life who you follow on Twit- ter than a close friend who doesn’t “tweet.” We will acquire a new level of “knowledge” about others and new expectations about participation in each other’s lives. Will having a lifestream become an expectation in your organization? 37
- 42. FORECASTS Increased desire for digital immortality. We increasingly document and archive our life streams—from photos and videos to various types of health and identity data. In the process, the data about individuals is becoming more easily aggregated, searchable, and measurable. In many instances, such data migrates from individuals into vast data banks on populations, making it possible to discern new patterns and provide more fine- grained simulations. With enough data we will be able to simulate not only group dynamics but also individual life scenarios. Would we be able to simulate how someone might approach a problem? What their reaction would be in a particular situation? Simulations are only as good as the data we feed them and algorithms we use. There is no question that our data is getting better and our pat- tern recognition tools are becoming more and more sophisticated, so it is reasonable to expect that our simulations will become more realistic and accurate. Emergence of a mediated oral and visual culture. As the Web continues its metamorphosis from a text vehicle to an image (or, moving image) vehicle, spurred by the availability and adoption of such technologies as the phonecam, the webcam, voice (and video) over internet protocol (VOIP), tiny handheld cameras like the Flip, and online video platforms like YouTube, it seems ever-more likely that video will overtake the written word as the predominant communication form for coming generations. We are seeing the emergence of a new digitally mediated oral society—one that will alter the way we shape our identities, communicate knowledge, create authority, and experience the world around us. A new public sphere, bringing together the semi-literate as well as the hyper-literate, will generate new channels for art, com- merce, politics, and education. Video comes with its own language, a language with a multitude of spontaneously generated and evolving vernaculars. We are in the early phases of evolving this vernacular and creating new ways of producing video content. As video becomes a predominant means of interaction, individuals and organizations will need to acquire new skills in creating, com- municating, and interacting with video content. Emergence of “better than real” worlds and environments. Using lightweight digital technologies, we will increasingly create our own worlds and structures that fit our individual needs and contexts. In many cases, the worlds we will be creating will be more emotionally charged, more appealing, more satisfying, i.e. more “perfect” for us. Institutions, practices, and rules created to serve the unmediated masses will undergo dramatic transformation or will simply disappear as individuals and groups increasingly chip away at them, re-creating rules, re-designing ways of working, learning, and socializing that suits them. 38
- 43. BLENDED REALITY superstructing superstructing Blurring of boundary between “us” and “that.” As computing, sensing, and imaging capabilities migrate from traditional communications and technology tools such as computers and cell phones, into everyday objects and environments around us, inanimate things around us will acquire identity and presence in our lives. This is what, in the past, we have called “sentient world.” These newly sentient objects will be producing their own life streams and will increasingly demand new levels of attention as well as bring new levels of awareness into our lives. Like the organizations concerned with animal rights, the number of other citizen groups organized on behalf of sentient beings will only grow. These groups will be newly empowered with data and documentation of what things around us need, “want,” and how they behave. By adding human-like features to these life streams (human voice, human features, Twit- ter streams, and Facebook pages) and many things we considered non-human will acquire more human-like identities. Growing sense of an individual as a node on many networks. As our networks become increasingly portable and we have access to members of our network (however we define the network), not on an episodic but on a continuous basis, we increasingly think of ourselves as one of many and act on that knowledge. Don’t know something? Post a ques- tion on your Twitter stream and within five minutes you might have a great answer. Watching a political debate when away from home and need some companionship to share reactions with? Go to a Twitter group created for just that purpose or create one on your own and get an immediate sense of being in a room in the company of others. Again, our institutions, particularly the educa- tion system, have not come to grips with the fact that people today, particularly young people, see themselves as not one individual but as a network. In many cases it is not as important what one individual knows but what she knows plus what her network knows. We need to develop new metrics for performance that acknowledge this networked view of self and recognizes that a metric such as “network intelligence” is a good measure of someone’s worth. Increased savviness in personal branding and promotion. Continuous feedback loops with our multiple audiences—our Twitter followers, Facebook friends, blog readers—will re-shape our sense of self. We are increasingly able to create multiple identities, multiple senses of self, in these different channels. For some, these channels will provide platforms for integrating multiple identities—home, work, hobbies. For others, they will provide platforms for carving out and tending different identity or self profiles. In either case, individuals will increas- ingly gain awareness of tools to shape identity and ways of manipulating their identities to achieve desired effect. Identity play will assume a whole new meaning as people truly express themselves and play with their personas in new ways. Advertisers possessed unique skills in shaping consumer desires, needs, and in achieving particular emotional impacts. Increasingly such skills are migrat- ing to individuals as they become skilled at creating performances, “packaging” their personas, and attracting audiences. The secret is out—we are all turning into advertisers. The result: selling products and services through traditional advertising means will be increasingly difficult as we are all media experts now and can deconstruct a packaged message. To reach desired audiences, companies will need to find ways of coming into our life streams in new and authentic ways. 39
- 44. FORECASTS Rise of New Taylorism. Frederick Taylor revolutionized the assembly line by taking careful measurement of many parts of the production process. Then he applied these measurements to designing assembly line plans and processes that minimized waste and increased productivity. Today, we are increasingly document- ing many aspects of our life and work far away from the assembly line. Much measurement and documentation also occurs without our knowledge—GPS trails in cars we drive, call logs on our phones, books we buy on Amazon, ads we click, how we long we look at ads, etc. As more and more behavioral data is collected, some of it visible, some invisible, the temptation to use such data to improve productivity or to optimize processes in areas far removed from assembly line—knowl- edge work, education, learning, will grow. Our reliance on software for many decisions previously based on intuition will also increase. Organizations will have to navigate the line between wanting to make their employees more productive while not making such attempts seem oppressive and “big brotherly” and taking away their employees’ sense of agency. 40
- 45. BLENDED REALITY superstructing superstructing F O U N D AT I O N P I E C E Blended Reality is the new reality. It is the new way we experience and understand life. The crucial filtering systems that mediate our interactions and activities are changing. The blending of physical and digital information and processes will permeate every area of our lives: health, social, financial, recreational, civic, and personal. This transformation will be driven by our technology: computing will become more mobile, broadband net- works will link people worldwide, sensemaking software will extend our senses by making information from widely diffused sensor networks available and comprehensible; additionally processor designs and network- ing will give the power of supercomputers to ordinary users. As a result, people will engage in new kinds of deeply immersive digital sensory experiences. This piece presents the key foundational technologies that are driving this transformation, they include: User Experience Technologies: flexible high-resolution displays, augmented reality, wearable devices, and tactile and gestural interface. Mobile, Local, and Cloud Infrastructures: always-on connectivity by fixed and wireless networks, supercom- puter-level computing power for everyone. Sensing Technologies: personal device positioning, sensor networks, image mining, pattern recognition, and motion tracking. Software Ecologies: code ecosystems for identity, sensemaking, mobile computing, and immersive worlds. 41
- 46. FOUNDATION PIECE Displays everywhere will dramatically alter the depth of sensory and tactile immersion experiences. Today, interaction with digital displays is a deskbound or device-dependent experience. However, develop- ments in display technology will enable a new form of interaction with digital media called ubiquitous com- puting. In ubiquitous computing, the physical location of data and processing power is not apparent to the user. Rather, information is made available to the user in a transparent and contextually relevant manner. A single display device restricts the repertoire of interactions between the user and digital media, so ubiquitous computing requires displays wherever the user might need one—in appliances, tabletops, public transport, walls, etc. Ambient displays will also emerge. These communicate on the periphery of human perception, requiring minimal attention and cognitive load. Sites for ambient display technologies include: Tabletop workspaces—horizontal flat displays that support multiple users moving around a common work surface Smart walls—large-format screens that seamlessly display users’ personal work environments over broadband wireless connections Chairtop work surfaces / control pads—seating with embedded digital controls for interacting with ambient displays Web signs—digital signs that are actually flexibly programmable Web displays for specific purposes Public display boards—displays similar to Web signs that serve a more general function as displays for news and mobile workers’ transitory interactions as they pass by Paper-thin digital displays—e-paper and textile displays enabled by OLEDs (organic light- emitting diodes) and OLEPs (organic light-emitting polymers) Over time, technologies like organic light emitting diodes will enable inexpensive paper-thin video displays to be used anywhere. By the time HDTV is pervasive, an new generation of much higher resolution TV-like a prototype 32 mega- pixel, 8K (7680 x 4320) from NHK will be used in special venues. . Credit card-sized, always-on personal devices will be ubiquitous. By 2015, nearly every person on earth will be able to afford a credit card-sized, always-on personal device capable of providing web, video, audio, and location services. In response to the success of the iPhone, Google Android, and Nokia nSeries, competitors like Samsung and LG are selling their phones for as little as $79.00. These prices are subsidized by network operators, but within a few years, these kinds of devices will be as cheap as voice/text phones are today. Camera viewfinders on our mobile devices will commonly be used as browsers for clickable web links that are attached to real objects and places. Nokia, Geovector, NHK, and Google are already developing capabilities to see digital labels and hyperlinks overlaid on real time video in the viewfinders of their handheld phones. There are indications that mobile augmented reality could be introduced this year, and by 2012 or 2015 this form of browsing and tagging could be common. 42
- 47. BLENDED REALITY superstructing superstructing Haptic Interfaces will emerge as new ways of interacting with the digital world. Within five years, a significant percentage of our digital interactions with will be through tangible or gestural means. The Nintendo Wii utilizes gesturing while iPhone users pinch and poke their touch screen displays. Samsung Electronics and LG Electronics are racing to popularize their new “haptic” phones that gently vibrate in vari- ous patterns depending on their settings. Groups like the Tangible Media Lab are prototyping a whole range of new interfaces like the Audiopad, a matrix of antenna elements which track the positions of electronically tagged objects on a tabletop surface. Large format 3D displays are already in development at labs like CalIT2 in San Diego. As our modern computing infrastructures evolve, fixed and wireless infrastructures will become ubiquitous. There are several new important technologies to watch here: WiMax has a number of carriers deploying its networks worldwide. It’s capable of delivering 70 megabits per second to devices within 30 miles (50km radius). Intel has announced that they’ll soon be offering WiMax enabled processors. [In the U.S. Intel, Sprint, and Google have announced they are working together to deliver WiMax services across the U.S. starting this year.] LTE (Long Term Evolution), which is envisioned as a competitor for WiMax, will offer over 100Mbps of throughput. Verizon and AT&T have competing formats currently (CDMA and GSM respectively). LTE will start appearing in the U.S. in 2010 with mass coverage by 2012. Femtocell, according to Wikipedia, “is a small cellular base station, typically designed for use in residential or small business environments. It connects to the service provider’s network via broad- band (such as DSL or cable); current designs typically support 2 to 5 mobile phones in a residential setting extending service coverage indoors, especially where access would otherwise be limited or unavailable.” According to Technology Review: 37 Similar in concept to the Wi-Fi routers that many people use to blanket their homes with wireless Internet access, these little boxes instead provide a network for carrying the voice and high-speed data services of mobile phones. They’re designed to give band- width-hungry cell-phone subscribers the strongest possible connections at home. Wireless HDMI (High-Definition Multimedia Interface) is a connectivity protocol that delivers uncompressed HD digital video and audio signals to a TV or projector. Early generation chipsets are available from compa- nies like Tzero, using WiMedia, an Ultra Wide Band (UWB) system. Over the next 15 to 20 years, computational resources will also become widely available and abundant in power. Companies like AMD and Intel have already started building and selling chips with multiple proces- sor cores and have announced multiple plans for 8, 80, and even 800 Pentium equivalent processors on one chip. As processors become redundant on a chip, processing net useful power may increase at rates beyond 43
- 48. FOUNDATION PIECE Gordon Moore’s forecast 38 of a doubling of power every 18 months. Prices for computation will continue to drop correspondingly. Anant Agarwal, CEO of startup Tilera whose company currently ships a 64-core embed- ded processor forecasts a corollary of Moore’s law: Today, companies like Amazon, Google, Yahoo, Microsoft, IBM, Sun, Hewlett Packard, and others are all offering utility cloud computing, which links networks of computers into grids and clusters of cloud computing ser- vices. Over time, supercomputer capabilities like pattern recognition, datamining, simulations and modeling, and high-resolution interaction will be widely used for mass-market web services and new applications in media, gaming, and ubiquitous computing. Over the next ten years, our mobile devices will be able to precisely calculate our position in three dimen- sions. Personal devices, like iPhones, already include a robust toolbox of sensors that can tell applications, with some precision, whether we are using GPS and their wireless sources. These devices can even calculate motion using built in accelerometers. Software like Fireagle allows us to fuse information from these sensors, and to selectively decide how to share information about our locations with other people and services. Research labs and startups are prototyping finer grained positioning using photographic techniques that recognize patterns in image and can compare and match those patterns with the real world captured in a mobile camera. Earthmine, a company in Berkeley, California, is already building point-cloud databases of 3D models of cities [where each pixel can be positioned within 20cm of the exact location.] In the future, we’ll utilize sensors by embedding them in our environments as well as wearing them. A new projection says that by 2012 more than 3.4 million senior citizens in the U.S. will be using networked sensor applications to monitor and improve their health. We are beginning to see some software tools and technologies that enable the creation of startling new high-resolution, real-world virtual worlds that might serve as compelling frameworks for blended realities. These are a few of the new software ecologies beginning to emerge. Forterra is building a toolkit, called OLIVE, a virtual world modeled on the real world, for enterprise simulations. Both Google and Microsoft are building 3D versions of the real world. The Google version is a grassroots effort where ordinary people create models using a free tool called SketchUp. Microsoft is building digital 3D models by flying aircraft with special radars called Lidar that capture shapes as well as imagery. Sun Microsystesm engineers are building a digital replica of a building in Menlo Park, California using a toolkit Sun is building called Wonderland. 44
- 49. BLENDED REALITY superstructing superstructing At MIT, researchers from the Shadow Lab and X-Reality Project are building infrastructures for mixed digital and physical experiences: two-way blended environments. Avatars from a digital version of their lab in a virtual world are shown in place in displays around the real lab, and real people are shown in place in the virtual lab. Google has built the markup language KML enabling people to add hyperlinks to the real world in Google street view or elsewhere, the way people use html to add hyperlinks to a text page. At the same time, we are creating systems that allow us to capture our digital identities, agendas, intentions and itineraries, and other aspects of our lives. This information will be used to proactively compute the context of our digital experiences and to tailor the interaction precisely to our needs at any given time as illustrated in the personal identity ecology in Figure A . Life recording and lifeblogging will go hand-in-hand with other systems for capturing information. Life recorders will begin to record everything about individuals, and those recordings will be used as a tool to filter contextual data so that the information an individual uses is highly tailored to his/her context and needs. Figure A Personal Identity Ecology Experiences are filtered through personal digital identities and contexts Biometrics & Sensor Data Medical Records Dietary Logs Exercise Data Lifeblog Recordings Health Legal History & Friend Nets Licenses Fitness Address Books Registrations Distribution Lists Permissions Social Network Accounts Civics Networks Ownership Aliases & Handles Records Avatars Personal Identity Ecology Media Preferences Personal Profiles Bookmarks Workspaces Media Recommendations Data & Servers Work Information Subscriptions Training & Certification Education Web, Blog, and Wiki IDs Media Nets Accounts & Logins Games & Virtual Worlds Keys & Signatures Commerce Locative Media & Banking Subscriptions Preferences Accounts & Logins Purchases Biometrics Wish Lists Source: IFTF 45
- 50. FOUNDATION PIECE The Future: 3D Web of blended worlds. Technologies and infrastructures outlined here will enable us to create imaginative 3D environments and overlay them on the real world, creating a new kind of reality. The 3D worlds we will be creating will be highly context aware, i.e., able to sense variety of contextual clues about the person, object, or situation (people, roles, activities, times, emotions, location, etc.), and respond in context. What will enable this is a set of descriptive languages ranging from standard formal vocabularies (formal classification and categorization systems), and ontologies (map of relationships among objects or terms in a formal vocabulary). At the most informal end of the spectrum are the emergent practices of appli- cations like Flickr and Del.icio.us, where users assign their own tags to personal or shared data. When it occurs, coordination is achieved primarily by sharing lists of existing tags, which serve as folksonomies—or informal, emergent taxonomies. Attempts are underway to bridge these formal vocabularies with informal tagging, potentially making them more uniformly useful for contextual process. We will also be able to move seamlessly between the various worlds we create. One signal of this is ability to carry our avatars from one context to another, from conventional games to user-created worlds like Second Life and even into reality-based environments like Google Earth. IBM and Linden Lab research teams have recently successfully teleported avatars from the Second Life Preview Grid into a virtual world running on an OpenSim server, marking the first time an avatar has moved from one virtual world to another. It’s an impor- tant first step toward enabling avatars to pass freely between virtual worlds. 46
- 51. BLENDED REALITY superstructing superstructing E N D N O T E S 1 Flip Mino, http://www.theflip.com/products_flip_mino.shtml 2 Qik, http://www.qik.com 3 http://secondlife.com/ Yahoo! Live, http://live.yahoo.com/ 4 Justin.TV, http://www.justin.tv 5 BoingBoing, http://boingboing.net 6 Future Builders, http:www.future-builders.org 7 FutureBuilders MySpace, http://www.myspace.com/futurebuilders 8 Bloom, Paul. “First Person Plural.” The Atlantic, November 2008. http://www.theatlantic.com/doc/200811/ multiple-personalities 9 “End of Speculation: The Real Twitter Usage Numbers,” April 29, 2008, http://www.techcrunch.com/2008/04/29/end-of-speculation-the-real-Twitter-usage-numbers/ 10 BarackObama on Twitter, http://Twitter.com/barackobama 11 Yammer, http://www.yammer.com/ 12 “Yammer Takes TechCrunch50’s Top Prize,” September 10, 2008, http://www.techcrunch.com/2008/09/10/ yammer-takes-techcrunch50s-top-prize/ Dolnick, Sam. “Bloggers Provide Raw View Of Mumbai Massacre.” Associated Press. http://www.huffington- 13 post.com/2008/12/01/bloggers-provide-raw-view_n_147600.html 14 “Google’s goal: to organize your daily life.” Financial Times, March 22, 2007. http://www.ft.com/cms/s/2/ c3e49548-088e-11dc-b11e-000b5df10621.html “A Scrapbook: The Life and Times of Stephen Wolfram” http://www.stephenwolfram.com/scrapbook/index. 15 html 16 Memiary, http://www.memiary.com 17 National Digital Information Infrastructure & Preservation Program, http://www.digitalpreservation.gov/ 18 MyLifeBits, http://research.microsoft.com/barc/MediaPresence/MyLifeBits.aspx 19 Highland, Michael. “Can the virtual be enough?” http://blog.gamerthink.com/2008/10/can-virtual-be- enough.html 20 Pigeon Blog, http://www.pigeonblog.mapyourcity.net/blog/index.php 21 Deer Project, http://ep.faculty.brynathyn.edu/Deer/william.html; http://tinyurl.com/5vkv4x 22 Red Jets on Twitter, http://Twitter.com/redjets; Red Jets Gmaps: http://www.ais-live.co.uk/AIS%20Live/ solentferries.html 23 Botanicalls, http://www.botanicalls.com 24 Midori-san, http://plant.bowls-cafe.jp/index.php 47
- 52. END NOTES 25 “A Tour Led by the Plants of Manhattan,” http://confluxfestival.org/conflux2008/botanicalls-a-walking-tour- of-manhattan-led-by-the-plants-of-new-york/ 26 Andy House, http://Twitter.com/andy_house 27 Get Satisfaction “This is what the Songbird official release icon should look like,” http://getsatisfaction.com/ songbird/topics/this_is_what_the_songbird_official_release_icon_should_look_like 28 GeoJSON, http://geojson.org 29 SF0, http://sf0.org 30 YayHooray! http://www.yayhooray.com 31 Get Vanilla, http://getvanilla.com/ 32 Jamin Brophy-Warren. “The New Examined Life” Wall Street Journal, December 6, 2008 Technology Section http://online.wsj.com/article/SB122852285532784401.html 33 Project Health Design,http://www.projecthealthdesign.org/ 34 See IFTF Health Horizons “Global Health Economy Map of the Decade” SR-1002, 2006; IFTF Technology Hori- zons “A Model World” SR-1121, 2007 35 Jesse Foxx et al, “Virtual Self-Modeling: The Effects of Vicarious Reinforcement and Identification on Exercise Behaviors” pending publication in 2009 , See, for a number of relevant peer-reviewed publications on the topic, http://www.greenleafmed.com/ 36 wg%20publications.htm 37 Technology Review, http://www.technologyreview.com/read_article.aspx?ch=specialsections&sc=futurebiz &id=20293&a= 38 Gordon Moore’s forecast, http://en.wikipedia.org/wiki/Moore%27s_law 48