The World is the Screen - Convey UX 2014

THE WORLD IS
THE SCREEN
Convey UX 2014 | Seattle WA | Andrew Hinton | @inkblurt
Understanding Information Environments
(Note for PDF: this is a shorter version of a talk I’ve done over the last year; some slides and
content are newer, but some have been removed. For other versions, see slideshare.net/
andrewhinton )

Andrew Hinton | @inkblurt
INFORMATION ENVIRONMENT
http://commons.wikimedia.org/wiki/File:Derbyshire_Landscape.jpg
This is an environment. It’s a natural setting, full of physical, naturally occurring surfaces and
objects. It’s the sort of environment that humans as a species spent most of its evolutionary
history in.
Notice there’s a stone wall here. It’s something made by humans that adds structure to the
environment. It’s an object that impedes motion, but it’s also an object with cultural meaning
-- it’s telling us something about this place. Maybe it’s marking off owned property, or it
says “this is where my sheep are contained.”
>>
What if we think of this as not only an environment, but an information environment? In what
ways is that true?
That’s largely what this talk is about.
Because networked technology is becoming so pervasive, so much a part of the fabric of our
surroundings, that we need to understand the fundamentals of how humans comprehend
their environment, regardless of the speciﬁc technology.
And understanding that means understanding the sorts of information that are in the world,
and how they work together to create places, contexts, and meaning.
http://commons.wikimedia.org/wiki/File:Derbyshire_Landscape.jpg

INFORMATION
ENVIRONMENT
3
INFORMATION
ENVIRONMENT
These are two information environments that involve office supplies.
What is the difference, really, between shopping for them in a brick and mortar retail store,
and shopping for them through an online retailer like Amazon?
If we think of the physical store as an information interface, how much information is
conveyed, and of what kind, through one interface versus another? How useful is it for the
activity of deciding between one printer or stapler and another? There are things the digital
environment affords us that the physical one does not: we can compare things more easily,
sort them, search them; we can see more of a selection; we can see what others have said
about the products.

Subway station + Food store
Here we have a mashup where a place is simultaneously a subway station and a virtual store,
with pictures of products on the walls where shoppers can scan them for delivery or pickup
using the store’s mobile app. It stretches what we assume about how an environment works,
and even what a place is.

Google Glass
Leap Motion Microsoft Kinect
We’re being surrounded by sensors and digital agents that are trying to read our actions, or
layers of digital information added to what we see around us.
left glass image wired.com / others from google
Leap motion from their site
Kinect from blogs.msdn.com

Now, even your refrigerator can spam you.
http://www.bbc.co.uk/news/technology-25780908 bestbuy.com
Our environment is full of even mundane things that are networked and active in ways we
used to only joke about. Like the smart refrigerator that was recently caught being used as a
spam bot.

Understanding the nature
of our materials.
We’ve gone for a long time in our work being comfortable with knowing that what we make is
conﬁned to a speciﬁc screen -- a bounded interactive surface. But now *all* the surfaces
around us are potentially digital and networked. Walls and windows, coffee pots and shoes.
So we need to re-assess, what is the nature of our materials?
Potters have a deep understanding of clay; masons have a deep understanding of stone.
We need to deepen our understanding of information.

INFORMATION
Digital
Semantic
Physical
Information is a word we use a lot, but I don’t think that in our general practice we have a
common framework for what it is, and the role it plays in the environments we inhabit.
I tried ﬁnding a single, official deﬁnition, but there really isn’t one -- instead, there are many
different ways of looking at various things in our world that people call information.
The problem is, all of them are pretty useful and valuable. So rather than just pick one, I
decided to pick several and see how they work together and inform each other, so to speak.
These modes or ways we experience information are physical, semantic, and digital.

Physical Information
10100010
01001000
01110011
10100010
01001000
01110011
Digital systems transmitting
to & receiving from other
digital systems.
Digital
People communicating
with people.
Semantic
Animals (including
people) perceiving the
environment.
Physical
Physical information has to do with the ecological relationship between an animal and its
physical environment.
The green ﬁeld we looked at earlier was mostly made up of physical information that
creatures use to know where they can go and what they can do in those surroundings.
The way I’m framing this comes from something called ecological psychology, which has been
adapted for a branch of study called embodied cognition.

Ecological Information
"Pick-up"
Agent
Perception
Action
Ambient,
structured
energy arrays
Embodied cognition:
Body & Environment, not just Brain.
Embodied cognition sees the environment and the body as part of a larger dynamic, in which
the brain is a participant, but not necessarily the center-point. It may even just be a
supporting player.
A big part of this is *action*: we don’t perceive the world by sitting still and taking in sensory
data; we perceive by moving and interacting with the environment.
Much of our daily activity is unconsciously nudged and controlled by our environment.

“... there’s no such thing as
disembodied cognition [for]
living creatures.
All of our cognition has to
be embodied ...
Brains evolved in the
service of action.”
- Louise Barrett
http://podcasts.ox.ac.uk/embodiment-taking-sociality-seriously-audio
Louise Barrett, who wrote a great book on embodied cognition called Beyond the Brain, puts
it nicely -- -- there’s no such thing as disembodied cognition for living creatures, all of our
cognition has to be embodied because brains evolved in the service of action.
We didn’t arrive on the planet as brains that grew bodies to supplement their disembodied
brain activity. Brains evolved to supplement the abilities of bodies in environments. They’re
the product of the structures of their surroundings.

James J Gibson - Ecological Psychology of Perception
Now seen as pioneer of embodied cognition.
Some embodied cognition theorists have heavily adopted the work of James J Gibson, who
was a scientist of something called “ecological psychology” in the mid 20th century -- a ﬁeld
he was part of inventing.

Ecological Information
"Pick-up"
Agent
Perception
Action
Ambient,
structured
energy arrays
Gibson used the word “information” to mean the
elements we perceive in the environment that tell us
what it “affords.”
“INFORMATION”
Gibson purposefully used the word “information” for this body-environment relationship. He
argued that our bodies do something he called “information pickup” -- where the body’s
perceptual system detects structural cues in the environment that specify what actions the
body can take. And he argued that this happens directly, with little or no need for brain
processing.

The perceived functional
properties of objects, places
and events in relation to an
individual perceiver.
AFFORDANCE
Perception exists only insofar
as we perceive affordances.
Gibson also invented the concept of affordance.
For Gibson, affordance isn’t a thing you add to something.
It’s the result of a property in the environment that presents the opportunity for action to a
perceiving organism.
This is true in the built environment as well as the natural one; the structure of these stairs
interacts with energy in the environment that my perception picks up on, and my body
detects that the speciﬁc structure of the steps will take my body upward if climb them.
Affordance was and still is a radical idea, in the sense that Gibson meant it.
Affordance is the organizing principle behind *ALL* perception. We don’t perceive anything
unless it affords meaningful action for a given context.
Gibson even went on to discuss how higher-order combinations of affordances can be found
in the complexities of human culture. A mailbox, for example, has the affordance for sending
mail -- but only because we’ve learned a lot of stuff about the world around that box that
makes it a special sort of container connected to a broader cultural system.

We perceive environment as “nested,”
not in logical hierarchy.
We perceive elements in the
environment as invariant (persistent) or
variant (in flux). Invariance grounds our
experience (literally).
Invariance
Nestedness
Two of Gibson’s principles of
environmental perception.
Gibson created a marvelous system of principles and components that work as building
blocks for how we perceive our environment and its affordances. I can only barely touch on
them here, but I’ll mention a few important ones.
>> Invariance is the property of environmental structure that makes some of it persistent
over time, so that we can make sense of the whole. Terrestrial animals like us evolved on
stable ground, under a persistent sky. Substances and surfaces that persist allow us to learn
what properties they have, and what they afford us. This can be a useful idea in digital design
because in software, even though we can make everything fluid and variant, we need to
design invariant structures that ground the experience.
>> Nestedness: we perceive the environment as structures that relate to one another as
nested -- a stream is in a valley, which is between mountains, which are part of a range,
which are between earth and sky. There aren’t always definite boundaries, though, and this
isn’t a purely logical relationship -- there are overlaps and redundancies, and shifts in
meaning. A stick on the ground can be kindling or a club, or it can be part of a roof over our
heads in a shelter. It’s not just one defined thing in how it relates to the rest of the
environment.
This is a powerful idea when we look at something like faceted navigation, where the
meaning of a piece of content or an object shifts depending on what perspective a user is
bringing to it.

Surfaces
Objects
Layout
Place
Some of the components that
make up environments.
There are many components Gibson outlines, like surface, object, layout and place -- that
work together to form the environment.
There isn’t time to cover it all here -- but I think it’s a useful framework we can borrow from,
because all the environments we design should meet the expectations of people’s embodied
experience. Our bodies want and need environments to work in these ways.

Semantic Information
10100010
01001000
01110011
10100010
01001000
01110011
digital systems.
Digital
Animals (including
environment.
Physical
with people.
Semantic
Of course, humans don’t just exist in raw physical surfaces and objects -- we add to the
environment. In addition to buildings and roads and such, we add language to the world.
The semantic mode, in short, is language. But I mean language in the broad sense of things
we put into the environment to communicate with people. This can be all sorts of stuff:
speech, gestures, text, iconography, even buildings have semantic qualities.

LANGUAGE IS ENVIRONMENT
STRUCTURE THAT ENABLES US TO MAKE MORE STRUCTURE
Language is “a form of mind-
transforming cognitive
scaffolding: a ... symbolic
edifice [playing a] critical role
in promoting thought and
reason.”
- Andy Clark Supersizing the Mind
http://en.wikipedia.org/wiki/File:CantileverScaffold.jpg
Language is environment. In another excellent book on embodied cognition, Andy Clark
frames language as a form of cognitive scaffolding; it persists in the environment, as new
structure that allows us to make other structures. We wouldn’t have most of what we think of
as civilization without the scaffolding of language.

Language is the primary human interface to
complex systems.
Language is both the infrastructure we use for creating complexity, as well as the primary
interface we have for understanding and inhabiting complex systems.

WE LIVE IN LANGUAGE STRUCTURES
Language structures are an architecture that we live within together, whether we know it or
not. Whether these structures are deﬁned explicitly like in this early IBM management
diagram, or deﬁned tacitly through the collective assumptions within a shared culture, the
way we talk and write about our shared environment is also a structural feature of that
environment.

Physical Affordance +
Semantic (or “Linguistic”) Affordance
Physical
Semantic
Humans aren’t just perceivers of physical structure. We depend as much or more on the
symbolic workings of language – semantic information – as a part of our environment for
working together and figuring out our world, and making stuff – as well as just getting
around.
In this picture of a staircase at City Lights Bookstore in SF, we see stairs that have intrinsic
structural information that our bodies perceive as affording the ability to walk upward.
But there’s this other thing – a label – which signifies what where we will be going when we go
up the stairs.
We might call this semantic affordance. It’s an abstracted, symbolic version of how we
perceive and comprehend physical structure.
In linguistics we also hear of signification -- which I won’t go into in detail here -- but there
are signifiers at work in both the stairs and the label. The important point I want to get across
here is that affordance is at work even at the high-order level of meaning we have with
language, not just in physical structures.

Physical
Physical
+ Semantic
Simulated-
Physical
+ Semantic
Mostly
Semantic
Affordance across a spectrum
of physical to semantic.
Affordance is at the root of how we use language as well as the simulated physicality of the
stuff we put into interfaces.
>> The stairs intrinsically contain all the information needed for your body to know what will
happen when you climb them.
>> But look at this button. When people started making machines and systems that were
more complex, we had physical affordances that did more than we could tell just by touching
or looking at them. Because I’ve pushed similar buttons before, I know that the physical
information here means I should be able to push in the red part and it will probably spring
back out -- that’s all that the physical information is telling me. In order to know what the
button is going to cause elsewhere, I need a label.
>> When Microsoft updated Windows to add the Start button, they were basically adding an
invariant structure to the operating system so that users would know that, no matter what,
they could always “start” a new action from here. And like lots of other software, they used a
simulated button object, borrowing from what we were used to with buttons like the one on
the left. The button isn’t physical information, really ... it’s using semantic information to
simulate a physical object. That’s what graphical user interfaces do.
>> In other parts of interfaces, and especially on the web with hypertext, labels may not have
these simulated object qualities; so we have to rely a lot on the context of a label to know if it
just affords reading or if it’s something we can activate.
This is the issue at the heart of ﬂat versus skeuomorphic design -- it’s really not about
aesthetics, it’s about solving this issue of affordance.
Additionally, the web introduced broad usage of labels as paths -- tapping a hyperlink
doesn’t just activate some mechanism, it takes you on a journey to what feels to your
perception like a new place.

Which red x????
Looks like a
“confirm” action.
Simulated-Physical & Semantic Information
In Conflict
Here are some examples of where things perform both semantically and ecologically, in
confusing ways, in a UI.
In this email application, logically speaking, the red X’s are all very different -- but physically
they’re barely different at all; they require a lot of thought to disambiguate their semantic
meaning. When I was in a hurry, I would reach for the closest red X to do whatever I’m trying
to do - close the message, decline an invite, or delete it entirely. About half the time, I would
click the wrong one.
>>In an unsubscribe interface for fab.com, my wife discovered that she was apparently re-
subscribing without realizing it, because that big red button -- like a big berry you can’t help
but pick -- contextually feels like it’s a conﬁrmation, not a cancellation/re-subscription
action.

Animals (including
environment.
Physical
with people.
Semantic
Digital Information
10100010
01001000
01110011
10100010
01001000
01110011
digital systems.
Digital
So the examples we just looked at weren’t just any sort of semantic information, they were
semantic information driven by digital technology. And digital technology relies on digital
information.
Digital information is how the black boxes talk to the other black boxes. It’s the lifeblood of
information technology. The whole point of digital information is to strip human meaning out
of it, which makes it more efficient for machines to transmit, store, and retrieve.
This isn’t stuff we see face to face very much. Mainly we encounter its *effects* in the
environment.

Digital Information Mode Leaking into Semantic Environment
We see machines around us trying to get us to perceive what they are saying, or what they
want to hear from us. We see them murmuring to each other in weird, noisy machine-only
semantics that we do not comprehend either ecologically or semantically. The gas pump has
to have a sticker added to it that explains what “Enter Data” means.
>>The Twitter proﬁle with the iPhone coordinates expresses my location not in a semantic
way (the name of a city, for instance) but in a Cartesian grid that I have no contextual
orientation for, either semantically or ecologically.
>>The Delta app has information that I, as a human, can read, but it gives priority to the
machines that I encounter in the workﬂow of the airport.

Digital information enables
pervasive semantic place-making.
I don’t mean to paint digital information as a villain. It isn’t.
The ability to transmit, store and retrieve information in this way is a miracle.
An platform I like a lot is Avocado - it lets a couple keep in touch and share a place together,
pervasively.
It has nice touches that key into embodied experience of semantic information, like sending a
hug by touching the screen to your heart.
Another nice touch: the couple shares the same password - making a word into a very real
link of co-ownership of the place, like having the same keys to your home.
This sort of pervasively available place would be impossible without digital information in the
background. But it also requires a lot of discipline with semantic information structure to
make the place coherent.

Google Plus on Desktop Google Plus on Mobile
Easily post content in the circle I’m in.
Confusingly requires re-establishing the
circle where I’m posting.
Broken “invariants” across contexts.
Google Plus has invariance in how it establishes place between mobile and desktop versions.
When I’m on desktop, and I’m in my “Family” Circle reading posts, when I post something
new it automatically goes into that circle, just like I’d expect if I were having a conversation in
a room. But on mobile, if I’m in my Family circle, a new post defaults to viewable by everyone;
I have to work through a half dozen interactions to make it post just to the circle I perceived I
was already “in.”

The ontology of “news” is confused here.
• News Feed inside News
Feed?
• If All Friends, is it no longer
News? or Recent?
• Isn’t News “recent” by
definition?
• What, then, is News Feed as
a place? How am I changing
the kind of place it is here?
Ontology is the basis that we use for creating any sort of taxonomy - or relationship between
semantic invariants.
In the mobile version of Facebook’s “news feed” there are several options for how you want to
see the feed. Their meanings don’t make much sense together, though -- it’s not clear how
these are nested.
Is news feed another news feed inside news feed?
What does all friends mean? If I choose that, am I not seeing news anymore, or the most
recent?
Isn’t news recent by deﬁnition?
What is news-feed as a place? How am I changing the mode of the place when I select
something else?

What is “card” in this environment?
Lowes launched a service called MyLowes -- that requires the registration of a card. But they
also have a “Lowe’s Card” that’s a consumer credit card.
Conversations at checkout can end up like a “who’s on ﬁrst” routine -- “do you have your
Lowe’s card?” “My Lowe’s card? That’s what I’m paying with.” “No I mean your ‘my lowe’s’
card.” “This IS my lowe’s card!”

Shopping Simultaneously in a Store & Online
http://stevenglassman.de/2013/11/25/america-vs-deutschland/
In these omni-channel scenarios, the semantic information has to align with how it’s
presented both online and physically in the store’s structure. Knowing if a huge store has a
speciﬁc peanut butter or not is great; knowing exactly which aisle to get it from is even
better.

The
Thing
Ontology helps answer:
“What is a thing across multiple contexts?”
Presently our various ﬁelds are preoccupied with how to have content and functionality make
sense in various contexts.
Ontology is at the heart of this problem.
In many organizations and project teams, there’s an over-obsession with things like layout in
each of the instantiations of a thing, but not enough discussion about how to deﬁne the
nature of the thing in abstract. That requires an ontological perspective.
And, done properly, it forms the main structures of an information environment - the
invariant pillars, so to speak - that allow language to stitch together coherence across
channels.

LANGUAGE IS
INFRASTRUCTURE.
LABEL
LABEL
LABEL
LABEL
LABEL
RULES
Labels
Connections
Rules
All this stuff is essentially made out of language... it’s language working as infrastructure.
We essentially make things out of labels, connections and rules.
Too often, we assume the labels are something to add later - but in reality they’re the thing
we have to ﬁgure out ﬁrst.
This is why issues like ontology and taxonomy are so important - they establish the
“invariant” features of the environments we make.

• What are the invariants for perception
& action? What grounds the
experience?
• How is it structurally nested in relation
to a perceiver? Is it resilient for
different perspectives & contexts?
• Is it clear what is a place and what is
an object?
• How are semantic and physical
information working together, or not?
• Are the rules behind the environment
clearly expressed in its information?
Some of the questions we can ask ...
So in sum, this is just scratching the surface, but there are some questions we can ask.
>>What are the invariants for perception & action? What grounds the experience?
>>How is it structurally nested in relation to a perceiver? Is it resilient for different
perspectives & contexts?
>>Is it clear what is a place and what is an object?
>>How are semantic and physical information working together, or not?
>>Are the rules behind the environment clearly expressed in its information?
We have a long way to go to build heuristics and frameworks for responsibly creating the
broader environment that our users inhabit. Hopefully this is a helpful start.

THANKS!
Andrew Hinton | @inkblurt | andrewhinton.com
andrew@understandinggroup.com

The World is the Screen - Convey UX 2014

More Related Content

What's hot (20)

Similar to The World is the Screen - Convey UX 2014 (20)

More from Andrew Hinton (12)

Recently uploaded (20)

The World is the Screen - Convey UX 2014