The World Is The Screen

THE WORLD IS
THE SCREEN
Elements of Information Environments

IA Summit 2013 | Baltimore, MD | Andrew Hinton | @inkblurt

I’m Andrew Hinton, and I’m an information architect with The Understanding Group.

Today we’re going to be talking about information environments -- and as a way into that
conversation, I’ve titled this talk “the world is the screen” -- so let’s start by considering what
I mean by that.

Andrew Hinton | @inkblurt

We’re surrounded by screens these days. They’re proliferating to the point where we’re
interacting with them as often as any other objects in our surroundings.
So, in a way, we might say that screens are ﬁlling up the world to the point that it’ll feel like
the world is made of them.
There’s some truth to that. But it’s just one facet of the issue.

kiosk: kodak.com
table/phone: android.com
gps: garmin.com


Another way the world is the screen is because of technology like Google Glass, which
essentially lays a screen over the world around us, mediating between our perception and the
stuff we’re perceiving.
This is certainly worth considering, but it, too, is only a facet of how the world is the screen.
It’s also the case that these things -- all sorts of device screens and augmented displays -
are getting integrated into our environmental experience.

left image wired.com / others from google


The way we use these things isn’t conﬁned to the things themselves. They’re part of a larger
context.
Here, on an airliner, I surreptitiously snapped a picture while waiting for take-off. I did it
because it’s a good example of how we don’t just sit in front of screens alone, in a vacuum.
We do it as part of our activity in the world around us.


In this case, this guy was talking to friends about a football game in progress -- a game that
had a mirror-world of itself happening in a little avatar of itself on a smartphone.

There’s a relationship between the digitally generated “information” environments we use,
and the non-digital environments we live in. We live in both. I wondered, at the core of how
we understand the world, are there differences between them? Do they matter?

INFORMATION
ENVIRONMENT

INFORMATION
ENVIRONMENT

6 Andrew Hinton | @inkblurt

For example. What is the difference, really, between shopping for office supplies in a brick
and mortar retail store, and shopping for them through an online retailer like Amazon?
If we frame both of these as “information environments” -- does that help us understand why
one might be eating the other’s lunch?
If we think of the physical store as an information interface, how much information is
conveyed, and of what kind, through one interface versus another?

A CURATED, ENCLOSED, BLENDED
INFORMATION ENVIRONMENT


What about when information environments use all sorts of methods for communication, all
at once?
We’ve had blended information environments for a while. Here’s a marvelous exhibit at the
American Museum of Natural History in New York. It has a whole taxonomy -- the old-school
meaning of taxonomy, meaning an organized hierarchy of creatures. But this one is
instantiated on the wall. It also has it represented in written form, in a printed document, and
in digitally presented form, in a kiosk. (These were taken when I took my daughter there
some years back.)
This is a curated, complex information environment.
Physical objects, digital interfaces, lots of language and labels around. All connected together
to form a whole experience.
It’s is a highly controlled version of the world we now live in -- which is more emergent,
messier, but even more pervasively connected & digitally enabled.

photos by andrew hinton

AN OPEN-ENDED, CUMULATIVE,
BLENDED INFORMATION ENVIRONMENT


Here’s an everyday intersection in Dublin.
This is an environment that also has many different layers and modalities, but it’s not
controlled and curated in the same way as the museum. It’s been added to, streets have been
widened, signage added, infrastructure installed. And on top of that, lots of other information
is pouring through it in the form of newspapers, or advertisements on buses and vans.
The digital signs are something relatively new for our environments. It used to be that signs
said one thing, and you learned what they said, and then you could forget about them until
somebody put up a new one. These days, we can’t depend on surfaces being stable,
persistent homes for written information. The stuff is embedded in all sorts of places. This
street intersection in Dublin has digital signage mixed in with everything else.

Pervasive computing technology means that the world is only getting weirder and more
complex. We’re not talking about just consumer devices, but whole infrastructures, urban
networks, and wired economies.

photo by andrew hinton

What do we mean by
“Information Environment”??


For over a decade, we’ve been saying that IA is, in part, the structural design of shared
information environments. But what do we mean by that phrase? It sounded right at the time,
because even when most of what we were doing were static websites, we knew the scope
could be bigger, and that the world was going to change toward more complexity.
So, here we are in that spot we supposed we’d be in -- with all this pervasive information
complexity - and it seems high time to nail this thing down better.

Labels
Card Sorting
Mental Models Navigation
Methods,
Facets
Tools, Controlled Affinity Diagrams
Taxonomies Hyperlinks
Processes Vocabularies

Thesaurus
Task Analysis Hierarchies
Ontologies Context Models

What’s underneath
that makes these things work (or not)?


I’m actually a bit worried because I’m not sure our current tool sets are really up to the task.
We have a lot of methods, but not a lot of understanding about why or how they actually
work. (Kind of like antidepressants.)
We also tend to talk about a lot of things like “understanding” and “information” and whatnot
-- but what do we mean by those things? We need more rigor, more science - I don’t mean
information science but science about humans.
I’ve been working on a book about how information creates and shapes context. And in part
of that work, I’ve had the realization that we’re often looking at information and
environments the wrong way around, by starting with the technology ﬁrst.

(8 min)

A STRUCTURALLY DESIGNED ENVIRONMENT
^
INFORMATION

Especially since all the technology is becoming more and more pervasively integrated into our
surroundings, now I’m thinking we should start with something more basic -- how do we
comprehend our environment generally? What if we start with pre-digital structurally
designed environments?
>>
And even further: is something like this ﬁeld not just an environment, but an information
environment?
I believe it is. We’ll get to that, but ﬁrst something from ten years ago.

http://commons.wikimedia.org/wiki/File:Derbyshire_Landscape.jpg

STEWART BRAND
“PACE LAYERS”
IA SUMMIT 2003


Back in 2003, at the IA Summit in Portland Oregon, Stewart Brand gave the opening keynote.
One of the things he discussed was “pace layers” - the idea that some layers of human life
move more slowly than others. Nature changes very slowly, and all the stuff we’ve built up
from that foundation tends to move and change more quickly -- quicker and quicker still at
each concentric layer.

photo: Mike Lee
http://www.ﬂickr.com/photos/curiouslee/15238458/sizes/o/in/photostream/

PACE LAYERS OF
INFORMATION ENVIRONMENTS

Start
Here


Taking that idea and running with it, I’m working out a sort of pace layer stack about
information environments.

At the root is our perception and cognition of environment -- these are things that don’t, at
core, change much at all over millennia.
Then there’s spoken language, something we’ve had with us possibly for over a million years
-- to the point that it’s probably a shaping factor in our evolution as a species.
Writing and graphical symbolic language come later than speech. They’re technologies, in a
sense, for encoding, recording and sharing spoken language.
And only later do we get into information organization and design, or what we call
“information technology” -- digital computing, networks, & devices.
We tend to start our work through the lens of the upper two layers - but they’re the ones that
change and ﬂuctuate the fastest.
>> I think we should start with perception/cognition as the lens for understanding the rest.

Information in Three Modes

Digital 10100010 10100010
Digital systems transmitting
01001000 01001000 to & receiving from other
01110011 01110011 digital systems.

Semantic People communicating
with people.

Animals (including
Ecological people) perceiving the
environment.


There’s a long history of people trying to define information. I’m not into defining things so
much these days -- I’m more interested in describing them.
And that frees us up to understand a thing in more than one mode or dimension -- to be OK
with grasping something in all its facets.
Rather than defining information, I’d like to describe how it operates. I think information
affects perception and understanding in three major modes. Let me mention them all, then
we’ll look at each in more detail.
>> First is “ecological” information. It’s about how animals perceive their environment.
>> The second is “semantic” information: it’s the mode people use to communicate with one
another.
>> Third is digital information: digital information is information used by digital systems to
transmit to and receive from other digital systems. It’s what happens between the black
boxes of our digital infrastructure.
Like I said, we’ll look at each of these more closely. Let’s start with ecological information.

(12 min)

Ecological Information

Digital 10100010 10100010

with people.

Animals (including
environment.


So, starting with ecological information. The word ecological means having to do with the
relationship between an animal and its natural environment. I’m using the term this way
because many of the ideas I’m using are based on ecological psychology and embodied
cognition, which is different from mainstream cognitive science.

Mainstream Cognitive Science

Assumes the brain ...
• Works like a computer to “process
information.”
• Uses symbolic logic, “images” &
representational models.
• Is primarily (if not exclusively)
responsible for cognition.


Mainstream cognitive science, which still forms the foundation for most HCI theory and
practice, assumes that the brain works like a computer as a sort of information processor.
The brain takes sensory inputs from a sort of dumb, robotic body, processes those inputs as
“information” -- representational images and symbols of the world, along with images and
symbols stored in memory -- and once it has ﬁgured out what to do, it tells the body how to
respond.
This is still the predominant way of seeing how the brain works. It’s part of the assumptions
built into many of our methods and training.

Embodied Cognition (not yet mainstream)


Then there’s embodied cognition theory. Embodied cognition argues that cognition is not
brain-exclusive, but actually uses the body and even the environment around the body for
cognitive activity.
There are many ﬂavors and schools of thought even within the embodiment movement; but
one in particular is what some call “radical embodied cognition” that says we should not try
to marry embodiment with the traditional cognitive science perspective, but replace it
entirely. Full disclosure: the ‘radical’ or ‘replacement’ camp is the one I ﬁnd myself aligning
with.

James J Gibson - Ecological Psychology of Perception

Long sidelined, now hailed as pioneer of embodied cognition.


The so-called ‘radical embodiment’ movement has adopted the work of James J Gibson, who
was a scientist of something called “ecological psychology” in the mid 20th century..
He started out studying WWII pilots - and found that centuries-old assumptions about how
people comprehend their environment were simply wrong. His ideas have been acknowledged
and quasi-appropriated here and there, but now many are starting to see his whole corpus of
thought more clearly -- he was really writing about embodied cognition (but calling it
ecological psychology).

A few key ideas from Gibson’s theories

We perceive elements in the environment as
invariant (persistent) or variant (in flux).

We perceive the environment in human-
scale terms, not scientific abstractions.

We perceive environment as
“nested,” not in logical hierarchy.


There’s no way to cover all the important stuff from Gibson in this talk, but here are a few
key ideas.
>> We perceive elements in the environment as invariant or variant. Invariant elements are
necessary for orientation of everything else. They include at the widest scale, the earth and
the sky. Or perhaps a mountain range. Or even the occluding edge of one’s nose. Variant
elements are things that are in flux that we don’t rely on for their persistent structures.
>> We perceive the environment in human-scale terms, not scientific abstractions.
Perception doesn’t grasp the abstraction of space or time. Our bodies don’t perceive a fallen
tree limb in terms of centimeters, but in terms of whether it will fit in the hand, or if it’s too
heavy to pick up.
>> We perceive the environment as nested. A stream is nested between banks, which are
nested between hills, which are nested within a range of larger hills, all of which is nested
within the canopy of sky. This is importantly different from strict hierarchy, though. It
overlaps and shifts depending on the activity of the perceiver. A cave might feel like “inside”
but then feel like “outside” when rain starts leaking in. A stone may just be clutter to me
when I walk by it the first time, but when I need a stone to pound something, it becomes an
object I can pick up. Then when I pick it up, it becomes an extension of my body.
All of these are important ideas for the structures we make for digital and other systems,
because our cognition expects the world to accommodate these ways of perceiving.

AFFORDANCE

“... the perceived functional properties of objects,
places and events in relation to an individual
perceiver.” - JJ Gibson

Perception exists only insofar as we
perceive affordances.


JJ Gibson invented the concept of affordance. Others have since popularized it, but gotten it
somewhat wrong -- mainly because they’re coming at it from a traditional cognitive-science
perspective, not an embodied perspective.
For Gibson, affordance isn’t a thing you add to something. Affordance is the organizing
principle behind *ALL* perception. We don’t perceive anything unless it affords meaningful
action for a given context.

Information Pickup Theory

Perception
Ambient,
structured
Agent energy arrays

Action

Ecological Information
"Pick-up"

The perception-action loop.


We perceive affordance through something called “information pick-up.”
A perceiver, or agent, takes *action* in an environment in order to discover its affordances.
The action part is very important. Gibson rails against traditional cognitive science
laboratories that strap people into chairs to keep their heads still -- cognition doesn’t
function from stationary positions. We evolved as active, moving, interacting creatures that
perceive through action.
And when we act in the environment, we perceive, which then affects our action, which then
affects what we perceive, in a continuous loop of cognitive activity.
This is a very different way of thinking about “information” - but it’s valid, and forms the
basis for all the other sorts of information in our lives.

Sigmund


I’ve seen this with my dog, Sigmund. When I try taking him for a walk, he’ll stop as if the
ground has grabbed him. Sometimes I’ll let him explore to see what’s up, and it’s almost
always something that I didn’t perceive the way he did - either because it wasn’t relevant or
because I physically can’t perceive it. I’ve learned a lot by watching my dog ﬁgure out the
world. It’s not that different from us. He just doesn’t have the rich layer of language draped
across the world like we do.
It’s that layer of language that humans have added to the environment that makes up the
next information mode.

(+7 = 19 min)

Semantic Information

Digital 10100010 10100010

with people.

Animals (including
environment.


The semantic mode, in short, is language. But I mean language in the broad sense of things
we put into the environment to communicate with people. This can be all sorts of stuff:
speech, gestures, text, iconography, even buildings have semantic qualities.

flickr - uicdigital

SEMANTIC INFORMATION CHANGES HOW
WE EXPERIENCE ENVIRONMENT

Information (in the sense we tend to mean it colloquially) is what creates and changes much
of what we consider to be contextual reality.
Look at this photo -- there’s information everywhere in this scene.
>>The lines on the road tell us where to drive; the traffic light is a virtual barrier that affects
our behavior; the road signs give us a layer of instruction that adds meaning to the city
around us. without the information here, it would quite literally be a different place.
Really, you could have civilization without cars, lightposts and buildings, but you couldn’t
have it without language. Language is our reality in many ways. And a city is as much a
construct made of language - speech as well as labels, signs, other semantic artifacts - as
one made of atoms.

http://www.ﬂickr.com/photos/uicdigital/5410417461/

WE LIVE IN LANGUAGE STRUCTURES


In a presentation last year, I heard Peter Merholz talk about how a cube farm in an office
building is like the org chart “made manifest.”
That’s due to the fact that language structures are an architecture that we live within
together, whether we know it or not. Whether these structures are deﬁned explicitly like in
this early IBM management diagram, or deﬁned tacitly through the collective assumptions
within a shared culture, the way we talk and write about our shared environment is also a
structural feature of that environment.

LANGUAGE IS
ENVIRONMENT
Language is “a form of
cognitive scaffolding...”
- Andy Clark - Supersizing
the Mind


Language is not information. Language is environment.
When I am speaking I’m vibrating the air - affecting the environment, putting structures into
it that weren’t there before. The same goes for writing - it’s environmental structure we’re
adding to the world.
We then pick up information about what those environmental features mean; you hear the
vibrations and, because you’ve learned what those words mean, they have affordance for you.

Ecological & Semantic Information In Conflict


Don Norman famously talks about the affordances of door handles.
In this case, a similar affordance situation can help us understand how different modes of
information can be in conﬂict.
I was walking into a store and did not even notice the sign.
The language of “Pull” had an intended affordance -- I’m supposed to read it and allow it to
control my action.
But the ecological information I picked up from the structure of the handle had a stronger
effect on my action. I was talking with someone as I was entering the store, so my language
perception was preoccupied; also I could see through the glass into the store toward the
context I intended to enter -- essentially seeing right past the sign.

Digital-Ecological & Semantic Information
In Conflict

Which red x????

Looks like a
“confirm” action.


Graphical user interfaces are essentially simulated ecological information. Objects with
affordances, simulated on screens.
But they’re also performing semantically. It can get very confusing.
Logically speaking, the red X’s in the ﬁrst example are all very different -- but ecologically,
they require too much thought to disambiguate. In this app I found myself always deleting
rather than declining, closing rather than deleting, etc. When I’m in a hurry, I just reach for
the closest red X to do whatever I’m trying to do - close the message, decline an invite, or
delete it entirely. About half the time, I end up clicking the wrong one if I don’t stop and
think about it explicitly.
>>In an unsubscribe interface for fab.com, my wife discovered that she was apparently re-
subscribing without realizing it, because that big red button -- like a big berry you can’t help
but pick -- contextually feels like it’s a conﬁrmation, not a cancellation/re-subscription
action.

Which of these will accidentally tweet publicly?

Very little
semantic or
ecological
information
about what
context I’m
in

Ecological
Information /
Affordance
for action.


The infamous Twitter “DM Fail” problem is largely caused by users responding to DMs via
SMS.
In this case, it’s hard to tell: which of these is a Twitter app that will safely allow me to DM
someone, and which is my SMS app that will tweet to everyone who follows me? The
physicality of the interface can easily override my perception of the semantic information’s
differentiating cues.

Digital Information

Digital 10100010 10100010

with people.

Animals (including
environment.


So the examples we just looked at weren’t just any sort of semantic information, they were
semantic information driven by digital technology. And digital technology relies on digital
information.
Digital information is how the black boxes talk to the other black boxes. It’s the lifeblood of
information technology. The whole point of digital information is to strip human meaning out
of it to make it efficient for transmitting and storing encoded information.
This isn’t stuff we see face to face very much. Mainly we encounter its *effects* in the
environment.

Digital Information Mode Leaking into Semantic Environment


We see machines around us trying to get us to perceive what they are saying, or what they
want to hear from us. We see them murmuring to each other in weird, noisy machine-only
semantics that we do not comprehend either ecologically or semantically.
•The gas pump above has to have a sticker added to it that explains what “Enter Data”
means.
>>The Twitter proﬁle with the iPhone coordinates expresses my location not in a semantic
way (the name of a city, for instance) but in a Cartesian grid that I have no contextual
orientation for, either semantically or ecologically.
>>The Delta app has information that I, as a human, can read, but it gives priority to the
machines that I encounter in the workﬂow of the airport.

Digital information enables
pervasive semantic place-making.


I don’t mean to paint digital information as a villain. It isn’t.
The ability to transmit, store and retrieve information in this way is a miracle.
An platform I like a lot is Avocado - it lets a couple keep in touch and share a place together,
pervasively.
It has nice touches that key into embodied experience of semantic information, like sending a
hug by touching the screen to your heart.
Another nice touch: the couple shares the same password - making a word into a very real
link of co-ownership of the place, like having the same keys to your home.
This sort of pervasively available place would be impossible without digital information in the
background. But it also requires a lot of discipline with semantic information structure to
make the place coherent.

INFORMATION MAKES PLACES,
KIND OF LIKE THIS PICTURE MAKES A PIPE.

IF YOU COULD SMOKE THE PIPE.


This is the famous Magritte painting -- it says “this is not a pipe”
The picture deﬁnitely shows a pipe but it’s not a real pipe you can smoke.
>>Information is kind of like this in the way it makes places.
>>Except for a key difference that, with
Information, you can smoke the pipe.

LABEL

LANGUAGE IS LABEL
LABEL

INFRASTRUCTURE RULES

LABEL
LABEL


And Language is Infrastructure.
We essentially make things out of labels, connections and rules.
Too often, we assume the labels are something to add later - but in reality they’re the thing
we have to ﬁgure out ﬁrst.
This is why issues like ontology and taxonomy are so important - they establish the
“invariant” features of the environments we make.

ONTOLOGY
What am I? What is my world?
How do I exist in it?

Please describe a formal,
explicit specification of a shared
conceptualization for purposes of
structuring semantic data.
00101011100100101110100101


Behind the scenes of all this is Ontology.
Ontology can be the philosophical sort -- about the nature of one’s being and the
relationship of the self to the environment.
Or it can be the information-technology sort - developed for digital information work, to
define the formal specification for data purposes.
A big part of what IA should be doing is bridging these two planes of existence.

ONTOLOGY

The
Thing


Presently our various ﬁelds are preoccupied with how to have content and functionality make
sense in various contexts.
Ontology is at the heart of this problem.
In many organizations and project teams, there’s an over-obsession with things like layout in
each of the instantiations of a thing, but not enough discussion about how to deﬁne the
nature of the thing in abstract. That requires an ontological perspective.
And, done properly, it forms the main structures of an information environment - the
invariant pillars, so to speak - that allow language to stitch together coherence across
channels.

What is “card” in this environment?


Here’s an ontology example.
Lowes launched a service called MyLowes -- that requires the registration of a card. But they
also have a “Lowe’s Card” that’s a consumer credit card.
Conversations at checkout can end up like a “who’s on ﬁrst” routine -- “do you have your
Lowe’s card?” “My Lowe’s card? That’s what I’m paying with.” “No I mean your ‘my lowe’s’
card.” “This IS my lowe’s card!”

Semantic-information “place” signified by “account”

Digital architecture determining ecological & semantic context.


If I walked into a bank and asked to access an account, it’d be clear what I meant. But online,
it can mean different things (my proﬁle-account represents me in the digital context -- and
needs a label, which happens to also be “account”). The digital systems behind the scenes at
Kohls require that these two things we call “account” be separate - requiring disambiguation.
The ontology of ‘account’ is in question here. It’s one of the many sorts of things we have to
sort out with language, when we’re working in an environment that’s made of almost nothing
*but* language.

Shopping Simultaneously in a Store & the Cloud


Now that retailers are trying to be in the cloud and on the ground at the same time, context
is especially confounding. It requires a great deal of work to situate the user’s perception of
place.
For many retailers, product price and availability are driven by location - yet shoppers online
tend to come to the experience as if it’s a cloud-based store, not thinking about geography
yet. It puts the user in a strange environmental position of being in a local store and in an
amorphous web-shop experience simultaneously. The ontology of place is dissonant.

Subway station + Food store


And here we have a situation where a subway station is also ﬁlled with pictures of products
that you can actually buy -- not unlike Magritte pipes that you can smoke.
With the QR code sprinkled throughout -- digital information wrapped in massive simulacra
of ecological information -- plus the semantic information of labels/brands. This could have
just been a list of words with QR codes next to them, but perhaps wisely, the retailer decided
to create the place in our image, to help bring the “reality” of shopping for groceries into
what would otherwise override perception as a subway station.

INFORMATION MODES & PACE LAYERS

Digital

Semantic

Ecological


The examples we’ve looked at are going to seem primitive in a matter of just a few years.
So we need ways of breaking down whole environments into their essential elements - and
those elements are bound up in human perception & cognition.
This has been a very cursory overview of what I hope are a useful beginning for principles
and frameworks for doing this work into the future.

THANKS!

Digital

Semantic

Ecological


The examples we’ve looked at are going to seem primitive in a matter of just a few years.
So we need ways of breaking down whole environments into their essential elements - and
those elements are bound up in human perception & cognition.
This has been a very cursory overview of what I hope are a useful beginning for principles
and frameworks for doing this work into the future.

The World Is The Screen

More Related Content

What's hot (20)

Viewers also liked (14)

Similar to The World Is The Screen (20)

More from Andrew Hinton (12)

Recently uploaded (20)

The World Is The Screen