Semantics in Visual Perception: Methodological Remarks, Issues and Perspectives

Semantics in Visual Perception:
Methodological Remarks,
Issues and Perspectives
Jurgis Skilters
Center for Cognitive Sciences and Semantics
University of Latvia, Latvia

Thanks to my collaborators and colleagues:
Baingio Pinna (Sassari, Italy),
4-5.11.2010, Aarhus,
Jānis Pencis (CCSS, Riga)
Methods in Language&Cognition 1
Denmark

The aim of my talk
• to provide some methodological
preliminaries and some parameters for
research on the interaction between visual
perception and semantics.

4-5.11.2010, Aarhus, Methods in Language&Cognition 2
Denmark

interference effects between
language and visual perception:
• Language is an access to human
cognition.
• Visual perception is another access to
human cognition
• ....

Denmark

interference effects between
language and visual perception:
• we do have representations of linguistic
stimuli
• we do have certain representations of
visual stimuli.
• are they the same representations?
• What does happen when we look at visual
stimuli and when we process (either
produce or comprehend) language?
Denmark

Outline:
1. Visual perception and the role of psychophysics
2. How does semantics arise in perception?
3. Invariants
4. Affordances
5. Events
6. Experience
7. Objects: the basic principles of their perception
8. Perceived causality, motion perception
9. Semantics in Vision: Multimodality effects
10. Remarks on spatial cognition
11. Methods from semantics and perception (brief overview)
12. Conclusion: Constraints

4- Methods in Language&Cognition 5
5.11.2010, Aarhus, Den

1. Visual perception and the
role of psychophysics

Denmark

• The methods we chose depend on our
research motivation.
• Relationships between stimuli and
physiological response Φ
• Relationships between stimuli and
perceptual reaction ψ
• linking relationship (L) between perceptual
and physiological response.


Cp. Goldstein, 2005

Stimuli

Φ ψ

Physiological reaction L perceptual reaction

Denmark

Stimuli

Φ ψ

Physiological reaction L perceptual reaction

Denmark

• How are relations ψ studied and explored?
• methods formulated by Fechner (1860) to
determine thresholds,
• methods for magnitude estimation (above the
threshold formulated by Stevens (1961)),
• phenomenological observations
(Katz, 1935, Metzger 2006/1936);
• also identification
measures, recognition, reaction time research….
Also overlapping with Φ
Denmark

• there is no direct correspondence between
stimuli and perceptual result:
– different stimuli can cause the same perception and
(physically) the same stimulus in different contexts
can cause different perceptual results.
– we can perceive something without the sensorial
basis for it,
• e.g., we can perceive movement / causality / motion where
there is no movement at the level of physical stimuli.
And this has nothing to do with semantics but with
the principles of our visual perception.

Denmark

the binding problem:
• how is a unitary perceptual experience
generated that combines qualities such
as color, shape, location, orientation
(Goldstein, 2005, 12)?

Denmark

Most of the models are hierarchical:
• top level encodes the whole / figural representation;
• subsequent / lower levels encode parts and relations
between them (Palmer, 1975, 1977, 1978, Carlson-
Radvansky & Irwin, 1995)
• 1. Grouping
• 2. shape assignment
• 3. meaning assignment
• there are also parallel models assuming that
object memories critically determine object recognition
and also grouping and shape assignment factors.
• NB: grouping and shape assignment processes function
without attention, intention and awareness.

Denmark

a. Classical serial hierarchical theory (Marr) (picture from Peterson,
2005)

Denmark

b. interactive hierarchy model (picture from Peterson, 2005)

Denmark

c. Parallel interactive model (picture from Peterson, 2005)

Denmark

experience

stimuli Grouping Shape
meaning
assignment

situation

Denmark

2. How does semantics
arise in perception?

Denmark

1. visual field is segmented, the components are grouped
together;
2. shape assignment takes place
3. semantics is assigned.
• And finally….. the result is lexicalized.
• NB all these processes are interactive and there is not
one way of bottom-up or top-down determination.
• NB we „translate‟ continuous perceptual material into
discontinuous language. As soon as we lexicalize we
make the continuous discontinuous.

Denmark

Gouping Shape
Concrete stimuli

assignment Concepts language

Meaning
assignment
meanings
Visual processing

Continuous Discontinuous /
4-5.11.2010, Aarhus, Methods in Language&Cognition discrete 20
Denmark

Grouping

Rows vs. columns and the rectangle illusion.

Denmark

The role of directional symmetry in forming
the shape

The inverted rod and frame illusion
Denmark

The directional symmetry influences the shape

The pointing illusion.
Denmark

• Further, we assign border / contour to the
figural regions. We discriminate figure
from ground.

Denmark

From grouping to shape

Denmark

• And finally we assign meaning to what we
see.
– We simply do not stop the perceptual activity
after grouping and shape-building

Denmark

Form of meaning: More complex results

The illusion of meaning: A square showing different kinds of happenings
Denmark

3. Invariants

Denmark

There is consensus among vision
scientists:
• at least some structures remain invariant across
changes (cp. Carlson-Radvansky & Irwin, 1995, Pinna,
Tanca, & Skilters, 2010).
• Every type belongs to a different level of perceptual
processing and every level contains invariants of a
corresponding type.
– According to certain invariants from type „grouping‟ the
perceptual material (from zero-level objects) is transformed to
higher level and processed according to the type „shape‟ and,
finally, to meaning.
• when one unit (structural invariant in our case)
determines another, the one must belong to the higher
level (type) than the other (cp. Jeffrey 1981).

Denmark

• invariants itself are not objects of the
particular levels; they are principles that
structure the organization of particular
visual material.

Denmark

• Every level of perception contains structural
invariants determining the processing of lower
level.
• The third-level invariants – meaning – are the
strongest
• the structural invariants at the level of meaning
assignment are the strongest
• dissimilarities and multiplicities in lower level
structures become units and their dependencies
become stronger with every higher level of
processing.
• strength of functional dependencies increases
from lower to higher level

Denmark

4. Affordances

Denmark

• meaning as an affordance-sensitive
structure.
• affordances are dispositional properties of
objects and events.
• „Affordances are the acts or behaviors
permitted by objects, places, and
events.“ (Michaels & Carello, 1981, 42).
• affordances differs in different organisms
Denmark

• two mutually connected aspects: action system
and environment.
– „The action system (effectivity structure) and the environment
(affordance structure) are in a relationship of mutual
constraint.“ (Michaels & Carello, 1981, 54, cp. also. Shaw,
Turvey, & Mace, 1982, 209)
• Affordances are processed to a large part un-
intentionally: objects potentiate a range of
actions associated to them but irrespective of
intentions (Ellis &Tucker, 2000).
• the affordances change from one moment or
situation to another.
– more permanent and more time-dependant („episodic“, cp.
Glenberg et al. 2009) affordances can be gradually distinguished.


• Meaning “consists of the set of actions the individual can
undertake in that situation. The set of actions is
determined by the goal-directed mesh of affordances.”
(Glenberg et. al., 2007, 223).
– In a particular situation an individual modifies the meaning
according to his/her experience and bodily configuration.
• Affordances constrain meaning but affordances are
constrained by the actual situation and experiential
factors of the agent as well.
• the situation determines meaning; differences in
affordances result in differences in meaning.

Denmark

5. Events

Denmark

Preliminary
• Have to be distinguished:
• optic flow: when observer moves through
environment, where entire field of view is
determining,
• and
• perception of object motion: looking at a
point in visual field, where only limited
segments of visual field are co-involved
(cp. Shiffrar, 2005).
Denmark

• Visual perception takes place according to
events:
• Events are time-space-linked structures
• “If events are the significant units of the
world, the world must be described in a
way that preserves their integrity. The
world must be described in terms of both
time and space.” (Michaels & Carello,
1981, 10)
Denmark

• Events are cognitively perceived as goal-
oriented patronymic hierarchies (Zacks,
Tversky, & Iyer, 2001)
• and later lexicalized in object/individual-
predicate (object-action) structure (guiding
hypothesis of Pencis, Pinna, & Skilters,
2010).

Denmark

• events as segments of time at a given location,
perceived as having a beginning and an end, cp.
Zacks, Tversky, & Iyer, 2001;
– based on partonymic hierarchies.
• Even material presented in pictures is perceived
in time and according to hierarchic event-
structure.
– recognition of intentional action on objects is needed
(Tversky, Zacks, & Hard, 2008, 461).

Denmark

Correlation of three processes
(1) decomposition of visual material in spatial parts
(2) decomposition of events in temporal parts
(3) lexicalization.
functional dependencies and relations are generated.
hierarchical bias hypothesis: people are spontaneously
disposed to actively encode ongoing activity in terms of a
hierarchical part-whole structure; Zacks, Tversky, & Iyer,
2001

Denmark

• Goal/sub-goal relationship
• a single goal in sense of motion, causality is assigned to
a visual representation even in case of minimal stimuli
(even in case where there is no objective goal or motion
or causality at all).
• In a narrative we can observe hierarchic event
perception more explicitly (cp. also Abbott, Black, &
Smith, 1985, Tversky, Zacks, & Hard, 2008). Faster
narrative production is if there is one higher level event
(Foss & Bower, 1986, Zacks, Tversky, & Iyer, 2001)
• parts as visual segments and parts as sub-goals.
• parts are functionally different
• fast inference from visual appearance to function
• link between perception and function is always present
even in case of minimal stimuli
– (for the perception-to-function hypothesis cp. Tversky, Zacks, &
Hard, 2008)

Denmark

• Objects are integral and crucial to perceive
actions and vice versa (Baldwin & Baird, 1999)
• actions and objects are comprehended together
(Zacks, Tversky, & Iyer, 2001);
• we always perceive object-activity-
constellations.
• actions and objects are mutually constraining
(Tversky, Zacks, & Hard, 2008)
– Object-action matrices are occurring in events and
allow only certain combinations of substituted
instances.
Denmark

Denmark

6. Experience

Denmark

(a) interrelations between visual perception and
experience vary according to the material under
consideration
(b) and more crucially – experience effects are not
atomistic / summative, simplistic but rather
holistic.
the question is not whether but rather how
experience influences visual perception.
• Meaning assignment as resonation of visual
stimulus with experiential structures.

Denmark

7. Objects: the basic principles
of their perception

Denmark

• what counts as an object?
• Conceptions of object recognition
1. we recognize objects by components. I.e., there are
some core elements of visual objects (geons) which are
basic for building together any complex object.
– research tradition by Biederman, Marr
– One consequence from Recognition by Components Theory is
that object recognition is viewpoint invariant. It might be true but
if this is the case we would perhaps see no difference between
pigs and dogs.
2. Multiple Views Theory
– tradition of Buelthoff, Edelman and Tarr
• there are multiple two-dimensional views that enable
recognition of object. Object recognition is view-
dependant because objects observed in new views have
to undergo some time-consuming process before they
are matched to stored views and recognized.
Denmark

8. Perceived
causality, motion perception

Denmark

• Even when there is not causality at the level of
stimuli
there is causality at the level of perception.
• Integration of motion
• there is a tendency to assign intentionality to
„moving‟ objects even if they are simple
geometric stimuli. (For a single stimulus
approach cp. Pinna & Skilters, for a classic study
of moving geometric objects cp. Heider &
Simmel, 1944.)

Denmark

• In case of animated objects
• Heider, F. & Simmel, M. (1944). An
experimental study of apparent behaviour.
American Journal of Psychology, 13,
1944.
• http://www.all-about-
psychology.com/fritz-heider.htm or
• http://www.youtube.com/watch?v=sZB
Ker6PMtM
Denmark

Unanimated / single stimuli

Denmark

Denmark

• One object according to the reports of naïve
perceivers is causing changes in another object.
• Our memory for the spatial location of an object
is biased towards objects motion even if there is
no motion or even if the object is static or
geometric (cp. Shiffrar, 2005, 257),
• moreover, the motion assigned to non-animated
objects is frequently like human-movement.

Denmark

9. Semantics in Vision:
Multimodality effects

Denmark

• Vision is influenced by other expressions
of cognition
– Vestibular system
– Motor system
– Auditory system
Also purely visual stimulus may induce some
e.g. quasi-auditory perception in simulating
collision

Denmark

Denmark

cognition

Visual Vestibular Motor Auditory
perception system system system


cognition

Visual Vestibular Motor Auditory
perception system system system

Denmark

10. Remarks on spatial
cognition


• language provides strong constraints – selectivity and
enrichment – modulating the expression of spatial
cognition
1. Language with the basic components – symbolic units –
noun and verbs and configurations of these give rise to
semantic and syntactic functions of agent, patient,
subject, object.
2. In contrast spatial system contain primitives such as
shapes, objects, locations, landmarks, geometric
layouts, angles and directions – represented in different
spatial reference systems
• (Landau, Dessalegn, & Goldberg, 2010)

Denmark

11. Minimal semantics
generated in visual perception

Denmark

• parallelism between visual and language
processing
• First, we have to build a complementary
methodology, using both psychophysical
methods and semantic methods.
• Second – what to start with?
• Primitive perceptual semantics. Very
rudimentary stimuli can generate a lot of
semantics
Denmark

Object and predicate / thing and
action structure

Denmark

action-resonance effects
- we see not separate
objects but action-
resonated objects.
- we do recognize events
(instead of separate
objects) as soon as we
assign meaning (even is
case of relatively minimal
and poor stimuli).

Denmark

11. Methods from semantics
Different biased and non-biased tasks (e.g.,
sentence completion tasks, priming
experiments, self-report measures). (Cp.
Hasson, U. & Giora, R. 2007).

Denmark

• Lexical decision tasks are used to determine
whether a letter string on the screen makes a
word.
• prime and target.

Denmark

faster

Related prime target

Unrelated control target
prime

Denmark

• Memory measures
• learning stage and a test stage
• free recall method.
• old / new recognition task
• Self report measures: Listing features.
The task is to write down properties that
capture the meaning of an expression (or
visual stimulus).

Denmark

• Effects of comprehension on subsequent tasks.
Priming paradigms. How comprehension of one stimulus
affects comprehension of another stimuli.
• word / sentence fragment completion task
• Negative priming (Christie & Klein, 2008, May, Kane,
& Hasher, 1995, MacLeod, Chiappe, & Fox, 2002)
• Different modifications: some distractor determines the
target and decreases activation time.
• Unconscious / masked priming (Dehaene, S.,
Naccache, L., Le Clec'H, G.., Koechlin, E., Mueller, M.,
Dehaene-Lambertz, G., van de Moortele, P. F., &
LeBihan, D., 1998)

Denmark

Perception

Denmark

• Central assumption: stimulus and sensory
are not directly corresponding and
proportional

Denmark

Classically: Threshold
determination
• Method of average error
• Method of minimal changes
• Constant stimuli method

Denmark

Classically: Magnitude
estimation
• Magnitude estimation with standard:
• A stimulus is presented (standard) and a number is
assigned to it (modulus, e.g., 30).
• The following stimuli are evaluated by subjects in that
they report the intensity relative to the standard in
assigning the stimuli numerical evaluation. The ratio
between numerical estimated and sensations.
• Magnitude estimation without standard:
• Subjects can choose their own standard (any number)
and assign it to the first stimulus. All following are
assigned by subject; preserving the ratio between
sensations and numbers

Denmark

• Cross-modality matching:
• Indirect methods to differentiate
between stimuli
• Direct scaling methods
• Methods for stimulus comparison
• Some remarks regarding the use of
semantic differential in

Denmark

• research on attitudes:
• Semantic differential is a fast way how to
measure attitudes. (The crucial point is that no
more than attitudes can be measured in using
semantic differential methodology.)
• Visual categorization and the use of
quantitative methods as complementary
tools in research of semantics in visual
perception.
• basic level prominence effects (Rosch, 1978,
Tversky, Zacks, & Hard, 2008).


12. Conclusion: Constraints

Denmark

• Lexicalization itself / dinscontiuty generation
• Selectivity: hierarchic – both in object encoding and in
encoding objects in events
• Attention
• How much can be actually perceived at a moment
– Saccades and fixations are mediated and correlated with
transsacadic memory (some information retained from one eye
fixation is invoked in processing that occurs during the next
fixation).
• Invariants
• Experiential constraints
• Situational constraints


Semantics in Visual Perception: Methodological Remarks, Issues and Perspectives

More Related Content

Viewers also liked (20)

Similar to Semantics in Visual Perception: Methodological Remarks, Issues and Perspectives (20)

More from nacionalaidentitate (20)

Recently uploaded (20)

Semantics in Visual Perception: Methodological Remarks, Issues and Perspectives