Modelling User Interaction utilising Information Foraging Theory (and a bit of Quantum Theory)

Modelling User Interaction utilising Information
Foraging Theory
(and a bit of Quantum Theory)
Ingo Frommholz
Amit Kumar Jaiswal
Haiming Liu
University of Bedfordshire
ingo.frommholz@beds.ac.uk
Twitter: @iFromm
University of Glasgow
January 27, 2020

Who am I?
2001 Research Assistant, Fraunhofer IPSI, Darmstadt, Germany
Annotation-based retrieval, IR + cultural heritage, text
categorisation, digital libraries
2004 Research Assistant, University of Duisburg-Essen, Germany
PhD in Information Retrieval, probabilistic logic-based IR models,
DB+IR integration, annotation-based IR, machine learning and
IR, digital libraries
2008 Postdoc, University of Glasgow, UK
Interactive Models for Information Access based on Quantum
Mechanics
2011 Lecturer/Senior Lecturer (since 2012), University of Bedfordshire,
Luton, UK
Interactive IR models, probabilistic clustering, user simulation,
recommender systems, text mining and cyber stalking,
cross-cultural IR, high throughput IR, DB+IR integration

Outline
Modelling User Interaction with Information Foraging
A Quantum Perspective
Conclusion

Context of our Work
MSCA-ITN-ETN - EU H2020 Marie Curie European Training
Network (~3m EUR)
13 Early Stage Researchers (PhD students) funded
48 months (started January 2017)
http://www.quartz-itn.eu/

QUARTZ Consortium Members and Partners
Consortium Members
University of Padua, Italy
Open University, UK
University of Bedfordshire, UK
Vrije Universiteit Brussel, Belgium
University of Copenhagen, Denmark
Brandenburg University of Technology, Germany
Linnæus University, Sweden
Partners
Websays SL, Spain
Queensland University of Technology, Australia
Signal AI, London, UK
Staatsbibliothek zu Berlin-Preußischer Kulturbesitz, Germany

Modelling User Interaction with Information
Foraging

Information Foraging Theory
Theory originating from cognitive psychology
[Pirolli and Card, 1999]
Study how users navigate and forecast their behaviour when
searching
Similarity between animals’ food foraging and users’ information
searching strategies
Users usually want to put less effort into their information seeking
process and expect to maximise their information gain.
Information Patch: what users seek, e.g. Web pages, images,
regions of images
Information Scent: a patch has a scent, determined by their
information need and given cues

IFT and Search Engines
Textual and visual representations in search engine result pages
(SERPs) can be perceived in the context of IFT
Users look at patches with the strongest scent
Scent strength determined by textual and visual clues
Examples:
IFT in content-based image recommendation
[Jaiswal et al., 2019a, Jaiswal et al., 2019b]
Query Auto-Completion [Jaiswal et al., 2020]

Content-based Image Recommendation Interface
User-Image-Cue Model
hematic architecture of Content-based Image
s [Jaiswal 2019]:
Based on Pinterest boards. Search for “zoodles”. Board
recommendations (”zoodles”, “noodles”, etc.). Each image includes
(textual or visual) cues associated with it.

Procedure
1. User types query initial ranking of images based on text
features
2. Clicking “Amuse me” generates recommendations
3. User clicks category (e.g., “zoodles or “noodles) images
categorised into category
Methodology/1 - Architecture
The schematic architecture of Content-based Image
RecSys [Jaiswal 2019]:

Procedure
Images are more clear after recommendation / processing user
preferences
ResNet finds objects/patches in images
Further classification is performed to figure out whether a patch
fits the required category, i.e. has a high information scent
Methodology/1 - Architecture
The schematic architecture of Content-based Image
RecSys [Jaiswal 2019]:

Image Representation and Features in Recommendation
Step
1,116 images belonging to two food categories
Image embeddings (image2vec)
Image features (content, texture, colour); text features
(description/title)
Pre-trained ResNet50 model trained on ImageNet to categorise
images

IFT Perspective: Cues and Information Scent
Images or image regions are information patches reached via
cues
Image I consists of n image patches I = {Ipi
,...,Ipn
}
Users follow cues across the information space depending on
their information scent
Quality of cue depends on how effectively is helps minimising the
information access and diet costs

Study 1: Information Scent of User Preferences
Recommendation
Spaghetti Bolognese Zoodles
User Preferences IS User Preferences IS
R1 Bolognese 10 Zoodles 9
R2 Spaghetti 7 Zucchini 8
R3 Recipe 6 Easy 6
R4 Sauce 6 Pasta 5
R5 Easy 3 Chicken 5
User chose 5 preferences for each of the two categories, system
generates recommendation R1,...,R5 of images and image patches for
each of them
Image with “bolognese” or “zoodles” are perceived as cues
More cues mean higher information scent (IS)
Cue images receive large degree of attention, provide maximum
information diet, have lower access costs

Study 2: Detecting interesting cues
Manual judgements for each image — interesting (high scent),
uninteresting (low scent) — based on two food categories
(Pinterest ”bolognese”, “zoodles”) or 10 art categories (WikiArt)
ResNet34 to categorise 1500 randomly selected WikiArt images
into above categories
GridSearch (GS) SVM & Random Forest for Pinterest, XGBoost
for WikiArt. Shape and colour features
Class
Model
Pinterest Collection WikiArt Dataset
GS-SVM GS-Random Forest XGBoost
Scores
Precision Recall F1 Precision Recall F1 Precision Recall F1
uninteresting (0) 0.77 0.85 0.81 0.80 0.89 0.84 0.81 0.62 0.70
interesting (1) 0.81 0.70 0.75 0.85 0.74 0.79 0.47 0.53 0.50

Query Auto-Completion (QAC)
Predict what could be the next character or query item right after
the user types something
Often driven by query logs
Textual features, RNNs, language models
Word embeddings to compute query similarity for QAC
Challenge: anticipate a query that has never been seen before

General QAC Framework
From [Fei and de Rijke, 2016]
2.1. Problem formulation 9
prefix query
completions
query log
prefix
user prefix
query
completions
prefix p query q feature fu feature fv
p1
q11 f11 f’11
q12 f12 f’12
Index
q1n f1n f’1n
…
…
…
pk
qk1 f k1 f’k1
…
…
Online ranking signals
…
time location behavior
Figure 2.1: A basic QAC framework.
In essence, the QAC problem can be viewed as a ranking problem. Given

Multimodal Image QAC
Using images for query expansion/suggestion in image search
Complete textual query with a list of images
Potentially better for image search (no textual formulation for
information need, dyslexia)

Our Model
pk ∈ P set of patches
User inputs a query preﬁx qp, an incomplete query to retrieve an
image
We auto-complete the expected query q as probability
maximisation
qa∗ = argmax
q
P(q|qp,I) = argmax
{t1t2...tn}
P(t1t2...tn|qp,I) (1)
where qa∗ is the query adapted on an image, ti ∈ S is the term in
position i in a sequence S.
qa∗ is used to compute patch probabilities

QAC process
Two-stage process:
1. Textual query auto completion (LSTM based
on [Jaech and Ostendorf, 2018] extended with image features
with Beam Search); feeds into
2. Patch selection and ranking (iBERT)

Multimodal Image QAC Architecture

IFT Explanation
User starts typing a query
Perceptual cues allow her to either continue typing or access
provided suggestion
Each suggestion is a cue
Information scent of cues might be low if query preﬁx is unknown
Beam search to generate query based on image features
Image is considered as a set of patches containing features such
as color, shape, texture, etc.
iBERT: Patch selection to predict images with highest scent

Evaluation
Evaluation of extended LSTM model
Visual Genome (> 100k images and regions with descriptions),
ReferIt (∼ 42k image regions with descriptions)
Comparison with different models (MRR reported in literature)
Model MRR (Seen+Unseen)
MPC [Bar-Yossef and Kraus, 2011] 0.171
Character n-gram (n=7) 0.287
Mitra10K+MPC+λMART [Mitra and Craswell, 2015] 0.278
Mitra100K+MPC+λMART [Mitra and Craswell, 2015] 0.298
NQLM(S)+WE+MPC [Park and Chiba, 2017] 0.345
NQLM(L)+WE+MPC [Park and Chiba, 2017] 0.355
NQLM(L)+WE+MPC+λMART [Park and Chiba, 2017] 0.354
FactorCell [Jaech and Ostendorf, 2018] 0.309
E-LSTM LM (Ours) 0.764

IR Models and Principles
Geometry, Probability and Logics
LUP
pDatalog
VSM LSI
LM
PRP
BM25
iPRP Boolean

LUP
pDatalog
VSM LSI
LM
PRP
BM25
iPRP Boolean
QM

LUP
pDatalog
VSM LSI
LM
PRP
BM25
iPRP Boolean
QMqPRP
QIA

A Language for IR
The geometry and mathematics behind quantum mechanics can
be seen as a ’language’ for expressing the different IR models
[van Rijsbergen, 2004].
Combination of geometry, probability and logics
Leading to non-classical probability theory and logics
Potential uniﬁed framework for IR models
Applications in areas outside physics emerging
Quantum Interaction symposia

IR as Quantum System?
An Analogy
Quantum System IR System
Particles, physical properties in
Hilbert spaces
Documents, relevance, informa-
tion needs in Hilbert Spaces
System state uncertain Information need (IN) uncertain
Observation changes system
state
User interaction changes sys-
tem state
Observations interfere (Heisen-
berg)
Document relevance interferes
Combination of systems Combination of IN facets,
polyrepresentation

Quantum-inspired Information Access
Quantum Probabilities [van Rijsbergen, 2004, Piwowarski et al., 2010]
R
p1
p2
p4
p3
p5
System uncertain about user’s
IN
Expressed by an ensemble S of
possible IN vectors :
S = {(p1,|ϕ1 ),...,(pn,|ϕn )}
Document relevance R as
subspace
Probability of relevance:
Pr(R|d,S) = ∑
i
pi ·Pr(R|d,ϕi )
=||R|ϕ ||2

User Interaction and Feedback
R∗
|ϕ1
|ϕ2
|ϕ5
|ϕ3
Outcome of feedback: Query,
relevant document, ...
Expressed as subspace
Project IN vectors onto
document subspace

User Interaction and Feedback
R∗
|ϕ1
|ϕ2
|ϕ4
|ϕ3
|ϕ5
Outcome of feedback: Query,
relevant document, ...
Expressed as subspace
Project IN vectors onto
document subspace
Document now gets
probability 1
System’s uncertainty
decreases
Also reﬂects changes in
information needs

Textual Representation
IN Space / Documents
|crashi (Term)
|cari (Term)
|jupiteri (Term)
|jupiter crashi
|car crashi
R⇤
topic
|'i
IN space based on term
space
IN vectors made of document
fragments
Weighting scheme (e.g., tf,
tf-idf,...)
Document is relevant to all
INs found in its fragments
Document subspace R
spanned by IN vectors

Polyrepresentation/Multiple Evidence
[Frommholz et al., 2010]
Content Author
Ratings
Comments
Polyrepresentation space as tensor product of single spaces
Probability that document is in total cognitive overlap:
Prpolyrep = Prcontent ×Prratings ×Prauthor ×Prcomments
User interaction may lead us into an entangled state (so far
unexplored relationship between polyrepresentation and
entanglement)

System Architecture
R∗
R∗
Quantum engine
Frontend / UI
User interaction
Subspace representation
State change
New state fed back
to frontend

QIA and IFT for QAC: Sketch of an idea
Feature/embedding space as IN space
QIA model can represent user interaction with the system
Subspaces can represent hierachies; image: subspace Si in
Hilbert space H ; patch: subspace Sip
of subspace Si
Each possible query auto completion represented state vector
|ϕi with probability pi
Each |ϕi gives us probability distribution over patches
(initialised with iBERT)
User chooses patch:
Orthogonal projection onto subspace representing the
patch
Changes probability distribution pi of possible auto
completions
Also changes probability distribution over patches for
each |ϕi
Yields new ranking of completions as well as patch
ranking
R
p1
p2
p4
p3
p5
Possible query completions
and probabilities
Patch probability
for one QAC

Conclusion
Information Foraging Theory
IFT examples: Recommender systems, query auto completion in
image search
Quantum-based IFT model (sketch)

Thanks for your attention!
Questions?

Bibliography I
Bar-Yossef, Z. and Kraus, N. (2011).
Context-sensitive query auto-completion.
In Proceedings of the 20th international conference on World wide
web, pages 107–116. ACM.
Fei, C. and de Rijke, M. (2016).
A survey of query auto completion in information retrieval.
Foundations and Trends in Human-Computer Interaction,
10(4):1–92.
Frommholz, I., Larsen, B., Piwowarski, B., Lalmas, M., Ingwersen,
P., and van Rijsbergen, K. (2010).
Supporting Polyrepresentation in a Quantum-inspired Geometrical
Retrieval Framework.
In Proceedings of the 2010 Information Interaction in Context
Symposium, pages 115–124, New Brunswick. ACM.

Bibliography II
Jaech, A. and Ostendorf, M. (2018).
Personalized language model for query auto-completion.
arXiv preprint arXiv:1804.09661.
Jaiswal, A. K., Liu, H., and Frommholz, I. (2019a).
Effects of Foraging in Personalized Content-based Image
Recommendation.
In Proceedings of the 2nd International Workshop on ExplainAble
Recommendation and Search (EARS 2019) at SIGIR 2019.
Jaiswal, A. K., Liu, H., and Frommholz, I. (2019b).
Information Foraging for Enhancing Implicit Feedback in
Content-based Image Recommendation.
In Proceedings of the 11th Forum for Information Retrieval
Evaluation (FIRE 2019), pages 65–69, Kolkata, India. ACM.

Bibliography III
Jaiswal, A. K., Liu, H., and Frommholz, I. (2020).
Utilising Information Foraging Theory for User Interaction with
Image Query Auto-Completion.
In Proceedings European Conference on Information Retrieval
(ECIR 2020). Springer.
Mitra, B. and Craswell, N. (2015).
Query auto-completion for rare preﬁxes.
In Proceedings of the 24th ACM international on conference on
information and knowledge management, pages 1755–1758.
ACM.
Park, D. H. and Chiba, R. (2017).
A neural language model for query auto-completion.
In Proceedings of the 40th International ACM SIGIR Conference
on Research and Development in Information Retrieval, pages
1189–1192. ACM.

Bibliography IV
Pirolli, P. and Card, S. K. (1999).
Information Foraging.
Psychological Review, 106:643–675.
Piwowarski, B., Frommholz, I., Lalmas, M., and Van Rijsbergen, K.
(2010).
What can Quantum Theory Bring to Information Retrieval?
In Proc. 19th International Conference on Information and
Knowledge Management, pages 59–68.
van Rijsbergen, C. J. (2004).
The Geometry of Information Retrieval.
Cambridge University Press, New York, NY, USA.

Modelling User Interaction utilising Information Foraging Theory (and a bit of Quantum Theory)

More Related Content

What's hot (10)

Similar to Modelling User Interaction utilising Information Foraging Theory (and a bit of Quantum Theory) (20)

More from Ingo Frommholz (7)

Recently uploaded (20)

Modelling User Interaction utilising Information Foraging Theory (and a bit of Quantum Theory)