Multilevel Modeling Data analysis Models

Copyright © E.Y.Li 1 03/05/2025
Multilevel Data Analysis:
Models and Applications
Eldon Y. Li
Chair Professor & Director
Ph.D. Program in Business
Chung Yuan Christian University, Taiwan
http://www.calpoly.edu/~eli
*** All right reserved. Reference to this document should be made as follows: Li, E.Y.
“Multilevel Data Analysis: Models and Applications”, unpublished lecture, Chung Yuan
Christian University, 2019 ***

Copyright (c) E.Y.Li 2 03/05/2025
Agenda
• About me
• Abstract
• Introduction to model analyses
• Why multilevel model?
• When multilevel model?
• How multilevel model?
• Application example 1
• Application example 2

3
About Me
Eldon Y. Li is a university chair professor and former department chair of
MIS at National Chengchi University, an adjunct chair professor of Asia
University in Taiwan, and a former professor and coordinator of the MIS
program at College of Business, California Polytechnic State University,
San Luis Obispo, California, USA. He was the dean of College of
Informatics and the director of Graduate Institute of Social Informatics at
Yuan Ze University in Taiwan, as well as professor and founding director at
the Graduate Institute of Information Management at the National Chung
Cheng University in Chia-Yi, Taiwan. He received his PhD from Texas Tech
University in 1982. He is the editor-in-chief of several international
journals. He has published more than 250 papers in various topics related
to innovation and technology management, human factors in information
technology (IT), strategic IT planning, software quality management, and
information systems management. His papers appear in Journal of
Management Information Systems, Research Policy, Communications of the
ACM, Internet Research, Expert Systems with Applications, Computers &
Education, Decision Support Systems, Information & Management,
International Journal of Medical Informatics, Organization, among others.
03/05/2025

Conventional survey studies usually collect individual
data from different groups and analyze them
independently in each group, unless there is no
significant group difference. In contrast, multilevel
model analysis allows individual data with
organizational differences to be included in one
regression model by treating these differences as
higher-level independent variables. Such kind of model
is known as hierarchical linear model (HLM). This lecture
introduces various research models and discusses why,
when, and how to perform multilevel model analysis.
The applications of multilevel model analysis are
elucidated using two published studies in information
Multilevel Model Analysis:
Method and Applications
Abstract

Introduction – Model Analysis
• Relational model (predictive)
• Causal model (static)
• Behavioural model (dynamic)
• Process model (staging)
• Mediation model (partial vs. full)
• Moderation model
• Moderated mediation (MoMe) model
• Mediated moderation (MeMo) model
• Multilevel model
• Mixed model

Relational model (predictive)
Source: Li, E.Y.* and Soenen, L. (1994) "
Dollar Value of the Yen and Stock Price Reactions in Japan," Journal of Global
Topix Small
Topix Large
Topix Composite
Nikkei 225
Yen/Dollar
Exchange (-1)
Yen/Dollar
Exchange (-1.5)
Yen/Dollar
Exchange (-2)
Yen/Dollar
Exchange (-3)

Relational
model
(predictive
)
Source: Li, E.Y.* (1994)
"
Artificial Neural Networ
ks and Their Business
Applications
," Information &
Management
(Elsevier), Vol. 27, No.
5, October, pp. 303-
313.

Causal model (static)
Source: Wu, Y.L., Li, E.Y.*, and Chang, W.L. (2016) "
Nurturing user creative performance in social media networks: an integration of habit
of use with social capital and information exchange theories

Behavioral model (dynamic)
Theory of Reasoned Action
Source: Fishbein, M., & Ajzen, I. (1975). Belief, Attitude, Intention, and Behavior: An
Introduction to Theory and Research. Reading, MA: Addison-Wesley.

Process model (staging)
Source: Bhattacherjee, A. and Premkumar, G. (2004).
Understanding changes in belief and attitude toward information technology usage: a
theoretical model and longitudinal test.

Partial mediation model
Source:
Huang, Y.H.*,
Li, E.Y., and
Chen, J.S.
(2009.3) "
Information S
ynergy As the
Catalyst Betw
een IT Capabil
ity and Innova
tiveness: Empi
rical Evidence
from Financial
Service Secto
r
," Information
Research: An
International
Electronic
Journal, Vol.
14, No. 1, pp.
1-11.
Figure 1: Research Model

Full mediation model
Source: Yen, H.J.R., Li, E.Y. and Niehoff, B. (2008.9).
Do organizational citizenship behaviors lead to information system success? testing the
mediation effects of integration climate and project management.
Information & Management, 45 (6), 394-402.

Moderation model
(conceptual)
B C
A
D

Moderation model (statistical)
B C
A
D D×
A D×
B
C = β0 + β1 A+ β2 B+ β3 D+ β4 D×
A+ β5
D×
B+

Moderation model
Source: Ajzen, I. “TPB Diagram”, available at
http://people.umass.edu/aizen/tpb.diag.html

Moderated mediation model
Source: Edwards, J. R., & Lambert, L. S. (2007). Methods for integrating moderation and mediation:
a general analytical framework using moderated path analysis. Psychological methods, 12(1), 1.
The MoMe model occurs when the treatment effect of an independent variable
A on an outcome variable C via a mediator variable B differs depending on
levels of a moderator variable D. Specifically, either the effect of A on the B,
and/or the effect of B on C depends on the level of D. But, there is no overall
moderation of A on C. (Wikipedia, 2019)
D
A C
B
D

Moderated mediation model
Source: Yen, H.J.R., Thi, H.P., and Li, E.Y.* (2021).
Understanding customer-centric socialization in tourism services. Service Business (Springer),

Mediated moderation model
Source: Baron, R. M., & Kenny, D. A. (1986). The moderator–mediator variable distinction in social psychological
research: Conceptual, strategic, and statistical considerations. Journal of personality and social psychology, 51(6),
1173.
The MeMo model occurs when the treatment effect of an independent variable
A on an outcome variable C via a mediator variable B differs depending on
levels of a moderator variable D. Specifically, either the effect of A on the B,
and/or the effect of B on C depends on the level of D. And, there is an overall
moderation of A on C.
D
A C
B

Source: Venkatesh, V., Morris, M.G., Davis, G.B., and Davis, F.D. (2003).
User Acceptance of information technology: Toward a unified view. MIS Quarterly,

Source: https://arxiv.org/pdf/1701.08862.

(generalized)
Source: https://arxiv.org/pdf/1701.08862.

Mixed model
Source: DeLone, W. and McLean, E. (2003). The DeLone
and McLean Model of information systems success: a ten-year update. Journal of
Management Information Systems, 19 (4), 9-30.

Mixed model
Source: Wixom, B.H. and Todd, P.A. (2005).
A theoretical integration of user satisfaction and technology acceptance. Information

Multilevel
model
• Source: Li, E.Y.*
and Shani, A.B.
(1991 Spring) "
Stress Dynamics of
Information Syste
ms Managers: A Co
ntingency Approac
h
," Journal of
Management
Information
Systems (ME
Sharpe), 7 (4), 107-
130.

Multilevel
model
• Source: Li, E.Y.*,
and Ko, S.-F.(2021).
Employee's market
orientation behavio
r and firm's internal
marketing mechan
ism
:
A multilevel perspe
ctive of job perform
ance theory
. Sustainability
(MDPI), 13(12),
6972, 1-25.

Why multilevel model?
Company
1
Company
2
Company
3
Company 30
Company
4

When multilevel model?
• Condition for applying multilevel
analysis
– Variance between companies  relatively
large
– Variance within a company (between
employees)  relatively small.
• If variance between companies is small,
merge data and use single level
analysis.
• If variance within a company (between
employees) is large, remove outliers or

When? (Cont.)
• Intraclass agreement index (rwg)
– Also called Within-organization agreement
index
– rwg (J) value indicates the degree to which the
responses to a measurement scale by
members of the same organization
converge.
– rwg (J) value >0.70 (James et al., 1984)
James LR, Demaree RG, Wolf G (1984) Estimating within-group interrater reliability with and without bias. J
Appl Psychol 69:85response

When? (Cont.)
• Intraclass correlation coefficients
– ICC1 compares the between-organizations
variance with the within-organization
variance to indicate the portion of variance
in individual responses (MSW) accounted
for by the between-organizations
difference (MSB).
MSW
*
1)
-
(K
+
MSB
MSW
-
MSB
ICC1 = > 0.12 (Bliese
2000)
K = the average sample size from a company
Bliese, P.D. Within-group agreement, non-independence, and reliability: Implications for data aggregation
and analysis. In K.J. Klein and S.W.J. Kozlowski (eds.), Multilevel Theory, Research, and Methods in
Organizations. San Francisco: Jossey-Bass, 2000, pp. 349–381.

When? (Cont.)
• Intraclass correlation coefficients (cont.)
– ICC2 reveals the reliability of the mean of an
organization-level variable. If low reliability,
multilevel analysis is not needed.
– That is, MSB should be large, MSW should
be smaller and no more than 40% of MSB.
> 0.60 (Bliese
2000)
MSB
MSW
-
MSB
=
ICC2

How multilevel model?
Source: Huang, M.H., Li, E.Y.*, and Wong, C.S. (2015) "
A Multilevel Model of Information System Success in the User Department: Integrating
Job Performance Theory and Field Theory
," International Journal of Information Systems and Change Management

Application example 1
"
A Multilevel Model of Informatio
n System Success in the User De
partment: Integrating Job Perfor
mance Theory and Field Theory
"

Application example 1 -
Originality
• The IS Success (ISS) model of DeLone and McLean
(1992, 2003)
IS’s
Qualitie
s
User’s
Satisfactio
n
Organization’
s Net
Benefits
IS
Developers
Users ???
03/05/2025
Copyright © E.Y.Li 35
User
Managers

Research questions
• What are the factors influencing net
benfits of user department’s IS
appcliation (evaluated as user
manager’s job performance)?
• What are the interaction effects of these
factors on net benefits?

Originality (Cont.)
• The ISS model of DeLone and McLean (1992, 2003)
prescribes IS’s quality (including information,
system, and service qualities), user’s satisfaction,
and organization’s net benefits as the three
integrated components. While IS developer is
responsible for IS quality, users are concerned with
satisfaction, leaving net benefits unattended. This
study proposes user department’s IS performance be
the surrogate of organization’s net benefits.

Underlying theories
03/05/2025

Method - measures
• Job performance  User department’s
IS performance.
• Opportunity  Top management
support.
• Capability  User manager’s
knowledge
about IS applications.
• Willingness  User manager’s attitude
toward IS applications.

Research model
Source: Huang, M.H., Li, E.Y.*, and Wong, C.S. (2015) "
A Multilevel Model of Information System Success in the User Department: Integrating
Job Performance Theory and Field Theory
," International Journal of Information Systems and Change Management
(Inderscience), Vol. 7, No. 4, pp. 286-307.

Solution model
UDISPij
represents the ith
individual score of UDISP in jth
organization.
TMSj
represents the aggregate score of TMS in jth
organization.
ISKij
represents the ith
individual score of ISK in jth
organization.
ISAij
represents the ith
individual score of ISA in jth
organization.
γkl
represents the slope of the kth
level-1 predictor interacting
with the lth
level-2 predictor.
Ukj
is a normal distribution and represents the residual of
slope of kth
level-1 predictor in the jth
organization.
rij
is a normal distribution and represents the residual of
regression model in individual level.

Method - subjects
• Data requirements:
– Using similar information systems: ERP
– In similar industry: Manufacturing
– System experience: At least 1 year
– At least 10 companies in the industry: 42
– At least 5 data ponits (departments) in each
company: Average 6-7 deparments, Total
283

Analysis
• Focus group was used to ensure face validity of
the survey questionnaire.
• Valid survey data were collected from 283 user
managers and 42 top managers of 42 different
Chinese manufacturing companies in which
ERP systems were being utilized.
• Each company sample=6 to 7  K=6;
ICC1=0.253 >0.12; ICC2=0.670 >0.60.
• The model can be validated by using
Hierarchical Linear Modelling (HLM) software.

Model specification in HLM
Level-1 Level-2
Coefficients Predictors
---------------------- ---------------
INTRCPT1, B0 INTRCPT2, G00
TMS, G01
ISK slope, B1 INTRCPT2, G10
TMS, G11
ISA slope, B2 INTRCPT2, G20
TMS, G21
Summary of the model specified (in equation
format)
---------------------------------------------------
Level-1 Model
Y = B0 + B1*(ISK) + B2*(ISA) + R
Level-2 Model
B0 = G00 + G01*(TMS) + U0
B1 = G10 + G11*(TMS) + U1
B2 = G20 + G21*(TMS) + U2

Results

Example 1 Conclusions
• Top management support, user-manager
knowledge, and user-manager attitude all
affect the level of UDISP significantly. (H1, H2,
H3 supported)
• Top management support significantly
moderates the relationship between user-
manager attitude and UDISP. (H5 supported)
• The interaction effect of top management
support and user-manager knowledge on
UDISP is not significant. (H4 not supported)

Application example 2
A Multilevel Approach to Examin
e Employees' Loyal Use of ERP S
ystems in Organizations

Application example 2 -
Originality
• In the IS literature, organizational factors have been
analyzed at the same level as individual factors. This
study intends to breaks through the single-level lens.
03/05/2025
Organization
al Factors
Individual
Factors
IS Success

Research questions
• What are the factors influencing
employee’s loyal use of ERP system at
the individual level and at the
organizational level?
• What are the interaction effects of these
factors on loyal use of ERP?

Underlying theories
03/05/2025
Individual Level
Organization Level
Interactionism Paradigm Situational Strength Theory
Employee’s
Perceived Workload
Social Information Processing Theory
Social Learning Theory
Rational Choice Theory
Cost–Benefit
Analysis
Loyal Use
Individual
Perceptions
Causal relation
Construct
Theoretical foundation
Bottom-up/Top-down process

Research model
Source: Yen, H.J., Hu, P.J.H., Hsu, S.H.Y., and Li, E.Y.* (2015) "
A Multilevel Approach to Examine Employees' Loyal Use of ERP Systems in Organization
s

Solution model
Department-level Model:
Employee’s Loyal Use: ELUij
= β0j
+ β1j
(EPBij
) +β2j
(EPWij
) +rij
,
Company-level Model:
β0j
= γ00
+ γ01
(OLSQj
) + γ02
(OLIQj
) + γ03
(OLSOCBj
) +U0j
β1j
= γ10
+ γ11
(OLSQj
) + γ12
(OLIQj
) + γ13
(OLSOCBj
) +U1j
β2j
= γ20
+ γ21
(OLSQj
) + γ22
(OLIQj
) + γ23
(OLSOCBj
) + U2j
The final formula is as follows:
ELUij
= γ00
+ γ01
(OLSQj
) + γ02
(OLIQj
) + γ03
(OLSOCBj
) +U0j
+(γ10
+ γ11
(OLSQj
) + γ12
(OLIQj
) + γ13
(OLSOCBj
) +U1j
) EPBij
+(γ20
+ γ21
(OLSQj
) + γ22
(OLIQj
) + γ23
(OLSOCBj
) + U2j
)EPWij

Solution model
ELUij represents the ith
individual score of ELU in jth
organization.
OLSQj represents the aggregate score of OLSQ in jth
organization.
OLIQj represents the aggregate score of OLIQ in jth
organization.
OLSOCBj represents the aggregate score of OLSOCB in jth
organization.
EPBij represents the ith
individual score of EPB in jth
organization.
EPWij represents the ith
individual score of EPW in jth
organization.
γkl represents the slope of the kth
level-1 predictor interacting with
the lth
level-2 predictor.
Ukj is a normal distribution and represents the residual of slope of kth
level-1 predictor in the jth
organization.
rij is a normal distribution and represents the residual of regression
model in individual level.

Method - measures
• Loyal use  Proactive, extended use
and
willingness to recommend
such uses to others.
• Benefits
• Workload
• System quality
• Information quality
• Service-oriented OCB

Method - subjects
• Data requirements:
– Using similar information systems: ERP
– Employee 39 scales; IS staff 13 scales.
– 15 employees and 5 IS staffs per firm
– Final sample: 485 employees, and 166 IS
staffs
– System experience: At least 0.5 year
– At least 10 companies in the industry: 47
– At least 5 data ponits (employees) in each
company: Average 10-11 employees, Total
485

Analysis
• Focus group was used to ensure face validity of
the survey questionnaire.
• Valid survey data were collected from 47
different Taiwanese companies in which ERP
systems were being utilized.
• Each company sample=10 to 11  K=10;
• rwg IQ=.94 ; rwg SQ=.86 ; rwg SOCB=.94 >0.7
• ICC1IQ=0.14 ; ICC1SQ=0.16 ; ICC1SOCB=0.28 >0.12;
• ICC2IQ=0.63 ; ICC2SQ=0.66 ; ICC2SOCB=0.58 >0.60.

Results
Individual Level
OL-IQ OL-SOCB
Employee’s
Perceived Workload
Employee’s Loyal Use
Organization Level
OL-SQ
H1
-0.11* ns -0.12*
ns
H2
ns
0.32***
ns
ns
0.47***
Employee’s
Perceived Benefits
IL-IQ
0.11*
-0.29***
IL-SQ
IL-SOCB
IL-SQ: 0.15*** IL-IQ: 0.13*** IL-SOCB: -0.02
0.14*
-0.07***
0.23***

Example 2 Conclusions
• IL-SQ, IL-IQ, and IL-SOCB have positive effects
on PB, while IL-SQ has negative effect on PW.
• IL-SQ and IL-IQ have positive effects on ELU.
• PB has positive effect on ELU, while PW has
negative effect.
• OL-SQ and OL-SOCB negatively moderate PB's
positive effect on ELU.
• OL-IQ positively moderates PW's negative
effect on ELU.

Overall Conclusions
• Multilevel model analysis overcomes the
group differences and analyzes samples from
multiple groups in one regression model.
• When group differences are significant, use
multilevel level analysis; when not significant,
use single level analysis.
• Test rwg, ICC1, ICC2 before multilevel analysis.
• Remove outliers in each group before any
analysis.
• Careful interpretation of results needs
relevant experience.

Huang, M.H., Li, E.Y.*, and Wong, C.S. (2015) "A Multilevel
Model of Information System Success in the User
Department: Integrating Job Performance Theory and
Field Theory," International Journal of Information
Systems and Change Management (Inderscience), Vol. 7,
No. 4, pp. 286-307. (EI)
Yen, H.J., Hu, P.J.H., Hsu, S.H.Y., and Li, E.Y.* (2015) "A
Multilevel Approach to Examine Employees' Loyal Use of
ERP Systems in Organizations," Journal of Management
Information Systems (T&F), Vol. 32, No. 4, pp. 144-178.
(SSCI; FT50 Journal List)
Extra readings

Multilevel Modeling Data analysis Models

More Related Content

Similar to Multilevel Modeling Data analysis Models (20)

Recently uploaded (20)

Multilevel Modeling Data analysis Models