An Assessment Of Human-Computer Interaction Research In Management Information Systems Topics And Methods

An assessment of human–computer interaction
research in management information systems:
topics and methods
Ping Zhang*, Na Li
School of Information Studies, Syracuse University, Syracuse, NY 13244, USA
Abstract
As an emerging subfield of Management Information Systems (MIS), Human–Computer
Interaction (HCI) or Human Factors studies in MIS are concerned with the ways humans
interact with information, technologies, and tasks, especially in business, managerial, organi-
zational, and cultural contexts. To date, few studies have either synthesized existing studies or
drawn an overview picture of the HCI subfield in MIS. This paper first provides a framework
of broad HCI issues and concerns. It then reports an assessment of a sample of published HCI
articles in two top MIS journals, MIS Quarterly and Information Systems Research, over a
period of thirteen years (1990–2002). It identifies the main topics studied, the main research
approaches utilized, the research publication patterns, and the needs for future research efforts
in this subfield. The results should be of interest to researchers in this subfield, in the MIS field,
and in other related disciplines for future research, collaboration, and publication.
# 2003 Elsevier Ltd. All rights reserved.
Keywords: Human–computer interaction (HCI); Human factors in information systems (HFIS); Litera-
ture analysis; Literature assessment; Research methods
1. Introduction
Research that synthesizes existing studies to provide an overview of an emerging
field is often scarce but extremely important to advance our understanding of the
current status and suggest future directions of the field. This situation is especially
pressing for an emerging subfield that is an overlapping or intersection of the
two important disciplines, human–computer interaction (HCI) and management
information systems (MIS).
Computers in Human Behavior 20 (2004) 125–147
www.elsevier.com/locate/comphumbeh
0747-5632/$ - see front matter # 2003 Elsevier Ltd. All rights reserved.
doi:10.1016/j.chb.2003.10.011
* Corresponding author. Tel.: +1-315-443-5617; fax: +1-315-443-5806.
E-mail address: pzhang@syr.edu (P. Zhang).

As rightly recognized by Hewett and colleagues (Hewett et al., 1992), there is
no agreed upon definition of the range of topics that form the area of Human–
Computer Interaction. In an attempt to derive and develop educational materials,
these scholars defined HCI as ‘‘a discipline concerned with the design, evaluation
and implementation of interactive computing systems for human use and with the
study of major phenomena surrounding them’’ (Hewett et al., 1992). It is widely
recognized as an interdisciplinary field where many traditional disciplines contribute
to the study of its main issues (Preece, Rogers, Sharp, Benyon, Holland, & Carey,
1994).
Management information systems (MIS) is a community of scholars interested in
the development, use, and impact of information technology and systems in social
and organizational settings (Zhang & Dillon, 2003). MIS has been through a steady
shift from what was a techno-centric focus to a better-balanced technology/organi-
zational/management/social focus (Baskerville & Myers, 2002). User attitudes,
perceptions, acceptance and use of IT have been a long standing issue and a major
theme of MIS since early days in computing (Lucas, 1975; Swanson, 1974).
Human–computer interaction (HCI) or Human Factors studies in MIS are
‘‘concerned with the ways humans interact with information, technologies, and
tasks, especially in business, managerial, organizational, and cultural contexts’’
(Zhang, Benbasat, Carey, Davis, Galletta, & Strong, 2002, p. 334). The emphases
have been on both the ‘‘design, evaluation and implementation’’ aspect and the ‘‘use
and impact in social and organizational settings’’ aspect of information technology
for human use. Since the 1970s, MIS researchers have published abundant studies
concerning HCI issues in many MIS journals. The active HCI-centered minitracks,
sessions and workshops in major MIS conferences, along with the newly formed
AIS Special Interest Group on HCI, have also shown high interests in HCI among
MIS researchers in recent years.
With the fast development and deployment of information systems, information
technology and communication technology (in this paper, we use IS or IT to repre-
sent them all) into every part of our lives, HCI issues become persuasive and
fundamental. They also prompt a need to re-examine what HCI is about and how to
conduct research in this area.
Several studies have been conducted to systematically assess the intellectual
evolution of the MIS field in terms of its theories, topics, research methods, etc.
(Alavi & Carlson, 1992; Culnan, 1987). However, very few studies have drawn an
overview picture of the HCI subfield up to date (Zhang et al., 2002). The purpose of
this study is to reflect on the current status of the subfield in terms of research topics
and research methods, to identify gaps that need to be addressed, and to point out
future research directions.
Specifically, this paper will first depict a new framework of HCI issues, which is
intended to capture the dynamics and richness of the interaction between human
and technology. Utilizing the new framework for subject topics and an existing
framework for research methods, the paper then reports an assessment of a limited
collection of the published HCI articles from two main MIS journals, Management
Information Systems Quarterly and Information Systems Research, on the following
126 P. Zhang, N. Li / Computers in Human Behavior 20 (2004) 125–147

aspects: (1) the research topics being studied, (2) the research methods being utilized,
and (3) the publication patterns of the HCI type research in the two journals over
time. From the findings, we identify and discuss areas of need for future research.
The results of this study can provide useful information to scholars and practi-
tioners in the MIS and the traditional HCI fields for future research, collaboration,
publication, and practice. They can also be helpful for interested doctoral students
to identify potential research topics as dissertation research.
The rest of the paper is organized as the following. Section 2 introduces a frame-
work of broad HCI issues. Section 3 explains the methodology used in this paper for
assessing the published articles. Section 4 presents the assessment results. Section 5
discusses and concludes the paper.
2. A framework of broad HCI issues
Only a few comprehensive frameworks of HCI issues and topics have been devel-
oped so far. This coincides with the observation that no agreed upon definitions of
the range of topics for HCI exist (Hewett et al., 1992). Among these few frame-
works, Eason (1991) proposed a three-level model of HCI model, as depicted in
Fig. 1.
In Fig. 1, Level 1 considers human–computer interaction as a form of conversa-
tion between two participants capable of processing information. Level 2 broadens
the framework to examine user, task, and environmental factors which may affect
task performance. The next level considers the factors which are important when
human–computer interaction takes place within a socio-technical systems frame-
work (Eason, 1991, p. 722). In Eason’s framework, environmental factors at Level 2
mainly refer to the physical environment such as visual display terminals and
physical settings in which computers are used. At Level 3, IT and the interaction
Fig. 1. A three-level model of HCI (Eason, 1991).
P. Zhang, N. Li / Computers in Human Behavior 20 (2004) 125–147 127

between human and computers have impacts on social life by changing the nature of
jobs, the way organizations operate, and the way people interact with each other.
How human–computer interaction related to the society and organizations is not
predetermined. Instead, it is shaped by the way the IT is designed and implemented.
We think this multi-level model captures the importance of contexts of HCI con-
cerns in real world settings. Much emphasis of the human, however, is on cognitive
ergonomics.
Preece and colleagues (Preece et al., 1994) outlined the components of HCI, as
seen in Fig. 2. The factors in HCI range from system constraints, system function-
alities, productivity factors, to task factors, user interface, health and safety factors,
comfort factors, the user, and organizational and environmental factors. The model
is quite comprehensive in identifying all factors that contribute to HCI design. It
also recognizes the user as a complex being with cognitive processes and capabilities
but also with motivation, enjoyment, satisfaction, personality, and experience.
From an education perspective, Hewett and colleagues (Hewett et al., 1992, p. 16)
proposed a framework (Fig. 3) that was intended to identify and tie together the
building blocks of HCI curricula for Computer Science students with a HCI
concentration or major.
Fig. 2. Factors in HCI (Preece et al., 1994).

The framework identifies the need to understand humans in terms of human
information processing, ergonomics, and language and communication. It also
includes use and context by outlining social organization and work, and human–
machine fit and adaptation. The emphases of the framework are computer techni-
ques for designing various system elements (devices, graphics, dialogs) for humans
to interact with, and the approach and process to design, evaluate and implement
interactive systems. This focus is inherent given that the framework was proposed by
the ACM Special Interest Group on Computer–Human Interaction (SIGCHI).
The above frameworks have all contributed to our understanding of the possible
topics in HCI. Yet, in order to understand the intersection or interaction of the two
disciplines (HCI and MIS) and the broad issues studied in the intersection, a new
framework is in need. This framework should be relatively parsimonious, thus easy
to use. It should demonstrate the major components and their relationships, the
dynamic and rich nature of HCI, and the contextual factors that play an important
role in HCI. Fig. 4 is our attempt for such a framework of an overview of the broad
HCI issues and concerns.
The first basic component is Human. There can be different ways of under-
standing humans in general and their specific characteristics pertinent to their
interaction with IT. One way of examining the human is as depicted in Fig. 4, where
four categories of issues can be explored: (1) demographics that are found in many
HCI studies; (2) physical or motor skills, as those investigated in traditional
Fig. 3. ACM SIGCHI curricula for HCI (Hewett et al., 1992).

Ergonomics; (3) cognitive issues that have been examined by many HCI researchers
in a good number of disciplines; and (4) emotion or affect aspect, which has started
to gain attention from HCI researchers recently. The second component is the
technology, broadly defined as including hardware, software, applications, data,
information, knowledge, and supporting personnel and procedures. Fig. 4 indicates
one way of examining technological issues when studying HCI. All the topics in
Fig. 4 are meant to be illustrative, rather than exhaustive.
The thick vertical Interaction arrow (the ‘‘I’’ arrow) between Human and Tech-
nology represents the ‘‘I’’ in the HCI. It is the core or the center of all the actions.
Traditionally, HCI studies were concerned with designing and implementing inter-
active systems for specified users, and the usability issues during the development
process. The box ‘‘Design/Usability’’ on the left side inside the ‘‘I’’ arrow indicates
this emphasis. A big chunk of existing HCI studies in the traditional HCI literature
fits in this box. Its primary focus has been the issues prior to technology release and
actual use. Ideally, concerns from both human and technology should influence
design and usability issues. Thus the labeling is meant to be bi-directional.
We argue that this view of HCI centered on design and usability is narrow and
limited. It misses the other half of the cycle that has a significant impact on this first
half and the entire interaction experience a human has. John Carroll and colleagues,
more than a decade ago, illustrated the task-artifact cycle by stating that a task
sets requirements for the development of artifacts, and the use of an artifact often
redefines the task for which the artifact was originally developed (Carroll, Kellogg,
& Rosson, 1991). This cycle idea supports the evolutionary view of examining HCI
Fig. 4. A framework of broad HCI issues and concerns.

design and usability, which is depicted in the box on the right side inside the ‘‘I’’
arrow, ‘‘Use/Evaluation/Impact’’ in Fig. 4. This second half of the Interaction is
concerned with the actual use of technology by users in real contexts, their eval-
uation of the technology, and the impact of such use and evaluation. It is important
to note that design and usability studies should be informed by what we have
learned from the use, evaluation and impact of the same or similar technologies.
Thus the latter has implications for the former. Historically, this half has been
the focal concern for the MIS field, along with organizational psychology, social
psychology and social science. In the MIS field, studies on individual reactions to
technology (Compeau, Higgins, & Huff, 1999), IS evaluation from both individual and
organizational levels (Goodhue, 1997, 1998; Goodhue & Thompson, 1995), and user
technology acceptance (Davis, 1989; Venkatesh & Davis, 2000) all fall in this area.
The picture with Human, Technology, and Interaction alone is still incomplete.
Nothing happens in a vacuum. The interaction experience is relevant and important
only when humans use technologies to support their primary tasks within certain
contexts, being organizational, social or societal. Normally, humans use technolo-
gies not for the sake of technologies but for supporting their primary tasks, being
job related or entertainment oriented. In addition, tasks are carried out in a certain
setting or context that imposes constraints or significance for doing and completing
the tasks. Three contexts are identified: organizational context, social context, and
global context. The task and context boxes add the dynamic and essential meanings
to the interaction experience the human has with technology. In this sense, studies
on interaction are moderated by tasks and contexts. The two horizontal arrows
connecting with Task and Contexts represent this fact.
3. Methodology
3.1. Article selection
Published research articles need to be selected in order for the authors to identify
research topics being studied, research methods being utilized, and publication pat-
terns of the HCI studies in MIS. Owing to the exploratory nature of this study, two
MIS journals for a thirteen year period of 1990–2002 were considered as the sources
of the articles for analysis: Management Information Systems Quarterly (MISQ) and
Information Systems Research (ISR). Only articles on research and education are
considered for the analysis. Thus the pool of candidate articles excludes editorial
introductions, editorial notes, executive summaries, or announcements.
Each paper in the pool was initially screened for HCI relevance by the criterion
that the paper should address one or more aspects that fit the HCI framework out-
lined in Fig. 4. A paper is excluded if (1) it is concerned with group support systems
or group decision support systems but does not approach it from human aspects
either at the individual or collaborative level; (2) it is about pure system design or
development methods or processes without linking to user’s considerations; or (3) it
is concerned with the personnel or human resource management aspect of human

factors issues related to IT. This initial screening yielded a total of 171 articles. Each
paper was then coded according to the schemes to be explained below. During the
coding process, each paper was evaluated again for its relevance. As a result, 20
papers were excluded and 151 were considered for the final analysis.
3.2. Classification schemes
Each of the selected articles was classified and coded in two ways: by research
topic according to the HCI framework proposed in Section 2, shown as Fig. 4, and
by research method or type according to Alavi and Carlson’s research type frame-
work (Alavi & Carlson, 1992). The latter was modified slightly to reflect the paper
collection in this study.
3.2.1. Topic
In the MIS field, there are several attempts in providing a classification of topical
subjects of studies. Culnan, for example, identified several research streams of MIS
research by examining MIS publications during the period of 1980–1985 (Culnan,
1987). Barki and colleagues developed the MIS keyword classification scheme by
examining the keywords in published MIS research literature (Barki, Rivard, &
Talbot, 1988, 1993). Despite the influence and wide use of these classifications, we
note two limitations of using them for this study. First, they are intended to classify
the entire MIS field, not for a single subfield. Second, they were developed more
than a decade ago and have not been updated lately to reflect the development of the
fields and changes of research foci.
In this paper, we classify HCI studies according to the broad HCI issues depicted
by Fig. 4. Specifically, at a higher level, we classify an article by whether it is about
issues or concerns that occur at the design/usability stage where the technology/
artifact is still ‘‘in house’’, or whether it is concerned with issues that occur after the
technology/artifact is released and in use in a certain context. Inside each category,
we further divide the topics into different aspects. In addition, we identify several
articles that are concerned with the general research issues such as methodology and
measurement issues, in this subfield. Thus we include this type (also considering
articles dealing with teaching or education aspect of this subfield) into the third
broad category. Table 1 represents the topic classification scheme developed and
used in this paper. It was initially pre-tested by a set of 34 papers and evolved
and refined during the rest of the coding process.
3.2.2. Type/method
Alavi and Carlson’s research type framework (Alavi & Carlson, 1992) is employed
in this study owing to its comprehensiveness and wide acceptance in the MIS com-
munity (Pervan, 1998; Romano & Fjermestad, 2001). Fig. 5 depicts the framework.
At the highest level, the framework distinguishes between empirical and none-
mpirical articles. The empirical articles capture the essence of research by relying on
observation and are further divided into those that describe objects and those that
describe events or processes. Nonempirical articles are those that are primarily based

Table 1
Research topic scheme
ID Category Description and examples
A IS/IT development Concerned with issues that occur at the stage
of IS/IT development and relevant with the
relationship between human and technology.
Focus on the process where IS/IT is developed.
The artifact is still ‘‘in house’’ and being worked
on before release.
A01 Development methods and tools Structured approaches, Object-oriented approaches, CASE
tools, Social-cognitive approaches for developing IS/IT
A02 User analyst involvement User involvement, User participation, User-analyst difference,
User-analyst interaction
A03 Software/hardware development Programmer/analyst cognition studies, Design and development
of specific or general applications or devices
A04 Software/hardware evaluation System effectiveness, System efficiency, System quality, Information
quality, System reliability, System flexibility
A05 User interface design & development Interface metaphors, Information presentations, Multimedia
A06 User interface evaluation Instrumental usability (e.g. ease of use, low error rate, ease of
learning, retention rate, satisfaction), Accessibility, Information
presentation evaluation
A07 User training User training issues or studies during IS/IT development (prior
product release and use)
B IS/IT use,
evaluation, impact
Concerned with issues that occur when humans
use and/or evaluate IS/IT; issues related to the
reciprocal influences between IS/IT and humans.
The artifact is released and in use in real context.
B01 Behavior Theory of Reasoned Action, Theory of Planed Behavior, Social
Cognition Theory, Self-Efficacy, Perception, Belief (expectation),
Intention, Behavior, Acceptance, Adoption, Resistance, Continue, Use
(continued on next page)
P.
Zhang,
N.
Li
/
Computers
in
Human
Behavior
20
(2004)
125–147
133

Table 1 (continued)
ID Category Description and examples
B02 Attitude Attitude, Satisfaction, Preference
B03 Learning Learning models, Learning processes, Training in general (different
from user training as part of system development)
B04 Motivation Motivation (intrinsic, extrinsic), Expectancy, Incentives
B05 Emotion Emotion, Affect, Hedonic quality, Flow, Enjoyment
B06 Performance Performance, Productivity, Effectiveness, Efficiency
B07 Trust Trust, Risk, Loyalty, Security, Privacy
B08 Ethics Ethical belief, Ethical behavior, Ethics
B09 Individual differences Personality, Trait, Cognitive style (e.g. visual vs. verbal oriented,
field dependent/independent), Locus of control, Learning style
B10 Individual demographics Age, Gender, Education, Cultural background, Experience,
Knowledge
B11 Interpersonal relationship Conflict, Interdependence, Agreement/Disagreement, Interference,
Tension, Leadership, Influence
B12 User support Issues related to information center, end-user computing support,
general user support
C Research & education Address research and education issues/concerns
C01 Research
C02 Education
134
P.
Zhang,
N.
Li
/
Computers
in
Human
Behavior
20
(2004)
125–147

on ideas, frameworks, and speculations rather than on systematic observation. The
three italic types under empirical for events/processes in Fig. 5 are modifications to
Alavi and Carlson’s original framework: the original ‘‘case study’’ is divided into
‘‘positivist case study’’ and ‘‘interpretive case study.’’ Individual-based ‘‘Interview’’
is added to the framework. Table 2 provides the descriptions and examples for the
detailed research types reflected in this study.
3.3. Procedure
Two researchers independently evaluated and coded an initial set of 34 papers
to refine the topic framework and to get familiar with the coding process. All
disagreements were discussed and resolved. Then, each of the remaining articles was
evaluated for relevance and coded according to the two classification schemes by the
two researchers independently. A coding worksheet was developed in Excel to
record coding results and explanations when necessary. An article may have multi-
ple subject topics as the essential inquiries proposed by the authors. Only the main
topics of the paper were considered when assigning codes for subject topics. The
number of topics per article ranged from one to six with a mean of 2.05 and a
standard deviation of 1.06. A paper could also be assigned more than one research
Fig. 5. Research type framework. *Modifications to Alavi & Carlson’s original work.

Table 2
Research type scheme and examples
ID Category name and description Example 1 Example 2
1 Non-Empirical
1.1 Conceptual
Orientation
1.1.1 Frameworks: Proposes a framework
for defining the content and scope
of HCI in MIS context, and provides
directions.
None
1.1.2 Conceptual model of a process or
structure: presents an integrated,
schematic representation of a HCI-related
process, structure, behavior, activity,
organization, method, etc.
George, J. F. (1996).
Computer-Based Monitoring:
Common Perceptions and
Empirical Results. MIS Quarterly,
20(4), 459–480.
Orlikowski, W. J., & Iacono,
C. S. (2001). Research
Commentary: Desperately
Seeking the IT in IT
Research–A Call to
theorizing the IT Artifact.
Information Systems
Research, 12(2), 121–134.
1.1.3 Conceptual overviews of ideas, theories,
concepts, etc.: contains an overview of
many concepts or theories in one or
more areas, and does not propound or
support any individual theory, idea,
or approach.
Gerlach, J. H., & Kuo, F.-Y. (1991).
Understanding Human Computer
Interaction for Information
Systems Design. MIS Quarterly,
15(4),
527–550.
1.1.4 Theory from reference disciplines: presents
theory or theories drawn from outside the
HCI sub-field but applied within an
HCI context
None
136
P.
Zhang,
N.
Li
/
Computers
in
Human
Behavior
20
(2004)
125–147

Table 2 (continued)
1.2 Illustrative
1.2.1 Opinion (pure, or supported by examples):
gives advice and guidance for practice,
in the form of rules and recommendations,
steps or procedures to be followed, hints
and warnings, etc. May be supported by
examples and applications.
Pitt, L. F., Watson, R. T., &
Kavan, C. B. (1997). Measuring
Information Systems Service
Quality: Concerns for a Complete
Canvas. MIS Quarterly, 21(2),
209–221.
Alavi, M., & Leidner,
D. E. (2001). Research
Commentary:
Technology-Mediated
Learning: Call for Greater
Depth and Breadth of
Research. Information
Systems Research,
12(1), 1–10.
1.2.2 Opinion (supported by personal experiences):
as for 1.2.1., but also describes the author’s
experience in some relevant context.
None
1.2.3 Description of a tool, technique, method,
model, etc.: usually highly specific and
detailed, as well as technically or
methodologically precise.
Tan, F. B., & Hunter, G. M. (2002).
The Repertory Grid technique:
A Method For The Study of
Cognition in Information Systems.
MIS Quarterly, 26(1), 39–57.
Gordon, M. D. and
Moore, S. A. (1999)
Depicting the Use and
Purpose of Documents to
Improve Information
Retrieval, Information
Systems Research,
10, 1, 23–37.
1.3 Applied Concepts
1.3.1 Conceptual frameworks and applications:
contains both conceptual and illustrative
elements. May present some concept or
framework and then describe an
application of it.
Sharda, R., & Steriger, D. M. (1996).
Inductive Model Analysis Systems:
Enhancing Model Analysis in
Decision Support Systems.
Information Systems Research,
7(3), 328–341.
Galletta, D. F., &
Heckman, R. L. (1990).
A Role Theory Perspective
on End-User Development.
Information Systems
Research, 1(2), 168–187.
P.
Zhang,
N.
Li
/
Computers
in
Human
Behavior
20
(2004)
125–147
137

Table 2 (continued)
2 Empirical
2.1 Objects
2.1.1 Descriptions of types or classes of products,
technologies, systems, skills, etc.
None
2.1.2 Descriptions of a specific application,
system, installation, program, etc.
Ahrens, J. D., & Sankar, C. S. (1993).
Tailoring Database Training for
End Users. MIS Quarterly, 17(4),
419–439.
2.2 Events/process
2.2.1 Lab experiment: manipulates independent
variable; controls for intervening variables;
conducted in controlled settings.
Weber, R. (1996). Are Attributes
Entities? A Study of Database
Designers’ Memory Structures.
Information Systems Research, 7(2),
137–162.
Yoo, Y., & Alavi, M. (2001).
Media and Group
Cohesion: Relative
Influences on Social
Presence, Task Participation,
and Group Consensus.
MIS Quarterly, 25(3),
371–390.
2.2.2 Field experiment: as for lab experiment,
but in a natural setting of the phenomenon
under study.
Alavi, M., Marakas, G. M., &
Yoo, Y. (2002). A Comparative
Study of Distributed Learning
Environments on Learning Objects.
Information Systems Research, 13(4),
404–415.
Piccoli, G., Ahmad, R.,
& Ives, B. (2001). Web-based
virtual learning environments:
a research framework and
a preliminary assessment
of effectiveness in basic IT
skills training. MIS
Quarterly, 25(4), 401–426.
138
P.
Zhang,
N.
Li
/
Computers
in
Human
Behavior
20
(2004)
125–147

Table 2 (continued)
2.2.3 Field study: No manipulation of independent
variables, involves experimental design but no
experimental controls, is carried out in the
natural settings of the phenomenon of interest.
Barki, H., & Hartwick, J. (1994).
User Participation, Conflict, and
Conflict Resolution: The Mediating
Roles of Influence. Information
Systems Research, 5(4), 422–438.
Agarwal, R., Sambamruthy, V.,
& Stair, R. M. (2000).
Research Report: The Evolving
Relationship
Between General and Specific
Computer Self-Efficacy? An
Empirical Assessment.
11(4), 418–430.
2.2.4 Positivist case study: investigates one or a
few cases in details from a positivist
perspective, assumes an objective reality
existing independent of humans, may
involve hypothesis testing to discover
the reality.
Yoon, Y., Guimaraes, T.,
& O’Neal, Q. (1995).
Exploring the Factors
Associated
with Expert Systems
Success. MIS Quarterly,
19(1), 83–106.
Lawrence, M., & Low, G. (1993). Exploring
Individual User Satisfaction Within User-Led
Development. MIS Quarterly,
17(2), 195–208.
2.2.5 Interpretive case study: studies one or a
few cases from an interpretive perspective,
assumes interactions between researchers
and the phenomenon under investigation,
attempts to understand the phenomenon
through assessing meanings.
George, J. F. (1996).
Computer-Based Monitoring:
Common Perceptions and
Empirical Results. MIS Quarterly,
20(4), 459–480.
Davidson, E. J. (2002).
Technology frames and
framing: A socio-cognitive
investigation of requirements
determination. MIS
Quarterly, 26(4), 329–358.
2.2.6 Survey: Involves large numbers of
observations; the research uses an
experimental design but no controls.
Compeau, D. R., Higgins, C. A.,
& Huff, S. (1999). Social Cognitive
Theory and Individual Reactions
to Computing Technology:
A Longitudinal Study. MIS Quarterly,
23(2), 145–158.
Chwelos, P., Benbasat, I.,
& Dexter, A. S. (2001).
Research Report: Empirical
Test of an EDI Adoption
Model. Information Systems
Research, 12(3), 304-321.
P.
Zhang,
N.
Li
/
Computers
in
Human
Behavior
20
(2004)
125–147
139

Table 2 (continued)
2.2.7 Development of instruments: description
of development of instrument/measurement
or classification scheme, validation
of instruments.
Agarwal, R., & Prasad, J. (1998).
A conceptual and operational
definition of Personal Innovativeness
in the domain of information
technology. Information Systems
Research, 9(2), 204–215.
Palmer, J. W. (2002).
Web site usability, design,
and performance metrics.
13(2), 151–167.
2.2.8 Ex-post description of some project or event:
interest in reporting the results of
the project develops after the project is
complete (or is partially complete)
None
2.2.9 Secondary data: Uses data from secondary
sources, i.e., data collected by
sources other than the researchers.
Chin, W. W., & Newsted, P. R.
(1995). Research Report: The
Importance of Specification in
Causal Modeling: The Case of
End-user Computing Satisfaction.
6(1), 73–81.
Beath, C. M., &
Orlikowski, W. J.
(1994). The Contradictory
Structure of Systems
Development Methodologies:
Deconstructing the IS-user
Relationship in Information
Engineering. Information
Systems Research, 5(4),
350–377
2.2.10 Interview: conducted on an
individual-base.
Straub, D. W. (1994). The Effect
of Culture on IT Diffusion:
E-Mail and FAX in Japan
and the U.S. Information
Systems Research, 5(1), 23–47.
Vandenbosch, B., &
Higgins, C. (1996).
Information Acquisition and
Mental Models: An
Investigation
into the Relationship Between
Behaviour and Learning,.
7(2), 198–214.
140
P.
Zhang,
N.
Li
/
Computers
in
Human
Behavior
20
(2004)
125–147

type if the authors utilized multiple methods. Twenty articles (13%) utilize two
methods while the rest of the articles use only one method.
The inter-rater reliabilities for the entire set of 151 final articles are 64% for topic
and 83% for type. 70% is normally an acceptable inter-rater reliability rate. The
relatively low agreement for topic has to do with the fact that many papers have
multiple topics and only the main topics should be included in the coding. For this
complex coding situation (each paper has one to six topics with a mean of 2 and std
of 1, compared to normal situations where each paper has one code), we believe that
64% agreement rate is reasonable. The number of papers with any disagreement is
61. All disagreement was resolved after discussions. Average independent coding
time is 14 minutes per paper per coder. Average time for resolving disagreement is
eight minutes per paper.
4. Results
4.1. Topic
Table 3 shows the frequencies and percentages of articles on research topics. Fig. 6
summarizes the article numbers into the three sub-areas.
Among this collection of papers, the dominating studies are on IS/IT use, eval-
uation and impact (82% of the overall topics). 14% of the topics fall in the sub-area
of IS/IT development. Only 3% of the subject topics address general research-rela-
ted issues such as methodology and measurement concerns, while no article involves
education such as curriculum or study programs.
The number of studies in the IS/IT use, evaluation and impact sub-area also
shows an increasing trend over the years. As shown in Table 3, the most studied
topic is behavioral, including studies pertaining to perception, belief, intention,
behavior, acceptance, adoption, use, resistance to use, and continued use. The topic
on attitude, satisfaction, and preference follows, which is followed by the topic on
performance and productivity.
The sub-area of IS/IT development takes only 14% of the collection. Among this
sub-area, the most interest has been on user analyst involvement, followed by soft-
ware/hardware development. User interface design, development and evaluation
studies are covered only 13 out of 46 papers of this sub-area.
Overall, the results on the research topic indicate that this collection of research
emphasizes a great deal on issues that occur during the stage where IS/IT artifacts
have already been released and in use.
4.2. Type/method
The frequencies and percentages of research methods utilized in the articles are
shown in Table 4 and Fig. 7.
Overall, the frequency of empirical methods (83%) dramatically exceeds that of
non-empirical ones (17%). The frequency of non-empirical studies has been fairly

Table 3
Results on research topic
Topic 90 91 92 93 94 95 96 97 98 99 00 01 02 Total %
IS/IT development 5 4 5 4 6 3 5 2 2 3 2 1 4 46 14.3
Development methods and tools 1 1 0.3
User analyst involvement 2 1 4 1 1 1 1 2 13 4.2
Software/hardware development 1 2 1 1 3 1 1 10 3.3
Software/hardware evaluation 1 1 0.3
User interface design & development 1 2 1 1 1 1 7 2.3
Interface evaluation 1 1 1 1 2 6 2.0
User training 2 2 1 1 6 2.0
IS/IT use, evaluation, impact 15 13 15 14 12 19 17 12 25 25 32 23 29 251 82.4
Behavior 5 7 4 5 4 7 3 5 7 8 12 7 11 85 27.7
Attitude 4 3 1 3 3 6 3 1 3 5 1 3 5 41 13.4
Learning 1 1 2 1 2 1 3 1 2 1 15 4.9
Motivation 1 1 1 2 1 2 1 1 10 3.3
Emotion 1 1 1 2 1 3 2 3 14 4.6
Performance 3 2 3 3 1 2 4 2 4 2 6 5 1 38 12.4
Trust 1 1 1 3 6 2.0
Ethics 1 1 2 4 1.3
Individual differences 1 5 1 1 2 1 4 2 1 2 20 6.5
Individual demographics 1 1 1 1 1 4 1 10 3.3
Interpersonal relationship 1 1 2 1 2 1 8 2.6
User support 1 1 2 0.7
Research & education 1 2 3 1 1 1 1 10 3.3
Research 1 2 3 1 1 1 1 10 3.3
Education 0.0
Total 20 17 20 19 18 24 22 17 28 29 34 25 34 307 100
Fig. 6. Results on research topic: sub-areas.

stable over the years. Among them, opinion with examples, applied concepts, and
conceptual models have been fairly utilized.
Empirical studies have been conducted almost entirely on events/processes (versus
on objects). In particular, lab controlled experiment and survey method, each
counting about 25% of all the methods utilized, dominate the collection. They are
followed by field study, field experiment, development of instrument, interpretive
case study, secondary data, interview, and positivist case study.
It is notable that four methods (Framework, Theory from reference disciplines,
Opinion with personal experience, and Description of types/classes of products)
have not been utilized in this collection of papers.
4.3. Publication patterns
In order to demonstrate the publication rates and trends of HCI studies in the two
journals, the total number of published research articles is counted. Fig. 8 depicts
the publication patterns of HCI studies in the two journals, and as a whole.
Table 4
Results on research type
Type 90 91 92 93 94 95 96 97 98 99 00 01 02 Total %
Non-Empirical 4 3 3 1 0 2 2 3 3 3 0 3 2 29 17.0
Conceptual orientation 1 1 1 2 1 1 7 4.1
Frameworks 0.0
Conceptual model 1 1 2 1 1 6 3.5
Conceptual overview 1 1 0.6
Theory from reference disciplines 0.0
Illustrative 1 1 3 1 1 3 1 1 1 1 14 8.2
Opinion with examples 1 1 3 1 1 3 1 1 12 7.0
Opinion with personal experiences 0.0
Description of tools, techniques 1 1 2 1.2
Applied concepts 2 1 1 1 1 1 1 8 4.7
Applied concepts 2 1 1 1 1 1 1 8 4.7
Empirical 5 6 9 11 14 13 12 8 9 13 12 13 17 142 83.0
Objects 1 1 0.6
Description of types/classes of products, technologies, etc. 0.0
Description of specific applications, systems, etc. 1 1 0.6
Events/Processes 5 6 9 10 14 13 12 8 9 13 12 13 17 141 82.5
Lab experiment 1 2 4 4 2 3 3 3 2 6 7 3 3 43 25.1
Field experiment 1 2 2 1 1 2 9 5.3
Field study 2 1 1 1 2 3 2 3 15 8.8
Positivist case 1 1 2 4 2.3
Interpretive case 1 1 1 2 1 1 7 4.1
Survey 2 3 3 5 5 4 2 2 3 2 7 5 43 25.1
Develop of instrument 1 1 1 1 1 1 3 9 5.3
Secondary data 1 1 2 1 5 2.9
Interview (individual based) 1 2 1 1 1 6 3.5
Total 9 9 12 12 14 15 14 11 12 16 12 16 19 171 100.0

It shows that there is an increasing trend of percentage of HCI studies in the two
journals over the years. The overall percentage of HCI studies is around 40% of all
published articles in these two journals during recent years. This is an encouraging
discovery for scholars conducting research in this subfield.
Fig. 8. Publication patterns.
Fig. 7. Results on research type: frequency of articles by methodology.

5. Discussion and conclusion
By examining existing HCI frameworks, proposing a new one to address the broad
HCI issues and concerns, and assessing published articles, this paper contributes to
our understanding of the current status of research topics, methods, and publication
patterns of the HCI studies in the MIS discipline. This understanding is important
for the identification and promotion of this emerging subfield in MIS. It should of
interest to researchers and young scholars for their future research, collaboration,
and publication.
This paper is the first attempt to draw such an understanding of HCI studies in
MIS based on the evidence of published articles. As such it is limited in scope due to
the time consuming nature of such studies. In order to provide an informative
picture, only the most recent 13 years of the two top MIS journals are selected as
paper sources. Although the 13-year time period is reasonable for this type of
research assessment, the journal selection may have had a strong influence on the
assessment results. This includes the potential biases of the two journals’ emphases
on publishable topics and methods. The HCI type research publication pattern may
also be affected by the two journals’ characteristics.
Another limitation of the paper is the omission of other possible aspects for
assessment. For example, the assessment of research can also include active
researchers and institutions as some of the MIS research assessment articles did
(Pervan, 1998; Romano & Fjermestad, 2001). This can provide useful informa-
tion for young scholars or doctoral students to identify potential academic
homes where collaboration is highly possible, and their research interests and
effort can be recognized and appreciated. We decided to omit this assessment
in this paper as we realized that limiting such a study to two journals might
provide a skewed picture.
Future research is planned to include relevant papers in more major MIS journals,
and to provide additional assessments such as specific technologies studied, contexts
where studies are conducted, and active researchers and institutions.
The fast development and pervasive use of technology prompt a need to re-
examine the broad HCI issues in light of the IS/IT development, actual use, and
impact on all aspects of our lives. The new framework proposed in this paper
emphasizes the entire interaction cycle between humans and technology, rather than
a stage or part of it. It also brings in the tasks and contextual factors. This view
is intended to show the dynamic as well as the evolutional aspect of issues and
concerns regarding the interactions between humans and technology.
The assessment of a limited collection of HCI studies in two top MIS journals
demonstrates a wide range of research issues and topics being studied by MIS
researchers over the past thirteen years. The dominating issues fall in the area of IS/
IT use, evaluation and impact. MIS researchers are more concerned with issues that
occur after IS/IT is developed. These concerns are closely related to humans’
perceptions, beliefs, behavior, attitude, satisfaction, performance and productivity,
and individual differences. Among the small percentage of studies focusing on the
development stage, MIS researchers are concerned with user involvement and

participation, user-analyst differences and interaction, programmer cognition
studies, and design methods.
The assessment also shows a broad range of research methods employed. The
dominating methods, however, lie in empirical studies utilizing lab experiments and
surveys to examine issues on events and processes. Due to the dynamic nature of
human interaction with technology, task, and context, it may indicate a need to
utilize more interpretive research methods such as phenomenology, action research,
ethnography, grounded theory, etc. It would help advance the subfield more if there
are general instruments being developed and validated, rather than many research-
ers reinventing the wheel. There are few studies focusing on providing frameworks
and high-level overviews of the subfields, which indicates potential research efforts
in the future.
Overall, it is exciting to see that there is an increasing number and percentage
of HCI studies published in the two top MIS journals over the years. The current
status of the subfield may indicate an emerging state, rather than a mature one.
Acknowledgements
The authors thank the guest editor, Dr. Henry H. Emurian, and two anonymous
reviewers for their helpful comments and suggestions on an earlier version of the
paper.
References
Alavi, M., & Carlson, P. (1992). A review of MIS research and disciplinary development. Journal of
Management Information Systems, 8(4), 45–62.
Barki, H., Rivard, S., & Talbot, J. (1988). An information systems keyword classification scheme. MIS
Quarterly, 12(2), 299–322.
Barki, H., Rivard, S., & Talbot, J. (1993). A keyword classification scheme for IS research literature: an
update. MIS Quarterly, 17(2), 209–266.
Baskerville, R. L., & Myers, M. D. (2002). Information systems as a reference discipline. MIS Quarterly,
26(1), 1–14.
Carroll, J., Kellogg, W., & Rosson, M. B. (1991). The task–artifact cycle. In J. Carroll (Ed.), Designing
interaction (pp. 74–102). Cambridge University Press.
Compeau, D. R., Higgins, C. A., & Huff, S. L. (1999). Social cognitive theory and individual reactions to
computing technology: a longitudinal study. MIS Quarterly, 23(2), 145–158.
Culnan, M. J. (1987). Mapping the intellectual structure of MIS, 1980–1985: a co-citation analysis. MIS
Quarterly, 11(3), 341–353.
Davis, F. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information
technology. MIS Quarterly, 13(3), 319–340.
Eason, K. D. (1991). Ergonomic perspective on advances in human–computer interaction. Ergonomics,
34(6), 721–741.
Goodhue, D. L. (1997). The model underlying the measurement of the impacts of the IIC on the
end-users. Journal of the American Society for Information Science, 48(5), 449–453.
Goodhue, D. L. (1998). Development and measurement validity of a task-technology fit instrument for
user evaluations of information systems. Decision Sciences, 29(1), 105–137.

Goodhue, D. L., & Thompson, R. L. (1995). Task-technology fit and individual performance. MIS
Quarterly, 19(2), 213–236.
Hewett, T., Baecker, R., Card, S., Carey, T., Gasen, J., Mantei, M., Perlman, G., Strong, G., & Verplank,
W. (1992). ACM SIHCHI curricula for human–computer interaction. Association for computing machinery.
New York, NY: ACM.
Lucas, H. C. (1975). Performance and the use of an information system. Management Science, 21(8), 908–
919.
Pervan, G. P. (1998). A review of research in Group Support Systems: leaders, approaches and directions.
Decision Support Systems, 23, 149–159.
Preece, J., Rogers, Y., Sharp, H., Benyon, D., Holland, S., & Carey, T. (1994). Human–computer
interaction. Addison-Wesley.
Romano, N. C., & Fjermestad, J. (2001). Electronic commerce customer relationship management: an
assessment of research. International Journal of Electronic Commerce, 6(2), 61–113.
Swanson, E. B. (1974). Management information systems: appreciation and involvement. Management
Science, 21(2), 178–188.
Venkatesh, V., & Davis, F. (2000). A theoretical extension of the technology acceptance model: four
longitudinal field studies. Management Science, 46(2), 186–204.
Zhang, P., Benbasat, I., Carey, J., Davis, F., Galletta, D., & Strong, D. (2002). Human–computer
interaction research in the MIS discipline. Communications of the AIS, 9(20), 334-355.
Zhang, P., & Dillon, A. (2003). HCI and MIS: Shared Concerns, Editorial Introduction. International
Journal of Human–Computer Studies, 59(4), 397–402.

An Assessment Of Human-Computer Interaction Research In Management Information Systems Topics And Methods

More Related Content

Similar to An Assessment Of Human-Computer Interaction Research In Management Information Systems Topics And Methods (20)

More from Rachel Doty (20)

Recently uploaded (20)

An Assessment Of Human-Computer Interaction Research In Management Information Systems Topics And Methods