Qualitative data analysis in design research
1 like206 views
The document summarizes a qualitative research study analyzing data from user tests of the Feeler prototype, a self-monitoring device. It describes the research methodology, including contextual inquiry, participatory design workshops, and tests with 6 graduate student participants. Thematic analysis of think-aloud and interview audio identified patterns in content. A coding scheme with categories like reflection and non-reflection was developed from theory and used to analyze the data. Challenges in coding like biases and fatigue are discussed. Results provide insight into how interactive technology can support different levels of reflection.
Qualitative data analysis in design research
- 1. Eva Durall, Ph.D. candidate Learning Environments research group Aalto School of Arts, Design and Architecture QUALITATIVE DATA ANALYSIS IN DESIGN RESEARCH: Feeler proof-of-concept research case
- 2. 1. Introduction The Quantified Self (QS) movement is a growing global effort to use new mobile and wearable technologies to automatically obtain personal data about everyday activities. Lee 2013
- 3. 1. Introduction Several scholars have highlighted the role of QS technologies for supporting reflection Li, Dey & Forlizzi 2011; Rivera-Pelayo et al. 2012 Self-monitoring is considered to be one of the first steps to self-regulated behavior Schunk, 2001
- 4. Research-based Design (Leinonen, 2008, 2010) 2. Research-based design methodology
- 5. 2. Research-based design methodology Research stage Description Contextual Inquiry 6 semi-structured interviews with graduate students 4 subject-expert interviews 4 days of field observations in a university library Literature review 3 focus groups (n=15) Questionnaires before and after the Participatory Design (PD) sessions Participatory Design 3 PD workshops 2 open sessions Product Design 4 prototypes, 2 of which are functional Software Prototype as Hypothesis User tests with functional prototypes: - Feeler v.1.0
- 7. 5. Feeler proof-of-concept research 5.1. Objectives Validate the design concept Explore to what extent the prototype supports learners’ awareness and reflection on study habits
- 8. Participants: 6 graduate students Different nationalities, all fluent in English Tasks: Reading an academic paper and solving a 3D puzzle. Duration: 30 minutes for independent study activities + 45 minutes: interview= 1h 15min Number of sessions: 1 test per participant (n=6) 5. Feeler proof-of-concept research 5.2. User tests
- 9. 5. Feeler proof-of-concept research Images of Feeler prototype user tests 5.2. User tests
- 10. 5. Feeler proof-of-concept research 5.2. User tests
- 11. 5. Feeler proof-of-concept research 5.3. Data collection techniques
- 12. Image published by Flickr user “Eelco”. 5. Feeler proof-of-concept research 5.4. Research data
- 13. 5. Feeler proof-of-concept research Thematic analysis is oriented to the identification, analysis and reporting of patterns (themes/categories) present in research data Braun & Clarke, 2006 5.5.Thematic analysis In the Feeler proof-of-concept research we conducted a thematic analysis of the think-aloud and the interview audio recordings in order to identify patterns in the content
- 14. 5. Feeler proof-of-concept research 5.5.Thematic analysis
- 15. 5. Feeler proof-of-concept research The coding scheme or codebook is a classification of the key themes used to categorise the observed data into the expected thematic areas of qualitative data analysis. 5.5.Thematic analysis / Coding scheme The coding scheme used in the thematic analysis of Feeler user tests interviews was developed following a inductive- deductive approach
- 16. C1/ Non-reflection c1a: No expectations C1b: Not understanding C2/ Recognition c2a: Integration c2b: Curiosity C3/ Reflection c3a: Puzzlement c3b:Appropriation C3c:Transformation Based on theory: - Dewey, 1933 - Kember et al., 2000 - Kolb, 1984 - Mezirow, 1991 - Peltier, Hay & Drago, 2005 5. Feeler proof-of-concept research 5.5.Thematic analysis / Coding scheme
- 17. 5. Feeler proof-of-concept research 5.5.Thematic analysis / Coding scheme Code Definition Example c1a: No expectations The person doesn’t express any particular interest, question or expectation about the data. “I hadn’t any particular expectation about the EEG data”
- 18. 5. Feeler proof-of-concept research 5.6.Thematic analysis /Validity & reliability Connected to accuracy The coding scheme provides rules for element orienting pattern recognition VALIDITY Refers to consistency with a standard RELIABILITY tradeoff
- 20. 5. Feeler proof-of-concept research 5.7.Thematic analysis / Challenges in the coders’ task Lack of consistency Bringing their own schemas into play (interpreting rather than recognizing patterns in the content) Saturation (by fatigue)
- 21. Distribution of the codes identifying behaviors connected to reflection 5. Feeler proof-of-concept research Identification of subthemes and discussion of the results in light of existing research Reflection levels that interactive technology can support (Fleck and Fitzpatrick, 2010) 1. Revisiting 2. Revisiting with explanation 3. Dialogic reflection 4.Transformative reflection 5. Critical reflection 5.8.Thematic analysis / Data analysis & results 1 2
- 22. 6. Qualitative data analysis Recommendations on coding Avoid introducing your own biases by carefully reading the data. Iterate, make sure your codes are consistent. Chose “good moments” for coding. Limit the amount of time for a coding session.
- 23. Thanks! eva.durall@aalto.fi Durall, E., Leinonen,T., Gros, B., & Rodriguez-Kaarto,T. (2017). Reflection in Learning through a Self-monitoring Device: Design Research on EEG Self-Monitoring during a Study Session. Designs for Learning, 9(1), 10–20. Published work