Qual, Mixed, Machine and Everything in Between
- 1. Qual, Mixed, Machine and Everything in Between Dr. Stuart Shulman Founder & CEO Texifter Prepared for the 3rd World Conference on Qualitative Research Lisbon, Portugal October 17-19, 2018
- 2. BACKGROUND: MY ROOTS Part One
- 3. Early 1990s: My Sustainable Agriculture Roots
- 4. Emergent properties found in very well read texts, such as the character type “extremist agent of the law”
- 5. Agenda-setting in the press
- 6. Relations between Classes Rates and Terms for Credit Farm Profitability Cost of Living Soil Fertility Education Exploration Speculation Coding Validation
- 7. Circa 1999
- 8. May 2001 Council for Excellence in Government June 2002 National Defense University
- 10. Qualitative Methods: Genes, Taste, or Tactic? • Qualitative by birth or choice? – Some look to words as an alternative to number crunching – Others rooted in rich and meaningful interpretive traditions • Another group is fluent in both qual & quant – Mixed methods open up rather than limits fields of knowledge • One central goal is valid inferences about phenomena – Replicable and transparent methods – Attention to error and corrective measures – Internal and external validation of results • Using computers for qualitative data analysis helps, but… – Rigor still originates with the research design, not the technology – Software makes better organization and efficiency possible – Coders enable the researcher to step back while scaling up
- 11. Purist A Spectrum of Methods Approaches deep immersion closeness to data antipathy to numbers credible interpretation in-depth analysis contextual subjective experimental mixed method adaptive hybrid flexible approach interdisciplinary open minded quantitative focus on error measurement critical validity and reliability replication & objectivity generalization hypotheses PositivistPluralist
- 12. COMPUTER ASSISTED QUALITATIVE DATA ANALYSIS SOFTWARE Part Three
- 13. An Incredibly Important Book for Me
- 14. Other Very Important Books
- 16. Nvivo
- 17. Atlas
- 18. MaxQDA
- 20. RapidMiner
- 21. Attensity
- 22. Published in 2008 Chi-Jung Lu and Stuart W. Shulman, “Rigor and Flexibility in Computer-based Qualitative Research: Introducing the Coding Analysis Toolkit,” International Journal of Multiple Research Approaches Vol. 2, No. 1 (2008), 105-117.
- 23. Positive Claims for Using Software • Convenience: Data is accessible & reducible • Efficiency: Computer-assisted tasks like search • Organization: Codes, memos, and teams • Patterns: Co-occurrences, frequencies, etc. • Outliers: Significant and otherwise • Scale: Testing of observable implications • Iteration: A continuous and evolving process • Transparency: Clarify methods & confirmability • Legitimacy: Accuracy, validity, & credibility
- 24. Concerns about Using Software • Convenience: Too many tempting short cuts • Efficiency: May undermine meaning making • Organization: Becomes an end itself • Patterns: Can be misleading • Outliers: May be undervalued as noise • Scale: Big data is not better data • Iteration: Bias may be inscribed in features • Transparency: Features that are a black box • Legitimacy: Research design is the actual key
- 25. COLLABORATION & MEASUREMENT Part Four
- 28. Text Classification A 2,500-year-old problem Plato argued it would be frustrating; it still is Software cannot remove the problem It can expose it more quickly
- 29. Grimmer & Stewart “Text as Data” Political Analysis (2013) Volume is a problem for scholars Coders are expensive Groups struggle to accurately label text at scale Validation of both humans and machines is “essential” Some models are easier to validate than others All models are wrong Automated models enhance/amplify, but don’t replace humans There is no one right way to do this “Validate, validate, validate” “What should be avoided then, is the blind use of any method without a validation step.”
- 31. Computer Science & NSF Influence: Measure Everything! How fast? How reliable? How accurate? Valid?
- 32. Inter-Rater Reliability is One Key Factor Understanding the landscape of human interpretation better prepares us to face the challenge of machine classification. Fleiss’ Kappa: The Level of Agreement Beyond Chance
- 34. “CoderRank for Enhanced Machine Learning” “CoderRank is to text analytics what PageRank was to search. Just as Google said not all web pages are created equal, Texifter argues that not all humans are created equal. When training machines, it is best to rely most on the humans most likely to create a valid observation. We proposed a unique way to rank humans on trust and knowledge vectors.”
- 35. HUMAN & MACHINE LEARNING Part Five
- 36. Labeling, Tagging, or Annotation Improves Machine Learning Over Time
- 37. Iterate Human Coding & Machine Learning
- 38. Word Sense Disambiguation (Relevance)
- 40. “Patriots” Football Versus Politics
- 42. Naturally Occurring Clusters of Free Text Can Be Discovered Automatically
- 43. • A free and open source software option • Web-based crowd source collaborative tools • Measurement innovation • Free real time Twitter data collection • Random sampling and keystroke coding • Advanced search and filtering • Deduplication and clustering algorithms • Custom machine-learning classifiers • Word sense disambiguation • CoderRank for enhanced machine learning What have CAT & DiscoverText contributed to the field of qualitative methodology?
- 44. Dr. Stuart W. Shulman Founder & CEO, Texifter, LLC Editor Emeritus, Journal of Information Technology & Politics Contact Information Email: stu@texifter.com Twitter: @stuartwshulman Thanks for Listening!