ACM WebSci 2018 presentation/発表資料
1 like2,848 views
This document presents the development of a Web Access Literacy Scale to measure users' abilities to critically evaluate information found online. The researchers conducted a study with 534 participants to develop and validate the scale. Factor analysis resulted in a 7-factor scale measuring logical approach, content verification strategies, inquisitiveness, bias tolerance, search skills, author verification, and objectivity. Scores were higher for those with information literacy experience. The scale can help identify weaknesses and inform the development of literacy training and search tools.
ACM WebSci 2018 presentation/発表資料
- 1. Web Access Literacy Scale to Evaluate How Critically Users Can Browse and Search for Web Information Yusuke Yamamoto1, Takehiro Yamamoto2, Hiroaki Ohshima3, Hiroshi Kawakami2 1: Shizuoka University, Japan 2: Kyoto University, Japan 3: University of Hyogo, Japan WebSci 2018: Session IIb
- 2. Background: Web information is not always correct The number of medical Web sites authorized by medical experts: < 50%* * E. Sillence et al., “Trust and Mistrust of Online Health Sites”, ACM CHI, pp.663-670, 2004
- 3. Possible approach in information science Obtaining correct information (Semi-) automatic analysis on information credibility
- 4. Examples of credibility analysis systems *2 Y. Yamamoto and K. Tanaka. Enhancing Credibility Judgment of Web Search Results. In Proceedings of the 29th ACM SIGCHI Conference on Human Factors in Computing Systems (CHI 2011), pages 1235–1244, 2011. *1 Yin, X., Han, J., & Philip, S. Y. (2008). Truth discovery with multiple conflicting information providers on the web. IEEE Transactions on Knowledge and Data Engineering, 20(6), 796-808. TruthFinder*1 Measures the consistency of a fact describing an object CowSearch*2 Provides credibility-related scores for credibility judgment These systems do not guarantee the correctness of information Limitation
- 5. Possible approach in information science Enhancing user’s skills to evaluate web information Obtaining correct information (Semi-) automatic analysis on information credibility
- 6. Research purpose Development of scales and questionnaire to self-evaluate the skills to “collect accurate web information” Web access literacy (WAL) 1 Understanding of people’ strong points and weak ones on web access literacy2
- 7. Web access literacy Information literacy Internet literacy Media literacy Skills to critically check web information and collect accurate information from the web, using information access system. Web access literacy
- 8. Elements of Web access literacy Critical thinking attitudes Verification strategies Skills to use web search engines Tolerance for cognitive biases KL : F A L : N: P KF: K : 2 P A PL K K : P L :K A K PL 7D L L KN N K L : NL L :K A L K KF: LL:K K K CN F . N L : N: KF: K : K K K:P : NL K ( 3 : : K : A N L LL C K : 8 K: K :L L PA :LL LL P KF: K ) : K F A :L L D L :L : LN: : : :L ( 8 2NLNF : - R LD K : LD 8: D- 8A :L A 1NDNLA F: 0: A 4N :K : : LDS LD .: : L L ) ( (&&* *3 G. Lindgaard et. al. An Exploration of Relations between Visual Appeal, Trustworthiness and Perceived Usability of Homepages. ACM TOCHI (2011) *1 UC Berkeley Library. Evaluating resources. http://guides.lib.berkeley.edu/evaluating-resources. ,
- 9. Examples of question item candidates Critical thinking attitudes (33) Verification strategies (20) Skills to use web search engines (5) Tolerance for cognitive biases (11) ? - - 5 9 9 ? - 9 ? - ? 5 ? - ? 9 9- - ? - ? 9 5? - - 95 - 9 5 A - ? ? 55 - - ? 9 - - ? 9 9 5 - ?
- 10. Online study $ Development of web access literacy (WAL) scale Analysis on the relation b/w WAL and external indicators Purpose Participant 534 05 30 Study method 32 0 0 25 ,280 , .85 / 5 8.2 1 08 2.0 Date 0 0 $ $ Reward ,8 2.2 , Language , , 0 0
- 11. Factor analysis results (#factor =7; promax rotation)Table 2: Factor analysis results of the web access literacy scale (promax rotation; N = 534). Items with an asterisk are reverse code scale items. Item M SD Factor 1 Factor 2 Factor 3 Factor 4 Factor 5 Factor 6 Factor 7 Factor 1: Logical approach (α = .88) I am good at thinking about complicated problems in an organized way 3.25 0.96 .835 -.041 -.036 -.031 -.097 .024 .028 I am good at summarizing ideas 3.27 1.02 .754 -.067 .047 -.011 -.073 -.034 .046 I set a roadmap when thinking about something 3.70 0.91 . 705 .036 -.065 .025 .024 -.124 .045 I am good at making constructive proposals 3.18 0.94 .700 -.187 -.001 .015 .034 -.012 .152 I am good at explaining so that anyone can understand me 2.92 0.98 .685 -.188 .035 -.047 -.021 .103 .157 I am confident about my ability to think precisely 3.25 0.89 . 679 .023 -.012 -.076 -.025 -.035 .045 I get confused whenever thinking about complicated problems (*) 3.74 0.92 .620 .125 .056 -.051 -.034 -.016 -.184 I can maintain concentration while working on a problem 2.72 1.04 .600 -.095 -.037 .089 -.054 .078 -.031 I can continue to challenge tough problems 3.26 0.99 .508 .073 .204 -.017 -.094 -.006 -.010 I am easily distracted (*) 2.84 1.09 .484 -.077 -.122 .051 .109 -.035 -.113 I can look into a problem carefully 3.70 0.87 .480 .212 .044 -.011 .058 -.025 -.097 My colleagues often ask me to make judgments because I am fair 3.04 0.95 .415 -.062 .094 -.064 .206 .056 .100 Factor 2: Content-based verification strategy for information credibility (α = .87) I try to spend as much time as possible on web searches 3.54 0.87 -.002 .804 .049 -.023 -.105 .011 -.175 I try to modify search queries to examine web page content more intricately 3.90 0.84 -.114 .741 .004 -.017 -.024 -.014 -.041 I try to compare multiple web pages 4.17 0.76 -.079 .741 .024 -.004 -.040 -.013 -.078 I try to issue multiple search queries to collect information in a broad perspective 3.85 0.86 -.069 .720 .067 -.027 .008 .001 -.128 I try to check other web pages or information resources to verify content credibility 3.73 0.93 -.012 .687 -.084 -.018 -.039 -.079 .160 I try to check similar or the same information on other web pages 3.54 0.90 -.042 .684 -.067 -.081 -.011 -.100 .143 I try to check to see that the information is complete and comprehensive 3.21 0.91 -.038 .492 .028 .024 -.014 .006 .263 I try to browse web pages in lower- and higher-ranked web search results 3.12 0.93 -.072 .490 -.021 .109 .016 .088 .066 I try to obtain evidence to verify the information on web pages 3.36 0.96 .017 .488 -.035 .071 -.046 -.011 .355 I try evaluating whether views represented on web pages are facts or opinions 3.79 0.98 .023 .428 -.002 .069 .086 -.147 .320 Factor 3: Inquisitiveness (α = .88) I want to learn a lot by communicating with various types of people 3.84 1.02 -.052 -.027 .785 -.105 .003 -.018 .083 I want to learn various cultures 3.90 1.04 -.040 -.010 .724 .006 -.002 -.030 .055 I want to continue learning new things over my lifetime 4.07 0.94 .136 .027 .703 .048 -.052 -.058 .021 I want to learn as much as possible even if I am unsure if the information will be useful 3.81 1.05 .029 .029 .666 .087 .033 -.038 -.034 I am interested in people with opinions that differ from mine 3.76 1.00 -.089 -.082 .658 .013 .176 .060 .029 I like to challenge new things 3.67 1.07 .182 .051 .635 -.051 -.104 -.042 -.031 I want to learn more about any topic 3.62 1.05 .038 -.001 .615 .067 -.064 .092 -.053 I like to discuss with those who have opinions that differ from mine 3.44 1.11 -.053 -.113 .576 -.009 .178 .074 .074 Learning how foreigners think is useful 4.13 0.88 -.097 -.001 .570 -.083 .087 .000 .052 I try to ask about what I do not understand 4.01 0.93 .085 .169 .432 -.026 -.134 -.126 .098 Factor 4: Tolerance for biases in information credibility judgment (α = .81) I trust information that is shared by many users or liked on social networking websites (*) 3.04 0.88 -.049 .063 -.071 .718 .067 -.104 .056 I trust information that my friends share on social networking websites (*) 2.96 0.85 .010 .034 -.071 .695 .071 -.148 .004 I trust information that my followers share on social networking websites (*) 3.35 0.85 .026 .082 -.025 .608 .041 -.171 .022 I trust information on easy-to-understand web pages (*) 2.51 0.71 .084 -.073 .016 .543 -.085 .036 .046 I trust posts that someone indicates are the best answers on Q/A sites (*) 2.80 0.93 -.016 .017 .020 .511 -.028 -.006 .043 I trust information on web pages with titles such as "the 20 best XXs" (*) 3.24 1.00 -.075 -.002 -.019 .507 .037 .084 .092 I trust information on web pages with a high rank on search engines (*) 2.66 0.81 -.068 .048 .002 .499 .014 .095 .068 I trust products or services with good reputations, on average, on review sites (*) 2.68 0.95 .071 -.118 -.027 .481 -.009 .068 -.003 I trust information on well-designed web pages (*) 2.77 0.68 .052 -.072 -.002 .474 -.027 .041 .038 Factor 5: Objectivity (α = .83) I try to adopt an objective attitude when deciding something 3.81 0.87 .041 -.107 .051 -.070 .710 .053 .040 I try to make fair judgments 3.64 0.90 .022 -.054 .068 -.072 .691 -.010 -.048
- 12. 7 factors (1/2) Logical approach (12) * 3 3 ( ) 1 ) * (1 3 3 3 * * 3 * Content-based verification strategies (10) 3* ) (1 3 ( ) * ) 3 ( ) 3 3 3 ) 1 Inquisitiveness (10) 3 1 3 1 ( ) 3 ) 3 1 3 1 3 ) 1 Tolerance for biases in web credibility judgment (9) 3 3 * 1 * ( 3 3 3 3 3* 3 ) * ) 1
- 13. 7 factors (2/2) Objectivity (7) 4 4 4 - 4 - 4 Skill level in using web search engines (6) 4 1 4 4 4 4 4 - - - 4 4 4 1 4 4 Author-based verification strategies (5) 4 - -4 1 4 4 1 4 4 - -4 4 -
- 14. Means of WAL score and factor scores 3.62 3.83 3.24 2.89 1.95 2.81 3.62 3.25 0 1 2 3 4 5 Objectivity Inquisitiveness Logical approach Bias tolerance Skill level using web search engine Author-based verification stragegy Content-based verification strategy Web access literacy Weak points are author-based verification strategy, bias tolerance, and search engine skill
- 15. WAL score and learning activity (1/2) No Yes Experience of information literacy class 3.15 3.20 3.25 3.30 3.35 3.40 3.45 3.50 Webaccessliteracyscore Not university-educated University-educated - 1 - 1 -1 - 1 1 -7 - -7 - Experience of information literacy classes had positive impact on the WAL scores (F = 8.82, p < .01 )
- 16. WAL score and learning activity (2/2) Educational background did not affect WAL scores against our expectation (F=7.7x10-5, p=.993). No Yes Experience of information literacy class 3.15 3.20 3.25 3.30 3.35 3.40 3.45 3.50 Webaccessliteracyscore Not university-educated University-educated 8 - 1 - 1 -1 - 1 1 - - - -
- 17. WAL score and experience in taking the literacy classes 0 1 2 3 4 Objectivity Inquisitiveness Logical approach Bias tolerance Skill level using web search engine Author-based verification stragegy Content-based verification strategy with experience without experience Any kinds of scores of participants with the experience was greater than those without the experience, except bias tolerance.
- 18. Implications for web access literacy (WAL) development ● People are likely to have weak points on author- based verification strategy, bias tolerance, and search engine skills ● Conventional literacy classes could contribute to improve some factors on the WAL, although bias tolerance could not be improved ● We need to re-design information literacy classes or to develop credibility-oriented search systems
- 19. Conclusion Web access literacy scale to evaluate how critically users can browse the web 2 3 2 3 -23 3 2 3 2 3 23 2 2 3 - 3 - A complete list of questions is available on our paper and my GitHub repository