Critical evaluation (web version)

Editor's Notes

• #3: The focus of this session is how you/we EVALUATE and RANK materials we locate, and how this is often a more complex process than we often appreciate… there are subtle processes going on, you are not just an empty vessel into which information is poured and knowledge is poured out.This is not a session looking at how to avoid bias, but to enable you to recognise it and to be prepared to incorporate it in to your research methodology.It is about taking a step backwards, being aware of the biases in play so you can acknowledge them in your work, especially in any viva or the equivalent. It is about YOUR role and responsibility as a professional researcher, and being able to state your awareness of it.The first half of the session is divided into two parts: Firstly, looking briefly at the importance of evaluation within the context of the research environment. Secondly, looking at ‘forms of value’ in the research context – a Marxist concept looking at how individuals or groups establish what as commodity is worth.
• #4: This first part focusses on self awareness of your role and responsibility, and what you bring to the table.We’ll explore:[much of the session will look at] Subjective forms of value: these could be personal, cultural or experiential influences on how we ascertain the value of information AND how it has already impacted upon the information we find.Cognitive bias – how various processes, both internal and external, affect our decision making processes, and those who we research or whose research we use.Objective forms of value – those not influenced by opinion or personal feelings, grounded in fact alone, but which may be impacted upon by the availability of and access to those facts.
• #5: Then the second part of the session will look at some practical tips and tools for evaluating information.
• #6: Image – self awareness, reflection on your role as a researcher.
• #7: Para 1: Just as a point of reference, for many (not all) first year undergraduates when we say ‘research’, by and large they “just find it”Para 2: But as a postgraduate researcher you must ‘find it’. You must ‘find all of it’ and then you must ‘read it’, ‘evaluate it’ and ‘synthesise it’.Para 3: The importance of critically evaluating information to you as a professional researcher is two-fold. - identifying and appropriate use of information, data and methodologies - awareness of your impact on both your own research, but also the research of others.Para 4: Focus to start is about what defines YOUR evaluative criteria, about an awareness of your role as a researcher in the evaluative process and less about developing a checklist of how to evaluate. - although we will look at this at the end of the session.
• #8: Being aware of your impact as a researcher is important when you consider what Rosemary Luckin, Professor of Learner Centred Design at the University of Sussex, defined as ‘the ecology of resources’. (reference in bibliography) - knowledge is not static - it is constantly changing and evolvingInformation resources are connected to a body of work linked by references and citations… - as we use them… - … we transform them into new knowledge… - … and this impacts upon and shapes what we, and others, go on to create.Prior knowledge shapes what we go on to find, how we interpret it and what we create.The chronology of what we learn impacts on our and other future decisions… what we read and where to search.How we evaluate information is the same, not a distinct clinical process. Our evaluation of information is subject to our own and others personal bias.Diagram… looking at the ‘original acorn’, that key text in any area of research. - You find it, and it moves you. - You think about it. You link it to other information and knowledge. And it grows, and changes. You shape it into a new form (the tree)… - others do the same, and it grows new branches… - others also find your work, and develop from your knowledge… and new stems grow - all of these branches and leaves grow and are shaped by us
• #9: Para 1: In theory we can select almost any information to complete a task.Para 2: In practice we filter it by selecting the resources we think are most appropriate.But are we always conscious of how, and why, we filter the resources which shape the knowledge we create? - would a researcher in the same discipline but from a geographic location, or with a different social, cultural or educational background select the same resources, and use them in the same way? - how have the authors of sources, or subjects of our research, filtered information to shape their knowledge and understanding?Its not just about the socio-cultural factors we bring in to our research, or which effect us, but also those we inherit from them.Para 3: In turn, motivation impacts upon what we find and how we use it. - can we be bothered to find it? - what about the authors we refer to? Could they? Did they search thoroughly and effectively? How did they filter that informatio they found? - What has influenced your choice of research area, your supervisor and work-style? How will that affect the information you look for, find and use?Image: how far do you sit back and accept what you find as the limits as to how far you will look?
• #10: Even before we make an active choice to filter information, this is already done for us...People – our family, other academics, friends,our supervisor... What do they recommend? Has your supervisor steered you away from any resources or information and towards something else? Could it be sometimes because the direction you might have been heading was slightly out of their comfort zone?
• #11: Technologies - how many of you will find the time to use all of the technologies and tools available to you? - how many of you have tried some of our databases that you aren’t already familiar with or are introduced to in a session or by a colleague? - how many of you will make do with Google Scholar – Google Scholar will not find everything.Also... - if you search on Google you will get different results to yahoo - if two of you search for the same keywords on Google, you won’t necessarily get the same results (location, search history, recently visited sites, browser version)And finally... - you are now the most information rich generation ever... But most of us access this through a 16” screen, maybe three paragraphs at a time. Does this limited window affect how we access, read, interpret and evaluate information?
• #12: Cost - many of the resources you access whilst at Durham are subscriptions we pay for you to access. - how much is your access or ability to search affected by the cost of resources, or even the cost of reaching resources if they are hidden in archives or private collections, need translating or restoring?
• #13: Skills – only as good as your search skills / language skills / social networking skills
• #14: CopyrightWhat can you not access, or use, because of copyright protection?Growth of the open access industry to allow access without restriction by cost – but also to allow use and re-use of both data and created knowledge. But there is still a long way to go.
• #15: In summary.... - you need to be critical and reflect on what you find. - you have a responsibility to do so. - but you must remember that you are using and creating knowledge from existing information which is not static... It has been created by others... And they will have applied their own evaluation process, will have filtered information and will have had different factors impacting upon them which has filtered their access to information. How will that have affected the knowledge you use and create?
• #16: Image: Graffiti: says value. Some people like graffitti, some don’t. Different people will place a different value on something based upon their own preferences. Those of the community they are part of and the society they live within.I lived in Easton, Bristol where there is a fair bit of graffiti. Some pieces are “perceived” to have a greater value because their author has become more well known than others (Banksy). So when a shopkeeper down the road went and painted over his walls to remove the graffitti, there was uproar from locals and the press when a ‘banksy’ piece (Gorrilla) was covered. But very little mention of the other pieces covered up – except by the local graffitti community.
• #17: 25 minutesForms of value: a Marxist concept looking at how individuals or groups establish what as commodity is worth.Para 1: At every stage of the information retrieval process, a value judgment is made.Peer review: tends to be a mark of academic rigour. Likely to be good. But it might not be... - some journals, including Nature, Proceedings of the National Academies of Sciences etc. Have been caught out with some articles quality later called into question (fraud, inherent bias) - some journals ‘peer review’ process varies. - above example, up until a couple of years ago. A very respected journal... But some articles the author could select his peer review panel.. And so may have picked those who advocated his ideas and thoughts rather than those who may have held different points of view. - open access arguments also opening up research more widely, meaning after it is made available the review by peers is broader and continuously developing.So, peer review is one way of measuring value... but not as clear-cut as just saying peer reviewed = good, non-peer reviewed = bad.Para 2: What forms of value are there, how should you use them together to help you strive to be an information literate researcher?
• #18: Para 1 and 2: Standard idea of evaluating is an objective evaluation... Determining a ‘value’ based upon measurable facts.We’ll come back to some measures in the final part of this session.Para 3: If we omit this objective evaluation, looking at the measurable facts (where there are any) we risk information becoming counter knowledge...1963 JFK assassination, one of the biggest conspiracy theories there has been, an entire publishing industry formed around it... And this was before the internet.Now, looking at pseudo science, on homeopathic medicines and then more recent conspiracies around 9/11.Also, 1998 research into MMR jab – published in the Lancet, later 10 authors published a retraction once it was revealed that the lead researcher was in fact being funded by those with a vested interest in proving a link to autism spectrum disorders. - Was the original research, and the studies selected, objective? - Was the subsequent press coverage objective?
• #19: Para 1-3: So if an objective form of value is evaluating an information source based upon its individual merit….Intersubjective forms of value looks at valuing something within the wider context of the needs and shared values of the community or society. Relativist – idea that there is no absolute truth. With your research, you need to embed this within a context and determine a ‘truth’ which is applicable. For example, does an information source has value within the framework or context of your own research. You may well have ‘objectively’ valued it highly, but you also need to evaluate it in the context of where it sits.Eg [NOT A BRILLIANT EXAMPLE THOUGH], research based on costs of managing and income tax system. Some blue-sky think tanks have done research to show that a single-standard level of income tax may generate more returns for the public purse than the present system of basic and higher rates. But would the majority of society accept that? Research is based on raw financial data (objective) but does not take into account society’s perception of value and applicability.
• #20: Para 1: We filter materials before we read them, both consciously and subconsciously... - affects the keywords we choose to search with - the authors we look for or identify with - the values we attribute to recommendations, the number of citations an article has received, editorial board and procedures... * but courses are available to write abstracts for your publications to attract more downloads... this then may lead to more citations, and thus a greater value being placed on that article by the academic communityPara 2: You are privileged as a researcher by being able to make those decisions... But with great power comes great responsibility.What bias do you bring to the table? How does this affect your research, and how will it then go on and impact on other people’s research?So should you avoid subjectively evaluating information you find?Para 3: But a key part of research is what you yourself bring as an individual... Your worth, your values, your judgments. Let people then evaluate your decisions themselves. If we omit this we get groupthink... We, as groups, naturally tend to minimise conflict and reach a consensus. Speaking out in a group to question the course of action the group is taken is the much more difficult task. Plenty of examples... Nazi Germany – one of the most advanced and most literate nations... Jonestown in 1976, where mass suicide and infantcide by most of a hundreds strong community following the example of a few core members of the community.Extreme examples, but replicated in a notorious psychology experiment in 1971...[Video break]
• #21: 30 minutes“stanford prison experiment” 10.29 minutes - group-think mentality quickly set in based around loose criteria given, no ‘authority figure’ in place so dominant personality came to the fore and no-one questioned their own or each other’s behaviour.“now I’m not saying that if you, in your studies and further research, forget your own individual assertiveness and give in to groupthink you will end up mentally and physically abusing your fellow research professionals… but, we’ll see a bit later if you are aware of any groupthink bias playing a role in your thinking.”http://youtu.be/oC5hl1BLUiw
• #22: We, as professionals and researchers are affected by bias... And so are those we may be researching or speaking to as part of our research.Tversky and Kahneman, professors at Stanford and Princeton respectively in the 70’s, demonstrated ways in which our own judgments and decisions differ from rational choices. - arises as a natural mechanism to prevent cognitive overload, eg when asking directions.Has been suggested we are subject to over 200 attempts to influence our opinions each day (adverts, news, newspapers, conversations etc.)I’m just going to quickly summarise a few examples of each of these four main groups... Then there is a short exercise for you to try and match examples from the literature to their respective groups...
• #23: Some examples...Forer effect:- people pick up on little snippets of what is otherwise general or vague information, and mis-apply it to themselves or another over-valuing its accuracy. For example horoscopes...Herd instinct... Following the majority decision without question.Does this happen in the academic community?
• #24: Halo effectHalo effect... See following screenshots. Example from a few years ago of fraudulent research that successfully duped the research establishment and caught media attention . Leading South Korean stem cell research scientist, Dr Hwang Woo-suk, who was found to have "faked all his research on cloned human stem cells". (2006. Cloner's Disgrace. New Scientist. Jan 10. p.4 )Highlighted section demonstrates that he had a number of influences – money and reputation.
• #25: Shock wasn’t so much that he had been fraudulent, but that he had got through the review process seemingly because his research wasnt’ questionned as thoroughly as it should have been, because of his esteem in the related field of cloning where he had been involved in the first successful cloning of a dog embryo.
• #26: Strange, Hayne and Garry conducted research in 2008 on false memories. Showed half a group of children doctored pictures of themselves in a hot air balloon and then later on, questioned them over if they had had a ride in a hot air balloon, and asked them to describe it.And over half of them did, and expanded on the memory... And were also when asked very confident that it had actually happened.
• #27: Childhood memories are often sunny... My summer holiday memories certainly are. But i lived in the north east of england, and my family holidayed in Scotland or the Lake District, not necessarily known for their sun, so can that really be true?
• #29: Gambler’s fallacy – future possibilities are affected by past events...Just because 17 research studies form one conclusion, and then a new study finds differently... Can you then just discount the anomally without evaluating it, and the previous 17 studies, further?Before we talk about the last example of cognitive bias two examples
• #30: How much larger is the yellow circle on the right of the screen compared to the yellow circle on the left of the screen?1) The Ebbinghaus illusion is an optical illusion of relative size perception. In the best-known version of the illusion, two circles of identical size are placed near to each other and one is surrounded by large circles while the other is surrounded by small circles; the first central circle then appears smaller than the second central circle.
• #31: How much larger is the yellow circle on the right of the screen compared to the yellow circle on the left of the screen?1) The Ebbinghaus illusion is an optical illusion of relative size perception. In the best-known version of the illusion, two circles of identical size are placed near to each other and one is surrounded by large circles while the other is surrounded by small circles; the first central circle then appears smaller than the second central circle.
• #32: Which of the grey columns is darker?2) White’s illusion is an optical illusion illustrating the fact that the same target luminance can elicit different perceptions of brightness in different contexts. Note, that although the grey rectangles are all of equal luminance, the ones seen in the context with the dark stripes appear brighter than the ones seen in the context with the bright stripes.
• #33: Which of the grey columns is darker?2) White’s illusion is an optical illusion illustrating the fact that the same target luminance can elicit different perceptions of brightness in different contexts. Note, that although the grey rectangles are all of equal luminance, the ones seen in the context with the dark stripes appear brighter than the ones seen in the context with the bright stripes.
• #34: Previous two slides metaphors... As we saw, the circles were the smae size / the grey bars were the same colour. But if we don’t evaluate something based upon its own merit, and allow our decision to be overly affected by other factors, then we may come to an imperfect conclusionhttp://chronicle.com/article/The-Undue-Weight-of-Truth-on/130704/ Haymarket riot and trial of 1886. (Timothy Messer-Cruse - Professor in the School of Cultural and Critical Studies at Bowling Green State University.)Academic, new research, identified earlier research had falsely claimed (in a case of deliberate bias) that there had been no evidence presented at the trial… when in fact over 100 witnesses had been called and gave evidence. He was the leading expert in the field… but was barred from having his correction added to Wikipedia because:-He could not cite reliable sources (his own work and the primary sources were not being considered ‘reliable’ as they were a ‘minority view’, because most work published up to that point had originated with that one source of information)Because each of the other sources had not questioned the original report, they created a body of knowledge that was given almost unquestionned value.Qualifier – he did, as a professor and an academic not approach the wikipedia community on its own terms, and expected it to conform to his own academic credentials. I am not arguing the right or wrong of how the disagreement with Wikiepedia developed, but this is a good example of how one piece of information was taken unquestionningly, and re-used repeatedly without any objective treatment of the research findings or motives of the original author, and created a body of knowledge whose ‘value’ had very shaky foundations.
• #35: After 1 hour (allow 15mins)At end: some are more difficult to assign than others, and may easily fall under more than one heading. Task was more to get you to think and discuss different bias which may or may not affect you and your evaluative criteria.
• #36: So – three forms of value we can apply to evaluate information. We need to use all three:-Omit objective, and we get counterknowledge.Omit subjective, and we risk battery cognition, groupthink... and damage the potential initiative and innovation in the academic community.Omit intersubjective and we risk relativism... Research which meets conclusions which do not fit with our societal and cultural framework and may lack applicability.
• #38: Image – pause and reflect on what might be affecting any practical decisions and actions you might take...
• #39: Worksheets handout – an example of different things to look at to inform any ‘value judgment’ you make on an information resource.Practical tips handout – highlight some tools to aid in this, and examples of those criteria mentioned in the worksheet.
• #40: Just going to highlight some tips for some of these…
• #42: See handout (images from handout)
• #45: Demo H-index and JCRs, comparing a journal within its subject field in a year an article was published. JCRs for journalH-index for author (mention comparison to other authors)H-index of journal, for the same year as article interested in as a comparison to the number of citations received by the article.Remind them – Impact factors are based on number of citations to the publication. The number of citations should not be assumed to indicate a level of quality. But, as many researchers still cling to these as a measure of prestige (wanting to publish in “high impact” journals), it is some measure of ‘perceived level of interest it may generate for the subject community’ at the point of publication.H-index is also a measure of the “no. of citations” and does not provide a clear positive or negative value.
• #46: No demo.
• #47: Self-awareness
• #48: If the article is peer-reviewed, be aware of differences in peer-review requirements (blind peer review, select your own reviewers, etc.)Editorial policy may give some indication of what influences choices I what is published?
• #49: If the article is peer-reviewed, be aware of differences in peer-review requirements (blind peer review, select your own reviewers, etc.)Editorial policy may give some indication of what influences choices I what is published?
• #51: How much time do you have? Thinking back to the unmotivated cat... Do you have more sympathy now thinking about all the different impacts upon your decision process, and the different things you could explore in great detail?But you do need to be responsible as a researcher... And be aware that others may not have been.
• #52: 30 minutes into session.Give 15 minutes hands on time