What I have learned from studying Wikipedia bias

Editor's Notes

• #3: I became a volunteer for CC in 2002. I helped organise the first Wikipedia Academy on the continent in 2005 as Executive Director of iCommons. It was hosted by CIDA City Campus in Johannesburg, attended by Jimmy Wales, Ndesanjo Macha and organised by the iCommons team including Kerryn McKay, Rebecca Kahn and Daniela Faris. As an Advisory Board member of the Wikimedia Foundation between 2007 and 2009,
• #4: the year is 2005. I had become executive director of icommons, an international NGO started by Creative Commons to drive the international movement around free and open soruce software and open content around the world. We organised a Wikipedia Academy in Johannesburg with Jimmy Wales, Ndesanjo Macha and others.
• #5: I helped to shape Majority World (Global South) participation in the open content and free and open source software movement. I worked with my colleagues at iCommons to collect memories from people in Johannesburg sitting in a shopping mall and asking people to bring their old photos of Johannesburg to scan to Wikimedia Commons. I was filled with the passionate belief that Wikimedia and the FLOSS movement could enable Africans to participate.  
• #6: https://geography.oii.ox.ac.uk/wikipedias-global-geography/#single/0 In 2009 I went bck to graduate school where I encountered my first map of Wikipedia’s gaps.
• #7: Big shock. In the next few years we learned through gaps research that
• #8: From these early days, many myths about Wikipedia’s biases were surfaced and remain predominant today.
• #9: But this assumes that Wikipedians are disinterested actors and that the platform is a neutral space in which facts are technically accreted over time
• #10: In my experience and research, I’ve realized that…
• #16: And measurement of gaps was really important to denote that we have a problem!
• #17: Measuring gaps demonstrates the outcomes rather than the sources of inequalities. The question is framed as: “What are the gaps?” rather than “What are the sources of gaps?” The answer we receive from studies that articulate Wikipedia’s gaps in content doesn’t demonstrate causal relationships. For example, women may be less well represented than men in our analyses but we don’t know whether that is because a) women are less well represented (or deemed socially as important) in the literature or in sources or b) because Wikipedia has determined that, despite the existence of reliable sources and external markers of notability, they will not grant them notable status inside Wikipedia (as Francesca Tripodi has recently discovered is happening on English Wikipedia for women’s biographies, 2021)
• #18: Measuring gaps doesn’t enable us to understand Wikipedia’s specific role in the production of inequality and exclusionary behaviour. Research on Wikipedia’s gaps shows how Wikipedia corresponds to existing social inequalities such as gender, geography and now race or ethnicity. This helps us to understand that Wikipedia can exacerbate existing inequalities but not how it uniquely produces such inequalities as a result of its own policies, norms, technologies, economic structures, position in the internet ecosystem etc. Gaps measurement research supports what I’ve called a “mirror theory” of bias – that Wikipedia is a mirror of the world’s biases, not a source of them – a theory that is prominent in Wikipedia culture and that doesn’t enable a recognition of the role that Wikipedia is playing specifically in excluding people and their knowledges.
• #19: The measurement of gaps is often established on flawed assumptions. If measuring gaps in Wikipedia’s coverage is the solution, the problem is being framed as an inability of Wikipedians to see the imbalances that they are producing through their editing practice. And that, therefore, surfacing gaps is the best way to solve the problem.   The role of research in this theory of change is limited to measuring gaps with the assumption that editors only need to know what the gaps are in order to fill them. This illusion of transparency as an automatic vehicle of accountability is well documented in internet policy and media literature and born out by evidence in my own experience.
• #20: In 2018, I worked on a South African primary school project with iolanda Pensa, I. Francis Devouard, and others. The premise of the project was that we would work with South African educators to devise a list of Wikipedia articles related to the South African primary school curriculum that needed to be improved or created. Once the list was created, the team set about sending notifications of the need to improve the articles to the related Wikiprojects. Notification was determined by Wikipedia literature as a strong motivator for the creation of articles. But we found that notification strategies failed to engage interest from either the existing editing community or readers who might be turned into editors. Most of our requests to WikiProjects were simply not considered by any editors even when the Wikiproject was clearly active. Instead, the team turned to an unconventional method, what has been called “expert reviews”. This required contacting local academic experts and asking them to conduct an expert review of the article. The reviews were uploaded to the talk pages of the articles and the team responded by making edits to the article suggested by the academic expert.
• #23: Gaps measurement requires looking for evidence of disparities according to existing (and easily accessible) sociological categories (namely cis gender) that can hide gaps that aren’t easily mapped (or are impossible to map at scale because they are only meaningful at the local level). The gender gap is the most widely studied by far. Predominantly cis gender because that is the data most readily available. Gender is also widely studied because it is a relatively stable concept globally. A handful of studies examines other gaps in coverage and participation including in terms of geography, language, sexual orientation, age, cultural background and socio-economic status. If this becomes the dominant means of evaluating progress towards equity and inclusion goals, we risk hiding and setting back groups that can’t be easily mapped e.g. Indigenous people who aren’t well represented in demographic data or are too small to count OR sources of inequalities in the practices e.g. of article deletion where even when successful or notable women are added in response to content drives, they are more likely to be deleted by Wikipedians who determine that they are non-notable (Tripodi, 2021).  
• #24: Finally, gap measurement rarely accounts for context necessary for enabling action. It results in large-scale, global studies that do not facilitate action by those working on filling gaps in the local/national communities to solve problems of representation where they matter and where they can be solved.
• #26: Instead of asking “What are the gaps?”, we need to ask “What are the sources of gaps?”   We don’t have to start from scratch in answering this question. Decades of scholarship in the history of knowledge can tell us that Wikipedia is a knowledge institution that, like any other, is shaped by questions of power, access and representation. The source of bias on Wikipedia is in relations of unequal power and authority in the socio-technical system that has baked in norms and rules that determine who gets to decide what can be included and what must be excluded on Wikipedia.   It is the practice of sorting – encyclopedic content from unencyclopedic content, knowledge from opinion, NPOV from POV, reliability from unreliability, original research from non-original research that we must focus on. As Tkacz writes: “there needs to be a much greater focus on the practice of sorting and classifying knowledge and the role it plays in both new and old forms of subjugation. “Wikipedia and the Politics of Openness” (Tkacz, 2015, p.12)  
• #27: Finally, we need research that empowers those who have the resources, power and expertise to act on recommendations. Even gaps measurement, when it is focused on the national or local level, is useful if it provides tools that can be used by editors to make change. (e.g. Tripodi’s gender study of articles for deletion or our wikihistories study on how English Wikipedians responded to external notability queues in the Order of Australia awards across all genders). If we are to measure gaps, we need longitudinal studies with data that is meaningful at a national level, because it is the chapters that have the resources to respond to local imbalances (a key component of the current strategy).