The Challenge of Digital Sources in the Web Age: Common Tensions Across Three Web Histories, 1994-2015
2 likes911 views
The document discusses the challenges historians face in utilizing digital sources from the web between 1994 and 2015, emphasizing the necessity of web archives for historical research. It presents three case studies highlighting the need for digital tools and metadata extraction to analyze the changing landscape of Canadian politics and user-generated content on platforms like Geocities. The document concludes that despite technical difficulties, leveraging web archives can democratize access to historical sources and enrich social history scholarship.
The Challenge of Digital Sources in the Web Age: Common Tensions Across Three Web Histories, 1994-2015
- 1. The Challenge of Digital Sources in the Web Age Common Tensions Across Three Web Histories, 1994-2015 Ian Milligan Assistant Professor
- 2. Why? The sheer amount of social, cultural, and political information generated every day presents new opportunities for historians.
- 3. Could one even study the 1990s and beyond without web archives?
- 4. No. Historians need to do this now, or we’re going to be left behind.
- 5. Nightmare Scenario • Wayback Machine won’t be enough. We won’t use that. • Historians rely uncritically on date-ordered keyword search results, putting them at mercy of search algorithms they do not understand; • Historians are completely left out of post-1996 research, letting everybody else do the work (a la Culturomics project/Nature magazine article); • Our profession gets left behind…
- 6. But what will web archives look like? • Three Distinct Case Studies • Wide Web Scrape, March - December 2011 (Internet Archive) (sample of 80TB WARC collection); • GeoCities End-of-Life Torrent, 2009 (Archive Team); • Archive-It Longitudinal Collections, Canadian Political Parties & Labour Organizations, 2005-2015 (Archive-It/University of Toronto)
- 7. Similarities - Windows into the lives of everyday people.
- 8. Differences - Incredible range of technical skills/no common platform!
- 9. Case Study One • The Wide Web Scrape (~ 80TB) - Snapshot of the Web • 85,570 WARC files, CDX metadata • Similar in some ways to traditional humanistic inquiry, just on a bigger scale.
- 10. ca,yorku,justlabour)/ 20110714073726 http://www.justlabour.yorku.ca/ text/html 302 3I42H3S6NNFQ2MSVX7XZKYAYSCX5QBYJ http://www.justlabour.yorku.ca/index.php? page=toc&volume=16 -‐ 462 880654831 WIDE-‐20110714062831-‐crawl416/ WIDE-‐20110714070859-‐02373.warc.gz
- 12. WARC File WARC-Tools/Lynx (warcfilter.py, warchtmlindex.py and filesdump.py) Indexing CDX Files (finding aids)
- 16. Problem is.. you need to know what you’re looking for!
- 17. Generated using Jimmy Lin’s Warcbase 622k .ca sites, 1,719,167 links
- 20. Countries Mentioned in .ca TLD (excluding Canada)
- 21. Provinces Mentioned in .ca TLD
- 22. Canadian Postal Codes visualized
- 23. Need longitudinal, but the size/intensity = extreme.
- 24. Wide Web Scrapes and the Dream of Social History.
- 25. Case Study Two • Archive-It Research Services: “Canadian Political Parties and Political Interest Groups” and “Canadian Labour Unions.” • 2005 - 2015 • WAT & WARC files
- 26. Pivotal Changes in Canadian Politics, 2005-2015 • Militarization of Canadian society? • Change from ‘natural governing party’ of Liberals to Conservatives • Major policy changes on foreign policy, environment, etc. • How to measure?
- 27. Current Interface • Very limited - simple search engine, some advanced options; no facets • Great collections.. but nobody uses them!
- 29. WAT Files? Potential sweet spot between the lightweight CDX and the heavy-duty WARC?
- 30. Do we want metadata or content analysis?
- 31. Two problems
- 32. Problem One: Historians want to work with content, but we can only use metadata on most computer systems.
- 33. (but that’s ok - we can use metadata to do great work)
- 37. Metadata Extraction • Results @ http://ianmilligan.ca/2015/02/05/topic- modeling-web-archive-modularity-classes/
- 38. Metadata Extraction • Conservative themes (2014): economic development, family, immigration, legislation, women’s issues, senior issues, Ukrainians, constituency offices, some prominent (and not-so- prominent) MPs, and of course, our economic action plan. • Liberal themes (2014): Justin Trudeau (the new leader), cuts to social programs, child poverty, mental health, municipal issues, labour, workers, Stop the Cuts, and housing.
- 39. Metadata Extraction • Conservative themes (2006): education, university, but tons of information on Aboriginal issues; • Liberal themes (2006): community questions, electoral topics, universities, human rights, child care support.
- 40. As well as short stories..
- 48. 2005 Canadian Federal Election
- 49. WATs help us find the files we need to use - and to contextualize them
- 50. Problem Two: You can do amazing things with the content (WARCs), but you need a cluster or powerful computer.
- 51. WARC Analysis • 2005-2009: 244 GB of content; 2.9 GB of plain text • 10,606,822 websites • On a local powerful node (3 Ghz 8-Core Intel Xeon E5/64 GB RAM, data on SSD), about three to four hours per query • On a cluster, about ~10-20 minutes per query, depending on traffic
- 52. Large-Scale Text Analysis • With Hadoop about 15-20 minutes to extract all plain-text from any specified queries: i.e. all pages belonging to Green Party, Liberal Party, Conservative Party, Council of Canadians, etc. • Compared to “out of memory”/ go home for an extended weekend on a local node
- 53. Large-Scale Text Analysis • NER/LDA/Keyword Frequency broken down by scrape date: i.e. scrape carried out 2005-10, see change over time; • Downside: not everything is optimized for parallel environment; if not, it crawls (there goes a day) • Downside: scrape date != creation date, requiring temporal analysis
- 54. Recipe Book Idea
- 55. Using Warcbase to analyze links and full-text
- 57. NER October 2005 62476 Stephen Harper 30234 Michael Chong 30109 Gwynne Dyer 28011 ami Entrez 26238 Paul Martin 22303 Harper
- 58. NER November 2008 3188 Stéphane Dion 2557 Stephen Harper 2471 Stephen HarperLaureen 2410 Dion 2356 Harper
- 59. Visualizing NER
- 62. Shine/WebArchives.ca • UK Web Archive’s Shine (https://github.com/ukwa/ shine) • Indexing as bottleneck • ~ 250GB of WARCs takes ~ 5 days on a single machine • Hadoop indexer available if data in HFDS • ~ 90GB index size
- 63. Five Things I’ve Learned • Political parties delete content • User-generated comments were more common in political parties • Absences can be more informative than presences • We can see the rise/fall of prominent people • Enabling user access is truly transformative
- 64. Shine • Advantages: accessible to the general public, easy to use, interactive trend diagram allows digging down for context, can move down to level of document itself. • Disadvantage: keyword searching requires you know what to look for; random sampling misleading when tens of thousands of records; etc. • Doesn’t take advantage of what makes web sources so powerful: hyperlinks
- 65. Building connections between Warcbase and Shine
- 66. Case Study Three • GeoCities: Archive Team End- of-Life Torrent • 2009, content dating back to 1996; can find sites created pre-1999 using neighbourhood structure
- 68. What was GeoCities? Why does it matter? GEOCITIES USERS: OCT. 1995: 10,000 USERS AUG. 1996: 100,000 USERS OCT. 1997: 1,000,000 USERS
- 69. “largest body of texts detailing the lives of non-elite people ever published”?
- 70. Massive experiment in user-generated content
- 73. Ethically Navigating the Records of Seven Million People
- 74. Selected Neighbourhoods Top Two Topics Athens “… based on education, teaching, reading, writing and philosophy”. people things time person sense life man work world human good mind soul make nature body case made point part parts goddess witch healing incense witchcraft love energy pagan shaman witches sun spirit protection light circle earth religion EnchantedForest “A place for and about kids. Games, stories, educational sites, and homepages created by kids themselves.” blue page school home day kids clues fun time year room birthday family mom jordan play great party friends jq battalion show st jonny horse battery armored lt artillery camp sailor army field col pingu war area quest Heartland “A family oriented neighborhood that represents Main Street in cyberspace. This is the place to find parenting, pets, and home town values.” people time children book years child information year work make life school person system state world books government good family county church home years information st city born state war school mrs history birth records great cemetery death Hollywood “Entertainment capital of the world. Movies, television, and our live video camera at the corner of Hollywood and Vine!” joey rachel ross monica chandler don yeah phoebe hey mike back gonna ll chris big uh guy guys rock frasier niles martin daphne roz don back ll door room scene ve dad turns takes crane good walks yeah Pentagon Military men and women. war people president government american world states power state united general military public soviet political clinton america make army fort war civil island iran world adams army british history badge rhode german french american forts walther cap newport WestHollywood “A community with a culture based on gay and lesbian identity.” gender women sex male female people men person woman sexual crossdressing feminine society identity transgendered marriage man children transsexual Topic Modelling Community to Test Coherence
- 75. Looking at millions of user- contributed & generated images
- 76. And the stories of significant users and meaningful experiences.
- 77. The possibilities of such digital scholarship
- 78. Shared Problems • Never have enough processing power or memory; • Web archive tools often designed for clusters - less than ten historians in North America probably can use one… • Tools • Some work on WARCs; • Some work on ARCs; • Some work on WATs; • And some work on live-web material;
- 79. End-user tools and co-operation with CS colleagues is key.
- 81. More voices, more people, the promise of social history achieved.