Making Sense of Abundance: Opportunity and Challenges Across Three Web Archive Case Studies
0 likes406 views
The document discusses the challenges and opportunities historians face with the vast amount of web-based information generated since the 1990s, emphasizing the necessity of utilizing web archives for contemporary research. It presents three case studies: a wide web scrape, an archive of Canadian political parties, and the preservation of Geocities, demonstrating varying methodologies and data extraction techniques. The author highlights the importance of collaboration with computer science colleagues and the need for advanced tools to analyze web archives effectively, advocating for a more inclusive approach to social history research.
Making Sense of Abundance: Opportunity and Challenges Across Three Web Archive Case Studies
- 1. Making Sense of Abundance Opportunity and Challenges Across Three Web Archive Case Studies Ian Milligan Assistant Professor
- 2. Why? The sheer amount of social, cultural, and political information generated every day presents new opportunities for historians.
- 3. Could one even study the 1990s and beyond without web archives?
- 4. No. Historians need to do this now, or we’re going to be left behind.
- 5. Nightmare Scenario • Wayback Machine won’t be enough. We won’t use that. • Historians rely uncritically on date-ordered keyword search results, putting them at mercy of search algorithms they do not understand; • Historians are completely left out of post-1996 research, letting everybody else do the work (a la Culturomics project/Nature magazine article); • Our profession gets left behind…
- 6. But what will web archives look like? • Three Distinct Case Studies • Wide Web Scrape, March - December 2011 (Internet Archive) (sample of 80TB WARC collection); • GeoCities End-of-Life Torrent, 2009 (Archive Team); • Archive-It Longitudinal Collections, Canadian Political Parties & Labour Organizations, 2005-2014 (Archive-It/University of Toronto)
- 7. Similarities - Windows into the lives of everyday people.
- 8. Differences - Incredible range of technical skills/no common platform!
- 9. Case Study One • The Wide Web Scrape (~ 80TB) • 85,570 WARC files, CDX metadata • Similar in some ways to traditional humanistic inquiry, just on a bigger scale.
- 10. ca,yorku,justlabour)/ 20110714073726 http://www.justlabour.yorku.ca/ text/html 302 3I42H3S6NNFQ2MSVX7XZKYAYSCX5QBYJ http://www.justlabour.yorku.ca/index.php? page=toc&volume=16 -‐ 462 880654831 WIDE-‐20110714062831-‐crawl416/ WIDE-‐20110714070859-‐02373.warc.gz
- 12. WARC File WARC-Tools/Lynx (warcfilter.py, warchtmlindex.py and filesdump.py) Indexing CDX Files (finding aids)
- 16. Problem is.. you need to know what you’re looking for!
- 17. Generated using Jimmy Lin’s Warcbase 622k .ca sites, 1,719,167 links
- 20. Countries Mentioned in .ca TLD (excluding Canada)
- 21. Provinces Mentioned in .ca TLD
- 22. Canadian Postal Codes visualized
- 23. Need longitudinal, but the size/intensity = extreme.
- 24. Wide Web Scrapes and the Dream of Social History.
- 25. Case Study Two • Archive-It Research Services: “Canadian Political Parties and Political Interest Groups” and “Canadian Labour Unions.” • 2005 - 2015 • WAT Files
- 26. WAT Files? Potential sweet spot between the lightweight CDX and the heavy-duty WARC?
- 27. Do we want metadata or content analysis?
- 28. Historians NEED content, but metadata can help us find and contextualize it
- 32. Metadata Extraction • Results @ http://ianmilligan.ca/2015/02/05/topic- modeling-web-archive-modularity-classes/
- 33. Metadata Extraction • Conservative themes (2014): economic development, family, immigration, legislation, women’s issues, senior issues, Ukrainians, constituency offices, some prominent (and not-so- prominent) MPs, and of course, our economic action plan. • Liberal themes (2014): Justin Trudeau (the new leader), cuts to social programs, child poverty, mental health, municipal issues, labour, workers, Stop the Cuts, and housing.
- 34. Metadata Extraction • Conservative themes (2006): education, university, but tons of information on Aboriginal issues; • Liberal themes (2006): community questions, electoral topics, universities, human rights, child care support.
- 35. As well as short stories..
- 43. 2005 Canadian Federal Election
- 44. WATs help us find the files we need to use - and to contextualize them
- 46. Some code/walkthroughs/ sample data available at https://github.com/ ianmilligan1/WAHR
- 47. Case Study Three • GeoCities: Archive Team End- of-Life Torrent • 2009, content dating back to 1996; can find sites created pre-1999 using neighbourhood structure
- 49. What was GeoCities? Why does it matter? GEOCITIES USERS: OCT. 1995: 10,000 USERS AUG. 1996: 100,000 USERS OCT. 1997: 1,000,000 USERS
- 50. Selected Neighbourhoods Top Two Topics Athens “… based on education, teaching, reading, writing and philosophy”. people things time person sense life man work world h soul make nature body case made point part parts goddess witch healing incense witchcraft lov shaman witches sun spirit protection light circle earth r EnchantedForest “A place for and about kids. Games, stories, educational sites, and homepages created by kids themselves.” blue page school home day kids clues fun time year room birthday family mom jordan play great jq battalion show st jonny horse battery armored lt artillery camp sailor army field col pingu wa Heartland “A family oriented neighborhood that represents Main Street in cyberspace. This is the place to find parenting, pets, and home town values.” people time children book years child information year school person system state world books government g family county church home years information st city b school mrs history birth records great cemetery death Hollywood “Entertainment capital of the world. Movies, television, and our live video camera at the corner of Hollywood and Vine!” joey rachel ross monica chandler don yeah phoebe hey mike back gonna ll chris big uh g frasier niles martin daphne roz don back ll door room scene ve dad turns takes crane good Pentagon Military men and women. war people president government american world state united general military public soviet political clinton am fort war civil island iran world adams army british histo german french american forts walther cap newport WestHollywood “A community with a culture based on gay and lesbian identity.” gender women sex male female people men person woman sexual crossdressing femin transgendered marriage man children transsexual Topic Modelling Community to Test Coherence
- 51. Looking at millions of user- contributed & generated images
- 52. And the stories of significant users and meaningful experiences.
- 53. Shared Problems • Never have enough processing power or memory; • Web archive tools often designed for clusters - less than ten historians in North America probably can use one… • Tools • Some work on WARCs; • Some work on ARCs; • Some work on WATs; • And some work on live-web material;
- 54. End-user tools and co-operation with CS colleagues is key.
- 56. More voices, more people, the promise of social history achieved.