Open Data Bay Area (OBDA) | Kurt Bollacker: Public Metadata Commons
0 likes398 views
The document discusses the need for a public metadata commons to preserve valuable digital data that is currently trapped or inaccessible due to technical, legal, or resource barriers. It proposes that a metadata commons, built from extracts of useful information from the web, can help ensure the long-term survival and usability of important trapped datasets by making their cultural context and metadata available, even if the original data becomes lost. The document encourages readers to create their own extracts from public web crawls and share them in an open metadata commons to help build this important public resource.
Open Data Bay Area (OBDA) | Kurt Bollacker: Public Metadata Commons
- 1. A Public Metadata Commons: What is it? Why do we need it? How do we get it? Kurt Bollacker Open Data Bay Area 2012 Nov 27 Wednesday, April 3, 2013 1
- 2. A long time ago, there was no “open” data. All of the media we used to create was physical. Wednesday, April 3, 2013 2
- 3. Then most (all?) of the media became digital. Wednesday, April 3, 2013 3
- 4. The Internet let us ship data around for (almost) free. Wednesday, April 3, 2013 4
- 5. And we learned how to connect it all together. So naturally, we started to build a Global Digital Data Commons! Wednesday, April 3, 2013 5
- 6. At first it was a “free for all” of academics and enthusiasts. Almost all data on the Web was considered to be “open”. Wednesday, April 3, 2013 6
- 7. And then folks figured out how to make money from our contributions, so they started to “lock down” part of the Internet that previously would have been part of the commons. Wednesday, April 3, 2013 7
- 8. Why is this bad? For the data archivist, centrally controlled data have far fewer (single?) points of failure. • Technical Failure • Legal Barriers • Incompetence Wednesday, April 3, 2013 8
- 9. A (Potential) Digital Dark Age "Those who cannot remember the past are condemned to repeat it" --- George Santayana Wednesday, April 3, 2013 9
- 10. How Do We Avoid This Lockdown Of Central Control, (And Hopefully A Digital Dark Age)? We Need A Practical Perspective On the Problem. Wednesday, April 3, 2013 10
- 11. Example Surviving Archives Wednesday, April 3, 2013 11
- 12. Data tends to survive if over the long term, it is: • Visible • Mobile • Well Loved These happen to also be the properties of data in a public commons. Wednesday, April 3, 2013 12
- 13. Historical • Bible / Torah / Koran Examples: • U.S. Constitution • DNA? • Wikipedia Present Day • Open Street Maps Examples: • Freebase • MusicBrainz Why? • There are many copies. (mobile) • Their use is mostly unrestricted. (visible) • Everyone can access and contribute. (well loved) Wednesday, April 3, 2013 13
- 14. But what about data that is still trapped by: • Technical Barriers? • Legal Restrictions? • Limited Resources? Wednesday, April 3, 2013 14
- 15. We build a metadata commons to hold the “cultural context” of our trapped data. Wednesday, April 3, 2013 15
- 16. How does a metadata commons work? Metadata Metadata Trapped Extraction Datasets Processes Metadata Metadata Metadata Even if the original contribution is lost or otherwise made unavailable, we still have its cultural context. Wednesday, April 3, 2013 16
- 17. The cultural context in a metadata commons might contain: • Indices and Tags (to find and organize) • Comments (to analyze and interpret) • Technical metadata (e.g. provenance, format info) • Transforms and Interpretations (to make something useful) Wednesday, April 3, 2013 17
- 18. Where is the trapped data that we care about? A lot of it is in The World Wide Web! But the Web is: • Very large (10TB - 100TB for accessible / deduped) • Very noisy (useless pages, partial duplicates) • Very diverse (in content, purpose, and target audience) How do we build a Metadata Commons from the Web? Wednesday, April 3, 2013 18
- 19. A Practical Place To Start: Common Crawl (and cheap cloud computing resources) make the Web far cheaper and easier to access and manipulate. • Can be downloaded wholesale • Can be processed and analyzed in situ. • Parts can be publicly referenced Wednesday, April 3, 2013 19
- 20. This foundation helps us scale up to “Web size”, but: • What is the useful “metadata of the Web”? • How to we extract that metadata? Wednesday, April 3, 2013 20
- 21. Useful Web Extracts Are • Interesting to many people (to me!) • Can be used to answer relevant questions. • Can be used to build useful products and services. Almost everyone will have an itch to scratch. Wednesday, April 3, 2013 21
- 22. Specific Examples Of Useful Web Extracts (From the Common Crawl code contest) • WikiEntities • Congressional sentiment • Reach of Facebook on the Web Wednesday, April 3, 2013 22
- 23. (A Few) General Shapes Of Web Metadata Extracts • Link graphs • N-gram counts • File Indices by domain or keyword • Mashups with interesting datasets • Wikipedia • Freebase • Location databases (e.g. Open Street Maps) We should all create an extract! Wednesday, April 3, 2013 23
- 24. How do I create an extract? An easy Recipe: • Ingredients: • A Web crawl snapshot • A little bit of programming skill • Access to a cloud computing resources (e.g. EMR) • Directions: • http://commoncrawl.org/mapreduce-for-the-masses/ Wednesday, April 3, 2013 24
- 25. What Happens Once I’ve Made This Awesome Extract? • Share the extracted data • Share the code you created / modified • https://github.com/commoncrawl/ commoncrawl-examples/ • Broadcast it to the world! Wednesday, April 3, 2013 25
- 26. And The World Is Saved! Thank you. Wednesday, April 3, 2013 26
- 27. Some Useful Links • https://github.com/commoncrawl • http://commoncrawl.org/mapreduce-for-the-masses/ • https://github.com/commoncrawl/commoncrawl-examples/ • https://aws.amazon.com/amis/common-crawl-quick-start • https://commoncrawl.atlassian.net/wiki/display/CRWL/About+the+Data+Set Wednesday, April 3, 2013 27