(Some of) Wikipedia's Open Data
0 likes221 views
The analytics engineering team at Wikimedia focuses on building analytics infrastructure to facilitate self-service data access for both WMF and various communities. They strive to make data publicly accessible by default, processing a significant number of requests and offering useful analytics tools like Wikistats 2.0 and live data eventstreams. Their goal is to enhance data utilization and encourage engagement from both researchers and the Wikimedia community.
![6th largest website[Alexa]](https://image.slidesharecdn.com/wikipediaspublicmetadata-170516215555/85/Some-of-Wikipedia-s-Open-Data-12-320.jpg)
(Some of) Wikipedia's Open Data
- 5. The Analytics Team sees as its primary responsibility making Wikimedia related data available for querying and analysis to both WMF and the different Wiki communities and stakeholders. We develop infrastructure so all our users, both within the Foundation as within the different communities, can access data in a self-service fashion that is consistent with the values of the movement.
- 6. We do not handle data requests (for the most part)
- 7. We try for (all) data to be public by default. The more accessible the data is, the more impact it can have.
- 8. But we are not there Yet
- 9. Public Data
- 10. Data that is Useful for the world at large.
- 13. Wikipedia reaches hundreds of millions of unique devices every month and, as such, are a good barometer of browser popularity.
- 14. The most popular browser
- 15. ?
- 16. The most popular browser in April 2017
- 17. Was Chrome 56 with 25% market share
- 19. Issues
- 20. IE7 making a comeback… up more than 1% last year
- 23. Bots ...
- 24. Data useful to WMF, Researchers and Community
- 25. Pageviews
- 26. We process about 200,000 HTTP requests / second at peak
- 27. At peak we process about 200.000 requests per second
- 28. Pageview API
- 29. Get a pageview count time series of en.wikipedia's article Albert Einstein for the month of October 2015: http://wikimedia.org/api/rest_v1/metrics/pageviews/per-article/ en.wikipedia/all-access/all-agents/Albert_Einstein/daily/20151 00100/2015103100 https://wikitech.wikimedia.org/wiki/Analytics/PageviewAPI
- 32. Issues
- 33. Bots, Bots, Bots
- 34. Data useful to WMF (mostly)
- 35. Unique Devices
- 36. Get the monthly number of unique devices for the mobile version of cs.wikipedia.org for the month of January, February and March 2016: http://wikimedia.org/api/rest_v1/metrics/unique-dev ices/cs.wikipedia.org/mobile-site/monthly/20160101 /20160301 https://wikitech.wikimedia.org/wiki/Analytics/AQS/Unique_Devices
- 38. Data useful to Community (mostly)
- 41. CHECK IN April 2007 TEAM/DEPT Analytics Wikistats exists to motivate our editor community. In Wikistats 2.0 we are not only updating the website interface but we are also providing new access to all our edit data in an analytics-friendly form. This much improves (and fundamentally changes) the way, time and resources it takes to calculate edit metrics, for WMF and community.
- 44. Live Data
- 45. EventStreams is a web service that exposes continuous streams of structured event data. Get live updates to Wikimedia projects.
- 47. Navigate to http://wikimedia.org in your browser and open the development console // This is the EventStreams RecentChange stream endpoint var url = 'https://stream.wikimedia.org/v2/stream/recentchange'; // Use EventSource (available in most browsers, or as an // npm module: https://www.npmjs.com/package/eventsource) // to subscribe to the stream. var recentChangeStream = new EventSource(url); // Print each event to the console recentChangeStream.onmessage = function(message) { //Parse the message.data string as JSON. var event = JSON.parse(message.data); console.log(event); };
- 49. Most things documented at: https://wikitech.wikimedia.org/wiki/Analytics/