Open Government Data and MongoDB
4 likes1,315 views
The document discusses using MongoDB to enable open government data. It describes how MongoDB can gather data from various sources and serve it through RESTful JSON APIs. MongoDB's flexible schema allows data to be stored in a format that closely matches the native structure without needing to transform it. This flexibility also enables the storage of custom fields without changing the database schema. Three open data projects are highlighted that use MongoDB: Poligraft, the Real Time Congress API, and the Open State Project.
![{
"title": "President Obama's climate 'Plan B' in hot water -
Darren Samuelsohn - POLITICO.com",
"slug": "EOsc",
"source_url": "http://www.politico.com/news/stories/
0810/40534.html",
"content": ".................",
"entities": [...] Text
}](https://image.slidesharecdn.com/mongodc2011-110629124114-phpapp02/85/Open-Government-Data-and-MongoDB-20-320.jpg)
![{
"title": "President Obama's climate 'Plan B' in hot water -
Darren Samuelsohn - POLITICO.com",
"slug": "EOsc",
"source_url": "http://www.politico.com/news/stories/
0810/40534.html",
"content": ".................",
"entities": [
{
Text
"name": "Barack Obama",
"type": "politician",
},
...
]
}](https://image.slidesharecdn.com/mongodc2011-110629124114-phpapp02/85/Open-Government-Data-and-MongoDB-21-320.jpg)
![{
"title": "President Obama's climate 'Plan B' in hot water -
Darren Samuelsohn - POLITICO.com",
"slug": "EOsc",
"source_url": "http://www.politico.com/news/stories/
0810/40534.html",
"content": ".................",
"entities": [
{
Text
"name": "Barack Obama",
"type": "politician",
"breakdown": {"indiv": "33", "pac": "67"}
"top_industries": ["Lawyers/Lobbyists","Finance/Insurance/
Real Estate","Misc. Business"]
},
...
]
}](https://image.slidesharecdn.com/mongodc2011-110629124114-phpapp02/85/Open-Government-Data-and-MongoDB-23-320.jpg)
Open Government Data and MongoDB
- 1. Open Government Data & MongoDB Luigi Montanez luigi@sunlightfoundation.com
- 4. Open Data + Open Source = Open Government Question? @LuigiMontanez
- 5. MongoDB enables open data Question? @LuigiMontanez
- 6. Opening Up Data ✴ Gather data from disparate sources ✴ Data dumps (SQL, Fixed-width columns) ✴ Web scraping ✴ Text/PDF parsing ✴ Serving RESTful JSON APIs Question? @LuigiMontanez
- 7. JSON ✴ Tree structure, not tabular ✴ Still relational ✴ JSON for data, XML for documents ✴ Closely resembles native data structures ✴ No manual parsing needed Question? @LuigiMontanez
- 8. Three Projects ✴ Poligraft ✴ Real Time Congress API ✴ Open State Project Question? @LuigiMontanez
- 9. Three Projects ✴ Poligraft ✴ Real Time Congress API ✴ Open State Project Question? @LuigiMontanez
- 10. App design drives schema design Question? @LuigiMontanez
- 15. { "title": "President Obama's climate 'Plan B' in hot water - Darren Samuelsohn - POLITICO.com" } Text
- 17. { "title": "President Obama's climate 'Plan B' in hot water - Darren Samuelsohn - POLITICO.com", "slug": "EOsc", "source_url": "http://www.politico.com/news/stories/ 0810/40534.html", "content": ".................", } Text
- 20. { "title": "President Obama's climate 'Plan B' in hot water - Darren Samuelsohn - POLITICO.com", "slug": "EOsc", "source_url": "http://www.politico.com/news/stories/ 0810/40534.html", "content": ".................", "entities": [...] Text }
- 21. { "title": "President Obama's climate 'Plan B' in hot water - Darren Samuelsohn - POLITICO.com", "slug": "EOsc", "source_url": "http://www.politico.com/news/stories/ 0810/40534.html", "content": ".................", "entities": [ { Text "name": "Barack Obama", "type": "politician", }, ... ] }
- 23. { "title": "President Obama's climate 'Plan B' in hot water - Darren Samuelsohn - POLITICO.com", "slug": "EOsc", "source_url": "http://www.politico.com/news/stories/ 0810/40534.html", "content": ".................", "entities": [ { Text "name": "Barack Obama", "type": "politician", "breakdown": {"indiv": "33", "pac": "67"} "top_industries": ["Lawyers/Lobbyists","Finance/Insurance/ Real Estate","Misc. Business"] }, ... ] }
- 25. Natural Schemas Question? @LuigiMontanez
- 26. Three Projects ✴ Poligraft ✴ Real Time Congress API ✴ Open State Project Question? @LuigiMontanez
- 27. Real-Time Congress API Credit: vgm8383 on Flickr
- 29. Politiwidgets
- 30. Requirements ✴ Aggregate lots of data Biographical, Bills, Votes, Earmarks, Video Clips, Floor Updates, Legislative Documents, Committee Schedules, Contributions, Interest Group Ratings ✴ Lightweight responses Question? @LuigiMontanez
- 31. {legislator: { in_office: true, title: "Rep", nickname: "", district: "9", bioguide_id: "L000551", govtrack_id: "400237", phone: "202-225-2661", website: "http://lee.house.gov/index.html", twitter_id: "", last_name: "Lee", name_suffix: "", last_updated: "2010/04/13 00:00:14 +0000", party: "D", chamber: "house", state: "CA", youtube_url: "http://www.youtube.com/RepLee", first_name: "Barbara", gender: "F", congress_office: "2444 Rayburn House Office Building", earmarks: { average_number: 20, total_amount: 10000000, average_amount: 22994535, total_number: 28, last_updated: "2010-03-18", fiscal_year: 2010, } ... }
- 32. // limit selection to a subset of fields db.people.find( { 'first_name' : 'john' }, { 'last_name' : 1, 'address' : 1 } ); // use dot-notation to dig into an object db.people.find( { 'state': 'CA' }, { 'address.zip_code': 1 } );
- 33. ?sections=last_name,first_name,state,earmarks {legislator: { last_name: "Lee", first_name: "Barbara", state: "CA", earmarks: { average_number: 20, total_amount: 10000000, average_amount: 22994535, total_number: 28, last_updated: "2010-03-18", fiscal_year: 2010, } }
- 34. ?sections=last_name,first_name,state,earmarks.total_amount,earmarks.total_number {legislator: { last_name: "Lee", first_name: "Barbara", state: "CA", earmarks: { total_amount: 10000000, total_number: 28 } }
- 35. Partial responses make payloads smaller Question? @LuigiMontanez
- 36. Three Projects ✴ Poligraft ✴ Real Time Congress API ✴ Open State Project Question? @LuigiMontanez
- 38. 50 States = 50 Formats Question? @LuigiMontanez
- 39. Schemalessness allows for granular control Question? @LuigiMontanez
- 40. Custom Fields ✴ Traditional RDBMS ✴ Update the schema for new fields, run a migration, feel icky ✴ Create a custom_fields table ✴ MongoDB ✴ Just store it Question? @LuigiMontanez
- 41. Speaking JSON natively Question? @LuigiMontanez
- 42. Python Source Scraped JSON PostgreSQL Transform
- 43. Source Scraped JSON MongoDB
- 44. Three Projects ✴ Poligraft ✴ Real Time Congress API ✴ Open State Project Question? @LuigiMontanez
- 46. Thanks! sunlightlabs.com @LuigiMontanez Question? @LuigiMontanez