Introducing JSONpedia
2 likes2,394 views
JSONpedia is a library and web service that parses MediaWiki markup (WikiText) into JSON format. It uses an event-based parser to build a JSON representation of wiki documents in a lightweight and tolerant way. JSONpedia differs from similar tools by producing JSON output for easier consumption and does not add semantics like DBpedia. It can link entities to sources like Freebase and extract specific structures from documents using processor components. The goal is to release JSONpedia as open source by the end of the year.
![WikiText Parser Events
// Document bounding. // Links
void beginDocument(URL document); void beginLink(String url);
void endDocument(); void endLink(String url);
// Error handling. // lists
void parseWarning(String msg, void beginList();
ParserLocation location); void listItem();
void parseError(Exception e, void endList();
ParserLocation location);
// Templates
// Tag handling. void beginTemplate(String name);
void beginTag(String node, Attribute[] void endTemplate(String name);
attributes);
void endTag(String node); // Tables
void inlineTag(String node, void beginTable();
Attribute[] attributes); void headCell(int row, int col);
void commentTag(String comment); void bodyCell(int row, int col);
void endTable();
// Sections
void section(String title, int level); // Generic parameter
void parameter(String param);
// References // parameter / text value
void beginReference(String label); void text(String content);
void endReference(String label);
mercoledì 10 ottobre 12](https://image.slidesharecdn.com/jsonpediarevised2-121010083731-phpapp01/85/Introducing-JSONpedia-11-320.jpg)
Introducing JSONpedia
- 1. WWW.SPAZIODATI.EU JSONpedia Facilitating consumption of MediaWiki content. Michele Mostarda <mostarda@spaziodati.eu>, TW: @micmos mercoledì 10 ottobre 12
- 2. What is JSONpedia? mercoledì 10 ottobre 12
- 3. “JSONpedia is a library and a web service meant to read WikiText markup as JSON.” mercoledì 10 ottobre 12
- 4. ‣ Initially conceived as a tool to produce data to train Machine Learning models. ‣ The REST service,inspired by Sweeble Crystalball,produces JSON, HTML and (coming soon) RDF data. ‣ Written over a context-dependent event based parser to be more performant than an Regex based parser (like the wikiparser) or a DOM based parser (like Sweeble). mercoledì 10 ottobre 12
- 5. Differences with Sweeble mercoledì 10 ottobre 12
- 6. ‣ Lightweight Event based parser. ‣ More tolerant to frequent syntax errors present within WikiText pages. ‣ Serializes to JSON output which is easier to consume! mercoledì 10 ottobre 12
- 7. Differences with DBpedia mercoledì 10 ottobre 12
- 8. ‣ JSONpedia doesn't add any semantic to the extracted data. ‣ JSONpedia could integrate the current DBpedia regex-based parser. ‣ JSONpedia is a not competitor of DBpedia but rather a complement. mercoledì 10 ottobre 12
- 9. JSONpedia Internals mercoledì 10 ottobre 12
- 10. Architecture Parser Structure Validator Input WikiText Extractor Splitter DBpedia API/ Linker Freebase Output JSON + mercoledì 10 ottobre 12
- 11. WikiText Parser Events // Document bounding. // Links void beginDocument(URL document); void beginLink(String url); void endDocument(); void endLink(String url); // Error handling. // lists void parseWarning(String msg, void beginList(); ParserLocation location); void listItem(); void parseError(Exception e, void endList(); ParserLocation location); // Templates // Tag handling. void beginTemplate(String name); void beginTag(String node, Attribute[] void endTemplate(String name); attributes); void endTag(String node); // Tables void inlineTag(String node, void beginTable(); Attribute[] attributes); void headCell(int row, int col); void commentTag(String comment); void bodyCell(int row, int col); void endTable(); // Sections void section(String title, int level); // Generic parameter void parameter(String param); // References // parameter / text value void beginReference(String label); void text(String content); void endReference(String label); mercoledì 10 ottobre 12
- 12. WikiText Processors Processors receive the stream of events generated by the parser and perform data construction and transformation. ‣ Structure ‣ Extractors ‣ Linkers ‣ Splitters ‣ Validator mercoledì 10 ottobre 12
- 13. Structure The Structure Processor receives a stream of WikiText parsing events and builds a 1-1JSON representation of the document DOM. mercoledì 10 ottobre 12
- 14. Extractors Extractors are specific Processors that collect a certain type of data from the event stream: for example the SectionsExtractor collects the list of all sections detected in the document stream. mercoledì 10 ottobre 12
- 15. Linkers A Linker is a Processor which links the current document entity to other informations acquired from external sources. An example of Linker is the FreebaseLinker which connects an entity to the same representation in Freebase if any. mercoledì 10 ottobre 12
- 16. Splitters A Splitter is a Processor able to cut sub trees of the JSON document built by the Structure processor. An example of Splitter is the TableSplitter which extract the JSON structures representing the tables declared in the document. mercoledì 10 ottobre 12
- 17. Validator A Validator is a Processor performing the check of data structures parsed from a document. mercoledì 10 ottobre 12
- 18. Forthcoming Features ‣ JSONpedia DB (based on MongoDB + ElasticSearch) can be queried online. Also JSONpedia dumps will be available. ‣ Online data model Exporter Tool (CSV) ‣ RDF output. mercoledì 10 ottobre 12
- 19. Release JSONpedia will be fully released OpenSource in by the end of the year. mercoledì 10 ottobre 12
- 20. Live Demo http://bit.ly/jsonpedia or http://json.it.dbpedia.org/frontend/form.html mercoledì 10 ottobre 12
- 21. WWW.SPAZIODATI.EU Thanks! Michele Mostarda <mostarda@spaziodati.eu>, TW: @micmos mercoledì 10 ottobre 12