Implementing a Corpus for Sinhala Language
Download as PPTX, PDF2 likes975 views
The document outlines the implementation of a Sinhala language corpus, detailing aspects such as resource gathering, data storage, and user interface design. It discusses the architecture, composition, and evaluation of various database technologies, highlighting Cassandra for its performance in data insertion and retrieval. Limitations include the lack of part of speech tagging and the need for new column families for additional information needs.
Implementing a Corpus for Sinhala Language
- 1. Implementing a Corpus for Sinhala Language Upeksha W. D. Wijayarathna D. G. C. D. Siriwardena M. P. Lasandun K. H. L. Dr. Chinthana Wimalasuriya Prof. Gihan Dias Mr. N. H. N. D. De Silva
- 2. OUTLINE • Introduction • Resource Gathering • Data Storage • User Interface Design and Implementation • Application Programming Interface • Limitations
- 3. INTRODUCTION What is a Language Corpus? ▪ A collection of authentic texts of the language ▪ Stored electronically ▪ Contains actual usage patterns of the language ▪ Covers a wide context of the language ▪ Can be used to discover information about language that may not have been noticed through intuition alone
- 5. IDENTIFIED SINHALA RESOURCES News Academic Creative Writing Spoken Gazette News Paper Text books Fiction Subtitle Gazette News Items Religious Blogs Wikipedia Magazine Mahawansa
- 8. DATA REPRESENTATION ▪ Raw data is separately stored as XML formatted files and converted to necessary formats according to information need ▪ Primary Metadata ▪ Article Name ▪ Author ▪ URL ▪ Date (year, Month, Day) ▪ Genre
- 9. DATA STORAGE SYSTEM Considerations ▪ Relational Databases ▪ Oracle DB ▪ H2 Database ▪ Indexed File Systems ▪ Apache Solr ▪ Column Store Databases ▪ Cassandra ▪ Graph Databases ▪ Neo4j
- 10. EVALUATION CRITERIA We considered performance for inserting data and for retrieving 12 different information needs.
- 11. Data Insertion time comparison
- 12. Information Retrieval Performance Comparison - Part 1
- 13. Information Retrieval Performance Comparison - Part 2
- 14. •Cassandra performed better than others in most of the scenarios, and its insertion time increased linearly. •So we chose it for implementing corpus.
- 15. • Cassandra Uses Query Based Data Modeling. • Uses a separate column family for each information need
- 16. User Interface Design and Implementation ● Web interface of Sinmin has been designed for users who would prefer a visualised and summarized view of statistical data of Sinmin. ● Visual design of the interface has been made in a way that any user without prior experience of the interface is able to fulfill his information requirements with little effort.
- 17. Searching the frequency of a word
- 18. Searching the most probable words comes after a word
- 20. APPLICATION PROGRAMING INTERFACE (API) •REST API to expose Corpus services •Much complex and customizable data retrieval and filtering •Interface for third party applications to consume
- 21. LIMITATIONS OF THE CORPUS ● words are not annotated with their Part of Speech (POS) tags and lemmas. ● If a new information need occurs, new column families may need to be created for them and data has to be inserted again.
- 22. QUESTIONS?
- 23. THANK YOU!