Amharic document clustering
5 likes1,474 views
This document discusses document clustering in the Amharic language for information browsing and retrieval. It introduces the challenges of searching and accessing information in Amharic due to the growing amount of digital documents. The document then describes the process of document clustering, which groups documents based on similarities to organize information. Key steps in the clustering process include document preprocessing, vector representation, and hierarchical clustering. Experimental results show that tuning the global support threshold is important for creating the desired hierarchy, and stemming affects cluster overlap. Future work could involve developing standard Amharic language resources and comparing different clustering and information retrieval methods.
![Document clustering process
2. Document pre-processing
- Indexing the documents
- Word identification (Amharic word separators considered)
- Smoothing( characters of the same voice were mapped to a single character)
- ጸሃይ፣ ጸኅይ፣ጸሀይ፣ ፀሃይ፣ ፀኃይ፣ ፀሀይ… ፀሐይ
- Stop word removal
- Words like [ለ፣ ወደ]=to, [ከ]=from, [የ] are removed [non-content bearing
words]
- Stop words in news domain such as [ገልጿል] disclosed, [አመልክቷል] ect.
- Stop words are validated against their frequency in the document
collection [a threshold of 100 is used]](https://image.slidesharecdn.com/amharicdocumentclustering-111208072442-phpapp02/85/Amharic-document-clustering-9-320.jpg)
![Document clustering process
3. Stemming of indexed terms
- Amharic language is morphologically complex
- Nouns have inflection [prefix, and suffix]
- አስተማሩ
- አስተማረ
- አስተማረች አስተማረ
- አስተማርኩ
- Verbs have inflection[prefix, suffix and infix]
- ሰበረ
- ሰበረች
- ሰበርክ ሰበር ስብር
- ሰበርሽ
- አሰበረ
- stemming brings the word into its common form](https://image.slidesharecdn.com/amharicdocumentclustering-111208072442-phpapp02/85/Amharic-document-clustering-10-320.jpg)
![Experimental result
• Tuning the global support to get hierarchical documents
– More than 10% global support gives flat hierarchy
– Less than 1% global support gives a single vertical hierarchy
– 5% global support shows a better performance
Global Support Width Depth Remark
>=20% < =9 0 Flat hierarchy
10% 61 2 1 level hierarchical(only for 2 classes
5% 92 10 10 level hierarchy for two classes 5 level hierarchy for five classes
<=1% >=120 25 25 level hierarchy[took too much time to cluster]](https://image.slidesharecdn.com/amharicdocumentclustering-111208072442-phpapp02/85/Amharic-document-clustering-14-320.jpg)
![Future directions
• Developing standard corpus collection
• Using ontologies as a concept map
• Standardization for Amharic language resources such as standard
stop word list
• Further research in stemming [cross domain research]
• Comparison with other document clustering algorithms
• Comparison with other information retrieval methods](https://image.slidesharecdn.com/amharicdocumentclustering-111208072442-phpapp02/85/Amharic-document-clustering-18-320.jpg)
Amharic document clustering
- 1. Document Clustering in Amharic for information browsing and retrieval Yalemisew Mintesinot Abgaz Yabgaz@computing.dcu.ie Dec 1, 2011
- 2. Introduction • The rate of production of information is growing exponentially • Documents produced in Amharic language are increasing available in digital format accessible online • Growing number of Amharic web documents than before • Growing number of Amharic language users • Increasing number of applications available in Amharic
- 3. Introduction • Challenges ahead – Searching and accessing the information in Amharic is difficult • From the language perspective • From the knowledge perspective • Availability of tools – Identifying the relevant documents from the available ones is challenging • Searching and Search results – Browsing the documents in a concept map is not available • The challenges call for a solution
- 4. Agenda Items • Introduction • Document clustering • Document clustering process • Experimental results • Conclusion • Future work
- 5. Document clustering • Document clustering is a process of identifying groups or clusters of documents with common features. • Groups documents based on similarities of the contents of the documents • Used for information organization and information retrieval • To design a retrieval mechanism for searching through the clusters • Can be – Hierarchical – None hierarchical • Is different from document classification
- 6. Document clustering • Hierarchical document clustering – Is a widely used method – Generates hierarchical classes with generalization at the top and specialization at the bottom • Clustering algorithms – Divisive – Agglomerative • Single link, complete link, group average link, ward’s method and • Frequent item based hierarchical clustering
- 7. Document clustering process Document collection Document Index words text Indexing collection Stemming Stemmed Stop index words Word list Vector Document term vectors Suffix list representation Cluster Clustering Representation Query Query Query‐Cluster Output processing Matching documents
- 8. Document clustering process 1. Document collection - Amharic news documents collected from Walta Information Centre - Similar documents were selected by previous researchers - The documents cover various domains such as - Governance - Market - Politics - Sport - Education etc.
- 9. Document clustering process 2. Document pre-processing - Indexing the documents - Word identification (Amharic word separators considered) - Smoothing( characters of the same voice were mapped to a single character) - ጸሃይ፣ ጸኅይ፣ጸሀይ፣ ፀሃይ፣ ፀኃይ፣ ፀሀይ… ፀሐይ - Stop word removal - Words like [ለ፣ ወደ]=to, [ከ]=from, [የ] are removed [non-content bearing words] - Stop words in news domain such as [ገልጿል] disclosed, [አመልክቷል] ect. - Stop words are validated against their frequency in the document collection [a threshold of 100 is used]
- 10. Document clustering process 3. Stemming of indexed terms - Amharic language is morphologically complex - Nouns have inflection [prefix, and suffix] - አስተማሩ - አስተማረ - አስተማረች አስተማረ - አስተማርኩ - Verbs have inflection[prefix, suffix and infix] - ሰበረ - ሰበረች - ሰበርክ ሰበር ስብር - ሰበርሽ - አሰበረ - stemming brings the word into its common form
- 11. Document clustering process 4. Representing documents using document vector - Term weighting is used to weight the term frequency - Weight(di,j) = Tfij* (logN- log n)+1 • Tf ij is frequency of term j in document i • N is the number of document in the collection and • n is the number of documents containing the term. – Weighted term frequency for index terms
- 12. Document clustering process 5. Clustering the documents - Constructing the initial clusters - Following the FIHC algorithm, initial clusters are constructed by setting the global support between 0 and 1 - The initial cluster groups similar documents together and creates a new cluster whenever it gets a different document - Used global support
- 13. Document clustering process 5. Clustering the documents - Making the clusters disjoint - The score function is used to measure how well a cluster fit the documents at hand. - Hierarchical tree construction - The cluster tree is built using inter cluster similarity - Centroid calculation - Tree pruning
- 14. Experimental result • Tuning the global support to get hierarchical documents – More than 10% global support gives flat hierarchy – Less than 1% global support gives a single vertical hierarchy – 5% global support shows a better performance Global Support Width Depth Remark >=20% < =9 0 Flat hierarchy 10% 61 2 1 level hierarchical(only for 2 classes 5% 92 10 10 level hierarchy for two classes 5 level hierarchy for five classes <=1% >=120 25 25 level hierarchy[took too much time to cluster]
- 16. Experimental result recall-precison globalsupport=10% 0.9 0.8 0.7 precision 0.6 0.5 10% 0.4 5% 0.3 0.2 0.1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 recall
- 17. Discussion of results • Tuning the global support threshold plays a significant role in creating the required clusters • Stemming affects the clusters and creates overlapping clusters • High precision can be achieved if frequent items(terms) are used • High recall can be achieved when the whole index terms are used but it greatly affect precision
- 18. Future directions • Developing standard corpus collection • Using ontologies as a concept map • Standardization for Amharic language resources such as standard stop word list • Further research in stemming [cross domain research] • Comparison with other document clustering algorithms • Comparison with other information retrieval methods
- 19. Thank you! Questions?