Aspect extraction using conditional random fields [SentiRuEval]
Download as PPTX, PDF1 like1,638 views
The document discusses aspect extraction for sentiment analysis using Conditional Random Fields (CRFs), focusing on explicit and implicit aspect extraction from reviews. It outlines various approaches, including supervised and unsupervised machine learning methods, and analyzes the performance of their systems compared to baselines across domains such as restaurants and cars. Despite some errors in aspect recognition, the performance was comparable to the best participants, highlighting the importance of using lemmas to enhance results.
![System description
example
w[0]=очень w[-1]=null w[1]=дружелюбное pos[0]=r O
w[0]=дружелюбное w[-1]=очень w[1]=место pos[0]=a O
w[0]=место w[-1]=дружелюбное w[1]=null pos[0]=n s-e
w[0]=с w[-1]=null w[1]=порога pos[0]=s O
w[0]=порога w[-1]=с w[1]=встречают pos[0]=n O
w[0]=встречают w[-1]=порога w[1]=симпатичные pos[0]=v s-e
w[0]=симпатичные w[-1]=встречают w[1]=работники pos[0]=a O
w[0]=работники w[-1]=симпатичные w[1]=тёплый pos[0]=n O
w[0]=тёплый w[-1]=работники w[1]=уютный pos[0]=a O
w[0]=уютный w[-1]=тёплый w[1]=интерьер pos[0]=a O
w[0]=интерьер w[-1]=уютный w[1]=и pos[0]=n s-e
w[0]=и w[-1]=интерьер w[1]=зажигательная pos[0]=c O
w[0]=зажигательная w[-1]=и w[1]=музыка pos[0]=a O
w[0]=музыка w[-1]=зажигательная w[1]=null pos[0]=n s-e](https://image.slidesharecdn.com/dialogue-d-150602093500-lva1-app6891/85/Aspect-extraction-using-conditional-random-fields-SentiRuEval-14-320.jpg)
Aspect extraction using conditional random fields [SentiRuEval]
- 1. Aspect Extraction Using Conditional Random Fields Yuliya Rubtsova Sergey Koshelnikov
- 2. SentiRuEval-2015 Aspect-basedsentiment analysistask A. Extract explicit aspects from the offered review B. Extract all the aspects from the offered review C. Perform sentiment analysis of the aspects D. Categorize the aspects terms by predefined categories E. Sentiment classification of the whole review on aspect categories
- 3. SentiRuEval-2015 Aspect-basedsentiment analysistask A. Extract explicit aspects from the offered review B. Extract all the aspects from the offered review C. Perform sentiment analysis of the aspects D. Categorize the aspects terms by predefined categories E. Sentiment classification of the whole review on aspect categories
- 4. Major approaches to extract aspects Frequency of nouns and/or noun phrases (Hu and Liu, 2004) Simultaneous extraction of both sentiment words (user opinions) and aspects Supervised machine learning (HMM, Jin et al., 2009 and CRF, Jakob and Gurevych, 2010). Unsupervised machine learning or topic modeling (Titov and McDonald, 2008; Brody and Elhadad, 2010)
- 5. Major approaches to extract aspects Frequency of nouns and/or noun phrases (Hu and Liu, 2004) Simultaneous extraction of both sentiment words (user opinions) and aspects Supervised machine learning (HMM, Jin et al., 2009 and CRF, Jakob and Gurevych, 2010). Unsupervised machine learning or topic modeling (Titov and McDonald, 2008; Brody and Elhadad, 2010)
- 6. Conditional Random fields (CRF) CRFs are a type of discriminative undirected probabilistic graphical model. It is used to encode known relationships between observations and construct consistent interpretations.
- 7. Conditional Random fields (CRF) Let G be a graph such that Y = (Yν) ν∈V, so that Y is indexed by the vertices of G. Then (X, Y) is a conditional random field when the random variables Yν, conditioned on X, obey the Markov property with respect to the graph P(yv |YV {v}, X)= P(yv |Yo(v), X),
- 8. Conditional Random fields (CRF) Where Z(x) is normalization factor, C – set of all graphs’ cliques, fc – set of features, 𝜆𝑖 – factors. P(Y | X) = 1 Z(X) exp( lccÎC å fc (yc, X)),
- 9. CRF advantages Relaxation of the independence assumptions CRFs avoid the label bias problem
- 10. System description “s-e” – start of an explicit aspect term, “c-e” – continuation of an explicit aspect term, “s-i” – start of an implicit aspect term, “c-i” – continuation of an implicit aspect term, “s-f” – start of an implicit aspect term, “c-f” – continuation of an implicit aspect term, “O” indicates not an aspect term. Pre-processing
- 11. System description To extract syntactic features (e.g. POS, lemma) we used TreeTagger for Russian (Sharoff, 2008) We also converted all the capital letters into lowercase Pre-processing
- 13. System description example Очень дружелюбное место, с порога встречают симпатичные работники, тёплый, уютный интерьер и зажигательная музыка Very friendly place where pretty staff meet from the threshold, warm and cozy interior and incendiary music
- 14. System description example w[0]=очень w[-1]=null w[1]=дружелюбное pos[0]=r O w[0]=дружелюбное w[-1]=очень w[1]=место pos[0]=a O w[0]=место w[-1]=дружелюбное w[1]=null pos[0]=n s-e w[0]=с w[-1]=null w[1]=порога pos[0]=s O w[0]=порога w[-1]=с w[1]=встречают pos[0]=n O w[0]=встречают w[-1]=порога w[1]=симпатичные pos[0]=v s-e w[0]=симпатичные w[-1]=встречают w[1]=работники pos[0]=a O w[0]=работники w[-1]=симпатичные w[1]=тёплый pos[0]=n O w[0]=тёплый w[-1]=работники w[1]=уютный pos[0]=a O w[0]=уютный w[-1]=тёплый w[1]=интерьер pos[0]=a O w[0]=интерьер w[-1]=уютный w[1]=и pos[0]=n s-e w[0]=и w[-1]=интерьер w[1]=зажигательная pos[0]=c O w[0]=зажигательная w[-1]=и w[1]=музыка pos[0]=a O w[0]=музыка w[-1]=зажигательная w[1]=null pos[0]=n s-e
- 15. System description System 1: CRF with all the above-mentioned labels. We used s-e, c-e and O labels for explicit aspect extraction to perform Task A and s-e, c-e, s-i, c-i, s-f, c-f, O to extract all the aspects for Task B. System 2: Combination of the results of two CRFs —CRF for extraction of explicit aspect terms and CRF for extraction of implicit aspect terms + sentiment facts terms (not explicit). Task A was performed using System 1 and Task B — using both systems.
- 16. Results Exact matching and partial matching. Macro F1-measure means in this case calculating F1-measure for every review and averaging the obtained values. Micro F – partial matching, the intersection between gold standard and extracted term was calculated. F-measure
- 17. Results Task A restaurant domain in comparison to baseline 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Precision Recall Fmeasure Exact matching baseline Word+POS +lemma 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 Precision Recall Fmeasure Partial matching baseline Word+POS +lemma
- 18. Results Task A restaurant domain in comparison to the best results 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Precision Recall Fmeasure Exact matching baseline №1 №2 Word+POS +lemma 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Precision Recall Fmeasure Partial matching baseline №1 №2 Word+POS +lemma
- 19. Results Task A car domain in comparison to baseline 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Precision Recall Fmeasure Exact matching baseline Word+POS +lemma 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Precision Recall Fmeasure Partial matching baseline Word+POS +lemma
- 20. Results Task A car domain in comparison to the best results 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Precision Recall Fmeasure Exact matching baseline №1 №2 Word+POS +lemma 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Precision Recall Fmeasure Partial matching baseline №1 №2 Word+POS +lemma
- 21. Results Task B restaurant domain in comparison to baseline 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Precision Recall Fmeasure Exact matching baseline №1 №2 System 1 Word+POS +lemma System 2 Word+POS +lemma2 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Precision Recall Fmeasure Partial matching baseline №1 №2 System 1 Word+POS +lemma System 2 Word+POS +lemma2
- 22. Results Task B restaurant domain in comparison to the best results 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Precision Recall Fmeasure Exact matching baseline №1 №2 System 1 Word+POS +lemma System 2 Word+POS +lemma2 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Precision Recall Fmeasure Partial matching baseline №1 №2 System 1 Word+POS +lemma System 2 Word+POS +lemma2
- 23. Results Task B car domain in comparison to baseline 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Precision Recall Fmeasure Exact matching baseline №1 №2 System 1 Word+POS +lemma System 2 Word+POS +lemma2 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Precision Recall Fmeasure Partial matching baseline №1 №2 System 1 Word+POS +lemma System 2 Word+POS +lemma2
- 24. Results Task B car domain in comparison to the best results 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Precision Recall Fmeasure Exact matching baseline №1 №2 System 1 Word+POS +lemma System 2 Word+POS +lemma2 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Precision Recall Fmeasure Partial matching baseline №1 №2 System 1 Word+POS +lemma System 2 Word+POS +lemma2
- 25. Error Analysis Not recognized excessively recognized
- 26. Error distribution Restaurants Car Not recognized 67,1% 63% excessively recognized 32,9% 37%
- 27. Error types 1. Technical errors 1.1 Special symbols: Etalon: Салат "цезарь" System: Салат "цезарь 1.2 Lower case: Can’t recognize ie “TO” (technical maintenance in car domain) and “то” (the particle)
- 28. Error types 2. Not recognized 2.1 Shortness Рублей –> руб. –> р. (rubles -> rub -> R.) 2.2 listings Овощи, салаты «Цезарь», лосось (Vegetables, salads "Caesar", salmon)
- 29. Error types 3. Partly recognition 3.1. Before head word “Добавляла вина” (pour wine) “Официант хамил” (The waiter was rude) 3.2. After head word “местечко в углу” (a place in the corner) 4. Excessively recognized 4.1 Not always good deal with named entities Александр (Alexander)
- 30. Conclusion • The performance of our systems was comparable to the best results of SentiRuEval participants. • Realization of these systems demonstrated that the use of lemmas for the Russian language as a CRF feature improves the overall F-measure. • Subsequently we are going to add statistical methods as a CRF feature.