Expert estimation from Multimodal Features
Download as PPTX, PDF0 likes618 views
This document presents research on estimating expertise using simple multimodal features. Three approaches to obtaining features from video, audio, digital pen data, and their combinations are discussed: literature-based, common-sense-based, and "why not?"-based features. Logistic regression and classification tree algorithms showed that features like percentage of calculator use, words mentioned, and writing speed discriminated experts from non-experts with over 80% accuracy. Estimating expertise was possible even from a small number of problems. The researchers concluded simple multimodal features can successfully identify expertise.
![Variable Value for Expert s Discriminat ion Power
P CU
> 0.41
4.44
C oeffi ci entNof t h e L ogi st i c M od el P r edi ct i ng Od ds for a St u dent Sol v i n g C or r ect l y
Ps M
> 34.74
3.19
ASP Variable < 38.05
Predict or L
B 2.86
W ald df
p value exp(B )
Number of Int ervent ions (N OI )
0.0682.86
24.019 1
0.001
0.934
N OR
< 0.13
T imes numbers were ment ioned (T N M )
0.175 23.816 1
0.001
1.192
T M T M > pronounced (T CP ) 0.3292.65
T imes commands were6.25
4.956
1
0.026
1.390
Analysis at problem level
Proport ion of Calculat or Usage (P CU)
Fast est St udent in t he Group (F W )
Constant
1.287
0.924
1.654
25.622
18.889
53.462
1
1
1
0.001
0.001
0.001
a
3.622
2.519
0.191
To calculat e t he probability of correct ly solving a problem
N um b er of P oi nt s ( A N P ) : Represent s, in
s. Classificat
provided
by of point s t hat composehe following sub-setR st at ist ical ion Trees,[21] for M ac, wer
a st udent (P ) t each st roke
formula should be used: by rpa
number
in t he
software
second part of t he analysis.
St r oke T i m e L engt − 11. 7−L0.1N O I + s for N M + 0.3T C P + 1. 3P C U + 0. 9F W
h A ST
A ccount
ehe (st udent) :needed, in 0.2T
f milliseconds t hat t
avP st=
plet e each roke. + e− 11.7− 0. 1N O I + 0.2T N M + 0. 3T C P + 1.3P C U + 0.9F W
1
St r oke P at h L engt h ( A SP L ) : Represent s
umber of pixels t hat t he t raject ory of st rokes
St r oke D i sp l acem ent ( A SD ) : A ccount s for
splacement defined by t he st art ing and ending
4.2 Expert prediction
(1)
4.1 Odds of a student solving c
problem
A Logist ic regression was run wit h St udent](https://image.slidesharecdn.com/expertestimation-131208172210-phpapp02/85/Expert-estimation-from-Multimodal-Features-23-320.jpg)
Expert estimation from Multimodal Features
- 1. Expertise Estimation based on Simple Multimodal Features Xavier Ochoa, Katherine Chiluiza, Gonzalo Méndez, Gonzalo Luzardo, Bruno Guamán and James Castells Escuela Superior Politécnica del Litoral
- 3. How to (easily) obtain multimodal features? What is already there?
- 4. Three Approaches • Literature-based features • Common-sense-based features • “Why not?”-based features
- 5. All approaches proved useful Proof that we are in an early stage
- 6. Video: Calculator Use (NTCU) • Idea: – Calculator user is the one solving the problem • Procedure: – Obtain a picture of the calculator – Track the position and angle of the image in the video using SURF + FLANN + Rigid Object Transformation (OpenCV) – Determine to which student the calculator is pointing in each frame
- 7. was on hat ved inand format ions capabilit ies provided by OpenCV . W hile t here were some frames in which t his mat ching was not possible due t o object occlusions or changes in t he illuminat ion of t he calculat or, in general t he described det ect ion t echnique was robust and provided useful posit ion and direct ion dat a. Video: Calculator Use (NTCU) ing by ent was core ven iffion, ex- at h t et F i gur e 1: D et er m i n at i on of w hi ch st u dent i s u si n g
- 8. Video: Total Movement (TM) • Idea: – Most active student is the leader/expert? • Procedure: – Subtract current frame from previous frame – Codebook algorithm to eliminate noise-movement – Add the number of remaining pixels
- 9. image out put by t he Codebook algorit hm. T his magnit ude, when comput ed for t he ent ire problem solving session, result s from summing up it s individual values obt ained from each frame t hat compose a problem recording. Video: Total Movement (TM) (a) Original frame (b) Difference frame F i gu r e 2: R esul t s of t he C odeb ook al gor i t hm .
- 10. Video: Distance from center table (DHT) • Idea: – If the head is near the table (over paper) the student is working on the problem • Procedure: – Identify image of heads – Use TLD algorithm to track heads – Determine the distance from head to center of table
- 11. lem d to cupar- sped as preodeant mall ndihere ned and t hen, t he average of t hese dist ances is obt ained by problem (see Figure 3). A ddit ionally, t he variance of t he average dist ance head t o t able (SD-DHT ), was calculat ed t o det ermine if a part icipant remains most ly st at ic or varies his or her dist ance t o t he t able. Video: Distance from center table (DHT) deary ude, reom F i gur e 3: C al cu l at i on of t h e di st an ce of t h e st u d ent ’ s
- 14. Audio: Features • • • • • Number of Interventions (NOI) Total Speech Duration (TSD) Times Numbers were Mentioned (TNM) Times Math Terms were Mentioned (TMTM) Times Commands were Pronounced (TCP)
- 15. Digital Pen: Basic Features
- 16. Digital Pen: Basic Features • • • • • Total Number of Strokes (TNS) Average Number of Points (ANP) Average Stroke Path Length (ASPL) Average Stroke Displacement (ASD) Average Stroke Pressure (ASP)
- 17. Digital Pen: Shape Recognition Stronium – Sketch Recognition Libraries
- 18. Digital Pen: Shape Recognition • • • • • • Number of Lines (NOL) Number of Rectangles (NOR) Number of Circles (NOC) Number of Ellipses (NOE) Number of Arrows (NOA) Number of Figures (NOF)
- 19. Features Variation • When the features were evaluated inside a group two variations were usually obtained: – Percentage of the total (e.g. Calculator Use) – Highest / Lowest (e.g. Faster Writer, Lowest Time)
- 20. Next step: Find predictive features
- 21. Prediction at two levels Problem and Group
- 22. Analysis at Problem level Solving Problem Correctly • All available problems were used • Logistic Regression to model Student Solving Problem Correctly • Resulting model was significantly reliable • 60,9% of the problem solving student was identified • 71,8% of incorrectly solved problems were identified
- 23. Variable Value for Expert s Discriminat ion Power P CU > 0.41 4.44 C oeffi ci entNof t h e L ogi st i c M od el P r edi ct i ng Od ds for a St u dent Sol v i n g C or r ect l y Ps M > 34.74 3.19 ASP Variable < 38.05 Predict or L B 2.86 W ald df p value exp(B ) Number of Int ervent ions (N OI ) 0.0682.86 24.019 1 0.001 0.934 N OR < 0.13 T imes numbers were ment ioned (T N M ) 0.175 23.816 1 0.001 1.192 T M T M > pronounced (T CP ) 0.3292.65 T imes commands were6.25 4.956 1 0.026 1.390 Analysis at problem level Proport ion of Calculat or Usage (P CU) Fast est St udent in t he Group (F W ) Constant 1.287 0.924 1.654 25.622 18.889 53.462 1 1 1 0.001 0.001 0.001 a 3.622 2.519 0.191 To calculat e t he probability of correct ly solving a problem N um b er of P oi nt s ( A N P ) : Represent s, in s. Classificat provided by of point s t hat composehe following sub-setR st at ist ical ion Trees,[21] for M ac, wer a st udent (P ) t each st roke formula should be used: by rpa number in t he software second part of t he analysis. St r oke T i m e L engt − 11. 7−L0.1N O I + s for N M + 0.3T C P + 1. 3P C U + 0. 9F W h A ST A ccount ehe (st udent) :needed, in 0.2T f milliseconds t hat t avP st= plet e each roke. + e− 11.7− 0. 1N O I + 0.2T N M + 0. 3T C P + 1.3P C U + 0.9F W 1 St r oke P at h L engt h ( A SP L ) : Represent s umber of pixels t hat t he t raject ory of st rokes St r oke D i sp l acem ent ( A SD ) : A ccount s for splacement defined by t he st art ing and ending 4.2 Expert prediction (1) 4.1 Odds of a student solving c problem A Logist ic regression was run wit h St udent
- 24. Analysis at Group Level Expertise Estimation • Data from group 2 was removed because there was no expert • Features were feed to a Classification Tree algorithm • Several variables had a high discrimination power between expert and non-experts • Best discrimination (6.53) result in 80% expert prediction and 90% non-expert prediction
- 25. ASD AST L ASP MD FW Analysis MP Highest value Lowest value Group Level Highest value at Expertise Estimation T abl e 4: C l assi fi cat i on t r ee sp l i t s w i t h nor m al i zed and non-n or m al i zed feat ur es Variable FW LP P CU MN PN M Value for Expert s > 0.5 > 34.74 > 38.05 > 0.13 > 6.25 Discriminat ion Power 6.53 6.53 4.44 4.03 3.19 classificat ion is maint ained and plat eau at t he final value around t he 12t h problem.
- 26. Expert Estimation over Problems Plateau reached after just 4 problems
- 27. Main conclusion: Simple features could identify expertise
- 28. Conclusions • Three strong features: – Faster Writer (Digital Pen) – Percentage of Calculator Use (Video) – Times Numbers were Mentioned (Audio) • Each mode provide discrimination power to establish expertise • These features maintain its discriminant power at lower levels (solving a problem correctly)
- 29. Gracias / Thank you Questions? Xavier Ochoa xavier@cti.espol.edu.ec http://ariadne.cti.espol.edu.ec/xavier Twitter: @xaoch