Developing and validating a document classifier: a real-life story - Marko Smiljanic
Download as PPTX, PDF0 likes219 views
Marko Smiljanić from Niri Intelligent Computing Ltd details the development and validation of a document classifier, addressing challenges related to ambiguous taxonomies and the need for efficient data handling. The solution involved creating a classifier to automate the processing of a vast number of job vacancies, resulting in significant reductions in manual effort and slight improvements in accuracy. The presentation emphasizes the importance of continuous learning and monitoring of classifier efficacy over time.
![Architecture of our solution
Feature
Extractor
Negotiator
Classifier 1
Classifier 2
Classifier N
…
Vacancy [Class,
Confidence]+
Vacancy Classifier
External
Services](https://image.slidesharecdn.com/2-161227230344/85/Developing-and-validating-a-document-classifier-a-real-life-story-Marko-Smiljanic-18-320.jpg)
Developing and validating a document classifier: a real-life story - Marko Smiljanic
- 1. Marko Smiljanić, NIRI Inteligent computing Ltd, CEO Developing and validating a document classifier: a real-life story
- 2. Developing and validating a document classifier: a real-life story Marko Smiljanić, CEO www.niri-ic.com
- 3. About us. NIRI: 10 years in Intelligent Computing Text Mining Knowledge Discovery and Management All about Data Science
- 5. Business Context The Challenge The Solution Effectiveness Laboratory measurements Impact estimation Reality Wrap up The flow
- 7. Business context Largest clients include Public Employment Services in EU, USA, and Asia Staffing companies in EU, USA
- 9. Business Context The Challenge The Solution Effectiveness Laboratory measurements Impact estimation Reality Wrap up The flow
- 11. Occupation Taxonomies ISCO (International Standard Classification of Occupations) ESCO O*NET and many more ISCO level 1 (10) ISCO level 2 (42) ISCO level 3 (124) ISCO level 4 (400) ESCO level 5 (5000) “Delivery service worker” Challenges (for humans) Knowing the taxonomy Ambiguous taxonomy Hybrid positions Vague vacancy
- 12. Client’s situation in 2014 Vacancy Aggregator and Classifier Correct Code? Publish Repair Code!NO 23% ОК 65% nohelp 14% OK 9% nocode 12% 2000-4000 per day (into >2000 taxonomy classes)
- 13. Business Context The Challenge The Solution Effectiveness Laboratory measurements Impact estimation Reality Wrap up The flow
- 14. The Solution: NIRI will build you a better classifier Vacancy Aggregator and Classifier NIRI Classifier Publish2000-4000 per day
- 15. Really? How accurate will it be? How will it fit our process? Reduce manual effort Increase volume Improve final accuracy Really. We will (try to):
- 16. But you need to give us training data > 1M vacancies No class 12% Not verified 14% Verified 74%
- 17. Long tail effect
- 18. Architecture of our solution Feature Extractor Negotiator Classifier 1 Classifier 2 Classifier N … Vacancy [Class, Confidence]+ Vacancy Classifier External Services
- 19. What to do with confidence? Vacancy, Code, Confidence Vacancy, Code, Confidence Vacancy, Code, Confidence Vacancy, Code, Confidence Vacancy, Code, Confidence Vacancy, Code, Confidence Vacancy, Code, Confidence Vacancy, Code, Confidence Vacancy, Code, Confidence Vacancy, Code, Confidence Vacancy, Code, Confidence … To check manualy Batch Processing CONFIDENCE High accuracy Low accuracy
- 20. Using confidence
- 21. Business Context The Challenge The Solution Effectiveness Laboratory measurements Impact estimation Reality Wrap up The flow
- 22. Measuring accuracy in the laboratory No class 12% Not verified 14% Verified 74% No class Incorrect Correct Test 20% Train 80% Train Test x 5 Vacancy Classifier
- 23. 74% 78% 80% 85% 14% 13% 12% 10% 12% 9% 8% 5% CORPUS CLASSIFIER CLASSIFIER 100 CLASSIFIER 1000 ONE OF MANY LABORATORY MEASUREMENTS Correct Incorrect No class Measuring accuracy in the laboratory
- 24. Measuring accuracy in the laboratory No class 12% Not verified 14% Verified 74% Vacancy Classifier No class 9% Incorrect 13% Correct 78% Original Classifier This is not relaity Biased train/test set Accuracy of test set unknown Inability to test against 26%
- 25. Business Context The Challenge The Solution Effectiveness Laboratory measurements Impact estimation Reality Wrap up The flow
- 27. This is what it actually looks like. Check Repair Reduce manual effort Increase volume Improve final accuracy We will
- 28. And we proposed this one. Bulk Accept Check Repair
- 29. Best/worst case analysis, some manual validation, careful assumptions: Bulk Accept Check Repair
- 30. Impact estimation showed that: Step 1 effort reduction 60% (due to bulk acceptance) Step 2 effort reduction 11% (due to bulk acceptance and top 5 offers) Significant published volume increase (almost to 100%) Accuracy slightly larger (+1%, to around 92%)
- 31. Business Context The Challenge The Solution Effectiveness Laboratory measurements Impact estimation Reality Wrap up The flow
- 32. No class 12% Not verified 14% Verified 74% How can we measure production accuracy? We can not, unless…
- 33. Golden Test Set
- 34. How was it built? Check & Repair 4 eye principle Vacancy Classifier Published Original Code & Top 5 VC codes Original Code & Top 5 VC codes Original Code & Top 5 VC codes Every single classification was marked as either Correct, Acceptable, or Wrong
- 35. Results 63.05% 73.91% 72.06% 74.38% 65.98% 77.56% 76.25% 78.69% CURRENT NIRI VC CURRENT (HQ SOURCE) NIRI VC (HQ SOURCE) GOLDEN TEST SET RESULTS Correct Acceptable Highest Quality Source (Training)
- 36. Business Context The Challenge The Solution Effectiveness Laboratory measurements Impact estimation Reality Wrap up The flow
- 37. Wrap up Clean semantic data, in real-life, can only be a myth. We are looking into data cleansing approaches. Measuring usefulness can be hard and expensive, but … … it can/must to be monitored after the system is deployed. It changes over time. Continuous learning, where possible is a great thing. 1) Implementing state-of-the-art machine learning algorithm is one thing. 2) Making it useful is another. 3) Explaining that to the end-user is the third. NIRI is a very cool company to work with! I hope you liked the story, and I thank you for your attention.
- 38. Developing and validating a document classifier: a real-life story Marko Smiljanić, CEO www.niri-ic.com
Editor's Notes
- #11: FIXME Write about challenges.. Size ambiguiotz, semantics, multi faceted vacanc ies