![Problem
ď§ Systematic call center monitoring and behavior coding can be costly, slow,
(somewhat) subjective, and is done on a small sample.
ď§ However, cloud based speech recognition and machine learning tools are
available at modest cost and elastic scale. Can these tools be used to
overcome monitoring challenges?
ď§ Experiment 1: What is the level of agreement between call center monitors
(humans) and tools to classify the subject response to a question?
ď§ Experiment 2: How accurate are these tools in computing interviewer
speech rate and articulation rate? [underway]
2](https://image.slidesharecdn.com/e-bermanifdtc2019-190603123211/85/IFD-TC-2019-Automating-Call-Center-Monitoring-2-320.jpg)
IFD&TC 2019: Automating Call Center Monitoring
- 1. Automating Call Center Monitoring IFD&TC 2019 Lew Berman, MS, PhD John Boyle, PhD Don Allen Josh Duell Matt Jans, PhD Ronaldo Iachan, PhD Josiah McCoy May 21, 2019
- 2. Problem ď§ Systematic call center monitoring and behavior coding can be costly, slow, (somewhat) subjective, and is done on a small sample. ď§ However, cloud based speech recognition and machine learning tools are available at modest cost and elastic scale. Can these tools be used to overcome monitoring challenges? ď§ Experiment 1: What is the level of agreement between call center monitors (humans) and tools to classify the subject response to a question? ď§ Experiment 2: How accurate are these tools in computing interviewer speech rate and articulation rate? [underway] 2
- 3. Solution â Speech recognition and machine learning 3 * Photos by Unknown Author and are licensed under CC BY-NC-SA. Changes have been made to diagrams. Speech Recognition Language Understanding Question Response Text Cleanup Phone Interview Auto-Parse Interview into Question Snippets Have you smoked 100 cigarettes in your lifetime? no 0.5s 1.2s 2.3s 2.7s 3.4s 4.2s 5.3s 8.5s Microsoft Azure Amazon Web Services
- 4. Data Validation Recognized Text Classified Question Calculated Sentiment Classified Response â Machine Learning Results ⥠JSON 4
- 5. Results: Response agreement between monitors and tools 5 BRFSS Question Type Agreement Would you say that in general your health is excellent, very good, good, fair or poor? Straight forward wording Multi-category Straight forward response 96% (N=23) Have you smoked at least 100 cigarettes in your entire life? Straight forward wording Multi-Category (yes/no) Straight forward response 70%/100% (N=23) Are you currently employed for wages, self-employed, out of work for 1 year or more, out of work for less than 1 year, a homemaker, ⌠Complex wording Multi-Category Complex response 61% (N=23) How often do you use seat belts when you drive or ride in a car - would you say always, nearly always, sometimes, seldom or never? Moderately complex wording Multi-Category Moderately complex response 70% (N=23) Please think of the past 12 months. How many times did you reduce or change your outdoor activity level based on the air quality index or air quality alerts? For example avoiding outdoor exercise or strenuous outdoor activity. Please do not include ⌠Complex wording Long question Multi-Category Complex response 20% (N=5) ďž ďž ďž
- 6. Speech & Articulation Rate Q. How many times did the child eat fast food or pizza at school, at home, or at fast- food restaurants, carryout or drive thru? 1: XX per day (range 1 - 15) 2: YY per week (range 1 - 84) Interviewer time = 6.45 seconds Items detected = 27 Words detected = 25 Speech Rate = # đđĄđđđ đđđĄđđđĄđđ đ đđđâđđ đĄđđđĄ) = 27 đđĄđđđ 6.45 â0.0) đ = đ. đđ đđđđ đ/đđđđđđ Articulation Rate = # đ¤đđđđ đđđĄđđđĄđđ đ đđđâđđ đĄđđđĄ) = 25 đ¤đđđđ 6.45 â0.0) đ = 3.88 đđđđ đ/đđđđđđ ⥠Item Timestamping ⢠Behavioral Coding 6 â Question
- 7. Automating Call Center Monitoring IFD&TC 2019 Lew Berman, MS, PhD lewis.berman@icf.com Office: 301-407-6833 May 21, 2019