![VIP callers in Asterisk
You query QueueMetrics before routing a call:
same => n,Set(CURLOPT(hashcompat)=yes)
same => n,Set(NUM=${CALLERID(number)})
same => n,Set(URL=http://my.qm/queuemetrics/numberLookup.do
?mode=hash&user=pbxapi:api123))
same => n,Set(HASH(resp)=${CURL(${URL}&number=${NUM})})
same => n,Set(CALLERID(name)={HASH(res,name})
same => n,GotoIf($["${HASH(resp,action)}" = "VIP"]?supervip)
same => n,GotoIf($["${HASH(resp,action)}" = "BLACKLIST"]?blklist)
same => ....
So with one single call you get:
●Name of the caller
●VIP or Blacklist
●Agent affinity
And you can route accordingly!](https://image.slidesharecdn.com/queuemetrics-tipstricks-160224074431/85/QueueMetrics-Tips-and-Tricks-74-320.jpg)
![VIP callers in Asterisk
You query QueueMetrics before routing a call:
same => n,Set(CURLOPT(hashcompat)=yes)
same => n,Set(NUM=${CALLERID(number)})
same => n,Set(URL=http://my.qm/queuemetrics/numberLookup.do
?mode=hash&user=pbxapi:api123))
same => n,Set(HASH(resp)=${CURL(${URL}&number=${NUM})})
same => n,Set(CALLERID(name)={HASH(res,name})
same => n,GotoIf($["${HASH(resp,action)}" = "VIP"]?supervip)
same => n,GotoIf($["${HASH(resp,action)}" = "BLACKLIST"]?blklist)
same => ....
So with one single call you get:
●Name of the caller
●VIP or Blacklist
●Agent affinity
And you can route accordingly!](https://image.slidesharecdn.com/queuemetrics-tipstricks-160224074431/85/QueueMetrics-Tips-and-Tricks-75-320.jpg)
QueueMetrics - Tips and Tricks
- 1. Presented by: Lenz Emilitri Founder, Loway @lenz Effective Call-Center monitoring and reporting QueueMetrics Tips and Tricks Asterisk Africa – Johannesburg, Feb 16, 2016
- 2. Contents ●Introduction –Who we are –What we do ●Running QueueMetrics ●Integrating QueueMetrics ●Open discussion
- 3. Who we are Since 2004, Loway is leading the way in development of advanced software solutions for the Asterisk® PBX. With QueueMetrics we set up modern standards for call center performance measurement. Our Mission is to put Swiss passion for precision and reliability at customers' service. Together with Loway, you can provide your clients with the most reliable, flexible, and sophisticated call-center management solutions available today.
- 4. What we do Our History: ● Started working with Asterisk in 2003 ● Developed QueueMetrics in 2005 ● Developed WombatDialer in 2012 ● Launched QueueMetrics Live in 2015 Installed base: QueueMetrics currently deployed in thousands of call-centers worldwide Average site: ~50 agents Largest sites: ~1000 agents live (on Asterisk clusters) WombatDialer deployed in ~300 sites Average site: ~80 channels Largest site: ~4000 channels Client base: ● 30% USA / Canada ● 25% Europe ● 20% LATAM ● 10% Africa ● 10% Asia ● 5% Middle East
- 5. Contents ●Introduction ●Running QueueMetrics –Monitoring & Performance –Configuration best practices –Extending call tracking –Quality Assessment (QA) ●Integrating QueueMetrics ●Open discussion
- 6. Contents ●Introduction ●Running QueueMetrics –Monitoring & Performance –Configuration best practices –Extending call tracking –Quality Assessment (QA) ●Integrating QueueMetrics ●Open discussion
- 7. The key process to improve run-time performace is: Monitoring → Assessing → Understanding → Fixing. ●Monitoring builds a baseline of data ●Assessing lets you define how the problem appears ●Understanding – that's the hard part! ●Fixing is often trivial. ●Simply throwing better hardware at the problem will not usually help in the long run. ●Don't wait for problems to start monitoring! Monitoring & Performance
- 8. QueueMetrics was designed to be run in call-centers with up to 1,000 live agents. Most performance problems in mid-sized CC's (up to 150 agents) are caused by: ●Java memory (mis)configuration ●QM data caching turned off ●Broken MySQL indexes ●Wrong audio storage model ●JSON or XML-RPC wallboards Performance problems?
- 9. Monitoring – Java Memory QueueMetrics is written in Java. ●Java uses a fixed memory pool that you need to configure. ●Target: no stop-the-world major collections. That's where the system appears to hang. ●The RPM defaults are NOT okay for larger systems ●Sensible defaults: -Xms4096M -Xmx4096M -server -XX:+UseParallelOldGC - XX:PermSize=512M -XX:MaxPermSize=512M ●But you need to monitor them!
- 10. It is very easy to monitor remotely a production JVM. Monitoring - jVisualVM ●Check heap size and parameters ●Monitor GC activity vs CPU usage ●Monitor threads: –1 thread ~ 1 active request –>5 users per thread ●Normal for memory to go up until GC'd ●Thread dump: what is QM doing now? –Standard QM scripts let you take thread dumps ●CPU Sampler: why is it taking so long?
- 11. Internal tools: Admin page → System diagnostic tools Monitoring – QM internals RAM caches –Data for all partitions –SQL cache efficiency > 90% –Cached strings < 1M Live inspector –See data being loaded –Recent activity –Quite expensive to run
- 12. Sometimes we find random MySQL issues. ●Symptoms –Performance degrades strongly and all of a sudden –Very high MySQL CPU usage and light Java CPU usage –Lots of open threads –Very high disk I/O on MySQL server ●Root cause –Access indexes on tables present but broken / unused. ●What to do: –Manual reindexing (drop indexes and recreate) –Might take a while on very large databases –..assuming I/O on the server is adequate (SSD!) Monitoring – MySQL issues
- 13. If your audio recordings are all stored in a single folder, scan times can get high... ●Symptoms –QueueMetrics very slow when opening the call pop-up window ●Root cause: –Improper file storage ●Solution –Split your recording in multiple folders, per day and queue, e.g. /mount/audio/2016-01/30/queue1/…. –Use LocalFilesPerDay as the audio playback PM You can also have multiple locations scanned in sequence, e.g. local first, NAS next. Monitoring – recordings
- 14. If you use a remote wallboard or any other piece of integration though JSON or XML-RPC webservices, it should be well behaved. ●Symptoms –Overall performance degradation –VisualVM shows tons of open threads. ●Root cause –Too many requests sent per unit of time ●Solution –Services should cache local results if they need to feed multiple consumers / wallboards at once –Services must wait for an answer before submitting another – if not, it's sure „Death by a Thousand Threads“. Monitoring – Services
- 15. QueueMetrics offers a plain service to be used by monitoring apps: http://my.qm:8080/queuemetrics/sysup.jsp ●Lightweight JSON response ●Checks memory and DB connectivity ●Easy to integrate with existing monitoring platforms (Influx, Riemann, Zabbix… you name it) Monitoring – Sysup
- 16. We run hundreds of QueueMetrics instances in our service QueueMetrics Live. Monitoring – what we do ●Chose very basic metrics: –total memory, –free memory, –web hits –loader hits –errors ●Easily spot 95% of issues ●Manually instrument if needed
- 17. Contents ●Introduction ●Running QueueMetrics –Monitoring & Performance –Configuration best practices –Extending call tracking –Quality Assessment (QA) ●Integrating QueueMetrics ●Open discussion
- 18. QueueMetrics is built around an all-encompassing security model based on keys: ●System keys (access QM functions and pages) ●Custom keys Users have a keyring: –Default keyring = security class –Can add an revoke keys for each user –Masterkey ●All functionalities have a required system key ●All objects have optional keys ( = locks) –Blank key = visible to everybody The security model
- 19. Create different classes for different kinds of users ●Different clients ●Different access roles ●Start with the default ones (AGENT, USER, etc.) Edit keys with the new editor: ●Plain-English view ●Set individual keys ●View inherited keys Do not use the master key Security best practices
- 20. Configuration All configuration stored in the system folder. ●You can see which one is used in the License page A text file holds the current configuration keys: ●File: configuration.properties ●General defaults ●General settings ●Picked up on user log-in A different configuration file holds licensing information: ●File: tpf.properties ●Picked up on system restart
- 21. Config. best practices Keep a backup of your current configuration! Edit the configuration.properties file through Home → Edit system parameters. Edit common properties through the Explorer GUI: Home → Explore system parameters View the current configuration through the Home → System Diagnostic Tools → View Configuration.
- 22. QueueMetrics was born to analyze inbound queues... ●In order to analyze outbound, we used special „outbound queues“ that used a piece of dialplan to do the tracking ●When you do outbound in a call-center context, you need a „reason“ to distinguish calls ● You can originate outbound calls through phones or though the GUI. Outbound calls in QM
- 23. For best results... ●Use an underlying physical queue for outbound. –Presence –Pauses –Hotdesking –No calls! ●Name the queue „q-XXX“ where „XXX“ is the inbound queue ●Use the Icon page to originate, not phones ●If you need to place a lot of calls, use a dialer solution that works with QM (e.g. WombatDialer) Outbound – best practices
- 24. Agent configuration Agents are used by QueueMetrics to map names and properties to a PBX code. An agent has... ●An agent code ●A name ●A supervisor ●A group ●A set of queues ●Etc... Users are used by QueueMetrics to allow access to the GUI. A user has... ●A login ●A password ●Belongs to a class ●Has a set of security keys ●Etc... What is the difference between an Agent and a User? In order to let an agent use the GUI, its Login must match the Agent code (e.g. Agent/10 The QueueMetrics licensing model is based on Agents, not Users.
- 25. Agents – best practices For best results... ●Use simple SIP phones –Configure max lines = 1 –Check agent state on queues ●If using Local channels, avoid excess dialplan ●Use hot-desking ●Use the Icon page –Easy log-on multiple Q's –Easy status tracking –Agent messages –Outcomes
- 26. Contents ●Introduction ●Running QueueMetrics –Monitoring & Performance –Configuration best practices –Extending call tracking –Quality Assessment (QA) ●Integrating QueueMetrics ●Open discussion
- 27. Asterisk logs the basic events for queue. QueueMetrics improves on this, by tracking: ●Start of call (as opposed to queueing of call) ●DNIS ●IVR traversal and menus ●IVR goals ●Music on Hold As they are not provided by Asterisk, they need to be added at the dialplan level by adding INFO records. Tracking more information
- 28. By producing a log entry like the following: 1353461650|1353461627.33271|NONE|NONE|INFO|IVRSTART|1234|5556777 You get: ●Actual start of call ●DNIS ●Call visible on RT before it's queued Tracking DNIS
- 29. QueueMetrics is able to track the full life cycle of a call. You need to: ●Send events when keys are pressed ●Send events when goals are reached ●Queueing is automatically considered a goal Tracking IVRs
- 30. How often were your IVR menus traversed? When and where did people stop? How many self-service goals were reached? Tracking IVRs: Reports How much did it take to traverse each menu?
- 31. For many call-centers, measuring Music-on-Hold on answered calls is important: ●Directly affects the perceived quality of service ●Anomalous MOH durations are an „alarm bell“ for underlying problems ●QueueMetrics fully supports it –Specific reports –Call details So… where is the catch? ●Asterisk does not produce these events for queues Tracking Music-on-Hold
- 32. So how do we do it? ●Patches available for 1.4 & 1.8 –Never accepted into Asterisk –Used in production in a number of 100+ channel CCs for years –Hard to upgrade ●Passive monitoring daemon - MohTracker –Uses AMI and works with any version of Asterisk –Available as alpha software, free of charge –Contact us if interested Tracking MOH (2)
- 33. Contents ●Introduction ●Running QueueMetrics –Monitoring & Performance –Configuration best practices –Extending call tracking –Quality Assessment (QA) ●Basics of QA monitoring ●Effective scoring ●Performance tracking ●Integrating QueueMetrics
- 34. Contents ●Introduction ●Running QueueMetrics –Monitoring & Performance –Configuration best practices –Extending call tracking –Quality Assessment (QA) ●Basics of QA monitoring ●Effective scoring ●Performance tracking ●Integrating QueueMetrics
- 35. Quality Matters! ●Track calls –Find issues before they find you. ●Track agents –Improve training –Assess strenghts and weaknesses ●Track queues –Are we doing what we are expected to be doing? –Can we show it? If you are not using QA now, you are losing out. QA – Introduction
- 36. Each call is graded on a set of metrics you define –Each metrics has its own Engagement Code –They map a numeric score (0-100) –Same metric be used in multiple forms Metrics are grouped into Forms –Up to 10 sections with up to 130 questions –Metrics are grouped into four grade bands (Issue, Req.Impr, Meets exp., Exceeds exp.) –For each form, a score is computed –Forms are immutable (more or less) –Each call can be graded on multiple forms QA – General ideas
- 37. ●Form input is flexible –Questions can be hidden or shown –Questions can be weighted in order to form a score –Shortcut questions ●Immediate agent feedback –Receive a task when a call is scored –Agents are immediately engaged –Agents must acknowledge or dispute it QA – General ideas (2)
- 38. QA is a process and requires some planning. ●Define what you expect –Don't make things too complex at first –Tell your agents! ●Plan a call review process –Don't oversample issues! ●Define initial QA review targets ●Plan in advance for corrective action to be taken QA – Getting started
- 39. A simple form to review issues ●Track whether the problem is solved ●Track the sex of the caller ●If problem not solved… –Track conversation metrics ●Different data input ●Non-scoring questions ●Show-hide questions QA – A real-life sample
- 40. Start by defining questions... ●Engagement code ●Description ●Weight ●Shortcuts ●Value types –Number –Yes/No –Combo (drop-down) QA – Creating items
- 41. Then create forms... ●Security keys ●Allowed queues ●Open / Locked ●Sections ●Thresholds QA – Creating forms
- 42. Define which items are a part of the form. QA – Creating forms (2) ●Group items into sections ●Move up and down ●Score contributions ●Visibility Once a form has items in it, it cannot be changed.
- 43. ●Form selector ●Call information ●Recordings ●Timer ●Items per type ●Overview per section ●Score ●Questions & notes on questions Anatomy of a form What does a form look like?
- 44. Notes ●General notes ●Notes per question Audio player ●Speed controls ●Playback controls ●Markers Anatomy of a form (2) What does a form look like? (continued)
- 45. Run Report → Queue details → Call Detail → QA Form appears → Listen to the call → Fill it. QA Input Entering QA is really simple.
- 46. Results: ●Per agent ●Per queue ●Per group ●Per analyst Showing: ●Averages ●Items by threshold QA Results To see results: Run QA reports → Criteria
- 47. Divided by section: ●By agent: –Calls –Average per question ●By queue: –Calls –Average per question QA Results (2) More results...
- 48. See all the graded calls that contribuited to a result. Click on a call to see its QA form (questions, notes, associated audio files). QA Results (3) How did a result came to be?
- 49. ●Overall summary ●Scoring questions ●N. forms scored –Averages –% per threshold –% shortcuts ●Non scoring –Averages –Histograms QA Results - Summary Overview of results
- 50. Contents ●Introduction ●Running QueueMetrics –Monitoring & Performance –Configuration best practices –Extending call tracking –Quality Assessment (QA) ●Basics of QA monitoring ●Effective scoring ●Performance tracking ●Integrating QueueMetrics
- 51. Deciding which calls to grade Just clicking on random calls is not especially effective. ●Poor ergonomics ●Selection biases: –Issues / „interesting“ calls –Short / long calls –First / last calls Solution: ●Create a policy –By call outcome –By agent group ●Use the Grader's page –Weighted random sampling QA – Grading calls
- 52. Find efficent sets of calls to be graded ●Random sampling in call space ●Prefers calls matching multiple criteria ●Multiple Quality Analysts can work on the same issue at once ●Shows target statistics ●First line of defence against quality issues –Ask for coaching –Ask for CBTs to be taken QA – Effective grading
- 53. Contents ●Introduction ●Running QueueMetrics –Monitoring & Performance –Configuration best practices –Extending call tracking –Quality Assessment (QA) ●Basics of QA monitoring ●Effective scoring ●Performance tracking ●Integrating QueueMetrics
- 54. Long-term agent performance management ●Agents need to be managed in order to work effectively –Performance targets –Quality policies –Agents need training ●Agent lifecycle –Training –Validation period –Review –Production –Periodic Review QA – Agent Performance
- 55. Comprehensive Performance Rules ●Performance –Sales, SPH –Contacts ●Call duration –Averages –Anomalies ●Quality results Red and yellow alarms with relative importance scores. QA – Agent performance
- 56. The Performance tracker lets you: ●Apply Performance rules and find anomalies ●Send training and coaching ●Move agents between groups ●Be reminded for further review QA – Performance tracker
- 57. Contents ●Introduction ●Running QueueMetrics ●Integrating QueueMetrics –Automated QA input –Queue recalls with WombatDialer –Handling VIP callers ●Open discussion
- 58. Contents ●Introduction ●Running QueueMetrics ●Integrating QueueMetrics –Automated QA input –Queue recalls with WombatDialer –Handling VIP callers ●Open discussion
- 59. QA is an effective process, but... –Reviewing calls manually is very expensive –Most calls are not very informative –Biased sampling –What if we could automate it? ●How can this work? –Agent tranfers call to IVR at the end of the interaction – 70-80% of callers will accept –IVR gathers simple information („Are you satisfied?“ - „Problem solved?“) –Asterisk pushes this information to QM as a QA Form ●Targets: –You have a way to review issues immediately –You can monitor quality fairly and continuously Surveys: Automating QA
- 60. ●Create a QA form called „SATISF“ –One section –One single yes/no question - „Are you satisfied with the interaction?“ (item SAT) ●Create a remote user in QM –User „qasubmit“ password „passw0rd“ –Class ROBOTS –Custom security key „QATRACK“ Surveys – QM set-up
- 61. Asterisk issue: IVR happen on a different channel: ●When calling the queue, store the original UniqueId of the call and the Queue used on inheritable channel variables ●Allow unattended transfers on the queue (# + xxx) ●Download script pushQA.sh from the Open QueueMetrics Add- ons on GitHub ●Make it executable and edits its configuration ●Create a simple IVR script to transfer to Surveys – Asterisk set-up
- 62. exten => _999,1,Answer exten => _999,n,Background(ivr_satisfaction) exten => _999,n,Wait(10) exten => _999,n,Hangup() ; Not happy exten => 0,1,Playback(beep) exten => 0,n,System(/var/lib/asterisk/agi-bin/pushQA.sh ${UID} ${QUE} 0) exten => 0,n,Playback(thank-you) exten => 0,n,Hangup ; Happy exten => 1,1,Playback(beep) exten => 1,n,System(/var/lib/asterisk/agi-bin/pushQA.sh ${UID} ${QUE} 100) exten => 1,n,Playback(thank-you) exten => 1,n,Hangup Surveys – Asterisk set-up exten => _900,1,Wait(1) exten => _900,n,Answer exten => _900,n,Playback(queue-welcome) exten => _900,n,Set(CHANNEL(musicclass)=mymu exten => _900,n,Set(__QUE=myqueue) exten => _900,n,Set(__UID=${UNIQUEID}) exten => _900,n,Queue(${QUE},t,,,300) exten => _900,n,Hangup Asterisk dialplan Push QA to QueueMetrics Store Queue & UID
- 63. Contents ●Introduction ●Running QueueMetrics ●Integrating QueueMetrics –Automated QA input –Queue recalls with WombatDialer –Handling VIP callers ●Open discussion
- 64. Your services have a known peak time. ●You need to staff agents based on the load at peak time ●Your wait time SLA is stressed at peak time ●Harder interaction with callers who have waited too long Solution: offer people the choice to be recalled ●Better SLAs – less staff required ●Happier customers ●Can be extended to web-based „call me back“ Automated recalls
- 65. Caller is offered the option to press '1' to be recalled ●If they do, a script starts to get their phone number ●When done, the number is piped to WombatDialer WombatDialer monitors the recall queue for free agents ●When agents are available on it, recalls starts ●Wombat knows how to handle busy, no answers, etc. When the call is started... ●A script plays a welcome message on connection ●The call is queued on the Recalls queue Auto recalls: the logic
- 66. QueueMetrics ●Situational awareness ●Agent management Asterisk ●Script to track caller's numbers ●Script to greet people being recalled WombatDialer ●Monitors presence on Asterisk queues ●Implements recalls ●Handles busy, no answers, invalid numbers, etc. Auto recalls: components
- 67. QueueMetrics configuration ●Create two queues – one for inbound and one for recalls ●Monitor them through the Real-Time page ●Supervisor moves agents to and from the Recalls queue based on the state of the Inbound queue ●Everything else happens automatically Auto recalls: QM
- 68. WombatDialer ●Runs multiple independent campaigns in the background ●Receives numbers to be called and extra information for handling them over HTTP ●Monitors agent continuously –Are they available? Paused? Busy? ●Handles recalls with sensible rules –Retry in 10 minutes if you get a busy –Retry in 30 minutes if the number is free –Call only during allowed times ●When successful, connects to the welcome script Auto recalls: WombatDialer
- 69. ●Solution can be implemented in very little time ●Does not require extensive changes to the PBX config ●Use the same tools you use for inbound ●Very extensible solution –Reverse IVRs –Voice synthesis ●Excellent ROI ● Recalls – all together
- 70. Contents ●Introduction ●Running QueueMetrics ●Integrating QueueMetrics –Automated QA input –Queue recalls with WombatDialer –Handling VIP callers ●Open discussion
- 71. How can we handle VIP callers? ●Special low-latency queues ●Different products ●Automatic routing of open cases ●How can we identify callers? Usually done with dedicated DIDs, but... ●Hard to scale to hundreds or thousands of cases ●We can definitely do better! VIP callers
- 72. VIP callers in QM QueueMetrics offers „Known numbers“, an easy-to -query database of known numbers. ●Associates an action and an identity to the caller ●Action can be time-limited ●Optional „agent affinity“ to first try the agent who is currently handling the case ●Database can be fed through JSON API
- 73. VIP callers in QM (2) For security reasons, you need a special user to acces QueueMetrics through the APIs. ●Create a remote user in QM –User „pbxapi“ password „api123“ –Class ROBOTS –Custom security key „PBXAPI“
- 74. VIP callers in Asterisk You query QueueMetrics before routing a call: same => n,Set(CURLOPT(hashcompat)=yes) same => n,Set(NUM=${CALLERID(number)}) same => n,Set(URL=http://my.qm/queuemetrics/numberLookup.do ?mode=hash&user=pbxapi:api123)) same => n,Set(HASH(resp)=${CURL(${URL}&number=${NUM})}) same => n,Set(CALLERID(name)={HASH(res,name}) same => n,GotoIf($["${HASH(resp,action)}" = "VIP"]?supervip) same => n,GotoIf($["${HASH(resp,action)}" = "BLACKLIST"]?blklist) same => .... So with one single call you get: ●Name of the caller ●VIP or Blacklist ●Agent affinity And you can route accordingly!
- 75. VIP callers in Asterisk You query QueueMetrics before routing a call: same => n,Set(CURLOPT(hashcompat)=yes) same => n,Set(NUM=${CALLERID(number)}) same => n,Set(URL=http://my.qm/queuemetrics/numberLookup.do ?mode=hash&user=pbxapi:api123)) same => n,Set(HASH(resp)=${CURL(${URL}&number=${NUM})}) same => n,Set(CALLERID(name)={HASH(res,name}) same => n,GotoIf($["${HASH(resp,action)}" = "VIP"]?supervip) same => n,GotoIf($["${HASH(resp,action)}" = "BLACKLIST"]?blklist) same => .... So with one single call you get: ●Name of the caller ●VIP or Blacklist ●Agent affinity And you can route accordingly!
- 76. Contents ●Introduction ●Running QueueMetrics ●Integrating QueueMetrics –Automated QA input –Queue recalls with WombatDialer –Handling VIP callers ●Open discussion
- 77. Thank you for attending! QueueMetrics www.queuemetrics.com Loway www.loway.ch A real programmer puts two glasses on his bedside table before going to sleep. A full one, in case he gets thirsty, and an empty one, in case he doesn’t.