Sigdial poster mpowers_final
0 likes176 views
MPOWERS is a platform that aims to align spoken dialogue system evaluation approaches through established norms and terminology. It implements user-friendly design conventions to ensure all evaluators share a common corpus of data and evaluation parameters. The platform provides customizable dashboards with commonly defined key performance indicators to give evaluators different analytical views of evaluation data in a goal-driven and data-driven process.
Sigdial poster mpowers_final
- 1. MPOWERS a Multi Point of vieW Evaluation Refinement Studio for Spoken Dialogue Systems Ambitions ‐ Alignment of evalua�on approaches (not standardiza�on) ‐ Enforcement of norms to describe the evalua�on protocols, common thinking models and terminology for evaluators to make their procedures explicit Er Ho go s n Pro ting l Ex om vid ica r pe ics Marketing er chn ppe Te lo rt Experts ve Custom- De made dashboards commonly d efined KPIs (IT U-T Rec. Context sup.24) ‐ No one size fits‐all‐evalua�on to cover all needs ‐ Nomadism of evalua�on approaches ‐ Need for coopera�on between stakeholders A unique Features corpus of data ‐ Implements design conven�ons in a user‐friendly interface for all ‐ Ensures all evaluator‐users manipulate a unique shared corpus of data ‐ A shared set of parameters for evalua�ons design and retrieval ‐ Supports the convergence of the evalua�on prac�ces ‐ Considers several analy�cal points of view Let's Go! ‐ Supports a both data‐driven and goal‐driven process system corpus ‐ Business‐intelligence solu�on based (CMU) Evaluation "Process through which one defines, obtains . and delivers useful pieces of informa�on to se�le between the alterna�ve possible decisions." Daniel Stufflebeam, 1980. Visual dashboards display pre-processed data according to Screenshots pre-defined evaluation profiles. Ex: Dashboard dedicated to a high-level view on usability performances A summary of high-level KPIs provides a general view on the evaluated system with ”red-light indicators”. Links to more detailed charts or analysis tools are displayed next to each of them. Marianne.laurent@orange-ftgroup.com, Philippe.Bretier@orange-ftgroup.com We thank Telecom Bretagne, Q. Jin, X. Chen, S.Zarrad, F. Agez Orange Labs, 2 avenue Pierre Marzin, 22307 Lannion, France and A. Bolze for their contribution to the platform deployment.