Using Digital Technology to Assess Quality in the Arts
- 1. Culture Metrics – developing and testing a digital platform for assessing the quality of the arts 3rd November 2015 Abi Gilmore, University of Manchester John Knell, Culture Counts #policyweek
- 2. 11 Welcome and Introduction to Culture Metrics Project. Abigail Gilmore, The University of Manchester & John Knell, Culture Counts 11.20 Using the Culture Metrics system at the Matthew Darbyshire exhibition 12.00 Lunch 12.30 Use of Culture Metrics data in organisational practice. John Knell, Culture Counts & Ronan Brindley, Manchester Art Gallery 12.45 Future research directions: Social Media and Culture Metrics, Kostas Arvanitis & Chiara Zuanni, The University of Manchester 13.00 Roundtable and Discussion. Chaired by Abigail Gilmore, The University of Manchester Alison Clark, Director Combined Arts and North, Arts Council England Nick Merriman, Director of The Manchester Museum Hasan Bakhshi, Director, Creative Economy in Policy & Research, NESTA Cimeon Ellerton, Head of Programmes, Audience Agency 13.45 Final remarks 14.00 End
- 3. About the project Ambitions: • To create a system that allows for the cost effective generation of large-scale data sets on what the cultural sector believes are the key dimensions of ‘quality’ • To explore how, and in what ways, such data will be relevant and useful to cultural organisations in their creative and commercial decision making
- 4. NESTA/AHRC/ACE Digital R&D Fund • 12-month project to: – refine the quality metrics – test the method across a wider range of artforms and settings (60 test events) – develop resources to help organisations run the system themselves and create a truly automated flexible data platform (we have to be in a position to offer a free trial to funded arts orgs at grant end point) – carry out academic research to test the value of the approach
- 5. Partners • Academic Partner – Abi Gilmore, Kostas Arvanitis, Franzi Florack, University of Manchester • Tech Partner – Culture Counts • Cultural Partners – 20 of them, including ROH, RSC, Halle, Contact, Royal Exchange Theatre, Home, Whitworth Art Gallery, Manchester Museum.
- 6. 2 main categories of metrics Quality • Quality of product • Quality of experience • Quality & depth of engagement • Quality of creative process Organisational Health • Financial metrics • Quality of cultural leadership • Quality of relationships & partnerships
- 7. Quality dimensions (self, peer, public) Dimension Metric Statement Rigour ‘It was well thought through and put together’ Distinctiveness ‘It was different from things I’ve experienced before’ Captivation ‘It was absorbing and held my attention’ Relevance ‘It has something to say about the world in which we live’ Meaning ‘It meant something to me personally’ Challenge ‘It was thought provoking’ Enthusiasm ‘I would come to something like this again’ Presentation ‘It was well produced and presented’ Local impact ‘It is important that it’s happening here’
- 8. Quality dimensions (self & peer only) Dimension Metric Statement Concept ‘It was an interesting idea / programme’ Originality ‘It was ground-breaking’ Risk ‘The artists / curators really challenged themselves with this work’ Excellence (global) ‘It is amongst the best of its type in the world’ Excellence (national) ‘It is amongst the best of its type in the UK’
- 9. How do we measure? • Triangulation of evaluation by self, peers and public • Online data capture through the Culture Counts system via a web app or via post event surveys (urls posted via email) • 9 dimensions evaluated by all 3 groups • 5 extra dimensions evaluated by self and peers
- 10. Simple interface
- 11. What types of insights are generated? • The triangulation element is vital to the insight and reflection process – allowing cultural organisations to judge the extent to which they are achieving their creative objectives • Being able to anticipate how audience members and peers will respond to a work is a reflection of the quality / maturity of creative and cultural leadership in an organisation
- 12. What types of insights are generated? • Integrating quality metrics with audience size and profile and financial data allows for a sophisticated assessment of – How successfully organisations balance creative, commercial and audience objectives – How different kinds of cultural experience create different kinds of value • Developing prompts to help organisations understand their evaluation results and use them to reflect on their creative practice
- 13. Research and data strategy Bring together test event-generated data and analytics with qualitative, collaborative research with technology and cultural partners to answer these questions: • What is the potential value of Culture Counts to the different stakeholder groups (arts organisations, policy makers, critics, investors, and ‘the public’)? • How does Culture Counts support and impact upon (DDD) decision-making within these stakeholder groups (for example, in relation to programming, funding and audience motivation to attend further experiences)?
- 14. 1. What does co-production bring to the appraisal of quality in arts and cultural experiences (and how)? 2. Do bigger data improve D-D-D (and how do we understand the processes which mediate and generate big data)? Research methods • Thematic literature and evidence reviews • Qualitative research - documented workshops, interviews and observation and reflective practice with research stakeholders • Critical friends group Further critical focus on main project innovations
- 15. Research findings: co-production Literature review findings These endorse Culture Counts as a methodology for performance measurement in terms of “artistic vibrancy” and quality – ownership and leadership by sector, brings together range of internal and external stakeholders. “It’s difficult to do that with everyone so the metrics system I suppose gives you a bit of safety ground if there are peers that are coming who you don’t know so well or have a relationship with. At least it’s managed and structured, it’s moderated by lots of other things, but it’s only really going to work if everybody does it.” Qualitative research with partners and peers • Cultural partners appreciated the chance to reflect collectively on their evaluation frameworks, and to add value to the data they already collect • They are ready and willing to integrate with other existing data and value frameworks (e.g. Audience Finder, their own box office data) However • comparison across organisations (benchmarking) rarely happens organically, and when it does it tends to be informal - there is interest in comparing findings and experiences across art form but the real value of the data is for internal evaluation • There are challenges to encouraging peer assessment, but the persuasiveness of digital platform and the standardised ‘metrics’ help to provoke interesting conversations
- 16. Research findings: big data Qualitative research: There is interest in integrating with/using bigger data – e.g. from social media: • Social media outputs can offer chance for ongoing conversations • They are considered the ‘real voice’ of audiences • But at the same time acknowledge as different to survey responses “it’s sometimes about getting the retweet or favourite from the organisation” • General reticence and lack of awareness about potential of big data “what we need is ‘just enough data” • There is general caution and wariness about the amount of data collection required by funders without clear rationale - “we don’t know what they do with it – probably nothing” Resource and capacity for data management & analysis is low (and ‘big data’ is virtually non-existent) in most arts organisations - Culture Counts is perceived to be an effective way to increase this capacity
- 17. Now: your chance to give it a go! • Alison Whittaker, Culture Counts