Evaluating your reference service
- 1. Evaluating your reference service A public library reference desk
- 2. Why evaluate ? To establish and demonstrate the value of the service, especially to those who fund it “ Reference librarians, in failing to provide the means for accurate judgement on their place and contribution in library service, run the serious risk of having their work undervalued or ignored” (Rothstein 467) To establish a basis for making improvements to the service “ At its best, evaluation provides input for planning and decision-making, provides high-quality qualitative and quantitative data describing library activities, and serves as a basis to constantly assess and improve library services” (McClure 167)
- 3. Establish the focus of your evaluation Service inputs: reference materials, physical environment, staffing levels and types of staff Service outputs: reference transactions - advice, instruction and assistance given, questions answered, staff-user interaction Service outcomes: benefit to users and the wider community; user satisfaction and perceptions of the service (Whitlach, Evaluating … a practical guide 5; Allen 247)
- 4. Identify goals and objectives Goals give the process direction and help establish the most appropriate methodologies to use Example goal: “To provide reference assistance that satisfies the needs of users seeking information” (Whitlach, Evaluating … a practical guide 11) Accompany each goal with specific objectives, thereby setting standards against which the performance of your service can be judged Example objective: “Provide information that 85 percent of users judge relevant to their query” (Whitlach, Evaluating … a practical guide 11)
- 5. Assessing service inputs Reference resources : compare materials against standard bibliographies and reference lists (allow for local needs); assess for currency, accessibility etc; examine usage statistics Physical surroundings : Is the reference desk well signposted and are staff easily identifiable? Consult users in surveys and interviews Staff: examine staffing patterns and try alternative arrangements to judge the effect on queues at the desk, particularly in peak periods (Cullen 13; Allen 249-251; Regazzi and Hersberger cited by Whitlach, Evaluating … a practical guide 115 )
- 6. Assessing service outputs Records of reference transactions made by staff - ideally using standard forms Peer or self-evaluation by staff using checklists to assess performance and reference skills. Failure analysis requires them to record the questions that they could not answer. Interviews with staff can investigate staff perceptions Unobtrusive observation: test questions delivered by “mystery shoppers”. Traditionally used to assess reference answer accuracy, but questioners can also assess other aspects such as staff interpersonal skills. (Allen 252-3 ; Schwartz and Eakin 7; Cullen 14; Whitlach, Evaluating … a practical guide 14-16)
- 7. Assessing service outcomes: user surveys and questionnaires Provide an insight into peoples’ perceptions and concerns, but don’t necessarily give an indication of how they actually behave Can target both users and the wider community – are there groups not being served or not aware of what the service offers? In-house, post, telephone or e:mail/online – each has pros and cons Use clear and unambiguous questions. Could adapt a published survey to local circumstances. Ideally conduct a test-run with a small sample to help identify any problems with clarity or length etc Short in-house questionnaires can focus on user experience of particular reference transactions (Cassell and Hiremath 318; Novotny 20-42; Allen 258; Cullen 14)
- 8. Assessing service outcomes: interviews and focus groups Can provide more in-depth information to help interpret or supplement that gained through surveys and observation Can be structured - following a predetermined set of questions, or unstructured - discussion is more user led Greater insight into common concerns and issues can emerge from the interaction and debate between participants in focus groups Interviewers and focus group facilitators must receive careful training (Kuruppu 375-6; Whitlach, Evaluating … a practical guide 70-80)
- 9. Analysing the data Statistical analysis of quantitative survey data Identify broad themes arising from qualitative data gathered through observation, interviews and focus groups, then refine into categories and subcategories. Review coding to ensure consistency Computer software can assist analysis, e.g. Statistical Package for the Social Sciences (SPSS), Ethnograph and Nudist (Whitlach, Evaluating … a practical guide 100-103)
- 10. Presenting the data Tailor presentation style to suit target audience Produce pithy and relevant reports; include an executive summary and clear recommendations Make effective use of graphs and charts to summarize findings; confine detail to appendices Avoid violating the privacy of staff and users: remove details that might identify individual participants Keep library users informed (McClure 170; McNicol 173; Whitlach “Evaluating … in the electronic age” 213)
- 11. Staff resources Staff time will be required to record transactions, create and administer surveys, conduct interviews and focus groups Save time by using standard reference recording forms, adapting published surveys and using computer software to analyse data Allow time to analyse the data and disseminate findings Some evaluative processes, particularly interviews and focus groups, may require staff training, Evaluation findings may reveal further training requirements