7/10/2010

Acting on Ongoing Assessments

If you put in the time, effort, and money to conduct an assessment of your reference department, it only makes sense to act on it.

And assessments should be ongoing. This ensures that librarians and managers always have a clear understanding of who they are as a department, where they're headed, and who they are serving.

Acting on assessments and making sure that the process is ongoing guarantees a department's survival within its institution, as it can prove its worth (ROI). More importantly, the assessment process helps keep the department relevant to its patrons' needs.

Case Studies

As an LIS graduate student, you have most likely encountered published case studies when doing research. A case study is a multi-tooled approach to exploring a single question.


As one example, this article
Wan, G., Clark, D., Fullerton, J., Macmillan, G., Reddy, D., Stephens, J., & Xiao, D. (2009). Key issues surrounding virtual chat reference model: A case study. Reference Services Review, 37(1), 73-82.
is a case study where specific aspects of virtual chat reference at Texas Christian University were analyzed.

Case studies yield the deepest understanding about a particular issue and their results are the most reliable. However, they are also the most complex of the available assessment methods. They require skilled staff (or doctoral students) to conduct the study and analyze the results. And because the focus is on a single issue, the results are rarely transferable and are difficult to use when constructing an argument that defends and markets your reference department as a whole.

Focus Groups

Focus groups can work great in certain situations. They can be used to explore and define issues during the early stages of a strategic plan, to clarify conflicting survey data, to supplement surveys when they are taking too long, and to spur and initiate interest about new or undersold services.


For a clear overview of how focus group research can be used by libraries, read:
Shoaf, E. (2003, March). Using a professional moderator in library focus group research. College & Research Library, 64(2), 124-132.
You'll notice in the article's title, however, the phrase "professional moderator." That is one of the drawbacks of focus groups; they can be complex undertakings that require carefully selecting participants, carefully constructing sets of questions, and either extensive training of staff or hiring outside consultants. The results can be very useful, the the investment of time and money can be extensive. For this reason, they are usually only undertaken by larger libraries.

Surveys

Surveys are a tricky species. They are one of the more trusted and ubiquitous assessment tools out there. However, writing good survey questions can be very difficult and time consuming. And one of the most frustrating things about surveys is that the gathered responses often don't reflect the true thoughts and behaviors of the respondents. People don't intentionally lie, but they will often "say one thing and do another." 


That doesn't mean surveys aren't useful. They are one of the more powerful and persuasive methods because they can gather a large number of responses from a diverse user population.


When it comes to surveys, the article
Eng, S. and Gardner, S. (2005, February). Conducting surveys on a shoestring budget. American Libraries 36(2), 38-39.
gives some good advice on how cash-strapped libraries can conduct reliable surveys. 
And for an example of how an online survey can be used for assessment, read:
Chang, H.R., & Holland, M.P. (2005). User satisfaction survey of Ask-A-Question service at the Internet Public Library. Internet Reference Services Quarterly, 10(2), 61-7.
To see how one virtual reference service's online survey looks, visit L-Net's post-chat survey. And if you're interested in building an online survey, Surveymonkey allows you to do so for free.

Observation

Observation, as an assessment tool, actually covers three distinct methods.

First, direct observation involves monitoring staff and/or patrons openly. It might be in a controlled or uncontrolled environment. It can be useful for gathering a lot of details about a certain delivery method or reference interaction, but we also know that people act differently when they know they're being watched.

The alternative, then, is hidden observation, where the librarian is watched and evaluated secretly. This might be from afar or by a "secret shopper" patron who approaches the librarian with a fake information need. The results from this are more "honest," but of course the method itself is not. If it can be done where the librarian's identity is guaranteed to remain anonymous, the ethical questions are not as stark, but it should probably not be used in smaller library systems where staff could be vulnerable. One of the best situations in which to use hidden observation is the electronic and online environment, where people can remain anonymous and the process is entirely unobtrusive:
Stacy-Bates, K. (2003). E-mail reference responses from academic ARL libraries: an unobtrusive study. Reference & User Services Quarterly, 43(1), 59-70.
Finally, the last observation method is self-imposed observation, which means librarians observing themselves individually. This is a very standard, easy to carry out method. Typically, some sort of log of number and type of questions it kept at the reference station (desk, computer terminal, telephone); logs are used either daily or on some sort of monthly or quarterly basis. The data and method is not very sophisticated, but it can be useful in tracking meaningful trends in usage. 


To see how it can be used on a large scale, check out the National Center for Educational Statistics' Library Comparison page, which allows you to search and compare academic libraries using certain parameters, including number of reference transactions.


You might also find the The READ Scale (Reference Effort Assessment Data) interesting. The READ Scale is a  metric for evaluating and weighting certain types of reference questions on a scale of 1-6 (1 being the most basic and simple and 6 being the most difficult and complex). It then provides a more sophisticated but efficient way to collect and analyze reference transaction data.

The Suggestion Box

Everybody knows the traditional suggestion box; we find them everywhere, from fast food restaurants to courthouses. It's great for gathering random, diverse, visceral, honest feedback, and it shows that the institution cares about what its patrons think. However, the kind of information from suggestion boxes usually only lends itself to making minor, incremental changes at best.  That doesn't mean they aren't useful; if you're interested in a suggestion box (actually, a "suggestion book") that was used to effect, read:
Lubans, J. (2001). “Where are the snows of yesteryear?" Reflections on a suggestion ‘box’ that worked. Library Administration and Management, 15(4), 240-245.
Maybe the transparency and responsiveness allowed by a "suggestion book" works better than the box?

Assessment Methods

When it comes time to begin official evaluation of a reference department, there are quite a few assessment methods to choose from. But if you're serious about really understanding your patrons' needs and how effective your department is, it usually requires using multiple methods at once. Just as a 3-dimensional image has more visual depth than a 2-dimensional image, a multi-dimensional approach to assessment will yield a more in-depth understanding than an approach that uses only one assessment method. Assessment methods range from the basic (and free) to the complex (and expensive):
  • Suggestion Box
  • Observation (direct, hidden, self-imposed)
  • Surveys
  • Focus Groups
  • Case Studies
Each has its strengths and its weaknesses, and that is why they work best when combined to complement each other.


In this article
Norlin, E. (2000). Reference evaluation: a three-step approach−surveys, unobtrusive observations, and focus groups. College & Research Libraries, 61(6), 546-553.
Norlin reveals how the multi-dimensional approach to assessment works.


And for an overview, read:
Logan, F.F. (2009, July). A brief history of reference assessment: no easy solutions. The Reference Librarian, 50(3), 225-233. 
The author discusses the different types of reference assessment that have been used in libraries over the years.

7/09/2010

What to Assess

To perform an effective evaluation of a reference department, you have to choose what part of the reference department you will focus your assessment activities. Usually, emphasis will be placed on one of three segments of the department:
  • Reference Collection: 
    • The collection includes print reference materials, but subscription databases must constantly be evaluated for usage, efficiency, and relevancy.
  • Reference Staff:
    • The assessment of staff has do be done with as much objectivity and honesty as possible; the goals and standards need to be clear to all staff, and the focus should be on helping staff improve their skills; the goal is not to make librarians fear for their jobs or begin ranking themselves.
  • Reference Services
    • As patrons are constantly finding and expecting new ways of accessing information, reference librarians need to constantly evaluate the different ways in which they deliver their services--desk, roving, virtual, "embedded," telephone, etc.
Here's an article about how an academic library evaluated the delivery of its reference services:
  • Jacoby, J., & O’Brien, N.P. (2005). Assessing the impact of reference services provided to undergraduate students. College & Research Libraries, 66(4), 324-40.
The authors conducted a study evaluating the impact of reference services on undergraduate students at the University of Illinois at Urbana-Champaign

Typically, that's the ultimate goal of most assessment projects: determine the impact on patrons.

Why Should We Assess Library Reference Services?

First, the assessment of reference services help libraries substantiate the need for certain resources and services, particularly when funding is low and there is a demand for electronic resources. To forestall budget cuts (or perhaps request budget expansion), a reference department needs to be able to show its ROI (Return On Investment).

And sometimes assessment is required. The U.S. Government Performance and Results Act requires federal agencies to have quantifiable and measurable performance goals. This has an impact on other agencies that seek federal grants.


Here are a couple articles that might be of interest:
  • Holt, G., & Elliot, D. (2003, Winter). "Measuring outcomes: applying cost-benefit analysis to middle-sized and smaller public libraries." Library Trends 51(3), 424-440.
    • In this article, Holt & Elliot argue for cost-benefit analysis (ROI, basically) in smaller public libraries, where such methods have traditionally been absent. They also describe how the process works in that environment.
  • Buschman, J. (2004, August). Staying Public: the real crisis in librarianship. American Libraries, 35(7), 40-42.
    • Buschman takes a provocative stance. Although he does not question library reference assessment directly, he explores the problems with porting private sector philosophies (such as the desire to quantify through ROI) into the public sector. Buschman argues, "We have unthinkingly imported business models on a massive scale into public enterprises" (p. 40-41).
In any case, assessment can still be useful. Even if funding and ROI are left out of the equation, a reference department should always be looking for way to provide the best services possible to its patrons.