Usage and Usability Assessment: Library Practices and Concerns by Denise Troll Covey January 2002

Copyright 2002 by the Council on Library and Information Resources. No part of this publication may be reproduced or transcribed in any form without permission of the publisher. Requests for reproduction should be submitted to the Director of Communications at the Council on Library and Information Resources.

Contents

About the Author

Acknowledgments

Preface

Introduction

1.1. Report Structure
1.2. Summary of Challenges in Assessment

2. User Studies

2.1. Surveys (Questionnaires)

2.1.1. What Is a Survey Questionnaire?
2.1.2. Why Do Libraries Conduct Surveys?
2.1.3. How Do Libraries Conduct Surveys?
2.1.4. Who Uses Survey Results? How Are They Used?
2.1.5. What Are the Issues, Problems, and Challenges With Surveys?

2.1.5.1. The Costs and Benefits of Different Types of Surveys
2.1.5.2. The Frequency of Surveys
2.1.5.3. Composing Survey Questions
2.1.5.4. Lack of Analysis or Application
2.1.5.5. Lack of Resources or Comprehensive Plans

2.2. Focus Groups

2.2.1. What Is a Focus Group?
2.2.2. Why Do Libraries Conduct Focus Groups?
2.2.3. How Do Libraries Conduct Focus Groups?
2.2.4. Who Uses Focus Group Results? How Are They Used?
2.2.5. What Are the Issues, Problems, and Challenges With Focus Groups?

2.2.5.1. Unskilled Moderators and Observers
2.2.5.2. Interpreting and Using the Data

2.3. User Protocols

2.3.1. What Is a User Protocol?
2.3.2. Why Do Libraries Conduct User Protocols?
2.3.3. How Do Libraries Conduct User Protocols?
2.3.4. Who Uses Protocol Results? How Are They Used?
2.3.5. What Are the Issues, Problems, and Challenges ith User Protocols?

2.3.5.1. Librarian Assumptions and Preferences
2.3.5.2. Lack of Resources and Commitment
2.3.5.3. Interpreting and Using the Data
2.3.5.4. Recruiting Participants Who Can Think Aloud

2.4. Other Effective Research Methods

2.4.1. Discount Usability Research Methods

2.4.1.1. Heuristic Evaluations
2.4.1.2. Paper Prototypes and Scenarios

2.4.2. Card-Sorting Tests

3. Usage Studies of Electronic Resources

4. General Issues and Challenges

4.1. Issues in Planning a Research Project
4.2. Issues in Implementing a Research Project

4.2.1. Issues in Sampling and Recruiting Research Subjects
4.2.2. Issues in Getting Approval and Preserving Anonymity

5. Conclusions and Future Directions

APPENDIX A: References and Selected Bibliography

APPENDIX B: Participating Institutions

APPENDIX C: Survey Questions

APPENDIX D: Traditional Input, Output, and Outcome Measures

Survey Instruments

Denise Troll Covey is associate university librarian of arts, archives, and technology at Carnegie Mellon University. In 2000-2001 she was also a distinguished fellow in the Digital Library Federation, leading the initiative on usage, usability, and user support. Her professional work focuses on the research and development of digital library collections, services, and software; and assessment practices, copyright permissions, and change management as they relate to digital libraries. Covey has academic degrees in theology, philosophy, and rhetoric. Her graduate work emphasized the history of information storage and retrieval.

The author and the Digital Library Federation sincerely thank the 71 individuals who participated in the DLF telephone survey. Their time, experiences, concerns, and questions about library use and usability made this report possible. If the report facilitates discussion and research and encourages the development of benchmarks or best practices, it is because so many talented people shared their rewards and frustrations in trying to understand what is happening in their libraries and how to serve their users better.

Making library services available online is not only expensive; it is also very risky. The library's roles there are not at all clear. Neither are its relationships with users or with other information services. There is little information about how library users behave in a network environment, how they react to online library services, and how they combine those services with others such as search engines like Google, bookstores like Amazon, Internet gateways like Voice of the Shuttle, and instructional technologies like WebCT or Blackboard. Digital libraries are still relatively immature—most are still at a stage where limited experimentation is more important than well-informed strategic planning. While libraries have excelled at assessing the development and use of their traditional collections and services, comparable assessments of online collections and services are more complicated and less well understood.

Against this backdrop, the Digital Library Federation (DLF) has committed to driving forward a research process that will provide the information that libraries need to inform their development in a networked era. The goals of this process are:

• to develop a better understanding of methods effective in assessing use and usability of online scholarly information resources and information services; and
• to create a baseline understanding of users' needs to support strategic planning in an increasingly competitive environment for academic libraries and their parent institutions.

This report is an initial step in achieving the first of these goals. It offers a survey of the methods that are being deployed at leading digital libraries to assess the use and usability of their online collections and services. Focusing on 24 DLF member libraries, the study's author, Distinguished DLF Fellow Denise Troll Covey, conducted numerous interviews with library professionals who are engaged in assessment. In these interviews, Covey sought to document the following:

• why digital libraries assessed the use and usability of their online collections and services
• what aspects of those collections and services they were most interested in assessing
• what methods the libraries used to conduct their assessments
• which methods worked well and which worked poorly in particular kinds of assessments
• how assessment data were used by the library, and to what end
• what challenges libraries faced in conducting effective assessments

The result is a report on the application, strengths, and weaknesses of assessment techniques that include surveys, focus groups, user protocols, and transaction log analysis. Covey's work is also an essential methodological guidebook. For each method that she covers, she is careful to supply a definition, explain why and how libraries use the method, what they do with the results, and what problems they encounter. The report includes an extensive bibliography on more detailed methodological information, and descriptions of assessment instruments that have proved particularly effective. Examples are available on the Web for all to see, and potentially to modify and use. The work concludes with a review of the challenges that libraries face as they seek to gather and use reliable information about how their online presence is felt. These concluding remarks will be of general interest and are recommended to senior library managers as well as to those more directly involved with assessment activities.

Given its practical orientation, Usage and Usability is an ideal launching pad for CLIR's new series, Tools for Practitioners. The series emphasizes the immediate, the practical, and the methodological. As it develops, it will include work that, like Covey's, appeals to and provides guidance for particular professional audiences.

Daniel Greenstein
Director, Digital Library Federation

Next

Return to CLIR Home Page >>