Search
Close this search box.
Search
Close this search box.

Introduction

Expenditures on journals comprise about 70 percent of the average academic library’s budget for materials. Research libraries in the United States spend more than $500 million on journals annually, and several large libraries estimate that they spend 20 percent of their budgets on electronic materials.

When scholarly journals are converted to electronic form, they are frequently offered as part of a database hosted by a publisher or an aggregator and made accessible through the Web. As users shift from using local print materials to using remote files, librarians seek to collect usage data that justify the library’s investment in electronic resources. Because the library licenses, but does not own, the content, it must depend on the provider for usage data.

Providing usage data is a new role for publishers and aggregators-one that requires not only much learning but also a financial investment. While it appears that the data would be as useful to publishers as to librarians, publishers must first develop the capability to serve their own purposes and then provide additional analyses and support to present the data so that librarians can use them.

Less than half of the publishers who offer journals in electronic form today are able to provide statistics on the usage of these journals. What is available varies widely among publishers, and librarians are often unclear about what to ask for and how they will use the data. Guidelines for compiling statistics are just emerging and have not been widely adopted.

Publishers are concerned that the data they share with librarians lack context. If, in the absence of such a context, usage data seem low, the publishers fear that librarians may use such information as a basis for canceling subscriptions. As both librarians and publishers become more familiar with the current state of usage statistics, the focus of the conversation will shift to what needs to be done to ensure consistency and to provide a valid context for understanding the data. There have been rapid developments in this area in the six months since this study began, and the author is encouraged by recent discussions with publishers who were previously reluctant to provide data to libraries and are now inquiring about what should be delivered.

This paper provides a snapshot of developments in the industry. It identifies issues that concern both publishers and librarians and suggests a context for further discussion between the providers and consumers of electronic journals.

Background

Since libraries that host electronic journal content locally face the same challenges in collecting usage statistics as do publishers, the author chose to interview librarians at OhioLINK, Los Alamos National Labs, and the Florida Center for Library Automation to determine how they provide such data to their consortia members. Publishers and providers were then interviewed to compare their approaches to collecting and presenting the data with those of the libraries. Finally, comments from both groups were solicited to identify their concerns and to establish a base for understanding and interpreting the data. Summaries of selected interviews are provided in Appendix A.

To provide a frame of reference for this study, current initiatives by other organizations on developing data collection policies were reviewed. In addition to guidelines published by the International Coalition of Library Consortia (ICOLC), which were based on the JSTOR initiative, there is substantial work being done in this country by the Association of Research Libraries (ARL), the National Information StandardsOrganization (NISO), and the National Commission on Libraries and Information Science (NCLIS), and abroad by the European Commission. The ICOLC guidelines may be found in Appendix B; information on related industry initiatives appears in Appendix C. While some of the studies focus on defining data elements and collecting statistics, others examine data in light of factors used to assess performance, such as the percentage of user population being served.

Issues Affecting Librarians and Publishers

Among the most important findings of this study is that librarians and publishers share a significant number of concerns about the development and interpretation of statistics. Both are seeking agreement on core data that are needed and are exploring an appropriate context for interpretation. Once publishers and providers discover how to produce comparable and reliable data, it will be possible to continue discussions about usage and value to the user.

Issues of Common Concern to Librarians and Publishers

All indicators of usage are steadily rising, in part because of the continued growth of electronic content available on the desktop. However, in the electronic world, there are more variables that affect the analysis of statistics and an understanding of the results than there are in the print world. For a balanced picture, librarians and publishers will want to consider how the following variables affect their data, assessments, and conclusions.

Lack of comparable data

The issue of greatest concern to publishers and librarians is the lack of comparable data. Variations in definitions and implementation procedures make it impossible to compare data from different host environments with any degree of reliability.

Unless data on multiple publishers are collected from the same platform (such as OhioLINK, HighWire, or Catchword) with common hardware and shared software, variations in how items are identified and counted will skew the results. What is counted (e.g., searches, abstracts displayed, HTML pages viewed, PDF documents downloaded) and how (whether internal use, such as demonstrations, and external use, such as spider hits or “robot contamination,” are excluded) will vary according to the software used.

Librarians currently receive reports with different data elements that are not clearly defined and that cover different time periods, making it impossible to analyze them in a consistent way. Publishers likewise find it difficult to reconcile internal data that are produced from different systems that count data in different ways.

Lack of context

With insufficient data from the print environment and insufficient experience in the rapidly changing electronic environment, it is not possible to establish a context for understanding data available on the current level of online activity. What little data librarians have on the use of print cannot serve as a basis for projections on the use of electronic journals.

Current measures are limited to data on the amount of activity, such as the number of downloads. To base comparisons on the use of large or very popular journals (such as Nature, Science, or Cell) sets an artificially high benchmark for other titles with fewer articles available for use. This raises the question of whether the measure of activity should be relative to another factor, such as the price of the journal or the number of available articles, which puts the measure in a context.

Both publishers and librarians emphasize that measures of the level of activity do not indicate the value of an article. It is dangerous to assume that a popular title that is used by many students is worth more than a research title that is used by only a few faculty members working in a specific discipline. Other factors need to be considered.

Known differences in information-seeking behavior among users in various scientific disciplines warrant additional study to identify usage patterns. As more data are examined on use and behavior, it may be possible to establish average levels of use for different subject areas or user groups.

Incomplete usage data

Constructing a complete picture of use is further complicated by the existence of journals in multiple formats that are available through multiple sources or distribution channels, e.g., directly from the publisher, onsite at a library, or through a vendor such as OCLC. This means publishers must combine usage data for journals that are mounted on a remote host, such as OCLC or OhioLINK, with data for journals kept on their own Web sites. Libraries are confronted with multiple sources of usage data, or the lack thereof, for different formats (print, electronic, microfilm) and for multiple copies of titles that are available from several sources.

Marketing

Publishers who make usage data available are aware that this information will be used to assess the value of their journals. Consequently, they want to ensure that usage is high so that the cost per-use is low compared with that of other publications. Publishers and librarians with experience in electronic databases agree that marketing to users-whether librarians or library patrons-and making them aware of the availability of the resource and its features have a noticeable impact on usage.

It can take from sixteen months to three years for users to integrate into their routines changes in how they access information. For that reason, the amount of time a database has been available influences usage rates (Townley and Murray 1999). Elsevier’s experience with The University Licensing Program (TULIP) and Pricing Electronic Access for Knowledge (PEAK) taught publishers that it is essential to promote the availability of a journal database to users and to allow time for user behavior to change.

Both librarians and publishers involved in the PEAK project acknowledged that publicity and promotion made a difference in levels of use. At Vanderbilt University, the medical school’s use of the electronic journals was disproportionately low because the medical library was reluctant to publicize the use of a system that its staff considered to be temporary (Haar 2000).

Content provided

The demand for specific electronic titles is affected by both the timeliness of content and the amount of content provided. Some publishers release articles in electronic form before publishing the print version or choose to delay the electronic version for a few issues or for a year so as not to affect current subscriptions.

A collection becomes more useful when the amount of archival content available online increases, especially if it is well indexed. When back files are included with the current subscription or basic service, the user has more articles to view, and this will affect usage.

Interface affecting usage

Barriers to access, such as requirements to register, have proved to be a major deterrent to use (Bishop 1998). Charging user fees also limits access. Only in an unrestricted economic environment can demand be measured accurately.

The presence of links that take the user directly to the full text of articles from the library’s online catalog or from a Web list of journal holdings results in higher usage levels. Access data from Elsevier and MCB indicate that a high percentage of current use reflects the behavior of researchers who browse by a familiar journal title rather than that of general users who are searching for information on a subject. Tom Peters, director of the Center for Library Initiatives at the Committee for Institutional Cooperation (CIC), believes that accessibility is one of the crucial and complex factors affecting use.

The user’s experience of the interface also will significantly affect the results. Both Academic Press and the American Institute of Physics (AIP) noted that they experienced surges in usage after they introduced new platforms that simplified navigation and access.

Economic model

As long as the journal, rather than the article, is the primary unit of sale, statistics will be collated by journal title. Academic Press pioneered a site license for consortia that includes all journals published by the Press and gives the user unlimited access to all articles. In this model, titles that are highly used will have a lower cost per-use and be perceived as a better value.

The trend toward offering a large database of journals from which the user selects articles gives rise to new economic models. The PEAK project, in which Elsevier titles were loaded at the University of Michigan, allowed users to access, for a nominal additional cost to the library, articles from journals to which the university did not subscribe.

Some librarians have begun to develop analyses based on article usage and cost per article. The hazard of pricing per-use is that value is associated with productivity of an article rather than with other measures. Pricing solely by usage may work with popular titles, but it ignores the importance of little-used titles that have an impact on research.

User privacy

The topic of privacy applies both to data collected on individual users and to data on libraries shared within consortia.

Data that publishers currently provide on journal use do not reveal specifics about any individual user, but present a summary of activity by journal title. However, publishers who offer personalized or customized services, such as e-mail alerts, must retain user-specific information in order to deliver such services, and this requires that they establish policies about how they intend to use the data.

Librarians have a tradition of protecting the privacy of users with policies regarding book circulation records. They are equally committed to protecting users’ rights in the electronic environment. Publishers are considered responsible for how they use data they collect. Protecting the user’s personal information is not just a courtesy: it is a legal obligation (Rothman 2000). Publishers that collect such data need to develop policies for how it will be used throughout their organizations. Moreover, after a policy is established, it is essential that the company monitor compliance internally.

While the U.S. Federal Trade Commission is concerned that companies adhere to the privacy policies that the companies themselves have defined, there are more stringent requirements in Europe. Publishers who sell electronic publications in Europe must have privacy policies that indicate what information is collected, how it is used, how the user can change it, with whom it is shared, and how users can opt out.

For example, Elsevier’s ScienceDirect product alerts users to the fact that they may be entering personal information when they take advantage of customized services or order documents. The company’s privacy policy states that “Your information is kept confidential, unaltered and is used only by ScienceDirect and its parent company Reed Elsevier to administer your ScienceDirect/Elsevier Science relationship.”

JSTOR is an industry leader in the area of statistics. Its privacy policy states that “No data are provided that would allow for the identification of the activity of individual users. Librarians can generate reports only for their own institution’s usage activity, although average usage activity at similar institutions is provided for comparison purposes.” The concern for privacy at some libraries extends to JSTOR’s own statistics, and the library may want its usage data shared only with its permission.

In its guidelines, ICOLC states that “Statistical reports or data that reveal confidential information about individual users must not be released or sold by information providers without permission of the consortium and its member libraries” (1998).

Library Issues

While usage statistics validate the library’s investment, they also provide insights into usage patterns that indicate the need to access a broader spectrum of titles than previously owned. This raises questions about the approach to building collections on a “just-in-case” basis compared with new models that incorporate on-demand acquisition.

Budget justification

Reference librarians lament that students act as if a resource does not exist if it is not online. Declining book circulation and rapid growth in the use of electronic resources indicate that users are shifting from print to electronic resources.

Libraries can tell which Web sites users are going to for information, but once users reach the publisher’s site, their activity can be tracked only by the publisher. This means the library is dependent on the publisher to provide it with data vital for its internal reports. High usage demonstrates a good investment to administrators who approve budget increases. For example, one library used statistics on after-hour use to show how the availability of electronic journals extended the library’s services.

Impact on selection

Recent data from OhioLINK show that more than half of the articles selected by users come from journals not currently held by the library (Sanville 2000). There is increasing evidence from both libraries and publishers that current holdings are too limited to meet user demand, a trend that points to the benefits of user-driven selection procedures. The emerging models for article selection from a database of electronic journals challenge libraries to restructure their approaches to collection development and create new models to meet their users’ needs.

Publisher Issues

Publishers who have experience with their own usage statistics are becoming less worried about cancellations because they see that librarians are still processing the data, rather than reacting to it. Many publishers are still concerned, however, that because there is no context for most usage data, it can be misunderstood.

Internal applications

As publishers come to terms with the costs of developing their capability to collect and analyze usage statistics, they find multiple applications for such information internally. For example, the systems staff uses such data to budget for new hardware. The product-development staff analyzes how users access content. Marketing is interested in how users find the site. The sales staff wants to know about the level of activity of their customers, and the editorial staff wants data on the most requested search terms.

Quantitative Measures

The establishment of accepted means for producing reliable and useful data can be viewed as a two-phase process. In phase one, publishers reach agreement with each other and with librarians about what data are required and what standards should be adopted for collection and delivery. Once comparable data are available, it will be possible to analyze and draw conclusions from the data-phase two of the process.

ICOLC guidelines were created to address a variety of files, including bibliographic databases, which are focused on simultaneous users. As a result, they reference turnaways and menu selections that do not apply to sitewide licenses for access to journal databases.

There is the potential to learn a great deal about users and their behavior; however, at this early stage, experienced librarians agree that it is best to focus on only a few measures. OhioLINK Executive Director Tom Sanville notes that of the criteria in the ICOLC guidelines, the only applicable measure of use of an electronic journal is the number of times an article is viewed, printed, e-mailed, or downloaded.

There are three steps in processing the raw data that servers produce on visits to the sites of Web-based journal collections.

  1. Data Collection: On the basis of the needs of internal and external users, each host site decides what data elements it will collect. For example, do downloads include both HTML pages viewed on the screen and PDFs downloaded? Sizable log files are reviewed to extract and summarize data. Rather than use locally developed software, systems staff often prefer commercially developed software because it usually offers more features, enhanced graphics, and customer support.
  2. Analysis: Decisions are made as to whether the analysis will be performed ahead of time on preselected fields or whether librarians can select the data elements, specify a time frame, and create their own reports.
  3. Presentation: Once the content of the report is determined, decisions must be made on the currency of the data (i.e., real-time or periodic uploads on a nightly, weekly, or monthly basis), whether the files can be exported, and whether the data are pushed to the library via e-mail or the library must retrieve them.

What Are We Measuring?

Once publishers agree upon basic data elements to be collected, analyzed, and presented in a standard way, they will be able to produce the first generation of comparable statistics. Typically, what is being used (content), who is using it (user), and how the database is being used (activity) are measured. When the content is used and how the data will be presented are other questions of interest.

What is being used?

For a full-text journal database, the ICOLC guidelines define the use of articles as viewing, downloading, printing, or e-mailing the full text. Summaries of data usage by journal title can help librarians decide what titles to add, change, or delete and can assist publishers in determining the health of the journal. Comparing statistics on the abstracts and tables of contents viewed with statistics on downloads may provide insights on how users navigate the database.

Who is using the content?

Analysis by IP address range can sometimes reveal the academic department that has requested the article and can be useful in assessing the need to train users or make them aware that a resource is available. When users are remote and are assigned a dynamic IP address, it is difficult to determine the user’s discipline. Some libraries attempt to combine usage data provided by the publisher with their own data to determine the extent to which they are serving remote users.

How is the database used?

A “hit” registers each time the server receives a request to act (e.g., to do a search, to view an abstract, or to download an article). The type of hit can indicate how the user approaches the system-for example, to browse a journal title or to search for specific information.Searches include requests to search by title, author, or subject. Browsing includes accessing the full text from the journal title, issue, and table of contents. In menu-driven systems, the menu items that the user selects need to be counted. Direct access with a citation or URL may be counted separately.

When is the content being used?

Measures of activities (hits, sessions, downloads) are summarized by the hour, day, week, month, and year. The systems staff analyzes data from server logs to determine the ability of the server to meet the load during periods of peak demand. When systems provide access to simultaneous users, the number of times users are turned away also needs to be captured to measure unmet need.

How will the data be presented?

The degree to which statistics are useful to a library depends on how the data are presented. Librarians want to be able to do the following:

  • Query the system and specify the time period covered
  • Access two years of data online to monitor growth
  • Download data as a “comma delimited” file to load into a spreadsheet
  • Graph usage across years or titles or compare usage with that of other libraries
  • Access data that are real-time or that are updated nightly
  • Establish a profile and routinely receive the results by e-mail

Once publishers have established a well-defined and consistent set of data, additional analysis will support exploration of data related to behavior and use and will attempt to address questions related to value.

Data Reliability

A standard methodology for collecting and analyzing data is necessary to ensure that both publishers and librarians have data that are comparable and reliable.

With a full-text journal database, the conversation centers on three measures: hits (equated to searches), sessions (equated to users), and documents used (equated to downloads). However, measuring hits or sessions can yield misleading information. The number of hits will vary, depending on network access and telecommunication factors. Likewise, the number of sessions will vary because of time-outs and other network protocols.

Conversations with the staff who implement the statistics function revealed a common process of learning related to the design and development of internal processes to produce valid data. Any given method of implementation can produce varied results, based on the software selected and the diverse nature of local systems architecture.

Caching: Caching allows frequently accessed Web pages to be stored on a server to improve performance. When users access cached materials, these actions are not counted as a hit on the host database. Consequently, for popular materials, statistics supplied by the host are likely to underestimate usage.

Log files: Web server log files are a good means of helping administrators gauge the demands on a server. Such logs measure requests for specific documents on a server, but they cannot show exact usage because caching is often employed and because users are assigned variable IP addresses (Bauer 2000).

Although log files are not designed to describe how people use a site, they do allow analysis of the source of links into a Web site and therefore can be used to determine which sites are generating traffic. Such information can be useful to publishers. For example, Science News Online (SNO) learned that one of its articles, which had been cited on another Web site that was linked to the SNO Web site, had brought in a high volume of visitors to the SNO site. As a result, SNO decided to mount the full text of the cited article (Peterson 2000).

Software: Bridget Pairaudeau, who handles the statistics function for Institute of Physics Publishing, noted that an off-the-shelf package such as NetTracker can be used to screen out robot contamination as well as data from internal testing, demonstrations, training sessions, and trials that skew the usage data. It was found that NetTracker records HTML articles viewed rather than PDFs downloaded; this could be a concern for publishers that offer both functions.

Summary and Next Steps

The evolution of electronic journals to publisher-hosted databases of journal articles shifts the burden of measuring use from libraries to publishers. Although publishers need to collect data for their own purposes, the associated costs are considerable. Additional work is required to produce data that are meaningful and useful for librarians.

Fears of conflicting motivations between publishers and librarians are diminishing as publishers become familiar with their own data and focus on the challenge of producing useful statistics. Concerns about comparability are valid and need to be addressed in a meeting where publishers who have already implemented statistical functionality can share what they have learned. Issues to be discussed at such a meeting would include producing useful data and interpreting the data.

Producing useful data

Publishers and librarians share concerns about the lack of standards for collecting and presenting data and the lack of context for drawing conclusions. Putting together a complete picture of use with data in multiple formats and from multiple sources is an additional challenge for both.

Given the variety of platforms and software packages, publishers need to learn from each other about the variables and to agree upon an approach that will produce consistent measures of use. If a group of individuals involved in producing statistics were to pool their intelligence and produce guidelines, it would greatly advance the state of the art. Once a critical mass of publishers is producing consistent data, it will be clear to others who are just beginning their work what data are needed and how to collect and deliver valid data.

The industry is at the first stage of creating the capability to gather statistics, establish standards, and deliver comparable and reliable data. In another year, new systems will emerge that rely upon data mining and analysis and that focus on understanding user behavior.

Interpreting the data

Users want and need access to a much broader range of material than that which can be owned affordably in print. Emerging pricing models and consortial arrangements that provide users with access result in data that show higher levels of use of nonowned titles. Interpretations of these data vary; for example, they include concerns that the right titles are not being bought as well as the recognition that the information industry is moving from a supply-driven model, with preselected packages of information, to a demand-driven model, where users choose what they need from a wide array of options. Making users aware of what is available and increasing the ease of access will require cooperation between publishers and librarians.

Librarians and publishers need to understand users and their information-seeking behaviors in ways that were not previously possible or necessary. As intermediaries between the author and the reader, publishers and librarians must learn how best to serve their users. Doing so will require further analysis.

Recommendations for next steps

Publishers are discovering what data they need to provide and how to provide it. There is no forum where staff working on statistics can share their understanding of the technology and make it easier for those who have just begun to tackle these issues. To facilitate the development of statistics in the industry, an organization such as the Council on Library and Information Resources might wish to sponsor an invitational meeting that would enable those involved in this area to discuss the issues they have encountered and to explore the development of guidelines for all participants.

Associations involved in creating standards and guidelines on data collection are focused on defining the data elements and determining what is currently being done. No one is working directly with publishers who have developed data, understand the variables, and are in a position to provide guidance so that those producing data can be consistent in their implementation. CLIR is well positioned to host such a meeting, which should include representatives from the publisher, vendor, and library communities. Preliminary feedback from publishers and aggregators has been favorable. The author welcomes additional input on the structure and desired results of such a forum.

Skip to content