ContentsPerformance
Indicators Partners' Workspace (password protected) |
EQUINOXLibrary Performance Measurement and Quality Management SystemDeliberations leading to the EQUINOX Performance Indicator SetThe EQUINOX project followed on from an earlier EC funded project called EQLIPSE, which developed a set of performance indicators for European libraries. These performance indicators were based on the work of the ISO committee, which produced the ISO standard ISO 11620: Information and Documentation – Library Performance Indicators. One of the aims of the EQUINOX project was to further develop and enhance these indicators by developing performance indicators for electronic library services, which received only scant attention in the EQLIPSE project and indeed in the ISO standard, the focus in both cases being on traditional library services. In this work we drew on partners’ experience from the EQLIPSE project (most partners were also involved in this project), Peter Brophy’s expertise from work on the MIEL2 project, and Roswitha Poll’s knowledge of ISO and IFLA work in this area. The work on the EQUINOX performance indicators began by taking each of the EQLIPSE PIs and considering what if any equivalent it would have in an electronic environment. Relevant publications in the field were also examined, especially the work of McClure in the USA and the MIEL2 project. All this information was synthesised to identify potential PIs for the electronic services of libraries. These were discussed by partners and considered in terms of a framework for libraries suggested in the report of the MIEL2 project, in order to identify any gaps in the initial list of indicators. The initial list of indicators comprised of 34 performance indicators for electronic services. Further discussion on these indicators recognised that many of these could more sensibly be subsumed into the traditional indicators defined in the ISO standards. Others were omitted from the list because it was not possible to collect the data required for them. The PIs omitted at this stage were as follows:
The result was a list of 14 performance indicators for electronic library services as follows:
Having defined the performance indicators and their collection methodologies, the list of 14 PIs was publicised through the EQUINOX website, relevant mailing lists, dissemination through the literature and at EQUINOX workshops. Feedback was received from a wide range of professionals throughout Europe and as far away as the US and Australia. At the same time, the library partners in the EQUINOX project tested these performance indicators in their own real library situations in terms of ease of data collection, applicability to the current situation in the library, relevance to library management, etc. Both the feedback received and the testing of the PIs showed that some further revision of the PI list was required. Some PIs were felt to be too demanding of staff time in relation to the value of the information gleaned. Others were felt to be confusing and difficult to calculate. In many cases, the data required simply wasn’t available from many of the ELS currently in use in libraries. Some areas caused particular difficulty: Terminology A major hurdle throughout this work was defining terminology. There was much debate amongst the partners, and as the EQUINOX PI set should fit in with other international work in the field, it was decided to accord to a large extent with ISO work on definitions for the revised ISO 2789 standard on library statistics. Since this is itself a work in progress, the terminology used during the development of the EQUINOX PI has changed according to changes made in ISO 2789 work. For example, the term log-in was replaced by session, reference enquiry was replaced by information request, and document delivery was replaced by documents and entries downloaded, to mirror the work on ISO 2789. Total number of the target population Queries raised included who exactly should be included and whether the target population should change depending on the actual service being considered (e.g. only medical staff and students being included in the target population for Medline). It was decided by the project team that it would be more relevant for some PIs to distinguish between the total population to be served by the library and the target population at whom a specific service is aimed. Difficulties were identified with distinguishing between members and non-members in the collection of some data. Redefinition of some of the indicators provided a solution to this problem by suggesting measuring accesses to resources from the library’s or institution’s IP addresses. Lack of statistics from some suppliers The biggest difficulty encountered during data collection was the lack of availability of many of the datasets required from some or all electronic library services. It had been recognised in advance of the development of the performance indicator set that this would be a problem. However, it was agreed that the performance indicator set should be defined according to what libraries need to be able to measure rather than confining it to statistics that are available for all libraries and all services at the present time, especially as what is available is changing rapidly. Disaggregation of Statistics For many PIs it was decided that the PIs were more useful if the data was considered separately per electronic library service, so that PIs could be compared more easily. For example, number of documents downloaded per session for a specific service compared to the cost per session of that service, etc. Obviously, libraries can very easily calculate the aggregated totals for each PI from these statistics, if required. As a result of these deliberations changes were made to some of the initial performance indicators while others were omitted altogether: PI 2: Number of log-ins to electronic library services per capita per month It had been suggested that a time-based measure might be more indicative of quality than number of log-ins. However, the number of outside factors affecting the time spent accessing a service is felt to prohibit such a measure. Use of a service for user education and log-ins by library staff will of necessity be automatically included in this measure. The project team felt that this is not a problem as the indicator is still indicative of uptake. It was agreed that this PI would be more useful if each electronic library service were considered separately, such that the data for different services could be compared. It was also felt that each service should be compared to its specific target population rather than the entire population to be served. PI 4: Number of electronic documents delivered per capita per month It was agreed that this PI was not really comparable between services, for example where one service delivers full text documents while another delivers citations. For this reason this PI was changed to consider each ELS separately. The definition of electronic document delivery also caused some difficulties. For example, documents or records viewed on screen are often equally useful to the user as those printed or downloaded, but it was unclear whether these would be included. The PI was redefined to allow this type of access to a document to be included if required. It was also felt to be more useful to consider the number of documents downloaded/delivered in relation to each session on an ELS rather than per capita, showing the extent to which useful information is found by each user. PI 7: Reference enquiries submitted electronically per capita per month It was felt that a PI comparing reference enquiries submitted electronically with those received in the traditional way would be more useful and interesting than this PI. This was therefore replaced with Percentage of information requests submitted electronically. PI 9: Number of library computer workstations per capita PI 10: Library computer workstation hours used per capita per month The value of these as indicators of quality was called into question, since library services are also accessed from PCs outside of the library and indeed from outside the organisation. The data for PI 10 was also found to be difficult and time-consuming to collect. These two PIs were therefore merged into one PI – Number of LCW hours available per member of the population to be served. This PI proved to be easier to collect, while still giving some indication of the extent to which user demand for computer workstations is being met by the library. PI 12: Automated systems availability This PI is already available in ISO 11620 and since the aim of the EQUINOX PIs is to complement rather than replace the ISO PIs, this PI was omitted. It was also felt that generally library management systems are a lot more stable and reliable than heretofore, and unavailability is no longer a major concern of libraries. PI 13: Mean waiting time for access to library computer workstations It was suggested that it might be simpler and just as valuable to change this performance indicator to the average number of users queuing for library computer workstations at specified times. However, it was also noted that where a booking system was in operation this PI was not relevant. Indeed it was found that very often users simply do not bother to queue but prefer instead to return later when a library computer workstation becomes available. This PI was therefore omitted. PI 14: IT expenditure as a percentage of total library expenditure Debate ensured as to what should be included in IT expenditure, e.g. hardware, software, network costs, subscription costs. It is also not clear whether capital expenditure should be included as well as ongoing expenditure. Difficulty arose also where some services are paid for by another unit but ‘provided’ by the library. It was decided that the objective of this PI, to assess the degree to which a library is focusing on electronic resources would be better served by a comparison between expenditure on traditional acquisitions and acquisition and licensing of electronic resources. This PI was therefore replaced by: Percentage of total acquisitions expenditure spent on acquisition of electronic library resources. It was decided at this point in the deliberations that an indicator in relation to user training (which was omitted earlier) should be reincluded in a slightly altered form. Thus the following list of Performance Indicators was produced:
This page was last updated by Monica Brinkley on October 02, 2000 |