This checklist gives information useful in discussing, negotiating, and (if need be) fighting against performance indicators (PIs); and advice on actions you can take when discussing, negotiating, and dispensing with PIs.

PIs may come as completely developed "accountability systems," or as discrete items and proposals, apparently unlinked and unrelated to each other. It makes a difference whether performance indicators are imposed by your local university administration, by the provincial government, by the federal government, or by a granting agency. This checklist contains items that may not work equally well in all cases.

Insist on a List of Good Reasons for PIs

PIs should be accompanied by a detailed list of the reasons for imposing those PIs.

Ask what the purpose(s) of the PIs is/are: If PIs are from provincial government, are they part of a wider plan to make the financial "reporting" of social services consistent across all ministries? That is, are PIs merely the basis of a new accounting system? If so, what good does the new system do? How does it improve educational practice? How does it improve administrative practice? Exactly?

If improved educational practice/student learning is the outcome: Insist on seeing the exact chain of cause and effect from PIs to improved practice. This is essential, whether the PIs come from government or from the local university administration.

In the world of education, we rely on professional judgement in figuring out how to improve practice -- not on naïve ideas about how A B C D, where "A" and "B" are PIs and reports based on them, "C" is improved educational practice, and "D" is increased student learning.

If the aim is to save money: Ask for detailed budgetary forecasts of savings. Insist on a balanced (negative and positive) list of the likely consequences of budgetary restrictions, especially in learning conditions (class size, library size and availability, time available for professors and librarians to consult with students), and in research conditions (time to do research, research assistance/assistants, physical and technical infrastructure, and so on). Be sure to ask about "Cost" on this checklist.

If the aim is "improved accountability": Remind the supporter of PIs that accountability means "the ability to say what you are doing and why." Some PIs may show what universities and colleges do; that is why CAUT is not opposed to PIs. But no PIs show why they do what they do. This means that PIs have little or nothing to do with accountability.

Ask how PI statistics would ensure that governance was transparent and fair. PIs are, after all, indifferent to politics. A system of PIs would work just as well in the German armament industry in 1935, or the old Soviet coal mining industry of the 1950s, as it would with universities now -- yet those historical examples were of accountability as control. Ask how PIs contribute to true accountability in a democracy.

In the name of "accountability," insist that the long-term aims and short-term reason for each PI be given each and every time a PI is developed or used in public. Insist that the chain of authority for each PI be described for each PI at the time it is developed, when it is applied, and when it is published.

Cost

Insist that the agent(s) or person(s) who propose PIs provide a detailed and complete cost break-down for the development, implementation, and execution of a PI system.

It requires many bureaucrats and much of our time to develop and to apply PIs. This is time taken away from policy development, from teaching, from research, from direct interaction with students and the public. PIs thus put a strain on the usual priorities of government departments, and the priorities of universities and colleges.

That strain would be "justified" only if supporters of PIs could give satisfactory answers to the questions outlined above.

But in any case, PIs are costly. The question is, Are they worth the cost?

In costing out PIs, don't forget to ask about:

• time spent by bureaucrats (provincial or university) travelling to see PIs in use in other jurisdictions, reading and researching PIs, and defining PIs.
  
• time and money allocated to information retrieval to "feed" the PIs.
  
• time and money for superintending and administering PI reporting systems, once in place.
  
• time and money spent communicating PIs to higher levels of government, and communicating PIs to the public.
  

Benefits

PIs are supposed to produce benefits.

Insist on seeing how each and every cost is linked to a benefit. This is likely to prove very difficult in practice. But this is just the point: the usual cost-benefit analysis simply does not work in education.

The cost of running universities and colleges is considerable, of course. However, the reasons for spending money on education are different in kind from those for spending money on industrial machines or human "capital" -- in education, it isn't a question of input versus output, or throughput rates, or profit margins. This means that the link between PIs and educational benefits will be extremely indirect, if it can be found at all.

Some PIs (for example, straightforward reports on the administrative cost of running a university or equipping a library) may suggest useful improvements in administrative and budget practice. The result might be more time and money that could be spent on the real business of universities and colleges -- education.

Transparency

Insist on knowing who is writing PIs, when, where, and why. If PIs are coming from provincial government, this information should be open and available at all times (why not put it on a web site?). The information on the time and money being spent by government should be available on an accumulating, regular basis (again, on a web site -- and preferably not just annually).

PIs developed by university administrations or (if we're lucky) university senates, should be transparent in just the same way.

Transparency also means, of course, that the questions outlined above, are constantly asked and answered.

Transparency also means that the definition, development, and implementation of PIs will be accompanied at all times by publication, and by some sort of public (or even peer) review. The standards of reasoning and evidence we expect to apply to ourselves should be applied at all times to government and to university administration.

The faculty association and the university administration should require these kinds of transparency in their dealings with government (and this should be the subject of written agreements), or in collective agreements (where PIs are coming from the university administration or senate).

Discounting External Factors

All of us know of PIs that measure the "employability" of graduates, or the number of books per graduate in the university library, and so on. These PIs are not measures of university performance. In the case of employability, for example, they are a measure of the performance of the economy. In the case of library books, they are a measure of the priorities (and possibly the competence) of the university's financial officers.

Insist on discounting external factors when governments or administrations bring in PIs.

• for PIs on accessibility (that is, the equal access of people from all social backgrounds to publicly-funded higher education), the relevant factors are external to a high degree: general economic conditions, attitude of government and private sector to education and culture, and the level of funding offered to universities (with more funding, it is possible to ensure more access). Narrow university-based PIs on accessibility should be carefully discounted on these several grounds.
  
• for PIs on climate and governance inside the university: these are very largely within the power of university senates, administrations, and faculty associations. However, it will be a wise "user" of PIs who insists on checking whether external discounts may apply here, too.
  
• for research PIs: discount increases and decreases in funds allocated to the granting councils; discount fiscal and political fac-tors that affect the number of journals and presses which may publish the results; discount economic facts that may encourage/ discourage the creation and registration of patents.
  
• for PIs based on student evaluations of teaching: discount rising and falling popularity of certain subjects, programs, and degrees (which may lead to negative/positive evaluations of teaching, no matter what the reality); discount rising/falling levels of general student morale (for example, at a time of quickly rising student debt loads); discount the general unpopularity/popularity of the university administration.
  
• for all PIs: discount rising/ falling public funding; discount overall government attitude to public education and to public services more generally.
  

PIs for Administrators & Government

Few PIs are likely to improve educational practice and provision in Canada. For this reason, we think that PIs should be limited in number and kind, and very carefully justified in all cases.

PIs of similar kinds should, nonetheless, be applied to administrators and to bureaucrats/politicians -- for the very reasons that lead them to insist on PIs for us.

Collective Bargaining & Negotiation of PIs
Some supporters of PIs will argue that PIs are "system-wide," and that PIs detect efficiencies and inefficiencies in the "mechanism" of whole systems. They say that PIs have no implications for the individuals and the evaluation of individuals in those systems.

This is false.

All PIs lead to changes in the way individuals are evaluated. For example, a system-wide PI on the "employability" of graduates may mean that professors in programs whose graduates are employed in fields outside their discipline(s), will come under criticism. If physics graduates turn up working in the agricultural industry, for example, then physics professors may find themselves asked to teach the "physics of agriculture," whether this is good physics or not! If the physics department does not change its curriculum in line with this particular PI, then it might face funding cuts (or some other sort of discipline). Individuals would find that their promotion and tenure depended very much on evidence that their teaching and research would contribute to an improvement in the all-important PI.

For that reason, where PIs become part of the information stream for promotion and tenure decisions, such PIs should be subject to all the usual provisions of the collective agreement, and should be bargained in the appropriate ways.

Comparisons of Universities & Colleges on PIs
Ask how inter-university and international comparisons of universities, using PIs, can be reliably linked to improvements in

• teaching practice and learning conditions
  
• levels of public funding for higher education
  
• levels of public confidence in higher education
  

Unless these linkages can be shown, and absent answers to the questions above, public funding should be withdrawn from the development and implementation of such PIs.

Competition & PIs

Supporters of PIs may claim that PIs encourage healthy competition among and between universities and colleges.

The underlying assumption is that competition produces improved educational practice, intensified research and development, increases in the energy and productivity of individuals in their private intellectual lives, levels of accessibility, greater employability, improved student learning and performance, and greater financial "efficiency."

Insist on seeing both arguments and evidence (cause-effect evidence, if you please) that competition produces each of these effects. Ensure that this evidence is published and subjected to peer review.

The argument for competition rests on several false assumptions. However, for the purposes of this checklist, I close this section with just one: the assumption that universities and colleges are not competing already, or not competing enough.

The evidence is that Canadian universities have been engaged in a massive and continuous competition for funds (private and public), for reputation, for the best professors, for the best students, for the most salubrious locations and buildings, and so on and on -- and that this competition has been continuous and very public for more than 150 years.

Insist on proof that any further competition in the academic marketplace would be a good thing.

Use of PIs to Attack Departments, Faculties, & Individuals

The great danger of PIs is that they will be used to attack departments, faculties, and individuals.

It should be an aim of administrative policy, academic policy, and collective bargaining that PIs be used only for public information or the formation of public policy on higher education provision and funding.

Professor Bill Bruneau is President of CAUT.

This checklist was prepared by Bill Bruneau as chair of the ad hoc CAUT Committee on Performance Indicators and Accountability. It is not an official document of CAUT. Professor Bill Bruneau can be contacted at william.bruneau@ubc.ca. Fax: 604-224-9242.