Student opinion surveys of instructors should never be considered or described as measures of teaching quality.

Student opinion surveys and how they are used have been hotly contested in academia for years. But new research points to growing evidence that the surveys are biased and discriminatory.

There is reason to be concerned about student bias, says Anne Boring, a postdoctoral researcher in econo­mics at the Paris Institute of Political Studies.

With colleagues at the University of California at Berkeley, Boring is the author of a new study on how students evaluate their teachers.

“Teacher evaluations measure a lot of other things besides the quality of teaching,” she says. “In particular, the grades the students give are biased towards the gender of the teacher. Women are evaluated more poorly, even though nothing shows they’re worse teachers.”

The ratings the students give their teachers are also strongly and po­sitively correlated to the grades they receive. “In the end, evaluations wind up as a kind of quid pro quo between students and teachers, with each giving a good grade to the other,” she adds.

In their study, the French and American researchers studied two sets of data. The first contained 23,001 evaluations of 379 instructors by 4,423 students enrolled in six mandatory first-year courses at a French university. The second was a randomized controlled study of 43 evaluations involving four sections of an online course given to students at a US university.

Boring and her colleagues came up with a number of findings: evaluations systematically discriminate against women; the bias is strong even in the most objective parts of the evaluation, such as students’ satisfaction with how quickly their homework is corrected and handed back; the bias varies by discipline and the gender of the students; it’s impossible to compensate for the bias; evaluations are more influenced by the gender of the students and the final grade they expect than by the quality of the teaching they receive; and, the bias can be so strong that even the most productive teachers wind up with a lower evaluation than those who are less productive.

Biases aren’t limited to the gender of the teacher. Racialized faculty also face discrimination in student surveys, according to a recent analysis of the United Kingdom’s National Student Survey. The study, published in January, found that undergraduates systematically give lower evaluations to black or ethnic minority academics than they do to whites. The University of Reading researchers attributed the results to “unconscious bias” on the part of the students.

“Student opinion surveys of instructors should never be considered or described as measures of teaching quality,” says David Robinson, executive director of CAUT.

In 2014, a committee of the American Association of University Professors surveyed almost 9,000 faculty members to evaluate the effectiveness of student evaluations. In its study, published last year, the committee concluded that evaluations should only be part of a range of tools universities can use to collect feedback. More effective ways to evaluate and improve teaching include class visits by peers, regular updates of course outlines, and programs of continual development.

Student opinion surveys remain popular, however, and more uni­versities and colleges are now moving to electronic versions in place of paper surveys that are handed out in class. Last June for instance, Simon Fraser University announced a pilot project to move away from paper-based evaluation forms and instead do everything online. Critics say electronic surveys will only compound the problems. The AAUP study found the response rate for online evaluations varies between 20 and 40 per cent, compared with 80 per cent for evaluations done on paper.

Boring believes student opinion surveys should be eliminated entirely because they are biased and can negatively impact a teacher’s career. Quantitative criteria are particularly problematic, she says: the “grade” that instructors receive has little or nothing to do with the quality of their teaching.

This is especially true at universities and colleges that place a lot of emphasis on teaching. “The more universities decide to create these teaching positions, the less they should rely on (student) evaluations when it comes time to renew their staff,” Boring says. “If the only thing their jobs depend on is evaluations, then professors will do everything they can to get good evaluations.”