150-year-old system ensures academic quality

Since the system of expert assessment in the appointment of professors was introduced in 1876, the procedure has been scrutinised, investigated, evaluated, developed and re-regulated numerous times. Every review and investigation has concluded that the procedure is sound on the whole and that it is an important part of the appointments process that should be maintained, but both evaluators and politicians have disagreed on exactly what the process should look like.

Today, it is taken for granted that the specialists involved in the assessment are external experts, but during the first few decades of the system, they were drawn from the recruiting institution’s own faculty. In the early 20th century, expert assessors were granted leave of absence for up to six months to carry out their review, whereas today they are expected to complete their assignment within a month or so while still carrying out their other duties. In the middle of the last century, experts were encouraged to consult with one another, but nowadays, higher education institutions set guidelines to prevent such collaboration.

The current legislation has been in place since 2011 and represents a deregulation compared with the rules that applied previously. The current legislation states that ‘expert assessments of applicants’ competence is to be obtained, unless this is clearly unnecessary for the assessment of competence’ when employing a professor or adjunct professor. Furthermore, it stipulates that, unless there are exceptional reasons to the contrary, men and women must be equally represented when several experts are consulted. Apart from these stipulations, it is up to each higher education institution to draw up its own frameworks and rules.

As a result of this deregulation, it is difficult to gain an overview of today’s expert assessment procedures. It is not uncommon at larger higher education institutions for appointment and promotion procedures to be regulated at subject or faculty level. The five representatives from three different higher education institutions that Universitetsläraren talked to – at Stockholm University, Uppsala University and Halmstad University – all report that their institutions make use of subject experts to a greater extent than required by law.

Jonas Olson, Vice-Dean and Chair of the Appointments Committee at the Faculty of Humanities at Stockholm University, is keen to discuss the system of expert assessors, as he feels there are many misconceptions about the process.
“The issue interests me,” he tells us.

At the Faculty of Humanities, two external experts are used in all recruitment cases. They need to have subject expertise and there must be one male assessor and one female. The department responsible for the recruitment presents suggestions of potential experts to be consulted.
“Formally,” he says, “it is then I, in my role as a member of the faculty management team, who appoint the experts.”

Lena Mäler, Dean of the Faculty of Natural Sciences at Stockholm University, describes how important the expert review process is in academic appointments.
“This is one of the most important things we do,” she says.

At the Faculty of Natural Sciences, external experts are always used when appointing all types of university teachers. The department or faculty proposes suitable subject experts, after which the dean or vice-dean decides which ones to consult.

Karin Berggren Bremdal, Administrative Director at the Faculty of Science and Technology at Uppsala University, says that external experts are vital to the credibility of the system, and they help the university to build up a more comprehensive picture of scientific developments at both national and international levels.
“There is a system in place for the whole of Uppsala University,” she says, ”but in practice, a great deal is regulated by the three disciplinary domains, or even at faculty level.”

When employing or promoting a person to the position of professor, three external experts are consulted. These are proposed by the section dean and appointed by the faculty board’s working committee. For recruitment at lower levels, such as senior lecturer, it is the section dean who appoints the experts, based on recommendations by the head of the relevant department.

Martin Wahlén, Deputy Administrative Director at the Faculty of Medicine and Pharmacy at Uppsala University, explains that they have a checklist which they send to the experts to ensure the quality of the review. Within his faculty, it is the head of the relevant department who recommends experts.
“Often, they want at least one of the experts to be from Sweden,” he says, “or at least from the Nordic region. This is usually because they want someone familiar with the Swedish education system, so that they can assess educational qualifications correctly.” 

Jenny Söderberg, a human resources strategist at Halmstad University, says that the higher education institution has policy documents that regulate the use of expert reviews, including how they should be used, as well as specific cases where expert assessments are not deemed necessary. Halmstad University has a common set of rules for the whole higher education institution. The university has a recruitment committee that is involved in appointment processes for most teaching posts, and the same requirements for expert review apply regardless of subject.

The precise requirements for potential experts vary slightly between higher education institutions. Professors are normally the most sought after, but in general, experts are expected to have at least the same academic level as the post in question. Several higher education institutions stipulate that experts should not be allowed to cooperate and should therefore not be working at the same higher education institution. Or at least not at the same department, according to the policy document at Halmstad University, says Söderberg.

It is not in the process of appointing experts, but in the use of experts and their reports, that the differences between higher education institutions are most evident. Of the three universities that Universitetsläraren compared, only Halmstad University has a recruitment committee that conducts an initial screening of applicants before the applications are forwarded for expert review. Only applicants deemed eligible and qualified for the position proceed to the review stage.

“If an excessively large number of applicants are deemed eligible, we send a smaller number for review,” explains Jenny Söderberg. ”In such cases, we carry out an objective assessment first, based on competence.” 

If the number of applicants meeting the basic requirements is around ten or more, the recruitment committee, in consultation with the manager conducting the recruitment, reviews which five to six applicants are considered to be the most highly qualified and should therefore proceed to the assessment stage.

“We also write explanations for our choices so that it is clear what criteria were used to make the selection. If the candidates cannot easily be distinguished from one another, all of them are assessed, regardless of the number.”

The expert reports then form the basis for deciding which candidates are invited to interview and a trial lecture. At Halmstad University, experts do not attend any recruitment meetings. Once the report has been submitted, their assignment is complete. The expert reports, together with the trial lectures, interviews and reference checks, form the basis for the appointment decision.

Similarly, within the field of medicine and pharmacy at Uppsala University, expert reports form the basis for the decision on which candidates to invite for an interview. The reports are usually unanimous in their assessment, but if they differ significantly, the recruitment panel generally concludes that one line of reasoning appears more robust and convincing, says Martin Wahlén.
“It may also be the case that the panel feels that one expert is slightly off the mark on one point and the other is slightly off the mark on another. In that case, the panel makes its own assessment, based on what the experts say as a whole.”

Within this field, the extent to which experts are involved in the process after submitting their reports has varied over time, but nowadays the general rule is that the experts’ remit is complete once their reports have been submitted.
“It is only if the recruitment panel has any questions regarding an expert’s report that they are consulted later. Otherwise, they have not been involved in recent years,” Wahlén explains.

In the field of engineering and natural sciences at Uppsala, however, the experts participate digitally in the recruitment panel’s deliberations. At least one expert is present when the panel meets the candidates.
“They attend and get an overall picture,” says Karin Berggren Bremdal.

At the Faculty of Science at Stockholm University too, the faculty wants the experts to be involved throughout the recruitment process.
“Their work is not done once they’ve written their report,” says Lena Mäler.

All higher education institutions conduct some form of quality control on the expert reports they receive. Jonas Olson, from the Faculty of Humanities at Stockholm University, says that the quality of reports varies greatly.
“It is also worth emphasising that the expert reports are not the decisive factor. They form part of the body of evidence on which the recruitment panel bases its decision, alongside application documents and interviews. The panel reads expert reports with the same critical eye as we read everything else in our professional roles.”

However, all the representatives that Universitetsläraren interviewed emphasise that it is unusual for expert reports to be so poor that they cannot be used at all.
“If an expert report is found to be a little substandard, the recruitment panel can either try to make up for this themselves in their assessment, or they usually request a supplementary report,” says Martin Wahlén.

He tells us that “perhaps three times in 15 years” has an expert report been so deficient that it could not be used at all.
“I had one example where an expert simply sent in a customised Excel spreadsheet of their own design with score assessments that we didn’t understand. We didn’t use it in the end, because we didn’t understand their reasoning and there was no qualitative assessment at all.”

Individuals who are dissatisfied with the procedure in connection with an appointment or promotion, for example with the quality or implementation of the expert review process, can appeal against the decision to the Higher Education Appeals Board (ÖNH). The Swedish Higher Education Authority (UKÄ) is the host authority for the Board. Pontus Kyrk, a legal adviser at UKÄ, writes in an email reply to Unversitetsläraren that, in his 26 years of professional experience, it is common for people to contact the Higher Education Appeals Board or UKÄ with objections regarding expert reports.

“This can range from complaints about the experts’ assessment to claims that they are biased. For obvious reasons, this mainly happens when the higher education institution has employed the person that the experts have ranked highest. If it is the complainant who has been rated highest by the experts, but the higher education institution has appointed someone else, the complainant usually, again for obvious reasons, emphasises the experts’ assessment.”

However, UKÄ does not keep statistics on the number or proportion of complaints about expert assessments that have been upheld following an appeal.

Earlier this year, Universitetsläraren reported on two separate cases where experts had incorrectly used AI in their reviews of promotion and appointment candidates, resulting in the reports being withdrawn by either the higher education institution or the expert. The issue of the use of generative AI in expert reports is being monitored by higher education institutions, but this work is still in its infancy, and the process of drawing up guidelines is ongoing.
“It’s not an AI report we’re paying for,” says Jenny Söderberg. “We could do that ourselves.”

Jonas Olson takes a similar view, saying “We engage a person because we want that person’s expert opinion. It is that person’s expertise we want. We are looking for a specific person’s intelligence, not artificial intelligence or someone else’s intelligence.”

At the Faculty of Humanities at Stockholm University, they have not yet come across any obvious examples of the misuse of AI, but Olson says it would be naïve to believe that it does not happen. They are currently putting together a section on AI for their guidelines for experts. At the same time, he argues that it is a matter of trust, which makes it particularly damaging when it comes to light that senior, established researchers have used AI in an improper manner. There is a danger that it will erode the widespread trust on which the system is based.

“If we stop trusting one another and just sit there looking for signs of AI in reports,” he says, ”then that is also a very unfortunate situation. We must have trust.”

At the Faculty of Medicine and Pharmacy at Uppsala University, the issue is viewed differently. The main thing, according to Martin Wahlén, is that experts take responsibility for what they submit.
“We don’t really care that much about how they produce it.”

The time it takes for experts to submit their reviews is one of the biggest challenges within the current system, according to the people we interviewed. At present, there are no formal rules about how long a review may take. Instead, it is up to each higher education institution to set its own time frames. An expert review of up to two applicants is generally expected to take around a month – this can vary by a week or so between higher education institutions – but it is not uncommon for experts to take longer.

“That can drag out recruitment processes, which is obviously not good for anyone,” says Karin Berggren Bremdal. “Neither for us as a higher education institution, which risks losing applicants along the way, nor for the applicants who have to wait a long time for a decision. So we do everything we can to ensure quality in these processes and to streamline them so that they run as smoothly as possible.”

Some higher education institutions have introduced a speed bonus, whereby the fee they pay is higher the more quickly the expert submits their report. At Uppsala University, this is not used in the discipline of engineering and natural sciences, but it is used within medicine and pharmacy, where, in response to problems with experts failing to respect the agreed time frame, a bonus was introduced a few years ago for reviews submitted on time.
“With this system, where you receive a higher fee if you submit your report on time, my experience is that almost all experts do so,” says Martin Wahlén.

He explains that reforms were implemented during the development of the expert review process to speed up the assessments and to generally remove steps in the recruitment process that were deemed unnecessary, but that these have rarely had the desired effect.
“Every time an attempt has been made to speed things up, the results have not been as hoped for.”

All the people we interviewed agree that it is usually easy to find experts, but that in certain niche subjects it can be difficult to meet the gender balance criteria. Some scientific fields have a skewed gender balance, particularly at senior level, which places a heavy burden on potential experts from the under-represented gender. In narrower fields, it can also be difficult to find experts who are impartial. 
“Everyone is constantly collaborating with everyone else. That means you have to look beyond the Nordic region, and even then, there can still be problems,” says Wahlén.

If it cannot be resolved in any other way, experts from outside the specific subject are engaged, as long as they have sufficient knowledge of the subject to assess the scientific merits, he continues. “In such cases, you might need three experts. All three might be slightly on the periphery of what we actually want, but together they cover the subject.”

Another situation where it can be difficult to find experts is when there are many applicants and a heavy workload.

“Sometimes we have a great many calls for applications,” says Lena Mäle, “and then it’s actually a bit harder to find people who say they have time to take on the assignment.”

Ultimately, however, things usually work out, says Jonas Olson, as everyone is aware that the system is based on people taking responsibility and making their contribution to the academic community by acting as experts. 

The entire academic system has a collegial structure that requires academics to agree to conduct peer review of one another, whether it is in research or in appointments and promotions procedures. It is also seen as a merit and a recognition of the individual’s knowledge and competence, which means that higher education institutions generally view their staff volunteering as experts positively, or even expect them to do so. However, it can be difficult to find the time to fulfil the role of expert alongside other work responsibilities.

“It’s a problem at many departments,” says Olson. “People have heavy workloads. They have teaching and administration, and then they’re supposed to devote time to research as well. And then they’re expected to do this. It can be quite time-consuming.”

Although people are expected to undertake expert assessment assignments as part of their role as an employee, it is a work task that competes for the individual’s time and energy against many others.

“Our collective agreement essentially divides professors’ time between teaching, research and administration,” says Martin Wahlén. ”Obviously this task competes with other types of research time, so it competes with the time one can devote to one’s own research. At the same time, a collegial system is based on people taking on this type of assignment.”

Although most researchers agree that everyone must be willing to peer-review each other, it has nevertheless been accepted that the system would be difficult to implement if experts were to do so entirely for free, says Karin Berggren Bremdal.
“The fees we pay are important for ensuring and maintaining quality.”

Fee levels for expert review assignments vary between higher education institutions and are based on the number of applicants, as well as the position to be filled. The remuneration is paid directly to the individual, not to their department or employer.

All five university representatives interviewed agree that the expert assessment system generally works well, even though it has its challenges. None of them would like to replace or abandon it.

“I would say that expert assessments are an incredibly important part of the recruitment process, but you couldn’t run a recruitment process based entirely on expert opinions. The whole picture is important,” says Karin Berggren Bremdal.

She explains that the management teams at higher education institutions sometimes discuss the procedure, such as how experts are appointed, the levels of remuneration and how to manage external collaboration skills.
“In a way, it’s in everyone’s interest to keep an eye on those issues.”

Lena Mäler explains that other higher education institutions sometimes contact her to ask how her faculty conducts recruitment procedures. She is happy to share and discuss details. Furthermore, as university teachers are involved in each other’s appointment and promotion processes in their capacity as experts, this naturally ensures that the system operates in the same way at all higher education institutions.

“There are no secrets or anything like that when it comes to conducting these processes,” she says. ”The expert review system, and the fact that we have our committees, exists so that we can have as open and transparent a system as possible to ensure that the person best suited for the position gets the position.”

Jonas Olson also generally believes that the expert assessment system works well, even though it can mean that appointment processes take time.

“It’s a system that guarantees legal certainty and meritocracy, which, in my view, is an effective way to combat nepotism. It is an impartial and, as far as I have seen and know, fair system. Perhaps one of the costs of achieving this is that it takes a long time, but I think that’s a price worth paying, because the benefits are considerable.”