Guide for appointed evaluators of SEAS applicants

After the call deadline the SEAS office sets up an expert panel for each SEAS position that has been announced. Each panel consists of a minimum of 3 international expert reviewers, with documented expertise in the relevant academic field. All experts will remain anonymous throughout the evaluation process to ensure that they will not be contacted by applicants or potential hosts. Both genders will be represented in the evaluation panels. A Rapporteur will be appointed to represent the views of the Panel.

All assigned experts must confirm that they have no bindings or conflicts of interests that can in any way influence their objectivity in assessing the applicants for the position in question.

Experts must also sign an agreement of confidentiality and to deliver their reports according to the timeline set up by the SEAS office. Unless any unforeseen problems occur, the external evaluators have four weeks to complete their individual assessments, to reach consensus and to deliver their final evaluation report to the SEAS office.

Expert evaluators are paid by the SEAS programme, after fulfillment of obligations (as outlined below), according to a payment scheme per evaluated application.

All eligible applicants are to be evaluated without any restrictions based on nationality, age, gender, ethnicity, disability/special needs, or social status. Evaluators are encouraged to look at the career as a whole and to consider non-academic work, career breaks (e.g., parental and long-term sick leave, compulsory military service, time spent for obtaining refugee status) and variations in chronological order as a potentially valuable contribution to the applicant’s profile rather than a penalty.

Candidates are to be evaluated on their:

• Research proposal
• Motivation letter
• CV including list of publications
• Any relevant publications - if included by the applicant
• Letters of recommendation from the graduating university or previous employers - if included by the applicant

The evaluation steps consist of:

Individual evaluations
All the expert members of a panel are to perform Individual Assessment Reports for each application following the SEAS evaluation guidelines, criteria, and score chart.

The evaluators will work remotely and be blind to one another’s reviews until their individual assessment reports are finished. On the basis of all application documents, including reference letters, the experts must assess the applications according to the main criteria: Excellence, Impact, Implementation, Quality of the candidate and (for outgoing candidates) Integration. Each criterion will be scored on a rating scale from 0.0 – 5.0.

The evaluators must also screen the proposals for ethical issues and state unresolved ethical issues in their assessment reports.

Consensus
Next, the experts will meet (remotely) to discuss their individual evaluations and to reach consensus. A Rapporteur will be appointed among the experts who will be responsible for building consensus on the score for each criterion and on the final score. The Rapporteur will draft an Evaluation Summary Report for each applicant which all experts must approve. When consensus on scores and qualitative comments is reached, the final score will be calculated by the Rapporteur as the weighted mean of the different scores of each criterion. The panel will deliver a final Evaluation Summary Report for each individual application, justifying the scores by summarising the strengths and weaknesses of each of the criteria. All experts must agree on the final scores (for each criterion and total) and the evaluation report. The final scores will be the basis for ranking and selection of the fellows.

The Evaluation Summary Report should also include notes on any ethical issues and state unresolved ethical issues.

If consensus cannot be reached, the Rapporteur must inform the SEAS office as soon as possible. The SEAS office will in such cases appoint an additional external expert as an adjudicative reviewer, to deliver an individual assessment of the respective application. In such cases, the Rapporteur must ensure that the opinions of all the experts are considered.

Quality control
The SEAS office will cross-read all Evaluation Summary Reports to ensure fair and equal treatment of all applications. The following checks will be applied: i) Are there qualitative comments for each evaluation criterion and sub-criterion? ii) Are the scores justified by clearly identified strengths and/or weaknesses for each criterion? iii) Have the different panels used the scoring scheme in a similar way? Any Evaluation Summary Reports that do not pass quality control will be sent back to the respective panel for corrections. Corrections must be done within 5 working days.

Ranking (performed by the SEAS office)
Proposals are ranked after all Evaluation Summary Reports have been checked by the SEAS office. After the rankings are finalised, all applicants will be informed of their results by the SEAS office. All applicants will receive their final Evaluation Summary Report.