AI-powered system to ease RSF review grind

29.11.2018

tags: 

A suite of automated tools was developed by the Russian Science Foundation to assist with peer review but expert council members are still engaged in the process.

Researchers have repeatedly indicated worrisome spot in the review process is how peer reviewers are assigned to the proposals submitted to the RSF. They are selected by the panel coordinators among the candidates suggested by the information management system. RSF tried to minimize this subjective component in the first AI-powered review management platform trial.

Each application in the recent competition “Conducting research based on the existing world-class research infrastructure” will be assigned all three reviewers automatically by the computer. The panel coordinator can make a correction if not satisfied with the result of the selection, highlighting the reason why such assignment correction was required.

Crucially, peer review by artificial intelligence (AI) is promising to improve the process, boost the quality of reviews - and save time of the panel coordinators considerably. Almost 1300 proposals were submitted for this trial competition, each one required the assignment of three reviewers. Normally it took days coordinators to finalize all the assignments whereas for the computer this is a matter of a minute. However, more trials are required to see if such approach is justified.

The Chairman of the RSF Expert Council Alexander Klimenko in the interview for the newspaper “Poisk” was optimistic about the first results of the experiment: “We already have automatic application checkers that can flag errors, inconsistencies and issues related to competition guidelines. Our IT services examine whether the application contents technically meet certain competition requirements. As a result, the proportion of the ineligible proposals is now less than a percent (it was up to 20% of applications five years ago). Now we focus on the quality of the review assignments. The AI solution doesn’t replace panel coordinators judgement but it makes it easier. We expected that due to the insufficiently accurate indication of the classifier codes and keywords in the reviewers’ profiles, the number of refused review assignments would increase because the applications could get inappropriate reviewers but this did not happen”.

The algorithm is going to need some time to perfect. For example, it not yet smart enough to eliminate biases and a potential conflict of interest fully. The computer can check whether the reviewer is the applicant in the same competition or if the reviewer is not employed with the applicant in the same organization. But AI is not capable yet to understand family ties and complex relations between different groups of researchers working in the same field.

Russian Science Foundation is piloting AI tools in peer review in Russia. If the trials prove successful, RSF is keen to integrate semantic analysis of the applications and machine learning to add further value to the review solution platform.

News