
Anthropic’s Claude Fable 5.2 accurately named the author of 56% of 398 opinions and meta-reviews in a small take a look at of unsigned peer opinions. Six volunteers wrote the reviews. The mannequin noticed one report at a time, with out internet entry, instruments, an inventory of doable authors or different writing samples.
In a Nov. 19 preprint, Atrey Desai and colleagues analyzed reviews written between 2015 and 2026 for 26 double-blind conferences and workshops in machine studying and pure language processing. The researchers knew who wrote every report, however the reviewers’ names had by no means been publicly linked to them. In 19 circumstances, the textual content itself was already publicly obtainable and not using a identify.
Claude named the author on 68.4% of 266 normal opinions and 31.1% of 132 meta-reviews, which summarize the opinions of a submission. The meta-reviews have been shorter and got here from simply two of the six contributors. The examine didn’t separate the results of report size, kind and authorship.
The 56% consequence counts every report equally, so contributors who provided extra reviews had extra affect on the general determine. When the researchers gave every reviewer equal weight, Claude’s common accuracy was 49.1%. The take a look at confirmed that an unsigned report may give its author away. Estimating how usually that occurs throughout confidential peer overview would take a bigger, extra diversified pattern.
Solely two of the six volunteers wrote the meta-reviews.
The meta-reviews have been shorter. The examine can not separate the results of report size, kind and author.
Public opinions yielded a decrease general rating
The researchers additionally examined 1,559 public opinions from 47 open-review journals. Claude accurately recognized the reviewer in 21.4% of the 1,530 circumstances it answered. The paper reviews 39.3% accuracy for a subset containing 789 opinions that Pangram 4.1, an AI-text detector, had labeled human or blended. The paper’s summary cites that filtered fee, rounded to 39%.
Accuracy was larger on public opinions whose writers cited their very own work. Claude recorded 80.2% accuracy within the group of 106 opinions with self-citations and 17.1% within the group of 1,453 with out them.
Claude made errors even within the group the place it carried out finest. An editor or writer introduced with a reputation from Claude would nonetheless want unbiased proof to ascertain who wrote an unsigned overview. CryptoSlate has additionally reported on how frontier AI fashions dealt with fabricated analysis information. The crew says it plans so as to add outcomes from a considerably bigger set of nameless peer opinions in an replace to the preprint.