
AI reviewers at an experimental journal caught flaws in 11 of 16 analysis drafts. Once they learn corrected variations of the identical papers, they raised the identical objection in 10 of 16 instances. The authors of the Nov. 21 preprint A Society of AI Scientists say the reviewers stored flagging the difficulty as a result of the revised papers brazenly described the limitation.
In a separate audit on the agent-run journal, a visitor editor turned reviewer objections into scripts and ran them on the authors’ uploaded knowledge. The checks overturned the principle statistical conclusion in 77 of 156 objections classed as main, the preprint reviews. A number of objections may concern the identical examine.
The authors describe how the Journal of Agent Analysis grew out of a 12-week examine of 936 AI brokers throughout 35 human-operated labs. Brokers in a single lab based the journal and launched a standing panel of automated referees on Nov. 4. Human operators arrange the computing methods, capped spending on mannequin entry and selected broad analysis themes for the examine.
How the journal examined AI reviewers’ objections
The panel learn manuscripts and raised issues. Authors may additionally formally submit a examine for an editorial determination. The editor may then recompute the principle numbers from the uploaded code and knowledge. A referee may level to one thing within the textual content that regarded like a flaw. An editor may check what occurred to the consequence when the criticism was utilized.
Discover the 2 workouts
The identical objection typically survived a correction. The preprint attributes this to revised drafts brazenly describing the limitation.
In a separate audit, a visitor editor ran scripts on uploaded knowledge for 276 objections throughout 43 paper teams.
Referees nonetheless flagged corrected drafts. The separate knowledge audit measured how typically a examined objection modified the principle consequence.
The text-only comparability used 16 chosen pairs. The audit counts objections, a number of of which may concern the identical examine. Bar widths examine shares solely inside every train.
The objection audit put that strategy to work on 276 criticisms throughout 43 teams of papers. Operating the scripts overturned the principle consequence for twenty-four of 120 minor objections, much less steadily than for main ones. In a separate case, the editor rejected a paper after discovering that its code changed failed makes an attempt with the precise reply. That substitution undermined the paper’s declare that its verification methodology had improved efficiency.
The researchers’ public archive consists of papers, code and agent discussions. The dataset card says the obtain excludes uncooked experiment knowledge, which the journal used to check objections. The authors describe an exploratory, nonrandomized examine of computational analysis. Every lab ran the brokers by itself infrastructure, so the authors relied on members to reveal whether or not they adopted restrictions on handbook work.
With a proper submission’s code and knowledge, an editor may examine whether or not the objection held up and whether or not it modified the consequence.