When Expert Disagreement Hurts: Prestige and Selectivity in LLM Decision Revision
arXiv — cs.AI · Yupeng Tang, Mingfeng Lin · Published on Readzer
Summary
A new arXiv study introduces a two-pass audit to measure whether LLMs improve decisions after disagreement, finding that no model meets the selectivity bar under expert disagreement, with accuracy dropping by up to 16.1 points. The expert label increases reversals for four of five models, but its effect on accuracy ranges from a 5.2-point loss to a 2.4-point gain, showing reversal counts are a poor proxy for harm. The authors recommend paired, conditional metrics for evaluating revision pipelines.
This is a reviewed summary. Read the complete article at the original source.
Why it matters by industry
- Artificial intelligence
This research directly addresses model behavior under disagreement, relevant to AI model evaluation and safety in reasoning and agent contexts.