AI editorial reviewArtificial intelligence
RAW: Measuring User Disengagement from LLM Refusals in Production Logs
arXiv — cs.AI · Pavel Kireyev · Published on Readzer
Summary
The paper introduces RAW, a method using LLM stochasticity as a natural experiment to measure the causal user-side effect of refusals in deployed models. Analyzing 15 LLMs in LMSYS-Chat-1M and WildChat, they find that for about half of the models, a refusal reduces subsequent re-engagement by 3 to 17 percentage points, while others show no distinguishable effect. The method runs on logs alone at about $0.30 per 1,000 conversations.
This is a reviewed summary. Read the complete article at the original source.
Loading reading state…
Why it matters by industry
- Artificial intelligence
For AI platforms, this provides a low-cost, log-based method to quantify refusal impact on user retention, informing moderation and UX design.