Skip to content
AI editorial reviewArtificial intelligence

RAW: Measuring User Disengagement from LLM Refusals in Production Logs

arXiv — cs.AI · Pavel Kireyev · Published on Readzer

Summary

The paper introduces RAW, a method using LLM stochasticity as a natural experiment to measure the causal user-side effect of refusals in deployed models. Analyzing 15 LLMs in LMSYS-Chat-1M and WildChat, they find that for about half of the models, a refusal reduces subsequent re-engagement by 3 to 17 percentage points, while others show no distinguishable effect. The method runs on logs alone at about $0.30 per 1,000 conversations.

This is a reviewed summary. Read the complete article at the original source.

Loading reading state…

Why it matters by industry

  • Artificial intelligence

    For AI platforms, this provides a low-cost, log-based method to quantify refusal impact on user retention, informing moderation and UX design.