Skip to content

READZER / INDUSTRY INTELLIGENCE

Industry desk

Follow important changes. Read the original evidence.

Nov 22 · UTC

Your reading desk
Newest first

Summary preview

Back to updates

RAW: Measuring User Disengagement from LLM Refusals in Production Logs

arXiv — cs.AI ·

What to know

The paper introduces RAW, a method using LLM stochasticity as a natural experiment to measure the causal user-side effect of refusals in deployed models. Analyzing 15 LLMs in LMSYS-Chat-1M and WildChat, they find that for about half of the models, a refusal reduces subsequent re-engagement by 3 to 17 percentage points, while others show no distinguishable effect. The method runs on logs alone at about $0.30 per 1,000 conversations.

Loading reading state…

Source and evidence

arXiv — cs.AIOriginal language · und ·

Original author: Pavel Kireyev · Source: arXiv and the named paper authors. Descriptive metadata: CC0 1.0.Source licence

Why it matters by industry
  • Artificial intelligence

    For AI platforms, this provides a low-cost, log-based method to quantify refusal impact on user retention, informing moderation and UX design.

Reviewed summary. The complete article remains at its original source.