Study Finds LLM API Relays Show Behavioral Differences, Tool Permission Violations, and Billed Token Discrepancies
arXiv — cs.AI · Xiaodong Wu, Zhimin Zhao, Qi Li, Chaklam Cheong, Xiangman Li, Jianbing Ni · Published on Readzer
Summary
A study tested 20 API relays offering GPT-6 Astra and Claude Opus 5, comparing answers to official responses. It flagged 7/10 low-price and 2/10 same- or higher-price relays for behavioral differences, with some flags changing on retesting. No injection was confirmed in 3,332 screened responses, but four relays returned tool calls despite instructions disabling them. Five relays reported 4,200 to 60,000 more input tokens than the official API for identical requests, and charges followed reported usage where balances were available.
This is a reviewed summary. Read the complete article at the original source.
Why it matters by industry
- Artificial intelligence
For AI model deployment and API procurement, this study highlights that advertised models and prices are insufficient to judge relay reliability, emphasizing the need for repeated testing and scrutiny of tool permissions and billing accuracy.