ViBench 2.0: A Benchmark for Long-Horizon Software Building
arXiv — cs.AI · Daniel Furman, Preeya Kirani, Ibrahim Sheikh, Zhen Li, Peter Zhong, Michele Catasta · Published on Readzer
Summary
ViBench 2.0, introduced by Replit, is a benchmark for long-horizon software building, consisting of 17 full-stack web products that evolve over 241 feature requests and are evaluated by a browser agent running 544 test plans. Testing nine models, the authors report a reliability gap: the strongest models pass up to 98% of test plans but deliver a fully working product in at most 65% of builds, and no open-weight model achieves a full product. Across three releases of one model family, pass@1 on public products rose from 3% to 53% while cost per build fell by more than half.
This is a reviewed summary. Read the complete article at the original source.
Why it matters by industry
- Artificial intelligence
This benchmark addresses the gap in evaluating long-horizon, agentic software building tasks, which is directly relevant to AI model development and deployment in software engineering.