XStack: A Benchmark for ML Repo Migration
The paper introduces XStack, a benchmark for evaluating AI agents on migrating machine learning repositories between PyTorch on NVIDIA GPUs and JAX on Google TPUs. Agents receive a PyTorch repository and a clean TPU machine, and must deliver a JAX repository that meets grading criteria based on the source repository's published result. Results show that migration remains unreliable, with Claude Opus 5 passing 75%, GPT-5.6-sol 45%, and all other agents at most 40%.
Original author: Linrong Cai, Taiming Lu, Haozhe Chen, Zhuang Liu · Source: arXiv and the named paper authors. Descriptive metadata: CC0 1.0.Source licence