Pulse-AI · Weekly AI research brief

This week in LLMs for code

This week’s papers suggest that improving the infrastructure around coding agents—such as co-adapting execution harnesses during training or deploying external trajectory monitors—can substantially boost performance and reduce inference costs on SWE-bench benchmarks.

Read in full

Training the Whole Agent, Not Just the Model

Co-adapting an agent's harness and tools alongside model training yields substantial performance improvements on SWE-bench benchmarks over optimizing the policy alone.

The authors introduce HarnessAdapt-RL, a framework that jointly trains a model policy and adapts its execution harness using teacher models during reinforcement learning. Tested on a 9B base model, the approach improved pass@1 over standard GRPO by 6.4 to 12.6 percentage points across SWE-bench Verified, Multilingual, and Pro. The results indicate that retaining the co-evolved tools and hooks during deployment is essential to preserving these performance gains.

Client angle, caveats & evidence

Client angle Relevant if a client is fine-tuning open-source coding agents and considering whether to co-evolve tool definitions and harness hooks during training.

Caveats Experiments are reported only on a single 9B base model architecture rather than across multiple model scales.

Lingxiao Tang, Wei Tao, Yiwen Guo et al.Preprint, not peer reviewed · 20 community upvotes · code released

Skim it

New benchmark

VerifierBench: Benchmarking the Generation of Executable Verifiers for LLM Agents

It reveals that agent-generated verifiers frequently suffer from false rejections on valid solutions, uncovering a significant reliability bottleneck for autonomous code validation and data synthesis.

Agents need reliable ways to validate their outputs, both when completing tasks and when filtering synthetic training data. We study verifier generation: from the instruction and environment available for solving a task, an agent writes an executable verifier that must accept every valid solution and reject invalid ones.

Client angle, caveats & evidence

Client angle Relevant if a client is designing autonomous software engineering pipelines that rely on LLMs to self-validate outputs or filter synthetic code training data.

Caveats The evaluation is limited to 222 tasks across existing benchmarks and does not assess whether alternative prompting or human feedback reduces false-rejection rates.

Jianing Yin, Ziyi Chen, Hao Wang et al.Preprint, not peer reviewed · 31 community upvotes · code released