AI agents · Pulse-AI

This week in AI agents

This week's papers suggest that autonomous agents struggle with practical execution challenges, showing failures in real-world deployment feasibility for operations research and an inability to autonomously recover from execution errors in computer-use tasks.

OR-License: Evaluating LLM Agents as Real-World Operations Research Practitioners

It benchmarks LLM agents against real-world client engagements and reveals that while some models beat human experts on conditional objective quality, zero match human performance once deployment feasibility is accounted for.

The authors introduce OR-License, a benchmark derived from 31 industrial operations research engagements totaling $3.68 million, paired with a client simulator (ORSim) to evaluate end-to-end problem translation and feasibility. Across 11 frontier LLMs, no model matched the human expert reference in Deployment-Adjusted Quality (DAQ), with the top configuration reaching an 88.39% feasibility rate and 0.87 DAQ against the 1.0 expert baseline. Trajectory analysis showed task-equalized research-chain completion rates ranging between only 24% and 45% due to formulation and implementation failures.

Client angle, caveats & evidence

Client angle Relevant if a client is considering deploying LLM coding agents to independently formulate and solve complex enterprise optimization or operations research problems.

Caveats The benchmark relies on a simulated client (ORSim) for information disclosure rather than live human stakeholders, and evaluates a relatively small sample of 31 problem instances.

“several exceed the expert reference in conditional objective quality, but none matches its DAQ; the strongest primary Claude Code configuration achieves 88.39% strict feasibility and a DAQ of 0.87”

Peisong Wang, Feixue Liu, Yize Meng et al.Preprint, not peer reviewed · 52 community upvotes · code released

Skim it

New benchmark

Agent World Model in the Wild: Computer-Use Agent Environment Synthesis

It establishes a fully open-source pipeline for generating controllable, multi-interface computer-use environments and shows that training exclusively on this synthetic data improves out-of-distribution agent performance.

The paper introduces WildAWM, a framework powered by open-source models that synthesizes executable environments supporting GUIs, CLI, and model context protocol services. The authors implement execution-based quality gates to create 341 long-horizon, multi-step environment-task pairs that challenge even top models. Training two Qwen models strictly within these synthetic worlds yielded substantial improvements across eight out-of-distribution computer-use benchmarks.

Client angle, caveats & evidence

Caveats The synthetic task suite is limited to 341 environment-task pairs, and top model performance is cited for an unverified or unreleased model ('GPT-6-Astra').

Zhaoyang Wang, Qianhui Wu, Haorui Wang et al.Preprint, not peer reviewed · 101 community upvotes · code released

Skim it

New benchmark

Lost in Recovery: Demystifying Recovery Behavior of Computer-Use Agents via Goal-Oriented Action Replay

It formalizes transition-consistent recovery for computer-use agents and demonstrates that while ideal error recovery substantially unlocks latent task completion, current agents fail to autonomously recover from execution failures.

The authors present ReTrace, an evaluation framework designed to analyze autonomous, model-based error recovery in computer-use agents using goal-oriented action replay and controlled error injection. The study tests whether agents can return an environment to an uncorrupted target state without relying on resets or rollbacks. Findings indicate that while effective recovery yields measurable end-to-end task gains, existing computer-use agents do not adequately exploit their latent capabilities to self-recover.

Client angle, caveats & evidence

Caveats The abstract does not report specific quantitative recovery rates or empirical performance numbers for the evaluated models.

Byeonghun Bang, Sangwoo Park, Jinheon Baek et al.Preprint, not peer reviewed