37 papers matching “benchmark”
VerifierBench: Benchmarking the Generation of Executable Verifiers for LLM Agents
It reveals that agent-generated verifiers frequently suffer from false rejections on valid solutions, uncovering a significant reliability bottleneck for autonomous code validation and data synthesis.
- New benchmark
Last Secure Code Benchmark: Is the Code You Vibe Secure?
Frontier coding agents frequently pass functional requirements but rarely produce code that is also secure when evaluated against real vulnerabilities.
- New benchmark
CLASHarness: Benchmarking Multi-Agent Conflict in Computer Use via a Controllable and Verifiable Generation Pipeline
Multi-agent computer-use workflows suffer severe reliability breakdowns during conflict resolution, with frontier models failing more than half of basic coordination tasks such as releasing shared locks.
- New benchmark
SWE-MISBEHAVIOR: Benchmarking and Mitigating Boundary Violations in Software Coding Agents
Coding agents frequently commit boundary-violating misbehaviors like corrupting environment integrity, but a lightweight intervention framework can substantially cut these errors while boosting task completion.
- New benchmark
A Real-World CAD Segmentation Benchmark: Comparing Rule-Based and Learning-Based Approaches through Hole Detection
It introduces a human-annotated engineering CAD benchmark for hole detection and reveals that learning-based methods degrade under domain shift while rule-based alternatives maintain robust generalization.
- New benchmark
DecepEval: A Benchmark for Evaluating Deception in LLM Agents
Evaluations across nine frontier LLMs reveal that external inducements like pressure, incentive, opportunity, and conflict consistently increase agent deception across tasks, even for models with low baseline deception rates.
- New benchmark
Correct Code, Broken Contributions? SWE-CC: Benchmarking Repository Policy Compliance for Coding Agents
It demonstrates that even when coding agents generate functionally correct patches, they violate repository-specific contribution and governance policies 43.1% of the time.
- New benchmark
- Surprising result
- Reframes the problem
Event Cameras for Melt-Pool Monitoring in Additive Manufacturing: A Benchmark and a Cross-Machine Transfer Analysis
Event-based monitoring matches dense-frame accuracy while providing a ~380x reduction in sensor readout and enabling spiking models with 15x lower modeled inference energy.
- New benchmark
Detecting Defects that Matter: An Application-Driven Benchmark for Anomaly Detection in Manufacturing and Retail Logistics (VAND 4.0 Challenge)
- New benchmark
OR-License: Evaluating LLM Agents as Real-World Operations Research Practitioners
It benchmarks LLM agents against real-world client engagements and reveals that while some models beat human experts on conditional objective quality, zero match human performance once deployment feasibility is accounted for.
- New benchmark
- Reframes the problem
MMEnterprise-Bench: Evaluating Enterprise Agents over Evolving Multimodal Records
Higher retrieval recall achieved through larger models or budgets fails to consistently improve answer accuracy when enterprise workplace data contains temporal and cross-source conflicts.
- New benchmark
Agent World Model in the Wild: Computer-Use Agent Environment Synthesis
It establishes a fully open-source pipeline for generating controllable, multi-interface computer-use environments and shows that training exclusively on this synthetic data improves out-of-distribution agent performance.
- New benchmark
Lost in Recovery: Demystifying Recovery Behavior of Computer-Use Agents via Goal-Oriented Action Replay
It formalizes transition-consistent recovery for computer-use agents and demonstrates that while ideal error recovery substantially unlocks latent task completion, current agents fail to autonomously recover from execution failures.
- New benchmark
Toward Agentic Retrieval-Augmented Generation for Enterprise Retail Workflows over Heterogeneous Data Sources
Chaining evidence across disparate structured and unstructured sources is the primary performance bottleneck in enterprise retail workflows, which model scaling alone fails to solve.
- New benchmark
DICL-Qwen3VL: Decoupled Confidence Calibration for Interpretable Workwear Detection
It decouples visual uncertainty from compliance reasoning ambiguity in vision-language models while outputting explicit evidence to aid human industrial safety checks.
- New benchmark
More Methods, Better Agents? A Reality Check on Self-Distillation in Agentic Reinforcement Learning
A large-scale evaluation demonstrates that complex self-distillation methods in agentic RL do not reliably outperform standard GRPO, with extra compute failing to guarantee better results.
- New benchmark
- Surprising result
FDE-Gym: The Last Mile of AI Delivery
Frontier models fail dramatically at real-world customer AI delivery tasks, successfully resolving only 14 of 95 scenarios when required to uncover unstated business constraints.
- New benchmark
- Reframes the problem
MAI-Gym: An Open All-in-One Platform for GUI Agent Evaluation and RL Training across Mobile and Computer Environments
Reinforcement learning trained on mocked GUI applications can scale to thousands of parallel instances and successfully transfer to physical devices, nearly doubling real-phone success rates.
- New benchmark
CheckerBench: Can Long-Horizon Agents Synthesize Static-Analysis Checkers?
Current coding agents achieve only a 32.30% mean Pass@1 when tasked with the end-to-end synthesis and refinement of reusable static-analysis checkers across real-world repositories.
- New benchmark
Accurate but Not Humble: Evaluating Epistemic Humility in LLM Agents under Knowledge Conflict
Higher agent task accuracy does not correlate with epistemic humility, as high-performing models often detect knowledge conflicts during execution but fail to escalate or report unresolved uncertainty in incorrect answers.
- New benchmark
- Surprising result
Learn2Play Bench: How Well Do LLM Agents Learn from Experience in Unfamiliar Environments?
Retaining raw action and feedback trajectories enables better in-context agent learning in novel environments than summarizing experiences into rules, though agents still lag behind human exploration capabilities.
- New benchmark
RIT-RAG: Navigating Document Corpora with Retrieval-Induced Trees
It improves agentic RAG accuracy on large enterprise documentation by dynamically constructing multi-document sub-trees from initial retrieval results to guide LLM exploration.
- New benchmark
Code Understanding is a Bottleneck for Coding Agents
It identifies that deep code understanding, rather than edit size, acts as the primary difficulty bottleneck and error driver for software engineering agents.
- New benchmark
VeriHarness: Scaling Agentic Verification for Long-Horizon Tasks
Equipping the generating LLM with workspace tools and verification skills enables autonomous verification and revision of long-horizon tasks without reference answers or rubrics.
- New benchmark
CUA-SWE: When Computer-Use Agents Meet Visual Software Engineering
A unified benchmark evaluates agents on realistic software engineering workflows requiring concurrent code editing and graphical user interface interaction.
- New benchmark
Re-ranking and Late Interaction Drive Retrieval Quality: A Controlled Comparison of RAG Strategies for Scientific Question Answering
- New benchmark
QuantCode Model: Specializing Language Models for Executable Algorithmic Trading Code
It demonstrates that domain-specific code specialization can significantly boost task and backtest success while simultaneously triggering capability-retention failures in general structured tool calling.
- New benchmark
Retrieval Augmented Generation Compositionality
RAG pipeline components do not reliably compound their isolated performance gains when combined, and the best-performing pipeline composition shifts as document corpus size expands.
- Surprising result
- Reframes the problem
Is Complex Agentic Memory Always Needed? Naive RAG Is Stronger Than We Thought
A naive dense retrieval and reranking baseline operating directly over raw conversational history outperformed complex agentic memory architectures across multiple long-term conversation benchmarks.
- Surprising result
Training the Whole Agent, Not Just the Model
Co-adapting an agent's harness and tools alongside model training yields substantial performance improvements on SWE-bench benchmarks over optimizing the policy alone.
Robust Logical Anomaly Detection under Capture Shifts via Synthetic Anomalies
It demonstrates that leading normal-reference industrial anomaly detectors suffer severe performance drops under simple capture condition shifts and provides a synthetic anomaly framework that retains high AUROC under perturbations.
- Surprising result
Inverting Multi-Vector Visual Document Indices
It demonstrates that multi-vector visual document indices stored in vector databases can be inverted to reconstruct document images, exposing 45% of sensitive tokens.
- Surprising result
UNREAL: Unifying Retrieval and Long-Context with a Single Model
It introduces a single model-internal selection mechanism adding under 500K parameters to a frozen LLM that improves both multi-hop corpus retrieval recall and long-context inference speed.
Sharpening Tax in Post-Training
RL post-training of LLM agents improves single-shot accuracy but systematically reduces solution coverage under repeated sampling.
- Surprising result
AGO AI Quality Gate: Evidence-First Release Decisions for Retrieval-Augmented Generation
It introduces an evidence-first RAG release framework that reduces unsafe production deployments by treating LLM judges as fallible and quantifying regression risk probabilistically.
- Reframes the problem
Beyond the Model: Demystifying Harness Effects in Software Engineering Agents
It isolates the concrete performance impact of specific agent wrapper components, proving that harness design choice can add over 7 percentage points of performance independent of the base LLM.
MCFR: A Mask-Guided Coarse-to-Fine Regression Framework for Robust Multi-Variant Board-to-Board Connector Assembly
It demonstrates an average real-world insertion success rate of 99.25% for multi-variant board-to-board connectors in smartphone assembly by combining mask priors with photometric refinement.