Retrieval-augmented generation

This week in Retrieval-augmented generation

This week's papers suggest that increasing model scale, retrieval budgets, or architectural complexity does not reliably boost RAG performance, as multi-source enterprise bottlenecks persist and naive dense retrieval can outperform complex agentic designs.

Top story this week Raise with clients

Surprising result
Reframes the problem

Retrieval Augmented Generation Compositionality

RAG pipeline components do not reliably compound their isolated performance gains when combined, and the best-performing pipeline composition shifts as document corpus size expands.

The authors created LASAGNA, an open-source evaluation framework, to test 450 RAG configurations across eight benchmarks and corpus sizes from 5K to 200K documents. They found that combining individually strong techniques yielded significantly lower improvements than expected, optimizing for retrieval could degrade faithfulness, and winning architectures varied with corpus scale. Consequently, isolated component benchmarks are poor predictors of end-to-end system efficacy.

Client angle, caveats & evidence

Client angle Relevant if a client is stacking multiple RAG components under the assumption that benchmark gains will compound additively at enterprise scale.

Caveats The study tests corpora up to 200K documents, which may not capture dynamics in ultra-large enterprise repositories.

“three retrieval techniques predict a +47% combined improvement but deliver only +17% when used together.”

Ido Levy, Segev Shlomov, Leshem ChoshenPreprint, not peer reviewed · 54 community upvotes · code released

MMEnterprise-Bench: Evaluating Enterprise Agents over Evolving Multimodal Records

Higher retrieval recall achieved through larger models or budgets fails to consistently improve answer accuracy when enterprise workplace data contains temporal and cross-source conflicts.

This paper presents MMEnterprise-Bench, a suite of 1,613 questions testing evidence reasoning, conflict resolution, and unanswerability over evolving multimodal records from simulated projects. Evaluating 10 multimodal LLMs revealed that even top models struggle with cross-source or temporal conflicts and multi-image evidence. The study demonstrates that expanding retrieval budgets or retriever scale does not guarantee better downstream accuracy.

Client angle, caveats & evidence

Client angle Relevant if a client is deploying multimodal agents across frequently revised collaborative repositories containing mixed text and image records.

Caveats The benchmark relies on simulated collaboration projects rather than actual enterprise workplace environments.

“increasing the retrieval budget or employing a larger retriever improves evidence recall but does not consistently translate into higher answer accuracy.”

Seohyun Lee, Jongha Kim, Dohwan Ko et al.Preprint, not peer reviewed · 33 community upvotes · code released

Read in full

Surprising result

Is Complex Agentic Memory Always Needed? Naive RAG Is Stronger Than We Thought

A naive dense retrieval and reranking baseline operating directly over raw conversational history outperformed complex agentic memory architectures across multiple long-term conversation benchmarks.

The authors evaluated Raw-RAG, a standard dense retrieval and reranking approach on raw text history, against complex agentic memory frameworks across four conversational benchmarks. Raw-RAG took first place in six out of seven benchmark-model setups, outperforming top agentic systems by 0.13 to 5.53 points. Analysis showed that keeping raw history preserves relational and dependency information that transformed memory stores lose.

Client angle, caveats & evidence

Caveats Evaluation is restricted to long-term conversation understanding tasks across four academic benchmarks and may not cover other enterprise memory needs.

Zhengye Bao, Shengtao Zhang, Haokai Zhang et al.Preprint, not peer reviewed · 23 community upvotes · code released

Skim it

New benchmark

Toward Agentic Retrieval-Augmented Generation for Enterprise Retail Workflows over Heterogeneous Data Sources

Chaining evidence across disparate structured and unstructured sources is the primary performance bottleneck in enterprise retail workflows, which model scaling alone fails to solve.

The authors introduce RetailBridge, a compositional benchmark based on persona simulations requiring agents to link information across structured records and unstructured text. They also design RetailRAG, a system incorporating schema briefing, routed plan chaining, and plan revision, which outperforms existing agentic baselines. Evaluation across ten LLM backbones shows that multi-source chaining remains the key hurdle regardless of model scale or provider.

Client angle, caveats & evidence

Caveats The benchmark is constructed from persona-driven simulations rather than production enterprise retail transactions.

Jeonghwan Choi, Jibin Hwang, Hayoon Park et al.Preprint, not peer reviewed