A new model or agent framework ships every week right now. Anthropic, OpenAI, DeepSeek, and the rest are releasing fast enough that just keeping up has become its own task. Everyone is vibe coding, pointing an agent at a codebase and letting it run. Alongside that, there’s a steady stream of stories about an agent given a sandbox to work in doing something nobody quite expected, wandering past the boundary it was supposed to stay inside of. That combination is what actually got me curious about something narrower than “is this safe”: when we hand an agent room to act on its own, how much does the shape of that access change the quality of what it actually produces, separate from the risk question everyone else is asking?
That curiosity pulled me into reading about agentic AI and AI4SE/MSR research, and I came across SWE-agent and SWE-bench. SWE-agent’s core finding stuck with me: giving an LLM a small set of well-designed tools beats giving it raw, unstructured access, at least for the task of fixing code. I’d recently completed a research internship under Mitacs GRI, and the research side of it intrigued me. It’s a different way of understanding a system, one that’s actually useful for building or using that system better later, not just shipping something that works. Finding a question this specific and this testable felt like exactly the kind of thing worth chasing on my own.
But fixing code and investigating code are different jobs. A patch is either right or wrong. An explanation of “why did CI fail more in August” has no single verifiable output, and the reasoning path matters as much as the answer. Whether SWE-agent’s finding transfers to this kind of diagnostic task was a real open question, not something I assumed going in. The literature actually disagreed with itself: MetaGPT’s Data Interpreter found the opposite pattern on data-analysis tasks, and a paper posted days before I found it argued the effect depends heavily on how capable the underlying model is.
So I built both versions and tested them properly.
What I built
Two versions of the same agent, sharing one reasoning loop (a standard reason-act-observe cycle) and the same five investigative questions. What differs between them is only what “act” means.
Variant A gets six narrow, typed tools: check CI history over a date range, fetch a commit’s diff, query issues and pull requests, compare a metric across two periods, look up which contributors’ commits are most associated with later bug fixes, and compute PR review-approval rates. Each one is a plain Python function against Postgres, returning a typed result, nothing more.
Variant B gets one capability instead: a Docker-sandboxed environment where it can write and run its own Python or SQL against the same underlying data, read-only, with a timeout.
Both were run against three real, actively developed repositories
(tokio-rs/tokio,
eslint/eslint,
sqlalchemy/sqlalchemy), the same five
test questions, and six different LLMs spanning four providers and multiple
capability tiers, from
deepseek/deepseek-v4-flash up to
openai/gpt-5-mini and
anthropic/claude-haiku-4.5.
Under the hood: repository history is pulled via GitHub’s GraphQL and REST APIs, staged in object storage, cleaned and aggregated with PySpark, and lands in Postgres with a hand-written schema (no ORM). The ReAct loop itself is custom-built rather than a framework like LangGraph, models are called through OpenRouter so the same loop works across providers unmodified, and every step is traced through Langfuse. No tool in Variant A ever builds SQL by string concatenation with caller-controlled values; everything goes through parameterized queries.
How I actually tested this, not just ran it once
A single run of a stochastic system proves very little, so a few things mattered to me here:
- The research question and method were written down and dated before any results existed, with an amendment log for anything that changed afterward, so I couldn’t quietly adjust the story after seeing the numbers.
- Every answer was graded by a blinded LLM judge against independently computed ground truth, never told which model or variant produced the answer.
- A second, independently-sourced judge re-graded a large overlap of answers, and I checked whether the two judges actually agreed before trusting either one. They agreed within a point 58% of the time on the full dataset, lower than I expected, and I’m reporting that plainly rather than hiding it. What matters more: the headline result held under both judges, independently, for every model tested.
- The entire 60-run comparison (6 models x 2 variants x 5 questions) was replicated from scratch partway through the project, after I found and fixed a real gap in Variant A’s tool design, so the final numbers reflect the tools working as intended, not a known-broken version of them.
The moment that made this feel like real research
Partway through, one specific question (“is code review actually catching problems, or are bugs slipping through?”) was going wrong for almost every model, regardless of which variant it used. My first guess was that the question itself was leading, and a follow-up test confirmed that mattered for some models. But it didn’t explain everything, so I went back and read an actual failed run step by step instead of theorizing further.
What I found: no tool in Variant A actually computed a PR approval rate. One tool returned a raw list of pull requests, each with its own approval count, and the model was expected to aggregate that itself across however many came back. Most models didn’t. It wasn’t a reasoning failure. It was a real gap in what the “curated” tools actually curated.
I added a sixth tool, then re-ran the entire study, because patching only the affected condition would have made the comparison unfair for everything else that was already measured under the old tool set. That single fix changed the entire headline finding: what looked like an effect concentrated in one model turned out to be a broad, consistent pattern once the tool gap was closed.
What I found
Averaged across all six models: Variant A scored 6.53 out of 8, Variant B scored 5.50 out of 8. Curated tools produced a real, consistent quality advantage, present in every model tested, not just one.
But the honest, more interesting part is that this effect is not uniform. The size of
the gap ranged from +0.2 to +2.0 points depending on the model, and it does not
follow a simple “weaker models benefit more” story. The single largest gap belonged
to gpt-5-mini, the best-performing model in the entire study under both conditions.
Giving the strongest model curated tools still helped it the most, in absolute terms.
That complicates any tidy narrative, and I think that’s more useful to report than a
clean story would have been.
Variant B also cost meaningfully more, in tool calls needed to reach an answer, ranging from about 1.1x to 4.2x depending on the model, tracking less with model “strength” and more with how well-suited a model is to writing its own investigative code versus over-exploring or investigating without an efficient stopping point.
What’s next
This is a pilot study, three repositories and five questions, not a large-scale claim, and I’m not going to pretend otherwise. What it is: a real, pre-registered, blinded, replicated comparison that answers a specific, previously untested question in a way I can defend. The natural next step is scaling it up, more repositories, more questions, and ideally human-rater validation alongside the LLM judges, aimed at a proper venue rather than a solo pilot.
Try it yourself
Live demo: ask the curated-tools agent a real question about one of the three repositories, or step through an actual saved trace from the study comparing both variants side by side on the same question.