RepoCortex
Plate ivMeasured agent eval / 30 Django issues

A shorter route
through the code.

Measured on the oldest 30 held-out Django fixes with claude-sonnet-5-5. Numbers are means per issue. Accuracy is about the same with and without the tool (top-1 function 0.43 vs 0.40, one issue). The tool cut input tokens by 16 percent, or 27 percent when the agent is told to call it first.

Measured

30 Django holdout fix rows, model claude-sonnet-5-5, same checkout and tools in every arm. The ticket number is never shown. This evaluation measures locating functions. Separate repair trials have not established a learning advantage in fixing bugs.

Without RepoCortex

search, read, repeat
Input tokens
63,885
Turns
9.6
Files read
2.8

With RepoCortex

-16% input tokens
Input tokens
53,888
Turns
8.3
Files read
2.4

Told to call it first

-27% input tokens
Input tokens
46,361
Turns
7.5
Files read
2.2

The eval record

Means per issue
Oldest 30 held-out fixes. The last column is locate_bug alone, with no model.
MetricWithoutWithCall it firstlocate_bug alone
Top-1 function correct0.400.430.430.47
Top-1 file correct0.630.630.630.67
Input tokens per issue63,88553,888 (-16%)46,361 (-27%)0
Tool calls per issue11.39.89.11
Files opened per issue2.82.42.20
Cost per issue$0.063$0.054 (-15%)$0.049 (-22%)$0

Accuracy is within noise across arms: one issue out of 30. locate_bug alone names the right function first 0.47 of the time against 0.40 for the agent searching by itself, in under a second and with no model call.

One run, #36277 Add GeneratedField(db_persist=False) support on PostgreSQL 18+

Measured trace
locate_bug
{
  "report": "Add GeneratedField(db_persist=False) support on PostgreSQL 18+",
  "k": 10
}

Measured run on this ticket. With locate_bug the agent made 21 tool calls over 13 turns, used 137,942 input tokens, and named the gold function DatabaseFeatures.is_postgresql_18 second. Without it: 23 tool calls, 15 turns, 166,134 input tokens, gold named third.