Edition 29: The AI trading winner that still lagged the market

An AI model can finish first on a trading-research leaderboard and still earn less than the market it studied. Quant-Arena makes that distinction unusually visible. Its highest-scoring model configuration returned 10.7% in the reported 2024 test, while the gross equal-weight market reference returned 17.1%.

There is a second surprise. Another configuration did beat that reference. It was not the configuration selected to represent its model in the headline comparison.

For a builder wondering whether a trading assistant has discovered something useful, these are different questions: did it win the benchmark, did its account grow, and did it justify the risk? This investigation shows how to read those answers without running a bot or buying a model subscription.

Three rows that change the story

Here are selected iterative-research results from the authors' public leaderboard, checked November 2, 2026. “Effort” is the model's reasoning setting. The score is a benchmark grade, not a percentage return. Maximum drawdown is the largest reported fall from a previous account peak.

Original selection and presentation of published results. Source: Quant-Arena's iterative leaderboard and paper Table 2. These are historical evaluation results, not independently verified funded trades. The strategy evaluator charges trading costs; the market reference is gross of costs.

The two Sol rows are especially instructive. The xhigh configuration has the higher benchmark score and the smaller drawdown. The medium configuration has the higher return. Saying “Sol returned 8.6%” without identifying the selected setting hides a materially different result elsewhere in the same experiment.

Neither row is automatically the right choice for a person. A return column cannot tell you how much temporary loss someone can tolerate. A low drawdown cannot tell you whether a cautious strategy will meet a return objective. The comparison is useful because it makes that disagreement explicit.

What the agent actually submitted

Quant-Arena's October 21 preprint tests a research loop, rather than asking a chatbot for tomorrow's stock tip. Models write stock-scoring programs, inspect feedback from 2022–2023 and freeze a strategy for a sealed 2024 assessment. The paper describes nine models, 38 iterative configurations and at most five candidate evaluations per run. The common portfolio targets 75 stocks from a selected A-share universe.

A-share means shares traded on mainland Chinese exchanges. A stock score is a way to order candidates; it is not an order fill. A backtest applies a strategy to historical observations under declared execution assumptions. Here those assumptions include cash accounting, lot sizes and simulated market impact, the estimated price effect of an order.

That is a more demanding task than generating plausible finance prose. It still differs from a real account facing a broker, delayed data, rejected orders and changing market conditions. This publication inspected the study and documentation; it did not rerun the authors' market-data experiment.

“No model beat the market” needs a qualifier

Read alongside Table 2, the paper's headline conclusion holds for each model's highest-total-score configuration. It does not hold for every setting: Sol at medium effort returned 18.4%, above the 17.1% reference. It also had the 35.9% drawdown shown above. Preserve both facts when describing the finding.

It would be equally misleading to announce that AI has therefore solved trading. Choosing a favorable configuration after reading the assessment results changes the question. You have identified an interesting historical observation. You have not demonstrated that a selection rule would have chosen it before seeing that outcome.

Imagine receiving only the training-period records and having to choose a setting in advance. What rule would you use? Highest research score? Lowest research drawdown? A fixed effort level? Write that rule before looking at a new assessment period. Otherwise the human selecting the winner can introduce hindsight even if the agent's own research access was restricted.

Why the grade and the return disagree

The benchmark documentation distinguishes total scores, individual window metrics and strategy outputs. Its scoring system combines several objectives. Its public research-data release is also narrower than the evaluation release: a documented quote subset is not the entire sealed benchmark.

Think of the total score as an answer to a designed evaluation question. It is not an account statement. If you want to compare return, inspect the return column. If you want to understand losses along the way, inspect the account path and drawdown. If you want to compare research capability, inspect the permitted tools, feedback and selection procedure as well.

This is not peculiar to finance. A coding agent can pass many tests while missing one critical behavior. A research assistant can supply several correct citations while failing to answer the question. Aggregation is useful for organizing evidence, but the individual requirements still matter.

Cash can be a strategy, and a cost

The project research page highlights a mechanism behind the leading model's outcome: avoiding exposure during unfavorable conditions. Holding cash can limit losses while also missing a recovery. It changes how much of the market's movement an account experiences.

That gives a builder a better next question than “Which model is smartest?” Ask what action generated the result. Did the program choose better stocks, avoid a falling market, trade less, or simply take less exposure? Those mechanisms call for different follow-up experiments.

For example, if a timing rule appears valuable, compare it with a simple fixed timing rule on the same future observations. Keep sizing, costs and available information aligned. If you change the model, execution assumptions and position sizes together, an improved outcome will not identify which change helped.

A relevant alternative is Agent Market Arena, revised October 30, 2025. That separate research team studies recurring decisions on incoming market information. Its execution protocol updates simulated portfolios. “Live” in that context describes the arrival of market information; it must not be silently converted into evidence of broker-confirmed funded profit. It is a different evaluation design, not another row to add to Quant-Arena's table.

Check the claim yourself, without installing anything

You need a browser and somewhere to keep a short note. Open the linked leaderboard, select the iterative section and locate the three model-and-effort rows above. Then read the paper's Table 2 caption, financial-evaluation section and the documentation's limitations.

Record this result in your own words:

The highest total score does not identify the highest-return configuration. The selected Astra row trails the gross market reference. Sol medium exceeds that reference but has a much larger drawdown than Sol xhigh. These reported historical results do not establish funded performance.

Now do one arithmetic check. Subtract the displayed market return from each displayed strategy return: 10.7 minus 17.1 is -6.4 percentage points; 18.4 minus 17.1 is +1.3 percentage points. The published excess column gives -6.3 and +1.4 respectively. Do not silently replace either set: subtraction of rounded display values need not reproduce an excess calculated before rounding. The precise reconciliation requires underlying unrounded values.

That small discrepancy is a useful completion check. You should be able to distinguish a copied metric from your own calculation and explain why neither proves a strategy's reproducibility. If a page changes, retain the inspection date and source version instead of combining a new leaderboard with an old screenshot.

The limit that a sealed year cannot remove

The platform documentation explicitly warns that restricting data access during a run does not establish the absence of historical information from model pretraining. It also requires an account and supplied SDK for participation. Reading the public tables is reproducible without those permissions; recreating the full experiment is not promised here.

There are further reasons to keep the conclusion bounded. This is a preprint from the benchmark's authors, who also list their own proprietary strategy, Kepler-1. A gross market reference and cost-charged strategy are not identical cost treatments. One assessed year does not reveal behavior across all market regimes. We have not audited the underlying data, obtained independent repeated runs or measured provider, setup, review and maintenance costs.

The useful next step is not to copy a winning row into a trading account. Choose one result you find persuasive and ask for the frozen code identity, complete execution assumptions and another genuinely unexamined period under a predeclared selection rule. That request separates an interesting research lead from a claim that an assistant can reliably earn money.