Commentary: Keeping Humans in the Loop

The Funds Getting AI Right Are Not the Ones With All the Hype

Article by Michael Sury, associate professor of practice in finance

A paper published last fall gave its findings a name that has stuck with me: “Profit Mirage: Revisiting Information Leakage in LLM-based Financial Agents.” The idea is simple and damaging. When you test a large language model (LLM) trading strategy on historical data, the returns look extraordinary. Sometimes thousands of percent. Then you move the test window one step past the model’s training cutoff, into data it has never seen, and the returns collapse. The best-performing agents lose half their performance or more. Many hit statistical zero. The alpha was never there. The model was reciting history it had already memorized.

That paper should be required reading for every venture investor pouring money into artificial intelligence-driven trading systems. It probably won’t be.

I run a graduate course in analytic finance and machine learning at The University of Texas at Austin. My students spend a semester doing something unglamorous and genuinely hard: processing hundreds of thousands of earnings conference call transcripts collectively totaling millions of sentences, using LLMs to score what management is actually doing with language. Tone. Sentiment. Internal contradictions between the prepared remarks and the Q&A. Whether the CFO sounds like someone who knows something is wrong. That work, tedious and methodologically demanding as it is, produces real signal. It has nothing in common with the papers claiming LLMs can predict stock returns.

The issue shows up very clearly when you try to replicate these results. In my class this year, one group tested an LLM-based strategy that “predicted” Nvidia’s 2023 run almost perfectly — until we moved the evaluation window just past the model’s training cutoff. The performance didn’t degrade gradually; it fell apart. What looked like signal was mostly recall.

That’s the core problem with much of the return-prediction work right now. These models have absorbed years of post-hoc explanations about why stocks moved. When they seem to anticipate a move, they’re often pattern-matching against narratives they’ve already seen rather than forming a forward-looking view. The “profit mirage” paper put numbers on this: performance drops of 50% or more out of sample. But the underlying issue becomes very clear once you run the experiment yourself.

More parameters and better prompts will not fix this. Fluency about the past and genuine prediction are different cognitive operations, and fine-tuning isn’t going to close that gap.

There is a second issue that worries me more than the bad backtests. Research by Alejandro Lopez-Lira, a finance professor at the University of Florida, presented to European regulators in June 2025, found that LLM trading strategies could introduce systemic risks precisely because agents built on similar architectures exhibit correlated behavior in response to comparable market signals. Dozens of funds building signal generators on top of the same two or three foundation models, all ingesting the same earnings releases and Bloomberg feeds, will converge on the same positions.

We saw what crowded systematic positioning looks like in slow motion during the summer 2025 quant wobble, when long-short equity funds lost an estimated 4% in less than two months while broad markets climbed 8%. LLM-driven crowding would not telegraph itself so politely. The Commodity Futures Trading Commission (CFTC) acknowledged the stakes when it launched a dedicated Innovation Task Force this past March, covering artificial intelligence and autonomous systems alongside crypto and prediction markets. Whether it moves fast enough is a separate question.

So, what are the funds getting right?

Man Group’s AlphaGPT is drafting trade rationales and surfacing anomalies in alternative data before they reach production signals. At D.E. Shaw, a federated system lets individual desks build custom tools while a central team enforces prompt logging and model-use policies. Bridgewater’s system uses dashboards that force portfolio manager sign-off on any suggested trades.

A theme emerges: These groups are not using LLMs to generate buy-and-sell signals autonomously. They are using them to do labor-intensive cognitive work faster. Reading, flagging, drafting, synthesizing. But they are not cutting the “human in the loop.”

This is exactly what my students are doing and why it works. The methodology requires some care, though.

When an LLM scores the tone and sentiment of a 2019 earnings call, the model is asked about linguistic properties of a text: hedging language, passive construction, whether the syntax grows more complex when the subject turns to inventory levels. The question posed to it has nothing to do with what a stock subsequently did. There is no direct feedback loop between the score and the price outcome. Think of it like a ruler: a ruler measuring the height of historical buildings does not know which buildings later commanded higher rents, and yet it produces a dataset a researcher can usefully test.

Of course, the contamination concern does not disappear entirely. An LLM trained on the web has absorbed not just original transcripts but also the subsequent commentary, analyst notes, and news coverage that discusses whether what management said turned out to be accurate. When the model scores a CFO’s guidance as “hedged” or “evasive,” it may be drawing partly on that downstream knowledge, importing a posterior about outcomes rather than reading the sentence cold. This is a real and underappreciated problem in the literature.

The contamination is, however, greatly reduced compared with direct return prediction. When you ask an LLM whether a stock will go up, the model pattern-matches against outcomes it has memorized at scale. When you ask it to identify internal contradiction between two sentences in the same transcript, or to detect unusual shifts in syntactic complexity, it is working primarily from the text itself. The connection to outcome knowledge is much more indirect and much noisier. And it varies: large-cap companies with dense retrospective coverage probably are more exposed to this problem than smaller companies with thinner analyst footprints. Researchers can design around it by weighting their validation toward less-covered segments of the market and by running diagnostics that test whether model scores perform materially worse in out-of-sample periods with thin coverage.

The honest claim is that the measurement approach substantially reduces the lookahead problem. It does not eliminate it. Responsible practice means understanding where the contamination risk is highest and building validation tests that probe for it directly.

Research acceleration is the other real application, and it too requires some candor. Multiagent systems that coordinate across the quantitative research workflow — handling data access, factor construction, code generation, backtesting, and report writing in sequence — can compress the cycle from economic intuition to tested evidence substantially. One of my graduate students and I have a recent preprint on exactly this kind of framework, designed to emulate parts of a quant research team while maintaining full computational audit trails of every step taken.

The p-hacking objection is fair and worth confronting directly. Faster iteration on the same dataset is a more efficient path to spurious factors if you are not disciplined about it. More tests, more false positives. The answer is preregistration, genuine out-of-sample validation, and multiple-testing corrections, none of which are new ideas. What well-designed agentic frameworks can add is auditability: every tool call, every intermediate result, every branch in the analysis is logged. The garden of forking paths has always been the dirty secret of quantitative research. At least now, it can be made visible. The Securities and Exchange Commission and CFTC have signaled that existing supervisory and recordkeeping obligations apply to AI-driven workflows in a technology-neutral way, which is another way of saying audit trails are no longer optional.

The funds threading all this successfully are treating LLMs as very fast, very well-read junior researchers. They do the heavy lifting or reading, flagging, and drafting. They surface things a human might have missed in a stack of 10,000 documents. But ultimately, they do not decide.

Of course, that is a less exciting pitch than 3,000% backtested returns. It is also, based on everything in the published research and in my own classroom, the version of this story that survives contact with real markets.