Guides · before you hand over an account
How do you verify an AI trading agent’s track record?
Before you connect an AI agent to money, ask one thing: can anyone verify what it did? Most claims you will meet cannot be verified. This guide gives nine questions that sort a claim you can verify from a claim you cannot. Each answer says what it proves, and what it does not.
1. Is it a live record, a paper record, or a backtest?
A live record lists real orders, placed with real money, in a real account. A paper record lists pretend orders in a practice account. A backtest replays a rule on old data to see what it would have done. Only a live record shows what actually happened.
Rules have a name for results that were not actually achieved: hypothetical. The CFTC (17 CFR 4.41) says hypothetical results “do not represent actual trading” and “are designed with the benefit of hindsight.” The SEC’s Marketing Rule (17 CFR 275.206(4)-1) defines “hypothetical performance” as results “not actually achieved by any portfolio.” A backtest fits that description. So does a paper record, because no real order was placed.
Paper records also leave out real costs. Alpaca’s documentation (updated 2026-07-07) says paper trading “does not account for” the “Market impact of your orders,” “Price slippage due to latency,” and regulatory fees. Some agent tools start in paper mode. Alpaca’s MCP server “defaults to paper trading,” per its docs dated 2026-09-09, so a record made in that mode is a paper record (Alpaca’s entry on our venue ledger).
What this answer proves: a backtest or paper record shows what a rule did on old data or in a practice account. It does not show what happens with real orders. Ask which kind you are looking at, and get the answer in writing.
2. Where can the trades be seen?
Trades can be seen in one of three places: on a public blockchain, through a read-only link, or only in a screenshot. The first lets you verify the record yourself. The second gets you part of the way. The third proves nothing to an outsider.
We tag each kind of evidence with one of three labels. Recomputable means the record is public and you can recompute it. Vendor-claimed means it is the seller’s own figure. Unverifiable means no public record exists. Each label comes with a line on what would prove the claim.
On a public chain. A record is on-chain when its trades sit on a public blockchain. Any address’s transactions are broadcast to the whole network, says ethereum.org (updated 2026-07-25). Swaps that settle on a public chain are public the same way. Label: recomputable. It has two limits. An address shows that the address made the trades, not who controls it, and not that it is the seller’s only wallet. And turning trades into dollar results needs an outside price source. What would prove a claim: the wallet address, its full list of transactions, and an outside price source.
Through a read-only link. MetaTrader’s “investor” login lets a viewer “see the account status, analyze prices ... but not trade.” Exchanges offer read-only API keys. They show that trades came from that account. They do not show that it is the only account, or what share of the seller’s capital it holds. Label: vendor-claimed, because the seller chose which account to show you. What would prove it: a read-only link for every account the seller runs.
In a screenshot or dashboard. Broker account statements are private: FINRA Rule 2231 requires quarterly statements to the customer. FINRA’s “Relationship Investment Scams” (2024-09-10) says “depictions of skyrocketing account balances often are fake.” The CFTC (2025-03-19) says generative AI makes it easier “to create false images, voices, videos, live-streaming video chats, social media profiles” and “malicious websites designed to look like financial trading platforms.” Label: unverifiable. What would prove it: a public record, or read-only access you can open yourself.
What this answer proves: only an on-chain record gives a stranger something to recompute. Every other kind depends on the seller.
3. Does it show the whole period, with the losses?
A record should show a long, unbroken stretch and its worst drawdown. A drawdown is a fall from a peak to the next valley. A record that starts after a bad stretch, or shows only a good one, is not a full record. Ask for the start date, the end date, and the worst drawdown.
US rules for people who market results set a standard. The SEC’s Marketing Rule asks for 1-, 5- and 10-year periods “with equal prominence.” CFTC rule 17 CFR 4.25 requires disclosing the worst peak-to-valley drawdown over five years. A professional reporting standard called GIPS (2020 edition, from the CFA Institute) asks for at least five years of results, building to ten. It also asks for three-year volatility, meaning how much results swing over three years.
Set periods, shown with equal prominence, leave less room to show only the flattering window. A new agent may have no record that long. If so, the honest answer is the exact dates it does have.
What this answer proves: a long, dated record with its worst fall shown tells you how the agent behaved in bad stretches. A short or undated one tells you little.
4. How many accounts, wallets or strategies were tried?
Ask how many were tried, not only how many worked. One good record can be the best of many tries. Without the count, you cannot tell.
Think of a contractor who hands you one reference. Is it their only job, or the best of twenty? You would ask. Ask the same here.
Researchers have put a number on it. Bailey, Borwein, López de Prado and Zhu (Notices of the AMS, May 2014) write: “After only twenty trials, the researcher is expected to find one specification that passes the AIC criterion.” In plain words, a specification is one version of a strategy, and try enough versions and one looks good. They add that “a backtest which does not report the number of trials N used to identify the selected configuration makes it impossible to assess the risk of overfitting.” Read it at the Notices of the AMS (PDF).
Bailey and López de Prado, in “The Deflated Sharpe Ratio” (2014-07-31), write that “researchers and investment managers tend to report only positive outcomes, a phenomenon known as selection bias.”
Professional standards close this gap. GIPS puts all fee-paying discretionary accounts in a composite, which is one combined record, and keeps closed accounts in the history. Its own words: “These requirements prevent firms from cherry-picking their best performance.” The FTC’s 16 CFR 437.4 says an earnings claim must state “The number and percentage of all persons who purchased the business opportunity” who achieved it.
What this answer proves: a record with the full count of accounts and strategies, closed ones included, is harder to cherry-pick. A record with no count could be the best of many.
5. Are fees, spreads and slippage included?
Ask whether the results are net or gross. Gross ignores costs. Net takes them out. Costs include fees, spreads and slippage. A spread is the gap between the best offer to buy and the best offer to sell. Slippage is an order filling at a worse level than expected because the market moved while it was sent.
The SEC’s Marketing Rule requires net results to be shown with at least equal prominence to gross. GIPS requires results to be labelled gross or net. A CFTC advisory (2024-01-25) says: “Also consider the impact fees, spreads, and subscription costs would have on returns.”
Paper records skip these costs. Alpaca’s docs (updated 2026-07-07) say paper trading “does not account for” market impact, slippage, and regulatory fees. Some research tests skip them too. The StockBench test (arXiv 2510.02209) does not model costs. LiveTradeBench (arXiv 2511.03628, 2025-11-05) does not account for fees and spreads.
What this answer proves: a net result with its cost assumptions stated can be compared with what you would keep. A gross result, or a paper result, cannot.
6. If it is a language-model agent, does the record start after the model’s training cutoff?
A language model learns from text up to a cutoff date. If a test covers dates before that cutoff, the model may already have read what happened. Ask two things: what is the model’s training cutoff, and does the record start after it?
Lopez-Lira, Tang and Zhu (arXiv 2504.14765, version 2 of 2025-12-15) write: “Large language models (LLMs) cannot be trusted for economic forecasts during periods covered by their training data.”
Li and co-authors (arXiv 2510.07920, 2025-10-09) write that “dazzling back-tested returns evaporate once the model’s knowledge window ends.”
Both are published research we have not re-run. What would confirm them: an outside re-run of each test. A live record that starts after the cutoff does not have this problem.
7. How does it compare with doing nothing?
Ask what the same money would have done if you had bought the same assets once and held them. That is the buy-and-hold baseline. A result is easiest to judge next to it.
StockBench (arXiv 2510.02209, version 2 of 2026-03-02) tested models over 82 trading days. It reports that “most models struggle to outperform the simple buy-and-hold baseline.” It did not model costs.
FINSABER (arXiv 2505.07078, version 6 of 2026-06-26) reports that “previously reported LLM advantages deteriorate significantly under broader cross-section and over a longer-term evaluation.” In plain words, the advantages shrink when the test uses more assets and a longer window.
Both are published research we have not re-run. What this answer proves: a comparison on the same dates, the same assets and with costs tells you whether the agent added anything. A result with no baseline cannot answer that.
8. Who is allowed to make this claim, and what must they disclose?
Some people who market results must follow rules. The rules say what they must show and what they may not say. Ask who is making the claim and which rules apply. We cannot tell you which rules cover a given seller. As of 2026-10-01, we found no regulator page that addresses a retail user’s own agent placing trades.
The SEC’s Marketing Rule says an adviser must have “a reasonable basis for believing it will be able to substantiate upon demand.” CFTC rule 17 CFR 4.25 also requires the results of other accounts to be disclosed. The FTC rule in question 4 (16 CFR 437.4) covers the same idea for earnings claims.
FINRA Rule 2210(d)(1)(F) says communications “may not predict or project performance.” Treat a forecast of future results as a reason to ask more questions.
Even GIPS verification, the standard’s outside check, covers a firm’s procedures, not one series of results.
What this answer proves: rules give you a list of things to ask a seller to show. They do not tell you whether the seller is covered, or whether the result is good.
9. What do regulators say the tell-tale signs are?
Regulators warn about three things: a promise of no risk, a promise of guaranteed results, and account balances shown as images. Each is a claim that a public record cannot back up.
A CFTC advisory (2024-01-25) says: “Fraudsters are exploiting public interest in artificial intelligence (AI) to tout automated trading algorithms, trade signal strategies, and crypto-asset trading schemes that promise unreasonably high or guaranteed returns.” It adds: “AI technology can’t predict the future or sudden market changes.”
The SEC, NASAA and FINRA (2024-01-25) give example pitches: “Our proprietary AI trading system can’t lose!” and “Use AI to Pick Guaranteed Stock Winners!” Label for both pitches: unverifiable. What would prove them: nothing, because no public record can show a future with no risk. The same notice says: “Claims of high guaranteed investment returns with little or no risk are classic warning signs of fraud.”
False AI claims are also enforced. In SEC press release 2024-36 (2024-03-18), two advisers settled for “false and misleading statements about their purported use of artificial intelligence,” with “$400,000 in total civil penalties.” Chair Gensler said: “Such AI washing hurts investors.” For fake balances and fake platforms, see question 2.
A short checklist to verify a record
Copy these nine lines.
- Is this a live record, a paper record, or a backtest?
- Where can the trades be seen: on a public chain, through a read-only link, or only in an image?
- Does it cover the whole period and show the worst drawdown?
- How many accounts, wallets or strategies were tried, and where are the closed ones?
- Are fees, spreads and slippage taken out?
- If it uses a language model, does the record start after the training cutoff?
- How does it compare with buying once and holding, over the same dates?
- Who is making the claim, and what must they disclose?
- Does it promise no risk, promise guaranteed results, or rest on a screenshot?
What this guide can’t tell you
This guide cannot tell you a good agent from a bad one. It tells you a claim you can verify from one you cannot. A record you have verified says what happened. The CFTC says: “AI technology can’t predict the future or sudden market changes.” We promise nothing, forecast nothing, and recommend nothing.
No independent measure exists for the whole field. No independent, rigorous, dated aggregate measurement of live, net-of-cost results for retail AI trading agents was found by anyone we asked to look. The only agent-level figures are research benchmarks, with the limits above.
Every rule, quote and finding on this page belongs to its source. We computed none of it.