The research question

Does the rule still look useful after we stop choosing it? An agent can appear to improve simply because we keep selecting the most attractive past result. We need a boundary between the observations used to choose a rule and the later observations used to evaluate it.

This lab uses two small sequences of invented, already cost-adjusted outcomes. It is a demonstration of evaluation discipline, not a statistical claim about a real strategy. We will deliberately select a challenger that wins on the development slice and disappoints on the later slice.

Time: approximately 30 minutes. You need: the standard-library Python environment. The script is self-contained.

Step 1 — Write the experiment before running it

Open a text note and record: Choose the highest development total among the declared candidates. Freeze that selection. Report its later total alongside the baseline. Do not change the rule after inspecting later results.

We are not defining a deployment threshold. A small lesson fixture cannot justify one. Our decision here is whether the evaluation remains honest when the interesting result becomes inconvenient.

The development observations come first in time. The later observations come afterward. Both candidates must use the same observation windows and accounting assumptions. Never compare one rule's easiest week with another rule's hardest week.

Step 2 — Run the frozen comparison

Save this script as lab_04.py.

from statistics import mean

# Invented net outcomes in currency units; no prices or future data are queried.
development = {
    "baseline": [1, 1, -1, 1],
    "challenger": [3, -1, 2, 1],
}
later = {
    "baseline": [-1, -1, 1, -1],
    "challenger": [-2, 1, -2, 0],
}
selected = max(sorted(development), key=lambda name: sum(development[name]))
print(f"selected_from_development={selected}")
print(f"development_total={sum(development[selected]):.2f}")
for name in ("baseline", selected):
    values = later[name]
    cumulative = 0
    peak = 0
    worst_drawdown = 0
    for value in values:
        cumulative += value
        peak = max(peak, cumulative)
        worst_drawdown = max(worst_drawdown, peak - cumulative)
    print(f"later {name}: n={len(values)} total={sum(values):.2f} "
          f"mean={mean(values):.2f} drawdown={worst_drawdown:.2f}")
difference = sum(later[selected]) - sum(later["baseline"])
print(f"challenger_minus_baseline={difference:.2f}")
print("conclusion=challenger_did_not_improve_this_later_fixture")

Expected output:

selected_from_development=challenger
development_total=5.00
later baseline: n=4 total=-2.00 mean=-0.50 drawdown=2.00
later challenger: n=4 total=-3.00 mean=-0.75 drawdown=3.00
challenger_minus_baseline=-1.00
conclusion=challenger_did_not_improve_this_later_fixture

The drawdown calculation begins with cumulative profit/loss of zero. It measures the largest fall from a previous peak in these currency-unit outcomes. It is not a percentage account drawdown, because we did not define an account balance.

Step 3 — Keep the disappointing result

Write a four-sentence research note. State the question, selection procedure, later result, and limitation. A useful version is: The challenger was selected using development totals. On the later synthetic slice it lost 3 units versus the baseline's loss of 2. This fixture does not support improvement. Four invented outcomes do not support an inference about a market strategy.

Do not relabel the later slice as development and reuse it as if it were still unseen. If you now invent a third rule after inspecting the failure, it needs a new later evaluation. Record how many variations you tried, including the ones you abandoned.

Step 4 — Audit for information leakage

For a real historical study, ask what was knowable at each decision timestamp. Were features calculated using future bars? Were instruments included only because they survived to the end? Did a later correction overwrite the version of a quote that was available then? Did you tune exits after seeing the outcome?

A chronological split helps preserve ordering; it does not automatically solve these problems. If positions overlap or labels extend into the evaluation window, a clean date boundary may still share information. State the observation and holding-window rules explicitly.

Troubleshooting

If the selected rule changes, check the declared development data and tie-breaking order. If drawdown is negative, you likely reversed peak - cumulative. If you want to show confidence intervals, stop and define the sampling unit first: repeated marks on one position are not independent completed examples. This lab intentionally avoids a misleading significance number.

Evidence and limits

The outcome is reproducible because the candidate list, selection rule, cost interpretation, and later slice are fixed. That is necessary for a fair comparison, but it is not sufficient evidence of skill. Future research would need more independent observations, multiple market conditions, realistic execution assumptions, and a record of all candidate rules considered.

Completion check: A reader can identify which values selected the rule, which values evaluated it, and why the later result must remain in the record.

Next edition: Give the agent a boundary, not a blank check.

Primary reference

  • Python statistics documentation documents the arithmetic mean used in the report. The split design, synthetic outcomes, and interpretation above are an original teaching exercise, not a published empirical trading result.