# MLP measurement and selection audit — 2026-09-21

This is a retrospective research audit. Promoted winners, thresholds, production defaults and Pine presets are protected. No candidate is promoted. Input hashes, the initial working-tree diff/status, prerequisite logs and characterization outputs are in `results/mlp_measurement_audit_2026_09_21/`.

## Evidence and restrictions

Parameter synchronization passed for all 20 files; Pine checks passed. The system Python lacks NumPy, so substantive checks use `.venv/bin/python`. Chart validation passed structurally for five BTC exports, with warnings. Their latest bars are June 25 or August 21, not September 21. Structural integrity does not prove freshness, publication-time feature availability, or TradingView execution parity.

On September 21 the available Chrome session opened TradingView's chart page and showed **Sign in**. It did not expose the user's strategy state. To obtain fresh parity evidence, sign in, load the unchanged current MLP Pine strategy and promoted presets for COINBASE:BTCUSD at 4H/6H/8H/12H/1D, load complete history, and export chart data and strategy trade lists together from each unchanged chart state. Record strategy settings, timezone, commission, date gates, export time, and whether the last bar is closed. Save them under new names rather than replacing these hashed audit inputs. Run chart validation first, then score/trade parity. The historical work below does not depend on that action.

## Confirmed defects

| Priority | Evidence in original code | Consequence / correction |
|---|---|---|
| High | `strategy_activation_scores.calculate_metrics`: years=`bars/365.25`, ratios scaled by `sqrt(365)` | Intraday annualization is wrong. Shared accounting uses observed calendar duration and observation frequency. |
| High | Same function starts running peak at first post-return equity | First loss disappears from drawdown. Include initial capital in the peak. |
| High | `mlp_results_table.run_backtest` uses realized equity and separate fee formula | Terminal open P&L disappears and floating peaks disappear. Reporting now shares the canonical marked-to-market return series. |
| High | `train_mlp.build_target/pretrain`: forward labels created before time-only split | Labels immediately before validation consume validation prices. Purge by label-end timestamp. |
| Medium | Reporting slices OOS signals before accounting | Carry-in holdings are lost. Compute full-history returns, then select the reporting window. |
| Medium | Date-only `<= TRAIN_END` | Last day's intraday bars are excluded. Research/reporting use explicit inclusive-date or half-open bounds. |
| Medium | `position.diff()` counts entries; exposure uses contemporaneous position | First-window entry can be missed; exposure differs from the intervals earning returns. Count execution events and lag exposure. |
| High | Current chart timestamps are Unix seconds; several loaders call `pd.to_datetime` without a unit | Interprets seconds as nanoseconds near 1970, potentially producing empty filtered data. Numeric loaders now specify seconds. |
| Medium | Training/sweep subperiods slice both bounds before computing returns | Boundary carry-in return and shifted fee disappear. Retain preceding history before applying the measurement start. |

Pre-fix characterization: prices `[100,90,90]` with an entry first and exit last produce −10.9475% net return but only −0.5% reported drawdown. An open `[100,200,150]` trade produces +49.625% canonical P&L while the former report shows zero. See `measurement_characterization.json` and focused regression tests.

Holding signals, data, gates and fee returns fixed, the old versus corrected Calmar values are 0.311→2.161 (4H), 0.214→1.004 (6H), 0.810→3.118 (8H), 0.874→2.025 (12H), and 1.596→1.596 (1D). The cross-timeframe order changes from 1D/12H/8H/4H/6H to 8H/4H/12H/1D/6H. This illustrates the daily-bar annualization bias; it is not a recommendation to select a timeframe on these already optimized results. `metric_version_comparison.csv` retains all before/after metrics, using the original function recovered from the recorded immutable Git commit.

## Execution and accounting contract

Signals and weights are unchanged. The canonical Python approximation uses close-to-close returns multiplied by prior-bar position; subtracts 0.5 percentage points on the first holding return after an entry and on the exit return; and marks terminal open positions to market without a fictitious liquidation fee. This preserves the existing additive commission convention, which differs from charging multiplicative execution-price fees. Drawdown is close-equity drawdown including initial capital, not intrabar-low drawdown. Calendar duration includes the actual elapsed interval represented by each selected return; the first available row uses the median bar interval. Exposure weights lagged positions by elapsed interval duration. Sharpe/Sortino use observed annual bar frequency; no IID or significance claim follows.

Continuous incumbent histories enter no earlier than SCORE_START (2017-12-01). Annual and rolling measurement windows retain carry-in positions; start-date-gated sensitivity replays are separate. Chronological outer experiments start flat with a frozen model and end marked to market. An execution date gate changes trades; a measurement start changes only the accounting window. They must not be silently interchanged.

The last row of each export is conservatively excluded from research because bar-finalization at export time is not proven. Missing leading feature warmup is recorded; required feature columns and internal missing values must pass explicit checks. Feature slot order, including duplicate names in the existing architecture, is preserved. Zero-valued inputs are not treated as proof of valid contemporaneous availability.

## Incumbent asymmetry and evaluation reuse

`run_mlp_deep_sweep.promotable` takes the greater of stored and recomputed incumbent Calmar, P&L/DD and OOS Sortino independently. Those floors can combine incompatible metric/data vintages into a reference no single replay produced. This is confirmed asymmetry, but the audit does not lower production hurdles or rewrite saved metrics.

`run_mlp_adaptive` also reads stored winner Calmar directly. The corrected metric code does not make those old baselines comparable automatically. Resuming adaptive production search against them would still mix vintages; this audit uses isolated comparison code and leaves the production artifacts intact.

Historical WFO periods participate in CMA weight/threshold optimization and later threshold ranking. The period called OOS participates in ranking and promotion and has repeatedly been inspected. Neither is an untouched full-pipeline test. Sweep robustness aggregates only sufficiently active folds and allows two qualifying folds; an omitted adverse or inactive fold can leave a misleading zero negative-fold count. Research outputs retain every period and show coverage separately from measured performance.

Adaptive search repeatedly includes archived seeds and reacts to stored improvements. Current BTC6H archives contain 66 dense bb50 plus 36 bull artifacts, versus three seeds per grouped/random-sparse/hybrid family. These are surviving artifact counts, not complete search ledgers. The hybrid report explicitly omitted a separate threshold sweep. The June architecture comparison also changed population, epochs, target horizon and sweep sample count. Equal additional tuning cannot erase historical selection advantages.

## Saved BTC6H reconciliation

Saved winner: +13,839.5253%, −33.4554% DD, 56 trades. Current full-end-day replay: +2,846.9437%, −50.4976% with SCORE_START entry gate; +4,113.5334%, −50.4976% without it. Removing the gate does **not** reconcile the result. The original archived score-structure runner already used that gate despite reporting the saved result.

The June 25 TradingView trade export contains 56 paired trades and approximately +13,818.22% cumulative return, supporting the existence of the older result under an older state. It does not prove parity with today's chart file.

Exact file: `data/mlp/MLPScores_COINBASE_BTCUSD_2026-06-25_6h.csv`; its last exit is January 16, 2026 at 06:00. All five available BTC trade exports use that June 25 filename date. The bounded historical-data search covered ignored data files, manifests, repo/worktree inventory and tracked BTC6H history. The two tracked June 18 chart versions (`ba6f2dd`, `485d01b`) have different hashes; no matching Git LFS snapshot was found.

| BTC6H data reference | SHA-256 |
|---|---|
| Incumbent training | `d42effbf2e64a6f33fd245d1c6447fd0ed85831fe4e6d463395e026802e5af1b` |
| Archived sparse training | `4262012479e260b95fee1f738a614650e311fdb4b8b35a1adc95ead8ee7ac27b` |
| Current chart CSV | `63107cd6218cb61e3d54a4086f70e23991bdefb7cca482a52f7b1edbedaef1db` |

These establish different source files, not which particular bars or features caused the divergence. Matching historical snapshots have not been recovered in the bounded search. Keep the discrepancy unresolved; do not repair it by overwriting the winner baseline. `legacy_replay_variants.csv` records all five timeframes with gate, final-row and date-end sensitivities before shared metric edits.

Feature-order verification independently reconstructed the exact BTC6H matrix. Artifact order equals current FEATURE_COLS. The duplicated `sopr_norm` slots 32/54 and `cvd_norm` slots 46/50 are intentional preserved artifact structure, not a newly introduced ordering error. Every feature's missing final row lies outside the historical IS discrepancy. Across all 40 legacy replay variants, the accounting correction alone preserves P&L within 2.3e−13 percentage points; changed annualization cannot explain the missing historical profits.

## Feature timing and generalization limitations

Pine imports current daily/weekly macro and on-chain values with `lookahead_off`, without a confirmed higher-timeframe offset. That setting alone does not guarantee equal realtime and reloaded historical values ([TradingView repainting documentation](https://www.tradingview.com/pine-script-docs/concepts/repainting/)). This establishes a risk requiring matched captures, not a measured magnitude of leakage in every feature. FRED/economic imports lack publication-vintage metadata. Historical exports cannot demonstrate what revised macro/on-chain values were available at each decision time. Lower-timeframe RSI and versioned published Pine libraries also require live/export parity evidence. Delayed RSI pivots without negative export offsets are not by themselves evidence of future leakage.

All promoted BTC artifacts record creation on June 25 and training through February 28. The nominal March-onward OOS includes pre-creation observations and selection use. Twelve-hour/daily exports end essentially at creation, leaving almost no observable post-creation evidence. Later 4H/6H/8H observations remain stale and repeatedly inspected. Historical profitability and reproducible contemporary predictive value are different claims.

The legacy activation CUDA scorer is outside the current CPU MLP comparison and remains incompatible: daily scaling, a fixed trade floor, multiplicative signal-bar fees, and fee-free moments differ from corrected CPU accounting. Do not combine its scores with the corrected research tables or infer GPU/CPU parity from this audit. No GPU training/evaluation is used here.

## Corrected incumbent results

The shared contract gives the following IS results through February 28, 2026. The old saved metrics remain unchanged.

| TF | Saved return % | Replayed return % | Calendar Calmar | Replayed close DD % |
|---|---:|---:|---:|---:|
| 4H | 1,601.93 | 1,601.93 | 2.161 | −18.98 |
| 6H | 13,839.53 | 2,846.94 | 1.004 | −50.50 |
| 8H | 35,649.62 | 30,630.24 | 3.118 | −32.16 |
| 12H | 9,447.75 | 9,447.75 | 2.025 | −36.45 |
| 1D | 14,828.23 | 6,420.57 | 1.596 | −41.33 |

Thus 4H/12H historical returns reproduce; 6H/8H/1D do not fully reproduce their saved baselines. The return differences predate the accounting corrections. Missing matching historical chart snapshots leave their exact causal attribution unresolved.

Since the recorded model-creation timestamp, within each timeframe's available export:

| TF | Incumbent return % | Entries / exits | Buy-and-hold % | Trend % | Cash % |
|---|---:|---:|---:|---:|---:|
| 4H | 21.80 | 3 / 2 | 27.00 | 11.09 | 0 |
| 6H | 0.00 | 0 / 0 | 28.75 | 10.04 | 0 |
| 8H | 16.76 | 1 / 1 | 29.81 | 10.08 | 0 |
| 12H | No coverage | — | — | — | — |
| 1D | No coverage | — | — | — | — |

These are tiny, stale, already inspected samples with different retained last bars. Returns include carry-in and open-position marking. In particular, 6H inactivity preserved cash but incurred an observed opportunity cost versus both invested benchmarks. The positive 4H/8H samples do not establish generalization or superiority to buy-and-hold. Empty post-creation windows are absent evidence, not zero-return successes.

The largest three positive equity-contribution trades account for 36.46%, 42.76%, 29.47%, 35.05%, and 42.97% of all positive trade contributions for 4H/6H/8H/12H/1D. Zeroing their paid return intervals leaves full-history returns of 802.47%, 1,431.12%, 14,389.23%, 3,621.86%, and 1,826.36%. These are concentration diagnostics, not feasible alternative strategies; original signal schedules/exposure remain fixed and compounding changes mechanically. The ledger records each trade's net return, initial-equity P&L contribution and open/closed status.

Canonical tables live in `results/mlp_measurement_audit_2026_09_21/baselines/final/`:

- `metrics.csv`: 3,728 incumbent/benchmark rows, including every annual period and complete month-end rolling 12/24-month window, with calendar Calmar, DD, exposure, entries/exits and coverage. Partial first/last annual periods carry exact bounds.
- `trade_ledger.csv`: 484 trades; contribution totals reconcile with full-history equity.
- `concentration_diagnostics.csv`: largest-one and largest-three removal for all five timeframes.
- `regime_attribution.csv`: above/below causal SMA and warmup attribution, conditional regime exposure/duration and entry/exit counts. Masked returns and whole-calendar ratios are explicitly noncontiguous attribution, not a standalone regime portfolio.
- `reconciliation.csv`: saved, replayed and June TradingView figures with source hashes.

## Bounded archived reevaluation

Nine current-schema grouped/random-sparse/hybrid artifacts (three seeds each), plus the incumbent, were replayed on the same current BTC6H frame. Each also received exactly 128 extra threshold configurations from the same seeded quantile-search rule. This is 1,280 evaluations, with selection based only on historical IS and each selection written before its later-window measurement. All historical folds and the already-inspected later period remain in the output, including inactive periods. Saved/recommended-threshold results remain alongside this comparison.

With saved thresholds, none of the nine beats the frozen incumbent's IS Calmar 1.004. With equal additional tuning, random-sparse seed9191 reaches Calmar **1.054**, return **4,709.96%**, and DD **−56.88%**, versus frozen incumbent **1.004 / 2,846.94% / −50.50%**. The equally retuned incumbent reaches only 0.672 Calmar: a small global budget need not rediscover its extensively optimized frozen configuration. The random-sparse candidate loses **12.84%** in the already-inspected later period, with four entries.

The broad claim that every alternative is intrinsically inferior does not survive this fairer measurement/tuning check. A deployable superiority claim also fails: drawdown worsens, later evidence is weak/adverse, weights were historically selected with unequal budgets, and the common data differs from every model's training snapshot. Do not promote. `archived_candidates.csv`, `archive_tuning_trials.csv`, ten frozen selection files and `protocol.json` preserve the complete bounded screen.

## Predeclared chronological comparison

Protocol: `docs/mlp_pipeline_comparison_protocol_2026_09_21.md`. BTC4H, unchanged 55→16→8→1 architecture, expanding/three-year/five-year histories, two fixed seeds, four outer calendar periods. Each preceding history allocates its final six months to threshold/seed calibration; supervised early stopping and ten-bar label purging happen strictly before that. The CMA objective uses preceding fit periods only. All twelve selected configurations were frozen before any outer evaluation.

Completed in **61.4 seconds** on deterministic, single-thread CPU: **24 fitted artifacts, 664 actual supervised epochs, 1,536 CMA evaluations, 3,072 calibration trials**. Budget maxima and all actual counts are saved. No architecture changes, new signals, broad adaptive sweep or promotion occurred.

| Outer period | Expanding return % | Three-year return % | Five-year return % | Trend return % |
|---|---:|---:|---:|---:|
| 2023 | 14.68 | −10.11 | 15.33 | 82.32 |
| 2024 | 59.79 | 4.12 | 144.74 | 70.24 |
| 2025 | −6.03 | −10.08 | −46.72 | −19.34 |
| 2026 through retained August export | 5.67 | −4.30 | 21.87 | 11.09 |
| Compounded reset-window diagnostic | 81.97 | −19.46 | 83.29 | 178.11 |

Every approach falls below trend in the compounded diagnostic. Expanding/three-year/five-year beat trend in 1/1/2 of four periods respectively; the predeclared requirement was at least three. Each also fails the requirement of three completed trades in every outer period. **All three are inconclusive on coverage and fail to demonstrate the required advantage.** This bounded comparison provides no basis to favor a trailing history or replace a promoted model.

The descriptive 30-day paired block bootstrap gives relative-wealth intervals versus trend of roughly [−85.7%, +164.7%], [−94.7%, +26.2%], and [−77.3%, +80.9%]. These wide intervals do not correct for prior inspection, repeated search or revised features. They support caution rather than an inference of equivalence. Current promoted-model results are separately labeled hindsight-selected and never enter candidate selection.

Each outer window starts flat and marks terminal positions without liquidation. Compounding independent windows is therefore a **reset-window diagnostic**, not a continuously executable portfolio return. Full 2022 bear-market evaluation is absent because that origin lacks five complete years of the eligible data; 2023–2026 coverage must not be described as exhaustive regime robustness. The BTC4H file also has a 16-hour gap ending August 10, 2018 at 16:00 and an eight-hour gap ending October 25, 2025 at 20:00. Calendar accounting spans them but cannot recreate missing decisions.

Detailed metrics, buy-and-hold/cash/trend/hindsight references, daily returns, every fitted artifact and frozen selection are in `results/mlp_measurement_audit_2026_09_21/pipeline/`. See `results.csv`, `decision_summary.csv`, `daily_returns.csv` and `input_manifest.json`.

## Verification, recommendation and reproduction

Independent arithmetic reconstructed all 28 outer model/benchmark rows from the frozen artifacts and matched returns, DD and execution counts. The verifier checks model/schema/selection hashes, 24 fits, 3,072 calibration trials, 1,280 archived trials and all 824 protected artifacts. No winner, data export, default configuration or Pine preset changed. Baseline CSVs reproduce byte-for-byte across reruns; initial/reviewed evidence remains available alongside the canonical final directory.

Recommendation: retain the current production state while resolving historical data provenance and collecting matched fresh chart/trade exports. Use the corrected accounting contract for future comparisons, but do not silently replace stored promotion baselines: adaptive and max-stored promotion logic still compare against old vintages. A separately authorized policy migration would need compatible replays and an explicit treatment of unresolved baselines. Future evidence should freeze model, tuning budget and feature snapshots before observation; this audit has not created an untouched future sample.

Reproduce with the project virtual environment, using new output directories:

```bash
.venv/bin/python tools/run_mlp_incumbent_audit.py --output /tmp/mlp_incumbent_reproduction
.venv/bin/python tools/run_mlp_pipeline_comparison.py --out /tmp/mlp_pipeline_reproduction
.venv/bin/python tools/summarize_mlp_pipeline_comparison.py /tmp/mlp_pipeline_reproduction
.venv/bin/python tools/compare_mlp_metric_versions.py
.venv/bin/python tools/verify_mlp_measurement_audit.py
.venv/bin/python -m pytest tests/ -q
```

The verifier checks this audit's canonical saved paths; the runners deliberately refuse nonempty output directories. Source snapshots and a consolidated evidence manifest capture dependencies beyond the initial runner manifests. Full test output and verification counts are stored beside the evidence.

Final verification: **209 tests passed, 2 skipped**, with three existing warnings (unregistered slow marker and intentionally older-schema test artifacts). `git diff --check` passed. No repository lint/typecheck configuration was found. Independent review found no outer-selection leakage; remaining source-provenance and data-gap limitations are disclosed above.