# MLP Training & Weight-Finding Runbook

A reference for **which script to run for which activity**. The MLP pipeline has
two distinct stages — *training* (produce neural-net weight artifacts) and
*sweeping* (find the best entry/exit thresholds for those artifacts and promote a
winner) — plus one orchestrator that runs both end-to-end and prints the results
table at the end.

> **You almost always want the orchestrator** (`run_mlp_train.py`). The standalone
> scripts below are for when you need finer control or are debugging one stage.

---

## TL;DR — the one command that does everything

```bash
# Train (3 seeds) → threshold sweep → promote winners → regenerate Pine presets
# → print regression table → ntfy when done. For ONE asset, all/selected TFs.
python3 tools/run_mlp_train.py \
  --assets COINBASE_BTCUSD --tfs 4H 6H 8H 1D \
  --seeds 3 --sweep --promote --presets --table --notify
```

This runs all four phases (training → sweep → presets → table) and is the script
that "does both and runs mlp_results at the end." Drop `--tfs` to do all 5.

Preview first without executing anything:

```bash
python3 tools/run_mlp_train.py --assets COINBASE_BTCUSD --seeds 3 --sweep --promote --dry-run
```

---

## Activity → script map

| I want to… | Script | Notes |
|---|---|---|
| **Do everything for an asset/TF** (train + find better thresholds + table) | `tools/run_mlp_train.py` | Orchestrator. Use `--sweep --promote --presets --table --notify`. |
| **Train new weight artifacts** (more seeds, new arch, regime filter) | `tools/run_mlp_train.py` (multi-seed) or `tools/train_mlp.py` (single run) | Produces `mlp_weights_{ASSET}_{TF}_{tag}_seed{N}.json`. |
| **Find better thresholds + promote** over already-trained artifacts | `tools/run_mlp_deep_sweep.py` | Sweeps entry/exit thresholds across ALL seed artifacts, promotes best. |
| **See what's trained / which artifact is winning** | `tools/mlp_status.py` | Source of truth for winner per TF + arch consistency + Pine sync. |
| **Compare backtest results across TFs** | `tools/mlp_results_table.py` | PnL %, Max DD %, Win Rate (IS + thru-date + OOS). |
| **Check score parity vs TradingView** | `tools/check_mlp_parity.py` | Run with `--asset/--tf` only (auto-detects winning artifact). |
| **Check trade-list + P&L parity vs TV** | `tools/compare_tv_trades.py` | Needs a TV trade-list CSV export. |
| **Validate chart-data integrity** | `tools/validate_chart_data.py` | Run after every TV re-export. |
| **Update Pine presets from winners** | `tools/generate_pine_mlp_presets.py` | Orchestrator does this via `--presets`. |

---

## The three scripts in detail

### 1. `tools/run_mlp_train.py` — orchestrator (preferred)

Runs up to four phases in order. Each post-training phase is opt-in via a flag:

| Phase | Flag | What it calls | Output |
|---|---|---|---|
| 1. Train | (always) | `tools/train_mlp.py` per (asset, TF, seed) | weight artifacts |
| 2. Sweep | `--sweep` | `tools/run_mlp_deep_sweep.py` per combo | winner CSV (with `--promote`) |
| 3. Presets | `--presets` | `tools/generate_pine_mlp_presets.py` | updated `.pine` |
| 4. Table | `--table` | `tools/mlp_results_table.py --asset …` | printed results table |
| Notify | `--notify` | `tools/ntfy.py` | phone push |

Key flags:

```
--assets COINBASE_BTCUSD ...   # default: all 4
--tfs 4H 6H 8H 12H 1D          # default: all 5
--seeds 3                      # seeds per combo
--seed-start 101               # first seed; subsequent += 101 (404 → 404,505,606)
--tag bb50                     # artifact name tag
--regime bull|bear|sideways|all  # filter Phase-1 to MVRV regime; auto-suffixes tag (bb50→bb50_bull)
--hidden 16 8                  # hidden layer sizes (BTC default 16 8 → 55→16→8→1)
--samples 80000                # sweep candidates per TF
--skip-existing                # don't retrain seeds whose artifact already exists
--dry-run                      # print commands only
```

> **Tag must match between train and sweep.** The sweep globs
> `mlp_weights_{asset}_{tf}_{tag}_seed*.json`. The orchestrator handles this for
> you (and auto-suffixes the tag with the regime, e.g. `bb50_bull`). If you run the
> two stages by hand, pass the matching `--extra-weights` glob to the sweep.

### 2. `tools/train_mlp.py` — single training run

Phase 1 (PyTorch supervised pretrain) + Phase 2 (CMA-ES fine-tune) for **one**
asset/TF/seed. Use when you want a dedicated, full-CPU, thoroughly-optimized run
(the orchestrator caps `--es-workers 2` so it can run many jobs in parallel; a
standalone run uses ~50% of CPUs and optimizes the weights more thoroughly).

```bash
python3 tools/train_mlp.py \
  --data "data/mlp/COINBASE_BTCUSD, 360.csv" --asset COINBASE_BTCUSD --tf 6H \
  --hidden 16 8 --fold-objective robust --l2 0.01 --seed 101 \
  --out strategies/params/mlp/mlp_weights_COINBASE_BTCUSD_6H_bb50_seed101.json

# Fast pipeline smoke test (tiny budget, ~seconds):
python3 tools/train_mlp.py --data data/COINBASE_BTCUSD-6H.csv --smoke
```

### 3. `tools/run_mlp_deep_sweep.py` — threshold finder + promoter

Sweeps the entry/exit threshold space across **all** the trained seed artifacts
you point it at, scores each (artifact, thresholds) candidate by the composite
metric with the parity-correct simulation, and (with `--promote`) writes the best
gate-passing one to the winner CSV. This is "finding better weights" — it picks
the best artifact *and* its best thresholds.

```bash
python3 tools/run_mlp_deep_sweep.py \
  --asset COINBASE_BTCUSD --timeframes 6H \
  --samples 80000 --promote \
  --extra-weights strategies/params/mlp/mlp_weights_COINBASE_BTCUSD_6H_bb50_seed*.json
```

- `--promote` only writes a winner if it **beats** the existing baseline (safe; no regressions).
- **First-ever sweep for a new asset** (no prior winner exists): use `--force-promote`
  instead — `--promote` alone never writes the first winner. See CLAUDE.md
  "New Asset Workflow".

---

## Recommended workflows

### Improve specific timeframes (e.g. all but 12H)

```bash
python3 tools/run_mlp_train.py \
  --assets COINBASE_BTCUSD --tfs 4H 6H 8H 1D \
  --seeds 3 --sweep --promote --presets --table --notify
```

To try the **regime-aware** lever (filters Phase-1 training to bull-regime bars;
broke the prior 4H/8H ceilings — see "Before you burn GPU hours" below):

```bash
python3 tools/run_mlp_train.py \
  --assets COINBASE_BTCUSD --tfs 4H 8H \
  --seeds 3 --regime bull --sweep --promote --presets --table --notify
```

### Add more seeds to an existing TF without retraining what's there

```bash
python3 tools/run_mlp_train.py --assets COINBASE_BTCUSD --tfs 6H \
  --seed-start 404 --seeds 3 --skip-existing --sweep --promote --table
```

### New asset, first ever sweep (ETH/SOL/LINK)

```bash
# Train first…
python3 tools/run_mlp_train.py --assets COINBASE_ETHUSD --seeds 3
# …then force-promote the first winners (no prior baseline to beat):
python3 tools/run_mlp_deep_sweep.py \
  --asset COINBASE_ETHUSD --timeframes 4H 6H 8H 12H 1D \
  --samples 80000 --force-promote \
  --extra-weights strategies/params/mlp/mlp_weights_COINBASE_ETHUSD_*_bb50_seed*.json
```

---

## After any run — verify (the self-verification loop)

```bash
python3 tools/mlp_status.py            # winner per TF, arch consistency, Pine sync
python3 tools/mlp_results_table.py --asset COINBASE_BTCUSD   # IS/thru/OOS metrics
# After re-exporting from TradingView:
python3 tools/validate_chart_data.py   # chart-data integrity (run FIRST)
python3 tools/check_mlp_parity.py --asset COINBASE_BTCUSD --tf 6H   # score parity
```

Objective metric for promotion = **composite (Calmar-based)**; cross-asset quality
metric = **Calmar ratio** (IS PnL is NOT comparable across assets). Guardrail =
TV parity (`check_mlp_parity` + `compare_tv_trades`) must pass before trusting a
winner.

---

## Before you burn GPU hours — read the ceiling note

Per CLAUDE.md "Shallow MLP Ceiling": for BTC, the **seed / architecture / objective
levers are largely exhausted**. More seeds or wider arches have repeatedly produced
zero promotions. The one lever that broke prior ceilings was **`--regime bull`**
Phase-1 filtering (4H: 0.229→0.254 Calmar; 8H: 0.727→0.877). Genuine further gains
most likely require **new orthogonal features** (order-flow, funding rate, hourly
on-chain) or a **different model class** (gradient-boosted trees, attention/LSTM) —
not another seed sweep.

**Current BTC snapshot** (as of 2026-06-24 — run `mlp_status.py` / `mlp_results_table.py`
for live numbers; the orchestrator's `--table` does this automatically):

| TF | IS PnL % | Max DD % | Win Rate | Winning artifact |
|---|---|---|---|---|
| 4H | +249.5% | −55.5% | 46% | bb50_bull_seed202 |
| 6H | +2,557.9% | −28.3% | 50% | bb50_bull_seed606 |
| 8H | +194.1% | −57.5% | 29% | bb50_seed303 |
| 12H | +10,515.5% | −32.7% | 47% | bb50_seed101 |
| 1D | +1,605.6% | −18.8% | 46% | bb50_bull_seed202 |

12H is the strongest; 4H and 8H are the weakest (highest drawdown, 8H low win rate)
and the most likely to benefit from a new lever — but mind the ceiling note before
expecting a seed sweep alone to move them.