# 4H MLP Parallel Robust Experiments - 2026-06-18

## Parallelization Decision

The slow part of `tools/train_mlp.py` was phase 2, where CMA-ES evaluates every candidate in the population sequentially. A Python process at roughly `100%` CPU with low total CPU load means one core is doing most of the work.

Subagents are not the right tool for this loop. CMA-ES needs synchronous generation-by-generation feedback: ask for a population, evaluate every candidate, then tell CMA-ES the ordered fitness values. The correct parallelization point is inside each generation, across independent candidate fitness evaluations.

Implemented for training:

```bash
--es-workers N
```

Behavior:

- default `--es-workers 0` uses auto sizing: `50%` of logical CPUs, capped by popsize,
- `--es-workers 1` uses the old sequential execution path,
- `--es-workers >1` evaluates each CMA-ES population with a `ProcessPoolExecutor`,
- the worker pool uses Python's `spawn` start method to avoid forking after PyTorch/MPS initialization,
- artifacts record `phase2.es_workers`.

Implemented for sweeps:

```bash
--workers N
```

Behavior:

- default `--workers 0` uses auto sizing: `50%` of logical CPUs, capped by candidate count,
- `--workers 1` uses sequential evaluation,
- `--workers >1` evaluates threshold candidates with a `ProcessPoolExecutor`,
- each worker keeps its own scored-frame cache so repeated weight files are not reloaded for every candidate,
- summary CSVs record the actual worker count used.

Observed result:

- Prior comparable 8x4 robust run without phase-2 workers took about `105s`.
- 16x8 robust with `--es-workers 8` took `23s`.
- 8x4 robust `target-k=6` with `--es-workers 8` took `16s`.
- 4H 10k deep sweep with `--workers 0` auto-resolved to `8` workers and took about `29s`.

## Validation

Parallel smoke command:

```bash
.venv/bin/python tools/train_mlp.py \
  --data data/COINBASE_BTCUSD-4H.csv \
  --smoke \
  --fold-objective robust \
  --es-workers 2 \
  --out /tmp/tradingbot25_4h_parallel_smoke.json
```

Result:

- Unsandboxed smoke passed.
- Phase 1 used `device=mps`.
- Phase 2 logged `evaluating CMA-ES population with 2 worker processes`.
- Artifact recorded `phase2.es_workers=2`.

Parallel sweep validation:

- Sequential 4H smoke sweep and `--workers 2` unsandboxed sweep produced identical top-25 CSV hashes: `501ece15513de029c9cee4772cb9a5a83e4256335fdef626687f0f27f58c9c65`.
- Sandboxed `--workers 0` falls back to sequential evaluation and reports `workers=1`.
- Unsandboxed 4H 10k sweep used `workers=8`.

The sandbox blocks Python process-pool semaphore checks, so parallel training and parallel sweeps should run unsandboxed in this environment. The tools fall back to sequential mode when the worker pool is unavailable.

## Experiment A: Robust 16x8

Command:

```bash
.venv/bin/python tools/train_mlp.py \
  --data data/COINBASE_BTCUSD-4H.csv \
  --hidden 16 8 \
  --phase1-epochs 250 \
  --es-generations 300 \
  --es-popsize 32 \
  --es-sigma 0.04 \
  --l2 0.01 \
  --fold-objective robust \
  --es-workers 8 \
  --seed 404 \
  --out strategies/params/mlp/mlp_weights_COINBASE_BTCUSD_4H_robust_16x8_seed404.json
```

Training result:

- Phase 1 device: `mps`
- Phase 2 workers: `8`
- Best fitness: `0.9738860296`
- Fold Calmars: `[0.0270, -0.2461, 0.2259, 0.1292, 0.2698, 0.0000]`
- Mean fold Calmar: `0.0676`
- Min fold Calmar: `-0.2461`
- Robust diagnostics:
  - drawdown penalty: `0.0718`
  - P&L/DD penalty: `0.5000`
  - trade penalty: `0.1667`
  - negative-fold penalty: `0.2077`
  - fragility penalty: `0.3860`
  - fragile bars at `0.05`: `2`
  - fragile bars at `1.0`: `143`

Sweep command:

```bash
.venv/bin/python tools/run_mlp_deep_sweep.py \
  --timeframes 4H \
  --samples 5000 \
  --top-k 2000 \
  --extra-weights strategies/params/mlp/mlp_weights_COINBASE_BTCUSD_4H_robust_16x8_seed404.json
```

Decision:

- Do not promote.
- The best sweep row came from the current canonical 4H artifact, not the 16x8 candidate.

## Experiment B: Robust 8x4, Target-K 6

Command:

```bash
.venv/bin/python tools/train_mlp.py \
  --data data/COINBASE_BTCUSD-4H.csv \
  --hidden 8 4 \
  --target-k 6 \
  --phase1-epochs 200 \
  --es-generations 300 \
  --es-popsize 24 \
  --es-sigma 0.04 \
  --l2 0.01 \
  --fold-objective robust \
  --es-workers 8 \
  --seed 505 \
  --out strategies/params/mlp/mlp_weights_COINBASE_BTCUSD_4H_robust_8x4_k6_seed505.json
```

Training result:

- Phase 1 device: `mps`
- Target horizon: `6`
- Phase 2 workers: `8`
- Best fitness: `-0.1776225532`
- Fold Calmars: `[1.2644, 0.0455, 0.5876, 1.4369, 0.2746, 0.0000]`
- Mean fold Calmar: `0.6015`
- Min fold Calmar: `0.0000`
- Robust diagnostics:
  - drawdown penalty: `0.0000`
  - P&L/DD penalty: `0.1667`
  - trade penalty: `0.1667`
  - negative-fold penalty: `0.1667`
  - fragility penalty: `0.0180`
  - fragile bars at `0.05`: `0`
  - fragile bars at `1.0`: `9`

Sweep command:

```bash
.venv/bin/python tools/run_mlp_deep_sweep.py \
  --timeframes 4H \
  --samples 10000 \
  --top-k 3000 \
  --workers 0 \
  --extra-weights strategies/params/mlp/mlp_weights_COINBASE_BTCUSD_4H_robust_8x4_k6_seed505.json
```

Best sweep row:

| Source | Entry | Exit | Conf | Trail | Trades | P&L % | Max DD % | Calmar | P&L/DD | Sub1 Calmar | Sub2 Calmar | WFO Min | WFO Neg | OOS Trades | OOS P&L % | OOS Sortino | Fragile 0.05 | Fragile 1.0 |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| `local_2` | `-65` | `90` | `-10` | `15` | `127` | `1993.28` | `-47.50` | `0.1335` | `0.0` | `0.1001` | `0.1725` | `0.1357` | `0` | `3` | `2.37` | `0.2613` | `0` | `9` |

Best positive `P&L/DD` rows did not clear WFO/subperiod/holdout gates. The sweep found `0` target-k6 rows clearing promotion-like gates.

Decision:

- Do not promote.
- This is the best 4H direction so far because it clears tight fragility, WFO-negative-fold, and holdout direction on the top row.
- It still fails the hard drawdown cap: `Max DD=-47.50%`, so `P&L/DD Ratio=0.0`.

The canonical 4H artifact remains unchanged:

```text
strategies/params/mlp/mlp_weights_COINBASE_BTCUSD_4H.json
```

No Pine preset regeneration is needed.

## Recommended Next Step

Continue from the target-k6 robust candidate direction and focus the sweep/search around drawdown-cap compliance:

- trailing stop around `10` to `25`,
- entry around `-80` to `-40`,
- exit around `60` to `120`,
- confirmation around `-40` to `40`,
- reject rows above `40%` drawdown earlier in ranking.