# Plan: Change Training Start Date from 2017-01-01 to 2017-12-01

## Context

The user is observing severe OOS degradation (all 20 combos negative or barely positive out-of-sample)
and hypothesizes that trimming early 2017 data — the "spectacular bull run" — would shift optimizer
focus toward bearish/flat regimes, producing weights that generalize better to the current 2025-2026
correction.

---

## What the Data Actually Shows

**OOS situation (from oos_dashboard.md, generated 2026-04-06):**
- 19/20 combos: negative OOS P&L
- Only LINKUSD 1D barely positive (+16.4%) but marked WEAK (38% of IS Sortino)
- OOS Sortino range: -3.17 to +1.36 vs IS Sortino range: +0.9 to +3.6
- Classic overfitting signature: strong IS, uniform OOS collapse across all combos

**Regime inventory if start changes from 2017-01-01 → 2017-12-01:**
- Jan–Nov 2017: Massive bull run ($1k → $20k peak). This IS the most "bull-heavy" segment.
  Removing it is materially correct in the user's framing.
- Dec 2017 onward: Starts right at the ATH peak, so training immediately hits 2018 bear crash.
- Net effect: ~11 months of pre-institutional bull data removed; training skews slightly more
  toward bear/sideways (2018-2019, 2022 bear are all retained).

---

## Honest Assessment: Will This Help?

### Reasons to be skeptical

1. **Scale is marginal.** Removing 11 months from a 7.75-year window is ~12% trim.
   The training already includes two full bear markets (2018, 2022) and two sideways periods
   (2019, 2023). The regime mix won't change dramatically.

2. **All 20 combos fail — that's an overfitting signature, not a regime coverage gap.**
   If the problem were "missing bear-regime data," we'd expect some combos (e.g., shorter TFs
   which have more trades) to still work. Uniform failure across all 20 points to the optimizer
   overfitting to whatever IS regime it sees, regardless of what that regime is.

3. **TRAIN_END may already be 2025-09-30** (config.py reads this value; CLAUDE.md still says
   2024-09-30 and may be stale). If true, the optimizer is already training on the 2025
   correction and STILL failing OOS in Oct 2025+. That is a much more alarming signal —
   the overfitting is occurring even within the correction period.

### Reasons the change is still reasonable to try

1. The 2017 early bull is genuinely anomalous — tiny market cap, no institutional participation,
   different volatility regime. It may be teaching the optimizer patterns that don't generalize.

2. It's cheap to implement (one config line change + reset).

3. Resets the DB anyway, which is overdue given all the WFO changes.

---

## Decision

User wants:
- `SCORE_START = "2017-12-01"` — trim pre-institutional 2017 bull run
- `TRAIN_END` stays at `2025-09-30` — keep the extra year of bear/sideways data
- Full DB reset required
- Pair with one additional change that actually moves the OOS needle

---

## Root Cause of OOS Failure

WFO currently selects winners by **mean Calmar across folds**. A bull-dominated solution can score:
- 2021 (bull): +5.0 Calmar
- 2022 (bear): -3.0 Calmar  ← fails
- 2023 (sideways): -2.0 Calmar  ← fails
- 2024 (ETF rally): +4.0 Calmar
- 2025 (late bull): +2.0 Calmar
- **WFO mean = +1.2** — selected as WFO winner despite failing the two bear/sideways folds

The current OOS regime (Oct 2025+) is correction/sideways — exactly the regime this hypothetical
winner fails. This explains all-20-combos failing uniformly.

---

## High-Impact Change: WFO Worst-Fold Gate

Change WFO winner selection from "highest mean Calmar" to:
1. Must have **all folds positive** (Calmar > 0 in each fold with sufficient trades)
2. Rank by **minimum fold Calmar** (worst-fold performance), not mean

This directly selects regime-robust solutions: they must work in the 2022 bear AND the 2023
sideways folds to qualify. Those are the historical analogues of the current OOS regime.

If no candidate passes all-positive-folds, fall back to the solution with the fewest negative
folds (most robust available).

**Files to change:**
- `tools/auto_optimize_loop.py` — WFO winner selection logic (lines ~641-677, ~1696-1701)
  - Change scoring key from `WFO_Score` (mean) to `WFO_Min_Fold` (minimum fold Calmar)
  - Add `WFO_Neg_Folds` column (count of negative-Calmar folds) for transparency
- `tools/wfo_rescore.py` — add `wfo_min_fold` and `wfo_neg_folds` to output

**Secondary change (low-code):** Stricter subperiod consistency gate
- Currently: Calmar > 0 in both P1 and P2
- Change to: Calmar > 0 AND Sortino > 0 in both P1 and P2
- File: `tools/auto_optimize_loop.py` lines 548-550
- One-line change; filters solutions that game one metric while failing the other

---

## Implementation Steps

### ✅ 1. Edit `config.py` line 8
```python
SCORE_START = "2017-12-01"
```
Propagates automatically to all tools via import. **Done 2026-04-07.**

### ✅ 2. Strengthen subperiod consistency in `auto_optimize_loop.py` (~lines 548-551)
Change gate from Calmar-only to Calmar AND Sortino:
```python
metrics['subperiod_consistent'] = (
    m1.get('Calmar Ratio', -10.0) > 0 and m1.get('Sortino Ratio', -10.0) > 0 and
    m2.get('Calmar Ratio', -10.0) > 0 and m2.get('Sortino Ratio', -10.0) > 0
)
```

### ✅ 3. Change WFO selection from mean → min-fold in `auto_optimize_loop.py`
- Compute `wfo_min_fold = min(fold_calmars) if fold_calmars else -99`
- Add `wfo_neg_folds = sum(1 for c in fold_calmars if c <= 0)`
- Store both in result dict for DB/CSV
- Change winner selection key: `max(top_for_wfo, key=lambda x: (x.get('WFO_Min_Fold', -99), x.get('WFO_Score', 0)))`
- Same change in `tools/wfo_rescore.py` for consistency

### ✅ 4. Update `CLAUDE.md` — fix stale date reference
- Currently says "2017-01-01 to 2024-09-30 (2024-10-01+ held out for OOS)"
- Update to: "2017-12-01 to 2025-09-30 (2025-10-01+ held out for OOS)"

### 5. ⏳ Reset results — `python3 tools/reset_results.py --apply`
Required: IS scoring window changed + WFO scoring logic changed = stale DB.

### 6. ⏳ Run optimization
Start with BTC short runs (~0.5h each) to validate before marathon.

---

## Verification
- After first optimization cycle, check that WFO winner CSV shows positive fold Calmars
- Compare OOS dashboard before/after — BTC combos first, then alts
- The WFO cheatsheet should show `WFO_Min_Fold > 0` for saved winners

## Verification

- Run `python3 tools/oos_dashboard.py` after first optimization cycle to check IS metrics
- Watch IS Sortino vs Composite to confirm optimizer is finding diverse (not degenerate) solutions
- After sufficient runs, compare OOS dashboard to current baseline

## Critical Files

- `config.py` — single source of truth for all date constants (`SCORE_START`, `TRAIN_END`, `OOS_START`)
- `tools/reset_results.py` — safe reset with dry-run guard
- `results/oos_dashboard.md` — before/after comparison target