# Next Steps — 2026-06-20

Session summary and prioritized plan after completing the shallow MLP ceiling experiments
(feat55, lean50, wide threshold sweep, wide32).

---

## Current State

**What holds:** bb50 (`50→16→8→1`) wins all 5 BTC TFs. No experiment promoted a new winner.
Active Pine strategy is untouched and correct.

**What's uncertain:** Our OOS evaluation framework has a known flaw that affected how we
interpreted experiment results. This needs to be fixed before drawing final conclusions
or starting new experiments.

---

## Priority 1: Fix the OOS Evaluation Framework

This is the most important thing to do next, and it should happen before any new training
or experiment work. Running new experiments under a flawed evaluation lens risks
misinterpreting results in the same ways we already did.

### Issue 1 — The -10.0 OOS sentinel

When a model produces 0 OOS trades, `OOS_Calmar Ratio` is set to `-10.0` as a degenerate
placeholder. This propagates into reporting and human analysis as "catastrophic failure"
when the honest value is `0.0` (neutral — no information).

The -10.0 sentinel is correct for **IS** 0-trade solutions (genuinely degenerate — the
optimizer found params that never trade). But OOS is different: the model can't control
whether the OOS market period has tradeable conditions.

**Fix:** Change the OOS 0-trade sentinel from `-10.0` to `0.0` in the sweep evaluation.
Small code change in `run_mlp_deep_sweep.py` (and anywhere else OOS Calmar is computed).

### Issue 2 — OOS evaluation has no regime awareness

Flat OOS Calmar collapses three distinct outcomes into one number:
- Model traded and made money → **good**
- Model traded and lost money → **bad**
- Model did not trade → **ambiguous** (good if market was adverse, bad if opportunity-rich)

For a **long-only** strategy, 0 OOS trades in a confirmed bear/sideways market is correct
behavior. The same outcome in a confirmed bull market is a structural failure. The current
framework treats both identically.

**Fix:** Add regime context to OOS evaluation. The MVRV data is already in the training
CSVs. For each OOS evaluation:
1. Characterize the OOS window by regime composition (% bull / bear / sideways by MVRV)
2. Report this alongside OOS metrics — "OOS was 70% bear by MVRV"
3. When `OOS_Total_Trades = 0`, flag whether OOS was regime-adverse for longs (neutral)
   or regime-favorable (suspicious)
4. Long-term: compute expected trades given IS trade frequency and OOS regime mix, to set
   a context-aware baseline for what "normal" OOS activity looks like

This is moderate complexity — the MVRV signal and regime thresholds already exist in the
codebase. The main work is plumbing regime context into the OOS reporting path.

---

## Priority 2: Re-evaluate Under Corrected Framework

After fixing the evaluation, re-run the deep sweep summary to get clean OOS metrics.
Most past experiment conclusions were driven by **IS composite failures** and won't change.
The specific case worth revisiting:

### Wide32 8H

This is the one case where our "done" conclusion was influenced by OOS interpretation:
- IS Calmar improved genuinely (+16%, 0.847 vs bb50's 0.727)
- IS composite fell short (4.891 vs 7.057) — primarily due to fewer IS trades (81 vs 105)
- OOS: 0 trades, which the flawed framework read as failure

With corrected evaluation:
- If OOS period was regime-adverse (confirmed sideways/down market), 0 OOS trades = neutral
- The IS composite shortfall from trade count remains — but the composite formula's
  trade-count bonus was designed to prevent cherry-picking, not to penalize conservative
  models that correctly sit out bad markets
- Worth running a clean sweep with corrected OOS framework and seeing if the story changes

**Do not run more training seeds** until the evaluation framework is fixed. More seeds
evaluated under a flawed framework compounds the problem.

---

## Priority 3: Future Experiments (Queue for After Framework Fix)

These should not start until Priorities 1 and 2 are complete.

### A. Sub-daily features (highest near-term leverage)

The 4H–8H ceiling is a **feature-set problem**, not a model problem. The current 50
features are a macro/momentum cluster — comprehensively represented at daily and
multi-day scales, but sparse on genuine sub-daily signal.

Features with orthogonal sub-daily predictive power:
- **Funding rates** (perpetual futures): captures short-term positioning; strongly
  correlated with short-term price direction; available on TV via exchange feeds
- **Real CVD** (cumulative volume delta from exchange order flow, not OBV proxy):
  actual buy/sell pressure at trade level, not price-proxied
- **Open interest changes**: momentum of aggregate positioning; leads price at
  intraday scale in ways OBV doesn't
- **Liquidation cascades**: directional events; not captured by any current feature

These are genuinely orthogonal from the macro cluster and exist at hourly resolution.
Path: find TV sources → add to Pine → re-export training CSVs → retrain bb50.
See `docs/experiment_wide_arch.md` for what architecture to use (bb50, not wide32).

### B. Regime-aware training sweep (low effort, try first)

`--regime bull` flag exists but has not been used at scale. Hypothesis: training only
on bull-market IS bars produces a model that better identifies bull entries and correctly
sits out bear/sideways conditions — exactly the behavior we want.

Run a full `--regime bull` sweep on the 4H–8H TFs where the ceiling is hardest.
If it doesn't beat bb50, rule it out quickly (1 afternoon of compute).

### C. Gradient-boosted trees (different model class, deployment question first)

XGBoost/LightGBM on the same pre-normalized features. Would find non-linear feature
interactions the weighted-sum MLP can't express. Fast to train, interpretable, no GPU.

**Open problem:** Pine can't implement a tree ensemble directly. Before investing in
GBT training, decide on a deployment path:
- Score-as-lookup-table (approximate, lossy)
- External signal service → alert → execution (higher infra complexity)
- TradingView webhook integration

Resolve the deployment question first; training is cheap once the path is clear.

---

## What Not To Do Yet

- **Don't start new training runs** until the OOS framework is fixed (Priority 1)
- **Don't delete wide32 artifacts or close the wide32 experiment permanently** —
  the 8H case deserves a clean re-evaluation under corrected OOS metrics
- **Don't change the Pine strategy** — bb50 winners hold, no regressions detected
- **Don't update the backlog** with speculative items from this session until we
  know which re-evaluations change direction (Priority 2)

---

## Progress Log

| Date | Status | Notes |
|------|--------|-------|
| 2026-06-20 | ✅ Priority 1 complete | OOS sentinel fix + regime context in oos_dashboard.py and run_mlp_deep_sweep.py |
| 2026-06-20 | ✅ Priority 2a complete | All 5 TF sweeps re-run — bb50 holds, conclusions unchanged |
| 2026-06-20 | ✅ Priority 2b complete | Wide32 8H: P&L/DD gate (176.7 < 434.7) still binding; OOS now shows 0.0 (neutral, 92% Bear) |
| 2026-06-20 | ✅ Priority 3a complete | Bull-regime training broke both ceilings: 4H 0.229→0.254 (+10.9%), 8H 0.727→0.877 (+20.6%). Both promoted (seed202). P&L/DD: 4H +162%, 8H +58%. Pine presets updated. |

## Summary

```
Priority 1 (do first):  Fix OOS evaluation                    ✅ DONE
  1a. Change 0-trade OOS sentinel from -10.0 → 0.0
  1b. Add MVRV regime context to OOS evaluation and reporting

Priority 2 (after 1):   Re-evaluate with clean metrics        ✅ DONE
  2a. Re-run deep sweep summary — what changes?               (bb50 holds, 4H unchanged)
  2b. Focused re-evaluation of wide32 8H                      (P&L/DD gate still binding)

Priority 3 (after 2):   New experiments
  3a. Regime-aware training (--regime bull, all 5 TFs)        ✅ DONE — 4H+8H promoted; 6H/12H/1D unchanged (strong baselines)
  3b. Sub-daily features (funding rate, real CVD, OI)
  3c. GBT — resolve deployment question first

Deferred:               Wide arch, more seeds, Pine changes
```