# TradingBot25 — Architecture Guide

## ELI10: What Is This Thing?

Imagine you're trying to decide whether now is a good time to buy Bitcoin. You can look at dozens of signals — is momentum accelerating? Is the stochastic oscillator oversold? Is on-chain supply flowing to exchanges? What's the Fed doing? — but combining 55 different signals into one "buy now" or "don't buy" decision by hand is impossible.

This system trains a tiny neural network to do that combination automatically:

1. **TradingView** computes 55 technical/on-chain signals, normalises each one to a −1 → +1 scale, and exports them as a CSV
2. **Python** trains a neural net (`55 → 16 → 8 → 1`) on that CSV — the net learns *which combination of signals predicts profitable trades*. The single output is a "market vibe score" from −1000 to +1000.
3. We then **search** for the best entry/exit thresholds: "enter when the score crosses −165 going down, exit when it crosses +50 going up"
4. Those learned weights + thresholds get **embedded back into TradingView** as a Pine Script strategy that auto-selects the right parameters for whichever chart you're on

The whole point of keeping all normalisation inside TradingView and only reading pre-normalised numbers in Python is that the two codepaths stay mathematically identical — which makes bugs much easier to catch.

---

## 1. What This System Does

TradingBot25 is a **quantitative crypto trading research system** with three jobs:

1. **Train** a neural network (MLP) that scores market conditions from −1000 to +1000
2. **Sweep** the threshold parameter space to find the best entry/exit rules on top of that score
3. **Deploy** the resulting strategy to TradingView via Pine Script

The key design principle: **all feature engineering stays inside Pine Script**. Python never touches raw OHLCV or computes indicators — it reads pre-normalised `_norm` columns that TradingView already computed and exported.

---

## 2. System Overview

```mermaid
graph TB
    TV["TradingView\nPine Script Strategy"]
    CSV["data/mlp/\nPre-normalised CSVs\n55 feature columns per bar (BTC)"]
    TRAIN["tools/train_mlp.py\nPhase 1: PyTorch pretrain\nPhase 2: CMA-ES fine-tune"]
    ART["strategies/params/mlp/\nmlp_weights_ASSET_TF.json\n{arch, layers, feature_cols}"]
    SWEEP["tools/run_mlp_deep_sweep.py\nThreshold random search\n80k–200k candidates"]
    WIN["results/winners/\noptimization_winner_*.csv\n{entry, exit, conf, trail thresholds}"]
    DB["results/sweep_database.db\nSQLite — all sweep rows\n(used for sign-stability, SHAP)"]
    CODEGEN["tools/generate_pine_mlp_presets.py\nWrites sentinel block\ninto .pine file"]
    PINE["strategies/strategy_mlp_scores.pine\nDeployed to TradingView\nAuto-selects preset by symbol+TF"]

    TV -- "Export CSV\n(plotchar columns)" --> CSV
    CSV --> TRAIN
    TRAIN --> ART
    ART --> SWEEP
    CSV --> SWEEP
    SWEEP --> WIN
    SWEEP --> DB
    WIN --> CODEGEN
    ART --> CODEGEN
    CODEGEN --> PINE
    PINE --> TV
```

---

## 3. Data Flow in Detail

```mermaid
flowchart LR
    subgraph TV["TradingView"]
        PINE_IND["Indicators compute:\nstoch_norm, macd_pred_norm,\nrsi_subtf_norm, bb_pct_b_norm\n...55 total _norm signals (BTC)"]
        PINE_STRAT["strategy_mlp_scores.pine\n• MLP forward pass\n• Score → entry/exit signals\n• plotchar() exports"]
    end

    subgraph DATA["data/mlp/ (CSVs)"]
        RAW["Per-bar rows:\ntime, open, high, low, close,\n+ 55 pre-normalised columns\nall in [−1, +1]"]
    end

    subgraph PY["Python"]
        PREP["_prepare_features(df, feature_cols)\n→ float64 (T × 55) matrix"]
        FWD["mlp_forward(X, layers)\n1000·tanh(W3·tanh(W2·tanh(W1·x+b1)+b2)+b3)\nScore ∈ [−1000, +1000]"]
        SIG["_score_to_signals()\nEntry: 1-bar crossunder ≥ entry_thr\nExit: 2-bar crossunder ≥ exit_thr\n+ trailing stop"]
        MET["calculate_metrics()\nCalmar = annualised return / Max DD\nComposite = Calmar × log-trade bonus"]
    end

    PINE_IND --> PINE_STRAT
    PINE_STRAT -- "Export via plotchar" --> DATA
    DATA --> PREP
    PREP --> FWD
    FWD --> SIG
    SIG --> MET
```

**Why this design?** Implementing 55+ indicators twice (Pine and Python) guarantees parity bugs. Instead: Pine is the single source of truth for normalisation; Python just reads the already-normalised values.

---

## 4. MLP Training Pipeline

```mermaid
flowchart TD
    D["data/mlp/COINBASE_BTCUSD, 240.csv\n20k+ bars, 55 feature cols (BTC)"]

    subgraph P1["Phase 1 — Supervised Pretrain (PyTorch, MPS)"]
        TGT["build_target(df, k=10)\ny = tanh(scale × k-bar fwd log return / past-vol)\nNo lookahead — vol is shift(1)"]
        NET["Sequential MLP: 55→16→8→1\nAll layers: tanh activation\nAdam, lr=1e-2, early stop (patience=20)"]
        VAL["Val set = last WFO fold OOS span\nTrain set = everything before that"]
    end

    subgraph P2["Phase 2 — CMA-ES Fine-Tune"]
        ES["CMA-ES searches:\n- All MLP weights (flattened)\n- 4 thresholds ÷ 100\nFitness = -mean(fold OOS Calmar)\n+ robust penalties"]
        WFO["WFO folds (6 regimes):\n2021 bull, 2022 bear,\n2023 recovery, 2024 ETF,\n2024 political bull, 2025 correction"]
        LOCK["L2 anchor = distance from Phase 1 weights\n(keeps CMA-ES near pretrained solution)"]
    end

    OUT["mlp_weights_COINBASE_BTCUSD_4H_bb50_seed606.json\n{version, asset, tf, arch, layers, feature_cols, training}"]

    D --> TGT
    TGT --> NET
    NET --> VAL
    VAL --> P2
    P2 --> ES
    ES --> WFO
    WFO --> LOCK
    LOCK --> ES
    ES --> OUT
```

The seed (`--seed 606`) controls PyTorch init and CMA-ES. Multiple seeds are trained; the one with best IS Calmar gets promoted to the active artifact.

---

## 5. MLP Layer Architecture

The network is intentionally shallow — it's a signal combiner, not a pattern recogniser. Deep nets would overfit on a ~20k-bar time series.

```mermaid
graph LR
    subgraph IN["Input — 55 signals (BTC)"]
        direction TB
        i1["stoch_k_norm"]
        i2["rsi_norm"]
        i3["bb_pct_b_norm"]
        i4["macd_pred_norm"]
        i5["nupl_norm"]
        i6["btc_dom_roc_sign"]
        i7["fed_net_liq_sign"]
        i8["sopr_norm · cvd_norm · ..."]
    end

    subgraph H1["Hidden 1 — 16 neurons"]
        h1["tanh(W1 · x + b1)\nW1: 55×16 = 880 weights\nb1: 16 biases\nLearns signal groupings"]
    end

    subgraph H2["Hidden 2 — 8 neurons"]
        h2["tanh(W2 · h1 + b2)\nW2: 16×8 = 128 weights\nb2: 8 biases\nCompresses to regime clusters"]
    end

    subgraph OUT["Output — 1 score"]
        o["score = 1000 × tanh(W3 · h2 + b3)\nW3: 8×1 = 8 weights\nb3: 1 bias\n∈ [−1000, +1000]"]
    end

    IN --> H1
    H1 --> H2
    H2 --> OUT
```

**Total parameters:** 880 + 16 + 128 + 8 + 8 + 1 = **1,041 weights** (BTC). This is deliberately small — CMA-ES can search 1k-dimensional spaces efficiently; gradient descent on a bigger net would need far more data than 20k bars.

**tanh everywhere (not ReLU):** The forward pass must be reproduced exactly in Pine Script v6, which has no built-in tanh. The closed form `x≥20 ? 1 : x≤-20 ? -1 : (exp(2x)−1)/(exp(2x)+1)` works in Pine; ReLU activation would also work but tanh keeps outputs bounded, which matters for the CMA-ES fitness landscape.

**alts (ETH/SOL/LINK):** Still at `50→16→8→1` (830 weights) pending a 55-feature retrain. Run `mlp_status.py` for the live arch per TF — it is the source of truth.

---

## 6. Threshold Sweep (Optimization) Pipeline

```mermaid
flowchart TD
    PARAMS["strategies/params/\nparams_strategy_mlp_scores_ASSET_TF.json\n{mlp_weights_file: values, entry: start/stop/step, ...}"]
    ART["mlp_weights_ASSET_TF.json\n(loaded inside generate_signals)"]
    DATA["data/mlp/ASSET, TF.csv"]

    subgraph SEARCH["run_mlp_deep_sweep.py — Random Search"]
        GEN["80k–200k random combos\n(entry, exit, conf, trail, flags)"]
        GSIG["generate_signals(df, **kwargs)\n→ score series + position + execute_entry/exit"]
        GMET["calculate_metrics(signals)\n→ Calmar, Composite, WFO, Fragile"]
        GATE["promotable():\n• Calmar ≥ current winner\n• P&L/DD ≥ current winner\n• WFO_Neg_Folds == 0\n• Fragile_0.05 == 0\n• OOS Sortino not degraded"]
    end

    subgraph OUT["Outputs"]
        WIN["results/winners/\noptimization_winner_mlp_scores_ASSET_TF.csv"]
        DB["results/sweep_database.db\nAll rows where Calmar > 0"]
        CHEAT["results/winners/\n..._cheatsheet.txt\nHuman-readable summary"]
    end

    PARAMS --> GEN
    ART --> GSIG
    DATA --> GSIG
    GEN --> GSIG
    GSIG --> GMET
    GMET --> GATE
    GATE -- "Passes" --> OUT
    GATE -- "All rows Calmar>0" --> DB
```

---

## 7. Score → Signal Logic

```mermaid
sequenceDiagram
    participant Score as mlp_score[t]
    participant State as Position State
    participant Out as execute_entry/exit

    Note over Score: Score ∈ [−1000, +1000]

    Score->>State: prev ≥ entry_thr AND curr < entry_thr?
    State-->>Out: execute_entry=True (1-bar crossunder)

    Note over State: Now in long position
    Note over State: trail_high tracked via max(high)

    Score->>State: pprev ≥ exit_thr AND prev < exit_thr<br/>+ optional confirm bar (score < conf_thr)
    State-->>Out: execute_exit=True (2-bar crossunder)

    Score->>State: close ≤ trail_high × (1 − trail_pct)?
    State-->>Out: execute_exit=True (trailing stop)
```

Two distinct exit paths: score-based crossunder (2-bar with optional confirmation) and percentage-based trailing stop from the highest close since entry. The trailing stop uses `high` to advance the peak but checks `close` to trigger.

---

## 8. Pine Script ↔ Python Roundtrip

```mermaid
flowchart LR
    subgraph TV["TradingView Loop"]
        IND["Compute indicators\n(stoch, MACD, RSI, OI, FedLiq...)"]
        NORM["Normalise each to [−1, +1]\nexport via plotchar(_norm, ...)"]
        MLP_FWD["MLP forward pass:\nread static weight arrays\ntanh(W3·tanh(W2·tanh(W1·x)))\nscale × 1000"]
        STRAT["Entry/exit crossunder\nTrailing stop\nPosition management"]
    end

    subgraph PY["Python (identical math)"]
        READ["_prepare_features(df)\nread *_norm columns from CSV"]
        PY_FWD["mlp_forward(X, layers)\nidentical tanh clamp at ±20\nidentical 1000× scale"]
        PY_SIG["_apply_trailing_stop()\ncalculate_positions()"]
    end

    subgraph CHECK["Parity Validation"]
        PAR["check_mlp_parity.py\nPass: max|Δ| < 0.01\nZero threshold-side disagreements"]
    end

    IND --> NORM --> MLP_FWD --> STRAT
    STRAT -- "Export CSV" --> READ
    READ --> PY_FWD --> PY_SIG
    PY_FWD -- "compare scores" --> PAR
    MLP_FWD -- "compare scores" --> PAR
```

---

## 9. File Structure (Annotated)

```
TradingBot25/
├── config.py                    ← TRAIN_START/END, SCORE_START, WFO_FOLDS, WEIGHT_COLS
│
├── strategies/
│   ├── library_activation_scores.py  ← Pure Numba functions: calculate_positions(),
│   │                                    _apply_trailing_stop(), candlestick detectors
│   ├── strategy_activation_scores.py ← Perceptron strategy (legacy / still active for
│   │                                    non-MLP assets), _prepare_features(), scoring
│   ├── strategy_mlp_scores.py        ← MLP strategy: FEATURE_COLS (55 BTC / 50 alts),
│   │                                    mlp_forward(), generate_signals(), artifact I/O
│   ├── optimize_strategy.py          ← Threshold sweeper: random/optuna, writes winners
│   ├── validate_strategy.py          ← Single backtest runner, --show-trades
│   ├── strategy_mlp_scores.pine      ← Pine Script: MLP forward pass + auto-presets
│   └── params/
│       ├── params_strategy_mlp_scores_ASSET_TF.json  ← Search ranges (20 active files)
│       └── mlp/
│           └── mlp_weights_ASSET_TF_TAG_seedN.json   ← Trained artifacts
│
├── data/mlp/                    ← TV-exported CSVs with 55 pre-normalised columns (BTC)
│   └── COINBASE_BTCUSD, 240.csv     (named: "ASSET, minutes.csv")
│
├── results/
│   ├── sweep_database.db        ← SQLite: all sweep rows (Calmar > 0)
│   └── winners/
│       ├── optimization_winner_mlp_scores_ASSET_TF.csv   ← Best thresholds
│       └── optimization_winner_mlp_scores_ASSET_TF_cheatsheet.txt
│
├── tools/
│   ├── train_mlp.py             ← Two-phase trainer (Phase 1 PyTorch + Phase 2 CMA-ES)
│   ├── run_mlp_train.py         ← Multi-seed training orchestrator
│   ├── run_mlp_adaptive.py      ← Adaptive loop: detects plateau → tries bull regime → retires TF
│   ├── run_mlp_deep_sweep.py    ← High-sample threshold sweep (80k–200k samples, promotable gate)
│   ├── generate_pine_mlp_presets.py  ← Writes weights into .pine sentinel block
│   ├── mlp_status.py            ← Artifact inventory + arch consistency check (source of truth)
│   ├── mlp_results_table.py     ← IS/OOS results table (compare to TV tester)
│   ├── check_mlp_parity.py      ← Pine vs Python score diff validation (use --asset/--tf)
│   ├── validate_chart_data.py   ← Catches truncated/partial TV exports before they corrupt results
│   ├── check_pine.py            ← Arch mismatch detection + hard-limit violations
│   ├── compare_tv_trades.py     ← Trade-level TV vs Python discrepancy root-cause
│   ├── refresh_winner_metrics.py← Recomputes winner CSV metrics in-place (fixes stale OOS rows)
│   ├── sync_params.py           ← Propagate WEIGHT_COLS changes to 20 params files
│   ├── lock_params.py           ← Sign-stability → lock near-constant params
│   ├── tighten_params.py        ← q5-q95 → narrow search ranges
│   ├── mlp_compare_weights.py   ← Compare feature sensitivities across assets/TFs
│   ├── shap_feature_importance.py← SHAP values from sweep DB
│   ├── auto_optimize_loop.py    ← Iterative optimizer (legacy perceptron strategy)
│   ├── profile_4h_trades.py     ← MFE/MAE trade classifier (diagnostic)
│   └── ntfy.py                  ← Push notifications via ntfy.sh
│
├── run_btc.py                   ← BTC-only batch sweep (5 TFs, time-budgeted)
├── run_all_crypto.py            ← All 20 combos batch sweep
├── run_marathon.py              ← Autonomous multi-cycle marathon (96h default)
└── mine_sweep_db.py             ← Sign-stability, range tightening, SHAP analysis
```

---

## 10. The 20-Combo Matrix

```mermaid
quadrantChart
    title IS Calmar by TF — BTC (run mlp_results_table.py for current numbers)
    x-axis "Shorter TF" --> "Longer TF"
    y-axis "Lower Calmar" --> "Higher Calmar"
    quadrant-1 Strong + Long
    quadrant-2 Strong + Short
    quadrant-3 Weak + Short
    quadrant-4 Weak + Long
    tf4H: [0.10, 0.13]
    tf6H: [0.28, 0.20]
    tf8H: [0.46, 0.28]
    tf12H: [0.68, 0.46]
    tf1D: [0.88, 0.97]
```

4 assets × 5 TFs = 20 combos. Current active assets: `COINBASE_BTCUSD`, `COINBASE_ETHUSD`, `BINANCE_SOLUSD`, `BINANCE_LINKUSD`. IS scoring window: 2017-12-01 (`SCORE_START`) → 2026-02-28 (`TRAIN_END`). OOS holdout: 2026-03-01+. **Source of truth: `config.py`.**

---

## 11. Adaptive Training Loop (`run_mlp_adaptive.py`)

The adaptive loop is the primary way to improve weights — it runs multiple training + sweep rounds per timeframe and stops automatically when further training stops helping.

```mermaid
flowchart TD
    INIT["Initialise\nRead winner Calmar per TF from results/winners/\nAll TFs start with phase='all'"]

    GATE{"Any active TFs\n(phase ≠ retired)?"}
    DONE["COMPLETE\nPrint before/after Calmar table\nSend ntfy push notification"]

    subgraph ROUND["Each round (up to max_rounds=10)"]
        direction TB
        TRAIN["train_round()  — all active TFs in parallel\nTrain seeds_per_round=3 new seeds each\nPhase 1 PyTorch pretrain → Phase 2 CMA-ES fine-tune\nArtifacts written to strategies/params/mlp/"]
        SWEEP["sweep_round()  — all active TFs in parallel\n80k random threshold candidates per TF\nCompetes ALL seeds ever trained (not just new ones)\nPromotes to results/winners/ if better than current winner"]
        CHECK["Improvement check  — per TF independently\nRead winner Calmar fresh from disk\nΔCalmar = new − old"]
    end

    IMPROVED{"ΔCalmar ≥ 0.02\nfor this TF?"}
    RESET["✅  Improved\nReset no_improve=0\nRecord phase_won"]
    TICK["— No improvement\nno_improve_rounds += 1"]
    PLATEAU{"no_improve_rounds\n≥ convergence_rounds (3)?"}
    PHASE{"TF phase?"}
    BULL["🔄  Switch to bull regime\nphase='bull', no_improve=0\nNext seeds trained on bull-only bars\nSeed counter restarts from scratch"]
    RETIRE["🚫  RETIRE TF\nSkipped in all future rounds"]

    INIT --> GATE
    GATE -- "none left" --> DONE
    GATE -- "yes" --> TRAIN
    TRAIN --> SWEEP
    SWEEP --> CHECK
    CHECK --> IMPROVED
    IMPROVED -- "yes" --> RESET
    IMPROVED -- "no" --> TICK
    RESET --> GATE
    TICK --> PLATEAU
    PLATEAU -- "no" --> GATE
    PLATEAU -- "yes" --> PHASE
    PHASE -- "all" --> BULL
    PHASE -- "bull" --> RETIRE
    BULL --> GATE
    RETIRE --> GATE
```

**Key non-obvious behaviours:**

| Behaviour | Why it matters |
|---|---|
| Sweep competes *all* seeds ever trained, not just the new batch | An old seed's weights may beat the new seeds if paired with freshly-optimised thresholds |
| Bull-regime is a second chance, not a replacement | `phase='all'` trains on all bars (2017–present); `phase='bull'` trains only on bull-market bars. The hypothesis is that the net may specialise better on the bull distribution. If it still doesn't improve after 3 rounds, the TF is retired. |
| Calmar baseline is re-read from disk each round | The sweep writes a new winner to disk if it's better — so the baseline automatically advances without any in-memory bookkeeping |
| TFs retire independently | A TF that plateaus early (e.g. 4H) stops consuming CPUs so faster-converging TFs (e.g. 8H) get more workers per sweep round |
| `--presets` regenerates the Pine sentinel block after any round that produced a promotion | Skip it if you plan to run multiple consecutive loops and only want one TV update at the end |

**When to run again:** After the Glassnode column addition (2026-06-25), force-promotes set the initial baselines from limited seeds. Three consecutive adaptive runs were required to reach the genuine ceiling — 8H improved from 0.55 → 0.74 → 0.81 and 12H from 0.91 → 1.16 across those runs before all TFs retired. If you add new features, change the architecture, or extend the training window, reset the baselines (`/reset`) and run again — the loop will find real gains. If nothing has changed since the last completed run, a re-run will retire all TFs quickly without improvement. The productive next lever is a `mlp-phase2` feature expansion or architecture change (wider hidden layers), either of which creates a new search landscape for the loop to hill-climb.

---

## 12. Installation

```bash
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
```

**GPU acceleration:**
- **Mac (M4/M3/M2):** MPS is used automatically by PyTorch for MLP training — no extra steps.
- **Linux/WSL2 (CUDA):** Only needed for the legacy activation-scores strategy (Numba CUDA kernel). Install `nvidia-cuda-toolkit` and set `LD_LIBRARY_PATH=/usr/lib/wsl/lib` — `auto_optimize_loop.py` sets this automatically for subprocesses.

## 12. Key Invariants

| Rule | Why it exists |
|---|---|
| Python never recomputes indicators | Divergent normalisation is the #1 source of parity bugs |
| `WEIGHT_COLS` in `config.py` is the single source of truth | `sync_params.py`, SHAP, and the DB schema all derive from it |
| Reset DB + winners together | Stale baseline scores block new winners from being saved |
| WFO folds cover 6 distinct market regimes | Prevents overfitting to one regime (e.g. 2021 bull only) |
| Phase 1 uses vol-normalised forward returns | Prevents the model from learning "big moves = good" regardless of volatility context |
| `stoch_peak_norm` must stay in `plotchar` | It gates `longExitCondition` — removing it causes Python to hold positions indefinitely |
| Trailing stop uses `high` to advance peak, `close` to trigger | Matches Pine's `fill_orders_on_standard_ohlc=true` behaviour |
| Use `--asset/--tf` not `--weights` for parity checks | Plain `mlp_weights_{ASSET}_{TF}.json` is often a stale non-winning artifact |