# TV / Python Parity Workflow

Whenever TradingView metrics (P&L, trade count, exits) differ from Python
backtest results, follow this runbook to diagnose and close the gap.

---

## Background: Why Parity Matters

The optimiser runs in Python.  The strategy is deployed in TradingView Pine
Script.  If the two compute different signals, the optimised params won't
produce the expected results in TradingView (TV) — the strategy is being optimised for
something it isn't trading.

**Accepted / expected differences (not bugs):**

| Difference | Why |
|---|---|
| TV entry timestamp = Python entry + 1 bar | Pine fills at open[T+1]; Python records signal at bar-close T |
| TV P&L slightly different per trade | TV uses open[T+1] fill price; Python uses close[T] |
| TV shows ≥1 margin-call trade | TV's micro-lot rounding; not modelled in Python |
| ±1 bar exit timing with trailing stop | TV trail_high uses intrabar highs; Python uses bar close |
| TV sometimes has no exit on last trade and shows it as "Open" | TV therefore calculates P&L for the last trade exit based on last known price from the latest realtime intrabar data as of now. This particular discrepancy is a TV difference we can ignore |
| ~10–15% TV-only / Python-only entry mismatch after fresh optimisation | Inherent score noise: Python `activation_score` vs TV `activation_score_poc` has mean ≈ 0 but std ≈ 6–7 units. On threshold-crossing bars the delta can reach ±15 units, causing one side to fire a crossunder the other doesn't. This is NOT a bug — it is an intrinsic difference between Python's `_norm`-column weighted sum and Pine's internal library normalisation. See §A below. |

**Flags that indicate real bugs:**

| Symptom | Likely cause |
|---|---|
| Large trade count mismatch (>20%) | Stale CSV (missing `_norm` columns) or params loaded from wrong preset |
| All TV-only trades have SCORE_DIFF + same direction delta | One `_norm` column missing or all-zero in Python |
| Cascading Python-only trades (Python extras >> TV extras) | Prior exit divergence leaves Python/TV in different position state |
| Huge P&L delta on matched trades | Trailing stop implemented differently |
| TV P&L vastly lower than Python P&L AND many TV-only losing trades | Short optimisation run — params overfit to Python's score; thresholds sit right in the ±6–7 unit noise band causing TV to fire extra (bad) entries |
| Python has many more trades than TV AND Python trades start years before TV trades | **na-propagation from a signal using a security with limited history** (see §E below). Python reads na CSV values as 0.0 (neutral), TV propagates na through math.max/min → entire activation_score_poc = na → no TV entries. Fix: wrap final `basis_norm` or similar in `nz(..., 0.0)`. |
| Python P&L >> TV P&L, ~40 "TRAILING_STOP" exit mismatches, Python holds far longer than TV | **`stoch_peak_norm` missing from CSV** (see §F below). Python exit crossunder never fires → Python only exits via trailing stop → holds much longer than TV. Cause: `stoch_peak_norm` plotchar export removed when freeing plot slots, but `stoch_is_peaking` is still the exit gate. Fix: restore `plotchar(stoch_is_peaking ? -1.0 : 0.0, "stoch_peak_norm", ...)` and re-export CSV. |
| Pine preset in exported CSV has wrong thresholds (e.g., `long_entry_threshold=345` but Python winner has 305) | **Stale Pine preset** — `generate_pine_presets.py` not run after latest optimization. Fix: run `python tools/generate_pine_presets.py`, paste updated Pine into TV, re-export both CSV files. |

---

## Step 1 — Generate the TV trade list

1. In TradingView, open **Strategy Tester → List of Trades**.
2. Click **Export** (download icon) → saves as a CSV.
3. Move/copy the CSV to this repo.  Naming convention:
   `data/ActivationScores_<ASSET>_<DATE>.csv`

---

## Step 2 — Run the comparison tool

```bash
python tools/compare_tv_trades.py \
    --tv-trades data/ActivationScores_COINBASE_BTCUSD_2026-03-26.csv \
    --data data/COINBASE_BTCUSD-4H.csv \
    [--params results/winners/optimization_winner_activation_scores_COINBASE_BTCUSD_4H.csv]
```

The tool auto-detects `--params` from the data filename if omitted.

For the MLP strategy, pass the strategy file so the tool loads
`strategy_mlp_scores.py` and auto-detects the matching MLP winner CSV:

```bash
python tools/compare_tv_trades.py \
    --strategy-file strategy_mlp_scores.py \
    --tv-trades data/MLPScores_COINBASE_BTCUSD_2026-06-17.csv \
    --data data/COINBASE_BTCUSD-6H.csv \
    [--params results/winners/optimization_winner_strategy_mlp_scores_COINBASE_BTCUSD_6H.csv]
```

Before interpreting MLP trade-list differences, run `tools/check_mlp_parity.py`
against a TradingView chart-data export that includes `mlp_score`:

```bash
python tools/check_mlp_parity.py \
    --data data/COINBASE_BTCUSD-6H.csv \
    --weights strategies/params/mlp/mlp_weights_COINBASE_BTCUSD_6H.json \
    --params results/winners/optimization_winner_strategy_mlp_scores_COINBASE_BTCUSD_6H.csv
```

MLP score parity should be much tighter than activation-score parity because
Python and Pine run the same forward pass over the exported feature columns.
If `check_mlp_parity.py` fails, fix the score export or Pine preset before
debugging trade-list differences.

**Reading the output:**

```
MATCH SUMMARY
  Total TV real trades  :   68  (coverage through 2024-09-20)
  Total Python trades   :   83
  Entry + exit matched  :   61  (89.7% of TV)
  Entry matched, exit Δ :    1
  Python-only (extra)   :   21  (IS: 9, OOS after 2024-09-20: 12 — OOS are expected, not bugs)
  TV-only (missed)      :    6
```

Key flags:
- `Entry + exit matched` — both sides agree.  Target ≥85% after a fresh optimisation.
- `Entry matched, exit Δ` — same entry crossunder, different exit.
- `Python-only (IS)` — Python fires extra trades within the TV coverage period (score noise or cascade).
- `Python-only (OOS)` — Python fires after the last TV trade date — **expected, not a parity bug**.  TV's export ends at the last bar; Python continues firing signals in the out-of-sample period.
- `TV-only` — Python misses entries TV fires (score noise or stale CSV).

Add `--verbose` to see score-context tables for all divergent trades.
Add `--exit-detail` to see score context on exit-mismatched bars.

---

## Step 3 — Identify the root cause

### A. Inherent score noise (expected ~10–15% mismatch rate)

Python computes `activation_score` from `_norm` columns (pre-normalised values
exported by Pine via `plotchar`). TV computes `activation_score_poc` via the
Pine library function `f_calculateActivation_poc` using the same normalisation
formulas, but floating-point differences accumulate across 33 components.

**Measured characteristics (BTC 4H, March 2026):**
- Mean delta (TV poc − Python): **≈ 0** (no systematic bias)
- Std: **≈ 6–7 units**
- Range: −19 to +19 units
- Highest correlation with stoch_norm (r ≈ 0.78) and macd_pred_norm (r ≈ 0.70)

**Consequence:** On threshold-crossing bars the delta can be 7–15 units in
either direction. This causes:
- **TV-only entries**: TV poc crosses the threshold but Python score doesn't (TV poc was higher on that bar).
- **Python-only entries**: Python score crosses but TV poc doesn't (less common since mean delta ≈ 0).

**This is not fixable without Pine library source access.** The mitigation is:
longer optimisation runs that find params whose thresholds are not sitting in
the ±6–7 noise band — i.e., the crossunder happens clearly (score difference >> 7)
rather than at the margin.

**Warning sign:** If TV P&L is vastly lower than Python P&L AND many TV-only
trades are losers, the params from a short run are likely overfit. TV fires
extra entries (from the positive tail of the delta) that Python's backtest
avoided, and those extra entries hit bad patches.  Fix: run a longer
optimisation (≥2 full iterations = several hours per combo).

### B. Stale or missing `_norm` columns (fixable)

If `compare_tv_trades.py` reports many SCORE_DIFF entries AND the delta is
consistently in one direction across many bars, a column is likely missing or
all-zero in Python.

Check with:
```bash
python tools/diagnose_missing_data.py data/COINBASE_BTCUSD-4H.csv \
    --params results/winners/optimization_winner_activation_scores_COINBASE_BTCUSD_4H.csv
```

The tool checks all `get_col()` calls in `strategy_activation_scores.py` against
the CSV.  With `--params`, it also distinguishes:
- **Non-zero weight missing** → exit code 1, score differences guaranteed → re-export CSV
- **Locked=0 missing** → no score impact → acceptable
- **Gate column missing** (e.g. `stoch_peak_norm`) → HIGH SEVERITY → see §F

Fix: re-export the TV chart data after any Pine changes, confirm tool exits 0.

### C. Cascading Python-only trades

If Python has many more extra trades than TV has TV-only trades, a single early
exit divergence is leaving Python flat when TV is still long.  Python then
re-enters on bars where TV is already in position.

Look at the **first** Python-only trade in the list — fixing that one often
clears the cascade.

Root cause of cascade (confirmed 2026-03-26): Python's computed score can be
2–5 units lower than TV poc on the exit-crossunder bar, causing Python to fire
the 2-bar exit crossunder one bar earlier than TV. Combined with `stoch_is_peaking`
exit condition, one early Python exit cascades into ~30 extra re-entries.

Zeroing the trailing stop does NOT fix this cascade — the exit is driven by
the score crossunder + stoch_is_peaking, not the trailing stop.

### D. Trailing stop exit divergence

When `i_trailing_stop_threshold > 0`, TV's trailing high is updated with
the intrabar high (which Python doesn't see — Python only has bar close).
On very large trailing stop values (30–50%) this can cause:
- TV exits earlier (intrabar high→stop triggers before bar close)
- TV holds longer (close never triggers stop; Python uses close)

This is **not a bug**.  It is an intrinsic fill-model difference.  The Python
trailing stop is a close-to-close approximation.

To quantify: run `--exit-detail` and look at the score and price context on
exit-mismatch bars.

### E. na-propagation from instruments with limited history (FIXABLE — **highest severity**)

**Symptom**: Python has significantly more trades than TV, Python trades start
years before TV trades, and TV shows zero entries in a historical period.

**Root cause**: A signal fetched via `request.security()` returns `na` for all
bars before the instrument existed (e.g. `BINANCE:BTCUSDTPERP` launched ~Sep 2019,
`CRYPTOCAP:USDT.D` may have a similar cutoff for older TFs).  If that `na` reaches
a `math.max()` or `math.min()` call in Pine v6 **without** being wrapped in `nz()`,
the entire `activation_score_poc` becomes `na` for those bars, silently blocking
ALL TV entries.  Python reads na CSV values as `0.0` (neutral), so it computes valid
scores where TV computes none — producing dozens of Python-only spurious trades.

**Example (confirmed 2026-03-28)**:
```pine
// BUGGY — _perp_close = na pre-Sep 2019 → _basis_pct = na → score = na
_basis_pct = (nz(_perp_close, _perp_close[1]) - close) / close * 100.0
basis_norm = math.max(-1.0, math.min(1.0, _basis_pct / 0.5))

// FIXED — nz(..., 0.0) forces neutral value, not na
basis_norm = math.max(-1.0, math.min(1.0, nz(_basis_pct, 0.0) / 0.5))
```

**Rule**: Every `_norm` variable that feeds into `activation_score_poc` **must**
be wrapped in `nz(..., 0.0)` (or equivalent) before being passed to `math.max` /
`math.min` or included in an arithmetic sum.  The plotchar export should also emit
the nz-wrapped value so Python's CSV reflects what TV computed.

**How to detect**: Run `compare_tv_trades.py` — if Python-only trade count greatly
exceeds TV-only count AND Python entries start years before the first TV trade,
this is the likely cause.  Confirm by checking which `request.security()` instruments
the strategy uses and when they launched.

**Checklist when adding a new `request.security()` signal**:
- [ ] Does the instrument have full history back to 2017?  If not, wrap the final
      `_norm` variable in `nz(..., 0.0)` before it enters any math or sum.
- [ ] After re-exporting CSV, run `compare_tv_trades.py` and confirm Python-only
      trades have not increased relative to the previous baseline.

---

### F. Missing `stoch_peak_norm` export (FIXABLE — **high severity**)

**Symptom**: Python P&L >> TV P&L, ~40 "TRAILING_STOP" exit mismatches, Python holds
positions far longer than TV.  TV trade count < Python trade count.

**Root cause**: `stoch_is_peaking` is used as a required gate in the exit condition
(`longExitCondition := ta.crossunder(score[1], exit_threshold) AND stoch_is_peaking`).
If `stoch_peak_norm` is absent from the CSV, Python reads it as all-zero →
`stoch_peak` is always False → Python's exit crossunder **never fires** → Python only
exits via trailing stop, holding for weeks/months longer than TV.

**Common cause**: `stoch_peak_norm` was removed from Pine's `plotchar` exports when
freeing plot slots (e.g., to add RSI divergence signals), without noticing it is still
needed for the exit condition (not just for the score weight, which is 0).

**Rule**: Any signal used as a **condition gate** (not just a score weight) must stay
in the plotchar exports regardless of whether its weight is locked to 0.  Currently:
- `stoch_peak_norm` — used in exit condition gate, MUST be exported
- `stoch_bot_norm`, `macd_bullish_norm`, `rsid_norm` — only used as weights (locked=0),
  safe to omit if needed to free plot slots

**Fix**:
```pine
// In the STRATEGY COMPONENT EXPORTS section — NEVER remove this line:
plotchar(stoch_is_peaking ? -1.0 : 0.0, title="stoch_peak_norm", color=color.white, display=display.data_window)
```
Then re-export the CSV and re-run `compare_tv_trades.py`. Exit mismatches should drop
from ~40 to the expected level (mostly trailing stop timing differences).

**Confirmed (2026-03-28)**: `stoch_peak_norm` removed in commit that added RSI
divergence signals → Python never exited via score crossunder → 39 "TRAILING_STOP"
exit mismatches → Python P&L 40,978% vs TV 453%.

---

## Step 4 — Fix and verify

After identifying the root cause:

1. **Stale CSV**: Re-export the data CSV from TV after any Pine changes.
   Run `tools/diagnose_missing_data.py` to confirm all `_norm` columns are present.

2. **Score noise causing TV-only entries + worse TV P&L**: This is expected with
   short optimisation runs. Run longer (≥2 full iterations per combo). The optimizer
   will find thresholds where crossunders are unambiguous (score swings >>7 units
   through the threshold, not just grazing it).

3. **Cascade from single exit divergence**: Trace back to the first mismatched
   exit. If it is a trailing stop difference, it is accepted. If it is a
   score-noise exit crossunder (Python score 2–5 units lower than TV poc on the
   crossunder bar), it is also accepted — longer optimisation will find more
   robust params.

4. **na-propagation (§E)**: In Pine, wrap the final `_norm` variable in
   `nz(..., 0.0)` before it enters `math.max/min` or the score sum.  Then
   re-export the data CSV and re-run the comparison.  Python-only trades in
   the pre-instrument-launch period should drop to near zero.

5. **Re-run the comparison** after each fix.  Target ≥85% full-match rate after a
   fresh optimisation.  ≥90% is achievable after a long optimisation run where
   the params are not on threshold boundaries.

---

## Reference: tool commands

```bash
# Full comparison (auto-detects TF and params)
python tools/compare_tv_trades.py \
    --tv-trades <trade_list.csv> --data <data.csv>

# Verbose: score context on every divergent trade
python tools/compare_tv_trades.py ... --verbose

# Exit detail: score context on exit-mismatched trades
python tools/compare_tv_trades.py ... --exit-detail

# Per-component breakdown for a specific date
python tools/diagnose_tv_parity.py \
    --data <data.csv> --params <params.csv> --date YYYY-MM-DD

# Show only bars with score delta > 10 vs TV poc
python tools/diagnose_tv_parity.py ... --max-delta 10

# Show all entry/exit bars in the full history
python tools/diagnose_tv_parity.py ... --crossunders-only

# Export full breakdown to CSV
python tools/diagnose_tv_parity.py ... --export /tmp/breakdown.csv

# Send ntfy notification (for Claude or scripts)
python tools/ntfy.py "Message here" --title "Title" --priority high
```

---

## Checklist after any Pine changes

- [ ] Re-export all affected data CSVs from TradingView
- [ ] Run `tools/diagnose_missing_data.py data/<ASSET>-<TF>.csv --params results/winners/...csv`
      Exit code 0 = all clear; exit 1 = missing non-zero column or gate column → do not optimise
- [ ] Run `tools/compare_tv_trades.py` on at least BTC 4H with the new winner params
- [ ] Check IS full-match rate ≥ 85% before starting a new optimisation run
      (OOS Python-only trades after the TV export date are expected — ignore them)
- [ ] If IS match rate < 75%, check for missing columns before optimising
      (optimising against a mismatched strategy wastes compute)
- [ ] If TV P&L << Python P&L with many TV-only losers: params from short run are
      overfit — run a longer optimisation (≥2h per combo for BTC 4H)

## Checklist when adding a new `request.security()` signal

- [ ] Does the instrument have full BTC history back to 2017?
      Perps (~Sep 2019), some DeFi tokens, and some dominance metrics launch late.
- [ ] If NOT full history: wrap the final `_norm` result in `nz(..., 0.0)` so na
      doesn't propagate through `math.max/min` into `activation_score_poc` (see §E).
- [ ] plotchar exports the `nz`-wrapped value — Python's CSV must match what TV computed.
- [ ] After exporting: run `compare_tv_trades.py`; confirm Python-only trade count
      has not increased, especially in the pre-instrument-launch period.