# Spec: Parallel CPU Verification

**Created:** 2026-03-13
**Status:** Draft
**Estimated effort:** 2–3 hours
**Motivation:** CPU verification is currently single-threaded. Parallelising it across the
i7-12700's 20 logical threads would provide ~10x speedup, allowing verification of 5,000+
candidates per iteration for the same wall-clock cost as 500 today.

---

## Current State

After each GPU sweep, `verify_top_results` in `auto_optimize_loop.py` takes the top N
GPU candidates (currently 500) and re-runs each through the full Python strategy logic
to get exact CPU-scored metrics:

```python
for index, row in df_sweep.iterrows():
    params = row.to_dict()
    df_res = strategy_module.generate_signals(df_data.copy(), **params)
    metrics = strategy_module.calculate_metrics(df_res, score_start=SCORE_START)
```

This loop is **sequential and single-threaded**. Each call to `generate_signals` is
independent — the loop could be fully parallelised.

**Estimated current timing (500 candidates, i7-12700):**

- Each `generate_signals` call processes ~4,000 bars of Python/Numba strategy logic
- Rough timing: 0.2–0.5 seconds per call
- 500 calls: **~2–4 minutes per iteration**

---

## Why This Is Parallelisable

Each verification is completely independent:
- Input: a fixed `df_data` (read-only), plus a unique `params` dict
- Output: a metrics dict
- No shared mutable state between calls

This is textbook embarrassingly parallel — identical to the GPU kernel's design
philosophy, just on CPU.

---

## Hardware Context

**i7-12700 in WSL2 Ubuntu:**

| Core type | Count | Logical threads |
|---|---|---|
| Performance cores | 8 | 16 (HyperThreading) |
| Efficiency cores | 4 | 4 |
| **Total** | **12** | **20** |

WSL2 exposes all 20 logical threads to the Linux process. A practical worker count
of **16** (reserving 4 for the system and GPU driver) should achieve near-linear
scaling on embarrassingly parallel CPU work.

---

## Expected Speedup

| Scenario | Sequential | Parallel (16 workers) |
|---|---|---|
| 500 verifications | ~3 min | ~18 sec |
| 1,000 verifications | ~6 min | ~36 sec |
| 5,000 verifications | ~30 min | ~3 min |

**Real-world efficiency note:** The 16x theoretical speedup will be partially offset by:
- Process startup overhead (~1–2 sec to spawn 16 workers)
- Numba JIT compilation on first call per worker (~1–2 sec per worker, cached after first run)
- DataFrame serialisation overhead (pickle of `df_data`: ~500KB, negligible)

Realistic effective speedup: **8–12x**. First run of a session slightly slower due to
JIT cache warmup; subsequent iterations within the same run are fast.

---

## What This Enables: Verification Count Scaling

The GPU sweep currently keeps the **top 1,000 results** via a min-heap in
`strategies/optimize_strategy.py` (`MAX_SWEEP_RESULTS = 1000`).

With parallelised verification:

```
Current  : 500 candidates verified in ~3 min (50% of sweep)
After    : 5,000 candidates verified in ~3 min (requires MAX_SWEEP_RESULTS = 5,000)
```

**Why more candidates matters:** The GPU and CPU compute different objectives.
The GPU scores Calmar over the full 2015–2025 window; the CPU scores over the
2019–2025 scoring window. A candidate ranked 2,000th by the GPU may rank 1st by
the CPU. With only 1,000 sweep candidates, that result is lost before verification
even starts. Increasing the sweep to 5,000 and verifying all of them dramatically
reduces this risk.

**GPU overhead of keeping 5,000 vs 1,000 in the sweep:**

The heap (`heapq.heappush` / `heappushpop`) runs on CPU, not GPU, and operates on
already-computed result dicts. Maintaining 5,000 entries instead of 1,000 adds
negligible overhead — O(log n) per push, and the heap is tiny relative to 50M GPU
results. The final CSV write grows from ~40KB to ~200KB. Both are immaterial.

---

## Recommended Combined Change

| Change | Where | Cost |
|---|---|---|
| Parallelise `verify_top_results` loop | `auto_optimize_loop.py` | ~2 hours |
| Increase `MAX_SWEEP_RESULTS` from 1,000 → 5,000 | `strategies/optimize_strategy.py` | 1 line |
| Increase `top_n` from 500 → 5,000 | `auto_optimize_loop.py` call site | 1 line |

Total: a ~2-hour engineering effort that turns the 30-minute-equivalent sequential
verification into a 3-minute parallel one, while covering 5× more of the GPU search
space.

---

## Implementation Plan

### Step 1 — Extract a worker function

```python
def _verify_one(params_dict, df_data_bytes, score_start, optimization_metric):
    """
    Worker function: deserialise df_data, run generate_signals, return metrics.
    Runs in a subprocess — must be defined at module level (not inside a function)
    for pickle to work on Linux fork.
    """
    import io
    import pandas as pd
    import strategies.strategy_activation_scores as strat

    df_data = pd.read_pickle(io.BytesIO(df_data_bytes))
    try:
        df_res = strat.generate_signals(df_data, **params_dict)
        metrics = strat.calculate_metrics(df_res, score_start=score_start)
        result = params_dict.copy()
        result.update(metrics)
        result['verified'] = True
        return result
    except Exception:
        return None
```

**Why pass `df_data` as bytes?** `ProcessPoolExecutor` on Linux uses `fork` by default,
so the child process inherits the parent's memory — `df_data` doesn't actually get
serialised. The bytes approach is a safe fallback if `spawn` is ever used instead of
`fork`, and the 500KB overhead is negligible either way.

### Step 2 — Replace the sequential loop in `verify_top_results`

```python
from concurrent.futures import ProcessPoolExecutor
import io

# Serialise df_data once (fork makes this cheap; just a safety net)
df_data_bytes = df_data.to_pickle(None)  # returns bytes when path=None

candidate_params = [row.to_dict() for _, row in df_sweep.iterrows()]

with ProcessPoolExecutor(max_workers=16) as executor:
    futures = [
        executor.submit(_verify_one, p, df_data_bytes, SCORE_START, OPTIMIZATION_METRIC)
        for p in candidate_params
    ]
    verified_results = [f.result() for f in futures if f.result() is not None]
```

### Step 3 — Increase GPU sweep size

In `strategies/optimize_strategy.py`, line 149:

```python
MAX_SWEEP_RESULTS = 5000   # was 1000
```

### Step 4 — Increase verification count

In `auto_optimize_loop.py`:

```python
best_row = verify_top_results(sweep_file, DATA_FILE, top_n=5000)
```

---

## Gotchas

### 1. Numba CUDA in forked processes
The GPU optimizer runs in a **separate subprocess** (`run_optimization` via `Popen`),
so it completes before `verify_top_results` is called. There is no active CUDA context
in the main process during verification — forking into worker processes for CPU work
is safe.

### 2. Numba CPU JIT warm-up
Each new worker process must JIT-compile Numba CPU functions on first call. Numba
caches compiled functions to `__pycache__` / `.nbi` files. After the first iteration
of a session, the cache is warm and startup is fast (<0.5s per worker).

### 3. `_verify_one` must be at module level
Python's `pickle` (used by `ProcessPoolExecutor`) cannot serialise functions defined
inside other functions. `_verify_one` must be a top-level function in
`auto_optimize_loop.py` (or a separate module).

### 4. `df.to_pickle(None)` returns bytes
Available since Pandas 1.4. On Linux with `fork`, this is effectively free since the
child inherits the parent's memory pages. On `spawn` (Windows / macOS), it avoids
passing a 500KB DataFrame through pickle implicitly.

### 5. Worker count tuning
`max_workers=16` is a reasonable default for the i7-12700. If the GPU is also active
(e.g., running the optimizer in a parallel process), reduce to 12 to leave headroom.
The verification step runs after the GPU subprocess completes, so in practice there
is no GPU contention.

---

## Acceptance Criteria

- [ ] `verify_top_results` with `top_n=5000` completes in under 5 minutes on i7-12700
- [ ] Results are identical to sequential verification on the same inputs (deterministic)
- [ ] No CUDA errors or Numba warnings from forked workers
- [ ] First iteration of a fresh session still completes within the 15-min iteration target
- [ ] `MAX_SWEEP_RESULTS = 5000` produces a valid sweep CSV and does not measurably
      slow down the GPU search phase