# Optimization Strategy: Critique & Thinking **Created:** 2026-03-13 **Status:** Thinking doc — no immediate code action required This document captures honest analysis of our current optimization approach, answers three specific questions raised in session, and maps out better directions for longer, more productive runs. --- ## The Scale Problem (Grounding Reality) Before addressing the questions, it helps to be concrete about the numbers. | Params file | Approximate search space | |---|---| | Template `params_strategy_activation_scores_1D.json` | ~688 quadrillion quadrillion (6.9 × 10^32) | | Current 1D asset file (AI-narrowed over many runs) | ~1.3 × 10^34 (paradoxically larger — fine steps × many params) | | Current 8H asset file | ~1.5 × 10^21 | At 610,000 samples/second (GPU), a 1-hour run produces **~2.2 billion samples**. Even against the 8H search space (10^21), that is **0.000000000002%** coverage. This is not a problem unique to this project — it is the fundamental reality of high-dimensional random search. The implication is that **we are never finding the global optimum**. We are finding a decent local pocket that happens to look good on the training data, and we are doing so largely by luck + repetition. --- ## Question 1: Why narrow the search space if coverage is already near zero? > "if we are only able to search such a small space, why would we be narrowing > our search by iteratively narrowing the start and end of each parameter?" **Short answer: narrowing can help, but only if the initial winner is meaningful. The current AI-driven narrowing has gone badly wrong on the 1D params.** Look at what the AI has done to the 1D asset params over many test iterations: ``` i_w_stoch 47.9 → 48.4 (range: 0.5, was originally 24–52) i_w_macd_pred 47.3 → 47.6 (range: 0.3, was originally 28–64) i_w_osc -1.96 → -1.50 (range: 0.46, was originally -1.9 – -1.6) i_long_exit_confirm_thresh 44.54 → 46.1 step 0.001 → 1,560 values (absurd precision) i_w_m3_momentum 4.141 → 4.143 (range: 0.002) ``` The AI is treating a 5-minute test run result as ground truth and zooming in around it. But a 5-minute run samples an infinitesimal fraction of the space — the "winner" it found is almost certainly not truly optimal, just the best of a tiny random sample. Zooming in around it locks out the rest of the space forever. **Does a short test run permanently affect a future long run?** Yes, because the AI modifies and saves `params_strategy_activation_scores_COINBASE_BTCUSD_1D.json` in place, and the next run (however long) uses whatever that file currently says. A 5-minute test run that narrowed `i_w_stoch` to `47.9–48.4` means an 8-hour follow-up run will never test values like 30, 35, 40 — even though those might be better for the 8H timeframe. **The narrowing approach is sound in theory but requires:** 1. A large enough initial sample that the winner region is meaningful 2. The ability to zoom out if the next run finds something outside the current range (the `ensure_winner_in_search_space` safeguard helps but only extends, doesn't restore) 3. A reset mechanism before starting a serious long run **Practical recommendation:** - Never use the AI-narrowed 1D asset params file as the starting point for a new timeframe (8H, 4H etc.) — always start from the template - Consider resetting the 1D asset params back to the template before any multi-hour serious run - The `--no-llm` flag can be used for "pure random search" runs that don't narrow at all — useful when you want max breadth --- ## Question 2: Are the parameter ranges well-chosen? > "maybe we should be generating them by looking at the data for that asset for > that timeframe then using some analysis of the ranges" **This is the right instinct. The current ranges are largely arbitrary.** ### How the current ranges were set The weight parameters were originally set as wide exploratory ranges (`i_w_stoch: 24–52`) without any principled basis. Over many AI-guided iterations they have been severely narrowed based on whatever happened to score well in the random samples seen so far. The threshold parameters (`i_long_entry_activation_threshold` etc.) control what activation score level triggers entry/exit. But the activation score's actual distribution — its typical values, range, and shape — **changes with every set of weights**. This means the right threshold range depends entirely on what weights you're using, and they can't really be optimized independently. ### The coupling problem The search space has a deep structural issue: weights and thresholds are **coupled**. If you double all weights, you double the score, and the threshold needs to double too just to get the same number of trades. The optimizer can solve this (it just finds a consistent weight+threshold combination), but it means there are many equivalent solutions and the search space is full of redundant near-duplicates that waste samples. One mitigation: **normalize the score to a fixed scale** (e.g., always maps to 0–1000) regardless of weights, so thresholds have a stable meaning. This isn't done currently. ### What data-driven range setting would look like For a given asset/timeframe CSV, you could: 1. **Compute the score distribution** for a grid of weight settings → see what threshold values are actually meaningful for that data 2. **Look at each component's actual distribution** in the CSV → set weight ranges that would make each component's contribution approximately comparable 3. **Use PCA or correlation analysis** on the sweep results (`optimization_sweep_*.csv`) to see which parameters actually drive score variance — many may be nearly irrelevant noise The sweep file (top 1000 results) is actually valuable for this. Running a quick correlation between each parameter and the Calmar score on the sweep data would immediately reveal which parameters matter most. ### Timeframe-specific concern When you run on 8H data with params copied from the 1D template: - There are ~3× more bars (8H has 3 bars per day vs 1) - Volatility per bar is lower - The score distribution will be different - Threshold values tuned for 1D may be completely off for 8H So the template approach (copy 1D → use for 8H) is a reasonable starting point for exploration but may need significant manual adjustment if 8H runs keep producing poor results. --- ## Question 3: Are we in a better position now to do longer runs? **Yes, meaningfully so. Here's what's changed:** ### Bugs now fixed that would have corrupted results - Timezone comparison failure for 8H data → verification never worked → global best never updated → entire AI loop was running blind - GPU kernel hardcoded threshold array size → would have silently used wrong trailing stop values for any param set after the 3rd threshold was added - Subprocess output not captured in log → couldn't diagnose issues after the fact ### What a longer run actually buys At 610k samples/s over 8 hours (3 iterations of ~2.7 hours each): - Samples per iteration: ~5.9 billion - Even against the cramped 1D asset space (~10^34), this is still near zero coverage - Against the 8H asset space (~10^21), it's still ~0.0000006% coverage The honest benefit of longer runs is not increased coverage — it's **more chances to get lucky in a good region**. Given truly random sampling, more samples means a better chance of finding a high-score point. The law of large numbers works in your favour, slowly. ### The bigger lever: better use of compute More hours of random search is the least efficient path to improvement. The same GPU compute budget could be used much more effectively with: **Bayesian Optimization** (e.g., `optuna` library): - Builds a probabilistic model of which regions look promising - Picks the *next* sample based on what it predicts will be best - Typically needs 10–100× fewer evaluations than random search to find a good result - The `optuna` integration would replace the GPU batch kernel with a CPU-side surrogate model, though the strategy eval itself can stay GPU-accelerated **CMA-ES (Covariance Matrix Adaptation Evolution Strategy)**: - Learns the shape of the fitness landscape (correlations between parameters) - Adapts a multivariate normal to focus on promising regions - Particularly effective when parameters are correlated (weights and thresholds are) - Also available via `optuna` or `cma` library **What we have now** is essentially "random search + human(LLM)-guided zoom", which is better than pure random search but far behind Bayesian methods. The LLM narrowing is also unreliable because LLMs are not optimization algorithms — they pattern-match on text and can (as we've seen) massively over-zoom. ### Recommended sequence for a serious long run 1. **Reset the 1D asset params** to the template (or a manually-reviewed wide range) 2. **Verify 8H params** are also starting from a reasonable range, not a past AI-narrowed state 3. Run with `--no-llm` for a pure random exploration pass (no narrowing, max breadth) 4. Analyze the resulting sweep file for parameter importance before doing another run 5. Then optionally use the LLM to narrow, but only on params shown to actually matter --- ## What ML techniques apply here Given your background: this problem is essentially **black-box function optimization** (we can evaluate f(params) = Calmar score, but can't differentiate through it). The main families: | Method | Sample efficiency | Good when | |---|---|---| | Grid search | Very low | Tiny space, exhaustive sweep needed | | Random search | Low | Baseline; surprisingly competitive per-sample vs grid | | **Bayesian (GP/TPE)** | **High** | Few thousand evaluations, moderate dimensions | | **CMA-ES** | **High** | Continuous parameters, correlated landscape | | Genetic algorithms | Medium | Discrete/mixed parameters, easy to parallelize | | Simulated annealing | Medium | Single-shot, avoids getting stuck in local minima | Our situation: 30+ mixed (continuous + discrete) parameters, fast evaluation (GPU), goal is to find good regions not prove global optimality. **CMA-ES or Optuna's TPE sampler** would likely outperform our current random search in wall-clock time to reach a given score threshold. The main obstacle to switching is that our GPU kernel does *batch* evaluation (a million at once), while Bayesian/CMA methods are inherently sequential (each sample depends on the previous result). The GPU advantage only exists in batch mode. A pragmatic hybrid: use Bayesian to pick a "center" region, then run our GPU batch within that region. --- ## Summary of Actions Worth Taking | Status | Priority | Action | Effort | |---|---|---|---| | Done | High | Reset 1D and 8H asset params to clean template before any serious run | 5 min | | Done | High | Analyze sweep file for parameter importance (correlation with Calmar) | 1 hour | | Done | Medium | Add `--no-llm` as the default for longer runs; only narrow after analysis | trivial | | Done | Medium | Score normalization: normalize activation score to fixed range so thresholds have stable meaning | 1 day | | Todo | Low | Replace LLM narrowing with CMA-ES or Optuna TPE | 1–2 days | | Todo | Low | Data-driven range initialization script: analyze component distributions in CSV | half day |