# Adaptive Lag Correlation Engine ## Purpose and scope This is the language-independent contract for a causal, rolling estimator of the lagged relationship between a `source` series (for example global M2) and a `target` series (for example BTC price). The Python implementation in `strategies/adaptive_lag_correlation.py` is canonical. A future Pine library must reproduce this behavior rather than reinterpret it. This engine estimates a relationship; it does not establish causality or make a trading decision. It deliberately has no lag smoothing, hysteresis, or persistence filter. Those belong in a consumer-side stabilization layer. ## Precise lag convention The engine operates on two series aligned to the same, regular time grid. Lag is an integer number of grid intervals. For a candidate lag `L` and target-grid position `i`, the paired observations are: ```text source[i - L] <-> target[i] ``` Equivalently, source at time `T` is compared with target at `T + L`. A positive result of `72` therefore means **source leads target by 72 grid intervals**. On a daily grid, this is 72 days. At evaluation time `E`, only target positions in the trailing window ending at `E` are considered; every paired source position is no later than `E - L`. The implementation slices data at `E` before calculating anything, so later observations cannot affect a historical estimate. ## Algorithm For each evaluation point `E`: 1. Apply the configured causal transformation independently to the aligned source and target series. 2. For every `L` in `range(min_lag, max_lag + 1, lag_step)`, form the pairs `(source[i-L], target[i])` across the last `window` target positions ending at `E`. 3. Drop only pairs containing a missing or non-finite value. A candidate is ineligible unless at least `min_observations` valid pairs remain and neither side is constant. 4. Calculate Pearson's sample correlation coefficient for each eligible lag. 5. Select a candidate by the explicit `selection_mode`: `strongest_positive` (default) maximizes `r`; `strongest_negative` minimizes `r`; `strongest_absolute` maximizes `abs(r)`. 6. Resolve scores within `tie_tolerance` deterministically by selecting the shortest lag. The full eligible `lag -> correlation` spectrum is retained. No candidate is returned if none has enough observations. ## Correlation and transformations Pearson correlation is the baseline because it is simple, inspectable, and can be reproduced in Pine. The current Python API exposes these transformations: - `levels` (default): raw aligned values. - `difference`: first difference. - `pct_change`: one-period fractional change. - `log_return`: first difference of `log(value)`; non-positive values become missing rather than being silently altered. There is no claim that levels are the right economic measurement for M2/BTC. Transformation choice is a configuration/research decision and must be reported with any result. Spearman or other correlation methods are intentionally not yet implemented; adding one requires a named configuration option, tests, and an equivalent Pine definition. ## Quality metric Raw correlation measures relationship strength, not whether a single lag is well identified. The selected quality metric is **relative peak separation**: ```text selection_score(r) = r | -r | abs(r), according to selection_mode margin = best_score - runner_up_score relative_separation = margin / max(abs(best_correlation), machine_epsilon) ``` The engine returns the raw `margin`, `relative_peak_separation`, runner-up lag and correlation, and observation count. This is a **nearby precision** measure: a value near zero means a neighboring candidate is nearly as good (a plateau/ambiguous exact lag). A tie has separation zero. If there is only one eligible candidate, separation is `None` because no comparison is possible. It also returns a **distinct local peak**. A candidate is a local peak when its selection score is no lower than either adjacent eligible lag (flat peaks are represented once at their shortest lag). The engine excludes every local peak within `distinct_peak_exclusion_radius` of the best lag, then reports the strongest remaining local peak and its raw/relative margin. The default radius is 7 grid intervals, an explicit initial experiment setting rather than an established statistical threshold. On a daily grid, this treats lags within ±7 days as one local peak. Set it to suit the frequency and use case. This is deliberately not presented as a probability or a formal statistical confidence interval. Alternatives worth testing later include a Fisher-z comparison with a dependence-aware correction, bootstrap stability of the chosen lag, and local-peak width. The exposed spectrum and runner-up make each approach testable without changing the core estimator. ## Alignment, missing data, and availability `align_to_regular_grid()` provides the deterministic preprocessing baseline: - Inputs use a unique, monotonic `DatetimeIndex`; timestamps should be UTC. - Each bucket uses its last observation. - Source can be forward-filled, optionally with a maximum fill count. Target is never forward-filled, preventing synthetic price observations. - Missing values are handled pairwise inside each candidate window. - An incomplete grid period must not be passed as final until its bar is known to be closed. The consumer controls this as-of boundary. - For macroeconomic inputs, index a value at the time it became available to the strategy, not merely its economic reference period. Revisions need a separately versioned vintage-data policy before they can be used in a causal backtest. For the supplied six-hour TradingView export, the diagnostic uses a daily grid: source column `M2 US EU CN` is last-observation then forward-filled, while BTC `close` is the final six-hour close of each UTC day. This is an explicit experimental policy, not a hidden property of the estimator. ## Python API and diagnostic Use `LagCorrelationConfig` and `AdaptiveLagCorrelationEngine.estimate_at()` for a single historical date, or `.estimate()` for every aligned timestamp. Each `LagEstimate` exposes `best_lag`, `best_correlation`, observation count, `confidence`, the complete `spectrum`, and `top_candidates()`. Run the supplied-data diagnostic with: ```bash .venv/bin/python tools/diagnose_adaptive_lag.py ``` It prints selected dates with configuration, selected lag, correlation, quality, and the top candidate spectrum entries. It is descriptive research output, not an optimization or a claim that its current lag is tradable. ## Research findings — 2026-08-21 The following is a dated, reproducible non-finding from the supplied `data/COINBASE_BTCUSD, 360-IndicatorM2LI_DebugArchive.csv` export. It exists to preserve the result and prevent a visual pattern from later being mistaken for validated trading evidence. The test reconstructed the visualizer on its daily calculation grid: last six-hour BTC close and last `M2 US EU CN` observation per UTC day, with M2 forward-filled. It used positive Pearson correlation of log returns, a 365-day trailing window, candidate lags 30 through 120 days in one-day steps, and a distinct-peak exclusion radius of seven days. A confidence value of `0.10` means the minimum of nearby-peak and distinct-peak relative separation was at least `0.10`; it does not mean 10% probability or 10% correlation. - High confidence was associated with BTC having already risen more over the prior 7, 14, 30, and 60 days. It was not a generally stronger next-7 to next-60-day BTC state than low confidence. - Fresh crossings above `0.10` had a visually interesting but inconclusive historical 12–48-hour return bump on the six-hour chart. The strongest short-horizon example was +0.28% over the next 12 hours, with a time-preserving circular-shift p-value of about 0.16; none of the tested 6–72-hour horizons was statistically convincing. - The apparent 12-hour crossing effect weakened from +0.44% in 2019–2023 to +0.08% in 2024–2026. That instability is incompatible with treating it as a validated entry signal. - Top 5% six-hour BTC up-bars were not concentrated after high confidence: 55.0% followed a high-confidence reading versus a 58.3% high-confidence base rate over valid signal bars. Conclusion: retain confidence, the correlation spectrum, and fresh-crossing markers as visual/research tools. Do not use confidence as a strategy entry or exit condition unless a predeclared rule succeeds on fresh prospective data with realistic availability timing and trading-cost assumptions. ## Validation contract `tests/test_adaptive_lag_correlation.py` verifies fixed known lags, changing synthetic lags, selection modes, deterministic ties, pairwise missing-data handling, regular-grid behavior, invalid inputs, and no-look-ahead invariance: mutating every observation after an evaluation time must leave that estimate unchanged. The future Pine implementation should first consume the same aligned exported data and match its Python spectrum on chosen historical bars before any indicator or strategy consumes it. ## Pine visualizer `libraries/LibraryAdaptiveLagCorrelation.pine` implements the causal rolling Pearson search as a pure calculation library. It returns the best lag, its correlation, the nearby runner, and the strongest distinct local peak outside an explicit exclusion radius. `indicators/IndicatorAdaptiveM2LagCorrelation.pine` is a visualization-only consumer. It forces a daily calculation timeframe, so lag inputs and plots are days even when attached to an intraday BTC chart. It plots the best lag, nearby runner, and distinct local peak with 42, 72, and 105 day reference lines. Correlation and peak-separation values are available through the Data Window and status line rather than a table, because Pine forbids table drawings in a script with a forced calculation timeframe. The best-lag line is confidence-colored: gray means the lag is ambiguous and aqua means its relative separation meets the configurable full-confidence threshold (default `0.10`). This is the minimum of nearby and distinct-peak separation, so it requires both a sharp local peak and no comparably strong distant peak. It is deliberately separate from correlation strength, which is shown in the status line. To use it in TradingView, publish `LibraryAdaptiveLagCorrelation` first under the `NiceOrbit` account as version 3, then paste the indicator. Its import is intentionally separate from plotting so future strategies can use the library without importing visual code. It adds no `request.security()` calls; M2 data comes only through the existing money-supply library. ## Walk-forward validation `strategies/adaptive_lag_walkforward.py` evaluates the estimator without reusing its selection window as proof. At each evaluation time `t`, it selects the lag using data through `t`, then records the matured pair `(source[t], target[t + selected_lag])`. Its score is calculated only over the collection of matured pairs. Fixed-lag baselines use the identical protocol. `source_availability_delay` shifts source observations later by a configured number of grid intervals before selection and scoring; it models publication delay but does not invent a release schedule. Run the comparison with: ```bash .venv/bin/python tools/walkforward_adaptive_lag.py --transform log_return ``` The command reports correlation, directional hit rate, mean selected lag, and a deterministic moving-block bootstrap interval. The bootstrap preserves short contiguous runs, making it more appropriate than IID resampling for serially dependent financial observations; it is still an uncertainty estimate, not a trading-performance result.