G=j/VdZddlZddlZddlZddlmZddlmZddlZ ddl Z ejjdeeej!j"j"ddlmZmZe j*de j*de j*d e j*d e j*d d ZdddddZdZdZdZdZdZdedefdZdedeefdZdedee ee ee!ffdZ"dZ#e$dk(rejJe#yy)u Chart-data integrity validator for TradingView exports ====================================================== Catches *broken or deficient* TV chart-data CSV exports BEFORE they distort backtests, parity checks, or winner-metric refreshes. Why this exists --------------- On 2026-06-23 the same MLP winners produced **34× different** backtest P&L (4H: 176% vs 5,987%) across TV exports taken minutes apart — because the exported price history differed (a truncated / partially-loaded chart). Score parity did NOT catch it (the recompute matched the stale score column). The only reliable signal was the data itself: a deficient export has too few bars / starts too late / has gaps. This tool checks the data directly. Five preconditions (per CLAUDE.md Primary Directive): 1. Objective metric : number of files failing integrity checks (target 0). 2. Measurement : the per-file structural checks below. 3. Action space : exit non-zero (blocks downstream use) + optional ntfy. 4. Stopping : one pass per invocation (run after each export, or wire to a launchd WatchPaths agent — see --help epilog). 5. Guardrail : callers (compare_tv_trades, refresh_winner_metrics) can import `validate_file` as a preflight and refuse to run on ERROR-level data. Checks (per data CSV) --------------------- ERROR (hard fail — data is unusable / will produce wrong backtests): - not a chart-data file (looks like a trade-list export) - missing OHLC/time columns - NaN / <=0 prices; highmin(open,close) - non-monotonic or duplicate timestamps - TRUNCATED history: first bar later than SCORE_START (IS window not covered) or row count << expected for the date span & timeframe WARN (review — may be benign): - first bar later than TRAIN_START (indicator warm-up may be short) - last bar far behind "now" (stale export) - large time gaps between bars - extreme single-bar moves (possible price glitch; legit crashes also land here) Usage ----- # validate every chart CSV in data/mlp/ (default) python3 tools/validate_chart_data.py # validate specific files python3 tools/validate_chart_data.py "data/mlp/COINBASE_BTCUSD, 240.csv" # treat warnings as failures; push an ntfy alert on any failure python3 tools/validate_chart_data.py --strict --notify Exit code: 0 = all clean (warnings allowed unless --strict); 1 = ≥1 file failed. N)Path)Optional) SCORE_START TRAIN_START)hours )days)4H6H8H12H1Drrrr)240360480720)COINBASE_BTCUSDCOINBASE_ETHUSDg?g?g?pathreturnct|j}|jddjjddjS)zHAsset prefix from a TV export filename, e.g. 'COINBASE_BTCUSD, 240.csv'.,r-)rstemsplitstrip)rrs D/Users/jameslopez/projects/TradingBot25/tools/validate_chart_data.py detect_assetr#XsD :??D ::c?1  # # % + +C 0 3 9 9 ;;cHt|jj}tD]B}|j d|s)|j d|s|j d|s@|cSt j D]\}}|j d|s|cSy)Nr_, )rrupper TF_DURATIONSendswith_MINUTES_TO_TFitems)rnametfminss r"detect_timeframer0^s :?? "D ==1RD "dmmatH&=QSTVSWyAYI#((*b ==2dV %I+ r$cg}g}i}t|}|js d|gg|fS tj|j j jj j_dj vsddhjj rdgg|fShd}|tj z }|r"|jd t||||fSt}||d <|d k(rd gg|fStjd dd} | jj!r9|jt#| jj%d| j&j)d} | j+j-j!r9|jt#| j-j%d| j+j.s|jd| j1| j3} } | | c|d<|d<dD]} tj4| d} | jj!r<|j| dt#| jj%d| d kj!s|j| dt#| d kj%dfddD\}}}}||kj!r.|jt#||kj%d||kj!s||kj!r4|jt#||k||kzj%d||kDj!s||kDj!r4|jt#||kD||kDzj%dtj6t8}tj6t:}t=|}tj>| r| |kDr@|t@vr8|jd| jCd |jCd!|d"nv| |kDr8|jd#| jCd |jCd$|d%n9| |kDr4|jd#| jCd&|jCd'tj6jEd(jGd)}tj>| rI|| z jHtJkDr3|jd*| jCd+|| z jHd,tM|}||d-<|rtj>| rtj>| rtN|}| j+jQj+}|||tRzkD}t|r8|jt|d.tRd/|d0|j3d1t#| | z |z d2z}||d3<||tTzkrD|jd4|d5|d6| jCd7| jCd8|d9||z d:d; |jWjY}||tZkD}t|r|j]}|jt|d| j^|r| j^|jCnd>d?|||fS#t$r}d|gg|fcYd}~Sd}~wwxYw)@uValidate one chart-data CSV. Returns (errors, warnings, stats). Empty errors → usable data. Importable as a preflight: `errs, warns, _ = validate_file(p)`. zfile not found: zcould not read CSV: Nz trade numbertypesignalz.looks like a trade-list export, not chart data>lowhighopentimeclosezmissing required columns: rowsrzempty file (0 rows)r7coerceT)errorsutcz unparseable timestampsz duplicate timestampsz×tamps are not strictly increasingfirstlast)r6r5r4r8r;z: z NaN/non-numeric valuesz values <= 0c3RK|]}tj|d yw)r:r?N)pd to_numeric).0xdfs r" z validate_file..s&bAaA"--1h77Aas$'z bars with high < lowz bars with high < open/closez bars with low > open/closezTRUNCATED history: first bar z is after SCORE_START u — zf has older history; the in-sample window is not fully covered (likely a partially-loaded chart export)z first bar u5 — in-sample window not fully covered (expected if z; is younger than SCORE_START; a truncated export otherwise)z is after TRAIN_START u# — indicator warm-up may be shortUTC)tzinfoz last bar z is zd behind now (stale export?)r.z time gaps > ×z (largest )r expected_rowszTRUNCATED/sparse: z rows but ~z expected for →z at z (z.0%z coverage)z bars move >z in one bar (worst ?u) — verify not a price glitch)0rexistsrAread_csv Exceptioncolumnsstrlowerr!issubsetsetappendsortedlen to_datetimeisnaanyintsumdt tz_localizedropna duplicatedis_monotonic_increasingminmaxrB Timestamprrr#notnaASSETS_FULL_HISTORYdatenowreplacer STALE_EXPORT_DAYSr0r)diff GAP_FACTORMIN_ROW_COVERAGE pct_changeabs EXTREME_MOVEidxmaxiloc)rr;warningsstatspexcrequiredmissingntr=r>colvohlc score_start train_startassetrir.intervaldiffsgapsexpectedretextremeworstrEs @r" validate_fileris FHE T A 88:"4&)*B559 [[ %%'++113BJ#'9'B'B2::'N@A2uLL7HRZZ(G 26'?2CDEx&& BAE&MAv%&E11 r&z(=Avvx||~ QVVX\\^,--DEF Axxz""$ Q\\^//1233HIJ 88: - - >?%%'15574E$)4!E'NE&M0 MM"S'( 3 668<<> MMSEC $7#88OP Q F<<> MMSECa $7#8 E F 0 cAabJAq!Q A{{} a!e[[]+,,ABC A{{}Q  q1uQ/446788TUV A{{}Q  q1uQ/446788STU,,{+K,,{+K  E xx ; 5,?#? MM/ ~=S##%&eE73WX  [ OOUZZ\N*@AQAQAS@TUGGLgNBC  [ OOUZZ\N*@AQAQAS@TU56  ,,  5 ! ) ) ) 6C xx~3:++.??)DIIK=cDj5F5F4GGcde $ BE$K bhhuo"((4.# !((*UX 223 t9 OOt9+]:,bJtxxzlRST u 01A5!)o x** * MM$QC{8*N::<.DIIK=RD1x<:LJX  ,,.   C# $%G 7| 7|nLc(:;ggi_D!&&QV-AX););)=^a(bc, - 8U ""M 9&se,-r5889s^'' _0^>8_>_c Xtjdtjd}|jddd|jdd d |jd d d |jdd d |j }|j }|sht tjjj}tdtjt|dz dz dz D}|s tdydx}}g}|D]r}t|\}} } | xr| xs |j } | r |j r9|s| sdn|rdnd} d} | j#ddd| j#dddt%j&| dj)dt%j&| d j)d!} t| d"t |j*| |D]}td#|| D]}td$||r+|d%z }|j-t |j*8| s<|d%z }|jsO|j-t |j*utd&t/|d'|d(|d)t1|xs|jxr t1|}|r2|j2r& dd*lm}|d+d,j7|d-d.d/g0|rd%SdS#t8$r}td1|d2Yd}~#d}~wwxYw)3Nz:Validate TradingView chart-data CSV exports for integrity.uBWire as an on-change guard (externally-created files) via launchd WatchPaths: ~/Library/LaunchAgents/com.james.tradingbot-data-validator.plist with WatchPaths=[/data/mlp] running: python3 tools/validate_chart_data.py --notify (alerts on ERRORs only) Logs → projects/logs/tradingbot-data-validator.log. ) descriptionformatter_classepilogfiles*z*chart CSVs (default: data/mlp chart files))nargshelpz--strict store_trueztreat warnings as failures)actionrz--quietzonly print files with problemsz--notifyzsend an ntfy alert on failurec3hK|]*}t|jjds|,yw) MLPScores_N)rr- startswith)rCfs r"rFzmain..s. F!7<<**<8 Fs02datamlpz*.csvz&No chart-data files found to validate.ru✅u❌u⚠️ r=z [r9rMz rows, rLr>] z ERROR: z warn: r  z file(s): z with errors, z with warnings only.)notifyzChart-data validation FAILED: r'zTradingBot data integrityr5warning)titleprioritytagsz(ntfy failed: rJ)argparseArgumentParserRawDescriptionHelpFormatter add_argument parse_argsrr__file__resolveparentrWglobrRprintrstrictquietgetrArerhr-rVrXboolr tools.ntfyjoinrP)apargsrrootn_errn_warn failed_filesrr;rtruokiconspanewfailedrrws r"mainrs  P << H BOOG3-YOZOOJ|:VOWOOIl9YOZOOJ|:YOZ ==?D JJE H~%%'..55 yyTF]U%:W%D!EF    67EFL "/"2%Z =\<_ $** "8u6%y 99W  )6#./wr||E'N7S7X7X7Z6[[^_a_k_klqrxly_z__`B_CCDED aQ ~dV,-A L$ %A L$ %  QJE   Q -  aKF{{##DGLL1), Bs5zl*UG>&AU VW %[ :T[[9T&\F $++ + ) 0<1H0IJ1[  1A + N3%q) * * +s%L L)L$$L)__main__)&__doc__rrsyspathlibrtypingrnumpynppandasrArinsertrRrrrconfigrr Timedeltar)r+rgrqrmrnrkr#r0tuplelistdictrr__name__exitr$r"rsM5n 3tH~--/66==>?+ ",,Q |r||!'< ",,Q   2(> ",,A  d4F =