J-jdZddlmZmZddlZddlZddlZddlZddlm Z m Z ddl m Z ddl Z ddlZddlmZddZddZd Zd ZGd d ZGd deZGddeZGdde ZGddZ[[y)zTools for (lean) experimentation and collecting data. See also cma.optimization_tools and cma.utilities.math.test*. TODO: write some running examples as doctests. )divisionprint_functionN) defaultdict namedtuple) literal_eval)warnings_and_exceptionscdt|zdzd|dk(zz}tjd|jtjj ddzS)z.a unique timestamp up to 1/10^decimals secondsz{0:.zf}.rz%Y-%m-%d-%Hh%Mm%Ss)strtimestrftimeformatsplit)decimalsss a/Users/jameslopez/projects/TradingBot25/.venv/lib/python3.12/site-packages/cma/experimentation.pyunique_time_stamprs\X%x1}(==A ==- . HHTYY[ ! ' ' ,Q / 11c|3tjt|tj|}}t|d|zkDr"|ddd}|ddd}t|d|zkDr"||fS)znreturn (index, x_down) if y is None, else (x_down, y_down). Example: ``plot(*down_sample(mydata))`` N)nparangelenasarray)xylen_s r down_samplersn  yyyQ "**Q-1 a&1t8  ccF ccF a&1t8  a4Krctj|}tj|}t|r0t |tj ||zt|z Stj S)a]return the average of finite entries in `data`, multiplied by the ratio ``Ndata / Nfinite == len(data) / sum(np.isfinite(data))``. This measure has been called Q-measure or success performance one, sp1. It is computed as the average over all finite entries times the number of all entries divided by the number of finite entries. If all entries are finite this is the average value. Details: the multiplier operates under the assumption that non-finite entries contribute the same as the average finite entry and resampling can be applied to get a finite entry. )rrisfinitesumrmeaninf)dataaidxs rsp1r(&sR 4A ++a.C25c(3q6BGGAcFO #c#h .FFrc<tjj|s%tjdj |y tj tjj|tj||y#t$rY"wxYw)z8copy src to dest and created dest folder(s) if necessaryz.{0} does not exist (yet), nothing to be copiedN) ospathexistswarningswarnrmakedirsdirname Exceptionshutilcopy2)srcdests r copy_filer69so 77>>#  FfSk #  BGGOOD)* LLd    s2B BBc$eZdZdZddZdZdZy) TimeWarnerzcontext manager to print the time spent iff it exceeds a threshold. Usage: ``with TimeWarner('saving'):``, the argument is used only for printing the message in case. c ||_||_yN)name warn_time)selfr;r<s r__init__zTimeWarner.__init__Ks "rc8tj|_|Sr:)r t0r=s r __enter__zTimeWarner.__enter__Ns))+ rctj|_|j|jz |jkDr=t dj |j |j|jz yy)Nz"TimeWarner: {} took {:.1f} seconds)r t1r@r<printrr;)r=exc_type exc_value tracebacks r__exit__zTimeWarner.__exit__QsW))+ 77TWW t~~ - 6==diiSWSZSZIZ[ \ .rNr)__name__ __module__ __qualname____doc__r>rBrIrrr8r8Es #]rr8c&eZdZdZdZedZy) ClassFromDictzMset class attributes from a `dict`, see also `cma.utilities.utils.DictClass2`cRt||_|D]}t||||yr:)dict_dictsetattr)r=dict_keys rr>zClassFromDict.__init__Xs'%[ C D#uSz *rc@tfdjDS)zrcollect only original attributes, check out ``.__dict__`` to see also the attributes later added. c3:K|]}|t|gywr:)getattr).0rWr=s r z(ClassFromDict.as_dict..asD#S'$,-s)rSrTrAs`ras_dictzClassFromDict.as_dict\s DDDDrN)rKrLrMrNr>propertyr]rOrrrQrQVs W+EErrQceZdZdZdej ej ej fdfdZdZ dZ dZ ddZ dd Z dd Zdd Zdd Zd ZddZy)Resultsaa container to communicate (changing) results via disk. The ``data`` attribute is a dictionary (`DataDict`) which contains all observed data, usually a list for each key. Any new container reads previously saved data (with the same name) on initialization. Before the data are saved, the old data are backupped under the current timestamp. Use case example:: import cma.experimentation # create or load 'sweep_data' results res = cma.experimentation.Results('sweep_data') # save data after each trial for dim in dimensions: res.backup() # in case we want fewer backups for ...: x, es = cma.fmin2(...) res.data[dim] += [es.result.evaluations if 'ftarget' in es.result.stop() else np.inf] res.save(backup=False) # like this we can never lose data Details: Saving/loading of nonfinite values with `ast.literat_eval` is covered with the ``values_to_string`` parameters. Some methods handle nested dictionaries, however it seems to be safer to use tuples as keys in a flat dictionary or use different `Results` instances, e.g. by dimension and/or popsize, like:: res = experimentation.Results('sweep_c_dim={}lam={}.format(dimension, popsize) In particular ``.data.clean()`` for `float` keys does (currently) not work with key tuples. Nr c B|dn||_d|jddvr|jdz|_tjj|j\}}tjj |\}}||_tjj |d|zdz|_||_t|_ td|_ tjj|jr|j|jr|d kDr|t|j}t!d j#t%|j|d |d |jt'|j)j+yyyy) N results_dictr iz.pydictz._z-backup)inforzHResults("{3}") loaded {0} data key entries ({1} ... {2}) with {4} values)filenamer*r+rsplitext _extensionjoinbackup_dirname_values_to_string_valsDataDictr%rS meta_datar,loadsortedrErrr"samplesvalues)r=r;values_to_stringverboser+extkeyss rr>zResults.__init__s9+/,D dmmBC( ( MMI5DMWW]]4==1 dGG$$T* c ggll4y1HI&6#J 2 77>>$-- ( IIKyyWq[dii()vc$))nd1gtBx"mmS1F1F1H-IKL)y )rc.t|j|S)z/access ``Results.data`` directly from `Results`)rZr%r=r;s r __getattr__zResults.__getattr__styy$''rcddl}|jjddddz} |j|}t d#t j dYyxYw)zPreturn dictionary from load json file with keys to `int` or `float`, if possiblerNr r z.jsonz?json file not loaded, this warning needs to be improved/removedz4how do we know whether to convert keys into numbers?) jsonrfrsplitrnr-r.NotImplementedErrorrointfloat ValueError)r=rzrfresk to_numberknews r _load_jsonzResults._load_jsons]==''Q/2W< ))H%C""XYY  MM[ \ s AAc~|jr|jt|jd5} t t |j |_ ddd|jd|jjdd|_ |S#t$r tdwxYw#1swYWxYw)z! GGJ  GGDM? > >s )#BB)TFr:)rKrLrMrNrr$nanmathr>rxrrnrrrrrrrrOrrr`r`cs_%N%'VVRVVTXX$>L.(.  * !#F   rr`ceZdZdZddZdZeejfdZ eddfdZ dZ ed d fd Z dd Z dd ZdfdZdfdZdfdZdZddfdZedZedfdZdZy)rlaA (default) dictionary like ``parameter_value: list_of_measures``, e.g. with float parameter or dimension as key and runtimes as value. This class is used under the hood in the `Results` class. A main functionality is the method `clean`, which joins all entries which have almost equal keys. This allows to have a `float` parameter as key. This class provides simple computations on this kind of data, like ``x, y = .xy_arrays() == sorted(keys), sp1(values)``. If the dictionary values are not lists, one may get rather unexpected results or exceptions. Details: this class allows to use `float` values as keys when `clean_key` and `set_clean` are used to access the data in the `dict`. Inheriting from `defaultdict` with `list` as default value, the syntax:: data = DataDict() data[first_key] += [first_data_point] without initialization of the key value works perfectly fine. Caveat: small values are considered as the same key, even if they are close to zero. Either use a different comparison via the `equal` keyword parameter, or use ``1 / key_value`` or `log(key_value)``. TODO: consider `numpy.allclose` for almost equal comparison? Nctj|t|Wt|dr|j|_t|dr|j |j y|j |yy)zUse ``dict(dict_.data or dict_)``, and `dict_.meta_data` for initialization. Details: `dict_.meta_data` are assigned as a reference. Nrmr%)rr>listhasattrrmrr%)r=rVs rr>zDataDict.__init___sZ T4(  uk*!&uf% EJJ' E" rc6|}|D]}||xx||z cc<y)z.normalize_in_place..s2\qt\srr )minrq)rmin_rs rnormalize_in_placez/DataDict.aggregated..normalize_in_placesJ2SZZ\22DQT)3q6!94AJr)ror~rsetitems) r=rrelativebyr%rrurr by_valuesres_by_rs r aggregatedzDataDict.aggregateds  d|?CDt!q5T!W&DG 55tD z"3' -00CqQrUC 0!#i.1C 2SYY[3a[TQTUVXTY]`T`AqD[3a bDI2  E 14bs*B6 B;C C c\|}t|Dcic]}|t||c}Scc}w)z1return a `dict` with the number of values per key)ror)r=r%rs rrpzDataDict.sampless/)/6A3tAw<666s)g?r c"|j|\}}tj|}|s||S|dvsJ|||z}|dk\rC|t|kr5|||||zkr'||z }|dk\r|t|kr|||||zkr'|||z S)zAslack_index_shift can be +-1, looking to the right/left of argmin)rer r)rrargminr)r=rslackslack_index_shiftrridxminrs rrzDataDict.argmins~~c"11 V9  G+>->>+ & &1fSV!& 0A(A " "A1fSV!& 0A(A&&''rc ddl}|}t|}tt||z Dcic]H}|||||zft |j j |||||||z|ddJc}Scc}w)zIreturn p-values of the `mannwhitneyu` test of entries adjacent with `gap`rN two-sidedmethod alternativer ) scipy.statsrorrr~stats mannwhitneyu)r=gaprscipyr%rurs rtestszDataDict.testssd| s4y3/ 10A a$qu+&KK,,T!W tD3K'8%;-@@AC)DD0 1 11sA A;c Tddl}|}t|}||vrtdj|||B|j |dz}|t |k\rtdj||||}||ft |jj|||||ddiS)z)return p-value of the `mannwhitneyu` testrNzkey {} not in data keys {}r z!key {} is the last key in data {}rr) rrorrindexrr~rr)r=rWkey2rrr%rur's rtestz DataDict.testsd| d?9@@dKL L <**S/A%Cc$i !D!K!KCQU!VWW9Dd UKK,,S 4:%;-@@ACDE Erc.|dz |cxkxr|dzkScSNgư>rOrrs rzDataDict.sq4x!'>a$h'>rcrt|jD]}||vr|j|||S)z+merge keys which have almost the same value)equal)rru clean_key)r=rrWs rcleanzDataDict.cleans8 $C$ NN3eN ,% rc.|dz |cxkxr|dzkScSrrOrs rrzDataDict.D10Gq4x0Grc|jdd}|j|||urA|j||}||k7sJ||xx||z cc<||=|j|||urA|||d<|S)zset similar key values all to be `key`, return `key`. Use method `set_clean` to access and change the clean-key dictionary *value* more conveniently. rmN)r _near_key)r=rWrrmrs rrzDataDict.clean_keys HH[$/ nnS%(3sE*A8O8 Ia IQ nnS%(3  )D  rc.|dz |cxkxr|dzkScSrrOrs rrzDataDict.sq4x!/Fa$h/Frc||vrgn t||}|g}|j||||ur7|j|||}||gz }|||z }|j||||ur7|S)zget the merged values list of all nearby keys. Caveat: the returned value is a new list :See also: `clean`, `set_clean`. )rr)r=rWrr done_keysrs rget_nearzDataDict.get_nears~tObd3iE nnS%33>sE95A ! I 47NCnnS%33> rc.|j|||S)ajoin all entries with similar `key` and return the new value, a joined list of all respective values. This is the same as `clean_key` which however returns the new key, not the values. Example:: data.set_clean(key) += [new_data_point] # same as data[data.clean_key(key)] += [new_data_point] # or more explicite, however with a different order of the data data[key] += [new_data_point] data.clean_key(key) # joins data, however in the "wrong" order # similar as data[key] += [new_data_point] data.clean() # cleans *all* keys )r)r=rWs r set_cleanzDataDict.set_cleans. sCyrc.|dz |cxkxr|dzkScSrrOrs rrzDataDict.rrc|g}t|jD]} |||r ||k7r||vr|cS|S#t$rY)wxYw)zRreturn a key in self which is ``equal`` to ``key`` and otherwise ``key``. )rorur)r=rWrexcluders rrzDataDict._near_keys_ ?G $A C=Q#X!72BH%    s< AAc t|Dcgc]}tj|s|}}tj|Dcgc]#}t tj ||%c}}tj|Dcgc]}t ||c}} tdgd}|tj|||||z }|Scc}wcc}wcc}w#ttj|||||z d}Y|SxYw)zreturn a class instance with attributes `x` (i.e. keys), `n`, `nsucc`, and `rate` as arrays. TODO: consider using cma.utilities.utils.DictClass2 instead namedtuple? Successes)rnsuccnrate) rorisscalarrr"r!rrrQ)r=rrurrrrs r successeszDataDict.successess"$<:<: tDt!C DG 45tDE JJd3dDG d3 4 ";#>@IBJJt,eQ BC ;D3  jj&C  s"CC(CC!/C!!(D dct)z-TODO:review percentile based on bootstrapping) r|rorrrrandomrandintrappend percentiler) r=prctilerrprrur bstrappedrr'r%s rrzDataDict.percentile6!!rc*tt|Sr:)rrSrAs r__repr__zDataDict.__repr__DsDJrr:)r auto)Nr)rKrLrMrNr>rr(rrrrrprrrrrrrrr^rrrrOrrrlrl?s> #  rzz5(!5T87 C2 ( 1E"?$H #G 4$H ,'*3 1 rrlceZdZdZddZdZedZddZddZ dZ dZ d Z iifd Z iifd Zd Zd ZddZedZdZdZy) ResultsPandasa WIP ad hoc code for writing (changing) results in a `pandas.DataFrame` and to disk, where "results" refers to end-results of experiment repetitions rather than the traces of single runs. Caveat: this is an ad hoc implementation, some interfaces may be incomplete, interface details are still versatile and in flux, some recent changes may be faulty. Main features, similar to `Results`, are * Intermediate savings of results which consequently can be loaded from a different shell while the experiment is running. * Backup under the current timestamp before each saving into a ``'backups-name'`` folder (optional but default). * Similar float values can be "equalized" for correct data aggregation, see the `check_close_values` and `equalize_close_values` methods. This uses `np.isclose` which considers 1e-8 to be close to 1e-9 and 1+1e-5 close to 1+1e-6. Compared to `Results`, this class is useful to store more information for each run, like the termination condition or a final condition number or constraints violations or meta parameter information. (In contrast, with `Results` the workaround for catching the final condition would writing a nonfinite entry when the target was not reached.) A guiding code example:: import cma.experimentation res = cma.experimentation.ResultsPandas('some-name') # reloads data to append/continue for dim in dimensions: es = cma.CMA(dim * [2], 1, {'verbose': -9}) # a tracker in case we want to track, say, a minimum over the trace as result es.optimize(cma.ff.rosen, callback=my_result_tracker) if es.opts.get('verbose') == -9: # do not save data while testing the setup res.append([es.N, es.popsize, es.result.evaluations, 1 if 'ftarget' in es.stop() else 0, repr(es.stop), es.condition_number, my_result_tracker.my_val_of_interest], colums=['dimension', 'popsize', 'evaluations', 'targethit', 'stopcondition', 'conditionnumber', 'myvalue']) res.save() # load the data: res = cma.experimentation.ResultsPandas('some-name') print(res.summary) # summary statistics of columns in a dict res.df # the pandas data frame Notes: ``self.df.drop(...)`` could be used to return a data frame with some entried dropped. c ddl}||_||_|dk(r ddl}tjj|j\}}tjj|d|z|_ tjj|j|j zr|jn|j|_tjdt j" y#t$rtjdwxYw#t$r tjdd|_Y!wxYw) zload data when `filepathname` exists. ``format='.csv'`` is human readible, however ``'.feather'`` is much more performant, ``'.parquet'`` should work too. rNzzResultsPandas.__init__s    !   Z  ) ww}}TYY'1 ggll1j1n= 77>>$))doo5 6 IIKllnDG ]7JJ L+  MMX Y   ) >?"( )sC4D4 D%E?Ec.t|j|S)z=access the `DataFrame` ``Results.df`` directly from `Results`)rZrrws rrxzResultsPandas.__getattr__stww%%rc |}i}|jjD]}|j|jj|j|jj }}t |t tjfr7t |t tjfrt |t |}}n t|t|}} t|j|j}|tt|j|||fg||<|S#t$rY`wxYw#t$r+t td|j|D}YpwxYw)znreturn number of finite entries, number of different values, and (min, max) for each column. c3$K|]}|dv yw))rrcNrOrs rr\z(ResultsPandas.summary..s!M 1!;"6 s)rcolumnsdropnarmaxrr}rintegerr~r1rr"r)r=rrpr;rmax_n_valids rsummaryzResultsPandas.summarys@ FFNND,,.224cffTl6I6I6K6O6O6Q$D$bjj 12z$bjjHY7Z YD d!&teDk$D OcffTl1134% SVVD\!23#Tl,GDM# ! Oc!Mt !MMN Os$ D8$&E8 EE1E;:E;c $|xr|j tjtjj |j |j ztd|5|j dk(rG|jdd|jj|j |j zfi|n|j dk(r5|jj|j |j zfi|ng|j dk7r.tjdj|j |jj |j dzfi|dddy#t$rY#wxYw#1swYyxYw) asave data, warn when saving takes more than `time_s` seconds. `kwargs` are passed to the saving method of the `pandas` data frame. Details: `save` is in essence just a shortcut for:: self.backup() self.df.to_feather(self.name + self.extension) # assuming .extension == '.feather' TODO: generalize by passing the `DataFrame` method name for saving, like ``.save('to_feather')``? Annoyingly, pandas does not add a proper extension by default. rr rFz.parquetr z8'{0}' extension not recognized, saving in feather formatN)rr*r/r+r0r;rhr1r8 setdefaultrto_csv to_parquetr-r.r to_feather)r=rtime_skwargss rrzResultsPandas.saves*  4;;=  KK DOO(CD E '&(!!'51tyy4??:EfEJ."""499t#>I&I??j0MM"\#)6$//#:<"""499z#9DVD( '    ' 'sA E6+DF6 FFFcddl}td5|jdk(r.|j|j|jz|_nj|j dk(r.|j|j|jz|_n-|j|j|jz|_ddd|S#1swY|SxYw)NrzResultsPandas.loadr z.parqet) r r8rhread_csvr;r _extention read_parquet read_feather)r=polishrs rrnzResultsPandas.loads , -&(++dii$//&ABI-//$))doo*EF//$))doo*EF . . s B(CC c|j}tt||}tt j |dd|ddD]/\}}|s |||j t||||dzk(|f<1y)Nrer )rror enumerateriscloselocrZ)r=columnrrrts requalize_close_valuesz#ResultsPandas.equalize_close_valuessu WW 3r&z? #bjj3B1278DAq@A!wr6*a!f4f<=9rc|jjDcgc]}d|vc}|jd<|jjdgjj }|jj Dcgc]<}||vr&t|jj|||ntj>c}|jd|z<ycc}wcc}w)z a quick hackcallbacksuccess dimensionbest_N) rstopdf_succgroupbyevalsrr8rZr1rr)r=r2rr'dims r_polishzResultsPandas._polishs7;ww||D|!jAo|D ll""K=177>>@.2gg.?.?%A.?sPSVYzWTWW[[S-BF%K_a_e_e%e.?%A& !E%As CACcttt|j|}t t j |dd|ddr&tjdj||yy)Nrer zdFound ambiguous double values in column {0} in {1}.Call .equalize_close_values() and .save() to fix.) rorrZranyrr0r-r.r)r=r2rs rcheck_close_valuesz ResultsPandas.check_close_valuess^ 3wtww/0 1 rzz!CR&!AB%( ) MMN!6&!, . *rc.|j|g|||y)zappend a single data rowN)extend)r=r%r kwargs_df kwargs_concats rrzResultsPandas.appends TFGY >rcddl}|jdd|j|j|jj |fd|i|gfi||_y)z0extend frame by data which is a sequence of rowsrN ignore_indexTr)r r"concatrr from_records)r=r%rrErFrs rrDzResultsPandas.extends[  6"))TWW %BLL % %d IG Iy IK]N[]rct)z)call ``self.df.drop`` and reassign result)r|rdrop)r=argsr's rrLzResultsPandas.drop rrc>|jjddy)z;caveat: the original index is lost which may be undesirableT)rLinplaceN)r reset_indexrAs rrPzResultsPandas.reset_indexs t4rc|ji}}|rAtt||D]%}tt||||k(|||<'|Stt||}|S)z7a sorted list of values in the column of name `column`.)rror)r=r2rrrr8s r column_valueszResultsPandas.column_valuesso''2C #C2K0 !'Br"v/B,CF,K(L!MI1 RZ)C rcjjDcgc]}d|jvs|j}}fd}jjDcgc]}|||s |jc}Scc}wcc}w)z>TODO: revise such that we have a boolean failed_setting columnr6c|D]c}|jt|fjj|jtjj|fk(scyy)NTF)r8rZrr1)dfailed_indicesrr2r=s rfailed_parameterz7ResultsPandas.failure_indices..failed_parameters`#KKF!34Q9Q9QSZ[_[b[b[f[fgh[ikqSr8ss$r)r itertuplesr:Index)r=r2rUfailedrWs`` rfailure_indiceszResultsPandas.failure_indicess|$(77#5#5#7T#7a:QVV;S!''#7T  "&!3!3!5U!5A9I!V9T!5UU U VsBB%B 4B cp|j}tdjt|t|j|jdj|t t jt jt jtt||y)zdo not use `iterrows` but `itertuples` https://stackoverflow.com/questions/16476924/how-to-iterate-over-rows-in-a-dataframe-in-pandas z.{0} data, {1} failures, failure_indices = {2} zmin {0} fraction = {1:.4}N) rrErrr[rrexpdifflogror)r=r2rs r print_checkszResultsPandas.print_checks%sWW ?FFs2wPSTXThThPikokkA)00RVVBGGBFFSYZ]^`ag^hZiSjLkDl=m9no rctjj|jt d|j z}t |j|j z|||_yr) r*r+rirjrrhr6r;rrs rrzResultsPandas.backup.sMww||D//-a04??BD$))doo-t4rN)r )Trr)r8)rKrLrMrNr>rxr^r rrnr4r?rBrrDrLrPrRr[r`rrOrrrrGs9tL>&2E> EA./1?/1]05VV rrrJ)Ni)rN __future__rrr-r*r2r collectionsrrastrrnumpyrcmarrrrr(r6r8objectrQr`rlrrOrrrhs 0 / C1 G& ]]" EF EZfZxF {F Pl l \ nr