IZj& dZddlZddlZddlZddlmZddlmZddlm Z ddl m Z ddl m Z ddlmZdd lmZd d lmZmZd d lmZejd ZeeddZededDZedd3dedede de!fdZ"de#edede#efdZ$d4dedede#efd Z%d4dedede#efd!Z&d5dedede'fd#Z(ed$d%edede#efd&Z)Gd'd(Z*d)edefd*Z+d+\Z,Z-Z.Z/Gd,d-Z0e0eZ1d.eddfd/Z2d0ed1ede!fd2Z3dS)6z>Code parts dedicated to duplicate removal and text similarity.N)SequenceMatcher) lru_cache)blake2b)add)RLock)Any)_Element)LRU_SIZE Extractor)trimz\.[^/?#]{2,63}$ bit_countcFt|dS)N1)bincount)xs e/Users/jameslopez/projects/MentorCore/.venv/lib/python3.11/site-packages/trafilatura/deduplication.pyrsSVV\\#5F5Fcji|]0}tjt|ddk-|d1S)rP ) unicodedatacategorychr).0is r rs@eeeak>RSVWXSYSY>Z>Z[\>]ad>d>d1c>d>d>drii)maxsize? reference new_string thresholdreturnctd|}td|}td|||kS)zIReturn the similarity ratio between two short strings, here domain names.N)STRIP_EXTENSIONsubrratio)r"r#r$s ris_similar_domainr+sQ ##B 22I $$R44J 4J 7 7 = = ? ?9 LLrtokens target_lengthcg}tdddD]*fd|D}t||dz kr|cS+|S)zCHelper function to get a sample of tokens based on length criteria.c:g|]}t|k|S)len)rtrs r z)_get_sample_by_length..%s$222s1vvzz!zzzr)ranger3)r,r-samplers @r_get_sample_by_lengthr9!se F 1b"  2222V222 v;;-!+ + +MMM , Mr@ inputstringlengthc|t}d|D}t||S)u This fallback implementation is used when the primary sample_tokens function generates an empty token list. This is mostly relevant for languages like mandarin where none latin-based punctuation is used e.g.: 。 c:g|]}||Sr2)isalnum)rr4s rr5z*sample_tokens_fallback..3s% ; ; ;Aqyy{{ ;a ; ; ;r) translate PUNCT_TBLsplitr9)r;r< clean_textr,s rsample_tokens_fallbackrD+sF&&y11J ; ;))++ ; ; ;F  0 00rc.g}|D]J}|tj}|r||Kt ||}t|dkrt||S|S)zbSplit input into list of tokens and adjust length threshold to make sure there is enough data.r) rBstripstring punctuationr?appendr9r3rD)r;r<r,tokenr8s r sample_tokensrK7sF""$$!! F.// ==?? ! MM% "66 2 2F 6{{a%k6::: Mrcdt|}t||S)z=Create a bag of words and generate a hash for a given string.r digest_size)joinrKrFrencodedigest)r;r< teststrings rgenerate_bow_hashrTHsR- 4455;;==J :$$&&F ; ; ; B B D DDri@rJctt|ddfdt |DS)zFToken's contribution to a Simhash vector, cached across all instances.rNbigc(g|]}d|zzrdndS)r r0r2)rr token_hashs rr5z"_vector_to_add..Ss+ F F F1qAv& .AAB F F Fr)int from_bytesrrQrRr7)rJr<rYs @r_vector_to_addr\OsZ A F F F M M O OQVWWJ F F F Ff F F FFrc eZdZdZddgZ ddedededzd dfd Zded efd Zd efd Z d ed edzfdZ d eezdzd edzfdZ ddd efdZ ddd e fdZdS)SimhashzAImplement a basic Charikar hashing approach of string similarity.hashr<r'r:Nr; existing_hashr%cr||_||p|||_dS)z&Store length and existing or new hash.N)r<validate create_hashr_)selfr;r<r`s r__init__zSimhash.__init__[s5 MM-00QD4D4D[4Q4Q rc dg|jzt||jD]8}tttt ||j9t fdt|jDS)zCalculates a Charikar simhash. References used: https://github.com/vilda/shash/ https://github.com/sean-public/python-hashes/blob/master/hashes/simhash.py Optimized for Python by @adbar. rc3:K|]}|dkd|zVdS)rr Nr2)rrvectors r z&Simhash.create_hash..ps/HHaa16HHr)r<rKlistmaprr\sumr7)rdr;rJrhs @rrczSimhash.create_hashes t{""; << P PE#c6>%+M+MNNOOFFHHHH5#5#5HHHHHHrc:t|jddS)z3Convert the numerical hash to a hexadecimal string.r6N)hexr_rds rto_hexzSimhash.to_hexrs49~~abb!!r inputhashcT t|dS#ttf$rYdSwxYw)z2Convert the hexadecimal hash to a numerical value.N)rZ TypeError ValueErrorrdrqs r _hash_to_intzSimhash._hash_to_intvs> y"%% %:&   44 s ''c\t|tr,dtt|cxkrdkrnn|St|trU|r,dt|cxkrdkrnnt|S||SdS)z9Validate the input hash and return it, or None otherwise.N) isinstancerZr3strisdigitrwrvs rrbzSimhash.validate}s i % % "C NN0C0C*I*I*I*Ir*I*I*I*I*I  i % % 0  "" &rS^^'A'A'A'Ar'A'A'A'A'A9~~%$$Y// /tr other_hashc:t|j|jz S)zJReturn distance between two hashes of equal length using the XOR operator.)BIN_COUNT_FUNCr_rdr~s rhamming_distancezSimhash.hamming_distancesdi*/9:::rcL|j||z |jz S)zjCalculate how similar this hash is from another simhash. Returns a float from 0.0 to 1.0. )r<rrs r similarityzSimhash.similaritys& d33J???4;NNr)r'r:N)__name__ __module__ __qualname____doc__ __slots__r|rZrercrprwrbrfloatrr2rrr^r^Vs_GG"I$( RRRRTz R  RRRR Is Is I I I I"""""ccDj #)d"2 sTz    ;9;;;;;OYO5OOOOOOrr^contentcDt|S)zACalculate a simhash hex value for meaningful bits of the content.)r^rp)rs rcontent_fingerprintrs 7   " " $ $$r)rr r6c`eZdZdZddeddfdZdedefdZd edefd Zd e d eddfd Z dd Z dS)LRUCachea Pure-Python Least Recently Used (LRU) cache using a circular doubly linked list Adapted from CPython functools.py lru_cache decorator implementation https://github.com/python/cpython/blob/3.9/Lib/functools.py#L524 First adapted by https://github.com/vbarbaresi r r%Nct|_||_i|_g|_|j|jddg|jdd<d|_dS)NF)rlockr cacherootfull)rdr s rrezLRUCache.__init__sIGG  +- !  49dD9 !!!  rlinkc|\}}}}||c|t<|t<|jt}|x|t<|jt<||t<|j|t<|S)N)NEXTPREVr)rdr link_prev link_next_keyresultlasts r _move_linkzLRUCache._move_links\-1* 9dF+4i( $4y'++T TYt_T YT  rkeyc|j5|j|}|r!||cdddS dddn #1swxYwYdS)zgTests if the key that is asked for is in the cache and retrieve its value from the linked list.Nr0)rrgetr)rdrrs rrz LRUCache.getsY - -:>>#&&D -t,, - - - - - - - - - - - - - - - - - - - - - - - -rs1AAAvaluec|j5|j|}|r+||||j|t<n|jrq|j}||c|t<|t<|t|_|jt}dx|jt<|jt<|j|=||j|<nb|jt}||j||g}|x|t<x|jt<|j|<t|j|j k|_ddddS#1swxYwYdS)z Stores a given key in the cache.N) rrrrRESULTrrKEYrrr3r )rdrrroldrootoldkeyrs rputz LRUCache.putsY @ @:>>#&&D @%%%*/ 3''9@"iG471GCL'&/!( DI!Ys^F9==DIcNTYv%6 6*'.DJsOO 9T?D $)S%8DEIIDJI44:c?!$DJ4< ?DIA @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @sD"D77D;>D;c|j5|j|j|jddg|jdd<d|_ddddS#1swxYwYdS)zDelete all cache content.NF)rrclearrrros rrzLRUCache.clears Y   J      Ity$=DIaaaLDI                  s;AAA)r)r%N) rrrrrZrerrrr|rrr2rrrrs   d    ssss#@s#@3#@4#@#@#@#@JrrrSct|}|dkr|dznd}t||dS)zImplement LRU cache.r0r N)LRU_TESTrr)rScachevalrs r put_in_cachersB||J''H$NNHqLLE LLU#####relementoptionscFtd|}t||jkrEt |}||jkr t ||dzdSt|dS)z(Check for duplicate text with LRU cache.rr TF) r rPitertextr3min_duplcheck_sizerrmax_repetitionsrr)rrrSrs rduplicate_testrschhw//112233J :333<< ++ g- - - LLX\ 2 2 24 5r)r!)r:)rL)4rrerGrdifflibr functoolsrhashlibroperatorr threadingrtypingr lxml.etreer settingsr r utilsr compiler(getattrrZrr| maketransr7rArboolr+rjr9rDrKbytesrTr\r^rrrrrrrrrr2rrrs@@ ######))))))))"*/00k+F+FGG MMee55??eee f f  4MMM#M%MRVMMMM$s)CDI 1 1 1S 1$s) 1 1 1 1sCc"EE3EEUEEEE 5G#GsGtCyGGGG :O:O:O:O:O:O:O:Oz%%%%%% %dCPPPPPPPPf 8H % % %$S$T$$$$ H y T      r