IZjdZddlZddlmZddlmZmZmZddlm Z ddl m Z ddl m Z mZdd lmZd e d e fd Zd ed eeeeffdZddeded efdZdS)z< Module regrouping baseline and basic extraction functions. N)Any)Element SubElement_Element) HtmlElement)BASIC_CLEAN_XPATH) load_htmltrim)delete_elementtreereturncHt|D]}t||S)z-Remove a few section types from the document.)r r )r elems `/Users/jameslopez/projects/MentorCore/.venv/lib/python3.11/site-packages/trafilatura/baseline.pybasic_cleaningrs/!$''t K filecontentczt|}td}||ddfSd}|dD]}|jrd|jvr t j|jdd}n#t$rd}YnwxYw|rid|vr5t|}|!t| nd}nt|}|t|d_||rd |zn|z }t|d kr||t|fSt|}d}|d D]W}t| }t|d kr!|t|d_||rd |zn|z }Xt|dkr||t|fSt} d}|d d ddddD]]} t| } | | vr6| t|d_||rd | zn| z }| | ^t|d kr||t|fStd}|d} | nt|d} d| D}dd|D| _|| jt| jfSt'|d}|t|d_||t|fS)a)Use baseline extraction function targeting text paragraphs and/or JSON metadata. Args: filecontent: HTML code as binary string or string. Returns: A LXML element containing the extracted paragraphs, the main text as string, and its length as integer. bodyNrz&.//script[@type="application/ld+json"] articleBodyz

p dz .//article blockquotecodepreqquote.//bodyc,g|]}t|S)r .0es r zbaseline..]s<<.^s < < S4ZZ$>  9~~C NN22 $  DI l33;; L--//00 t99s??-1Jx % % * y:td :I 8}}qC NN22eeGI99\63sGLLW))++,,   -2Jx % % *  <uu > B  B Tcontentr*ct|}|dS|d}|dS|rt|}d|S)zRun basic html2txt on a document. Args: content: HTML document as string or LXML element. clean: remove potentially undesirable elements. Returns: The extracted text in the form of a string or an empty string. Nrr!r)r r6rr8r1splitstrip)rFr*r rs rr9r9gs W  D |r 99Y  D |r $d## 88D%%''--// 0 0 6 6 8 88r)T)__doc__r-typingr lxml.etreerrr lxml.htmlrsettingsr utilsr r xmlr rtuplestrintrEboolr9r#rrrUs'  4444444444!!!!!!''''''""""""""L%#L%%#s(:";L%L%L%L%^99c9$9#999999r