IZj " < U d Z ddlZddlZddlmZ ddlmZ ddlmZ ddlZ ddl m
Z n
# e $ r dZ Y nw xY wddl
mZ ddlmZmZmZ d d
lmZ ej e Zd d hZee ed
< ej ej j ej ddg d Z! ej" e! Z# eddd d Z$ ej% dej& Z' ej% dej& Z( edd G d d Z)dee*z ez dz de+fd Z,de*de+fd Z-d e*de.e fd Z/d e*ez defd!Z0d"edefd#Z1d$ededz fd%Z2d&ede+fd'Z3d(ed&edefd)Z4d*ededz fd+Z5d*e*ez ez dedz fd,Z6d-ed.e.e defd/Z7d0edefd1Z8dS )2z7
Module bundling functions related to HTML processing.
N) dataclass)datetime)Any)detect)
from_bytes) HtmlElement
HTMLParser
fromstring )
MAX_FILE_SIZEutf-8utf_8UNICODE_ALIASES )i i i i i )totalconnectstatus_forcelist)retriesFT) collect_idsdefault_doctypeencoding
remove_pisz^< ?! ?DOCTYPE.+?/ ?>z()slotseqc F e Zd ZU dZeed< eed< eed< eed< eed< dS ) Extractorz0Defines a class to store all extraction options. extensivemaxminoriginalformatN)__name__
__module__ __qualname____doc__bool__annotations__r str Z/Users/jameslopez/projects/MentorCore/.venv/lib/python3.11/site-packages/htmldate/utils.pyr r / sC 66OOO MMM MMMNNN KKKKKr* r datareturnc 8 | pt | t k S )z6Check if the input object is suitable to be processed.)lenr r, s r+ is_wrong_documentr1 : s 8 0s4yy=00r* c V | d n# t $ r Y dS w xY wdS )zLSimple heuristic to determine if a bytestring uses standard unicode encodingzUTF-8FT)decodeUnicodeDecodeErrorr0 s r+ isutf8r5 ? sC G uu 4s
&& bytesobjectc V t | rdgS g }t >t | d }|'| | t | dd pt | }| d |D d |D S )zz#detect_encoding..X s :::1AJ:::r* c $ g | ]
}|t v |S r) )r )r9 gs r+ r; z#detect_encoding..[ s" ; ; ;!!?":":A":":":r* )r5 cchardet_detectappendlowerr extend)r6 guessescchardet_guessdetection_resultss r+ detect_encodingrE H s k yG"(55jA % NN>//11 2 2 2 #;vv#677R:k;R;R NN::(9:::;;; < ;w ; ; ;;r* filecontentc t | t r| S d}t | D ]K} | | } n2# t t
f$ r t d| Y Hw xY w|pt | dd S )znGuess bytestring encoding and try to decode to Unicode string.
Resort to destructive conversion otherwise.Nzwrong encoding detected: %sr
replace)r errors)
isinstancer( rE r3 LookupErrorr4 LOGGERwarning)rF htmltextguessed_encodings r+ decode_filerP ^ s +s## H +K 8 8 "))*:;;H
E /0 L L L NN8:J K K K K K L
Ks;KKKKs A,A21A2responsec Z t | d r| j n| }|rt | ndS )zRead the data from a response object exposing the body via ``.data``
(e.g. urllib3 or a compatible response) or from a bytestring, then guess
its encoding and decode it to return a unicode string.r, )hasattrr, rP )rQ r, s r+ decode_responserU q s6 $Hf 5 5 C8==8D $ ,;t ",r* urlc t d| d }|j dk r"t d|j | nwt |j r t d| nGt |j S # t $ r&}t d| | Y d}~nd}~ww xY wdS ) a, Fetches page using urllib3 and decodes the response.
Args:
url: URL of the page to fetch.
Returns:
HTML code as string, or Urllib3 response object (headers + body), or empty string in case
the result is invalid, or None if there was a problem with the network.
GET )timeout z!not a 200 response: %s for URL %sz incorrect input data for URL %szdownload error: %s %sN) HTTP_POOLrequeststatusrL errorr1 r, rU Exception)rV rQ errs r+ fetch_urlrb { s
2 $$UC$<<
?c ! ! LLRepair faulty HTML strings to make then palatable for libxml2.doctype
rS r )countzz\1> ) partition DOCTYPE_TAGsub enumerate
splitlinesendswith FAULTY_HTML)rg rc firstline_restilines r+ repair_faulty_htmlry s I ' 1 1$ 7 7 1d __R!_<E E#r elemlistc | | D ]R} | # t $ r. | }|| | Y Ow xY w| S )zDelete selected elements.)iter drop_treeAttributeError getparentremove)r r elementparents r+
clean_htmlr s 99X&& ' ' ' ' ' ' &&((F!
g&&& ' Ks .5A&%A&stringc P d | S )z7Remove superfluous space and normalize remaining space.r )joinsplit)r s r+ trim_textr s 88FLLNN # ##r* )9r% loggingre dataclassesr r typingr urllib3cchardetr r> ImportErrorcharset_normalizerr lxml.htmlr r r
settingsr getLoggerr" rL r setr( r' disable_warnings
exceptionsInsecureRequestWarningutilRetryRETRY_STRATEGY PoolManagerr\ r compileIrn rs r r r& r1 r5 listrE rP rU rb rf ry r r r r r) r* r+ r s ! ! ! ! ! ! 2222222 OOO ) ) ) ) ) ) 9 9 9 9 9 9 9 9 9 9 # # # # # # 8 $ $ $g .S . . . +B C C C ##
... $
G 7 7 7 juw4 bj0"$77 bj+RT22 % ! 1C%K+ 5 < 1 1 1 1 1
4 < <49 < < < <,LUS[ LS L L L L&-c -c - - - -3 3: :#s #t # # # #
3
3
3
t); 2%#+ 3 2 d8J 2 2 2 2j [ DI + $c $c $ $ $ $ $ $s * 44