AHj'< dZddlZddlmZddlmZmZddlmZm Z m Z m Z m Z m Z ddlmZddlmZdd lmZmZmZmZmZdd lmZmZmZdd lmZmZej>e Z!d d d d dddddddd Z"e"jGDcic]\}}|| c}}Z$hdZ%gdZ&dededefdZ'dUdede(defdZ)dVdede*e de+defdZ,de*ede-e.e.e.e*e(ffdZ/dVd ed!e(d"e+de-e+e*e(ffd#Z0d ede+fd$Z1 dWd%ed&e(d'e+d"e+def d(Z2 dXd)e ded*e+d+e+de dzf d,Z3d)e dede dzfd-Z4d)e ddfd.Z5d)e ddfd/Z6d!e(dzde+fd0Z7d)e ddfd1Z8d)e ddfd2Z9d)e ddfd3Z:d)e ddfd4Z;id5e5d6e5d7e5d8e8d9e8d:e8d;e8de9d?e9d@e6dAe6dBe6dCe:dDe:dEe:dFe;iZd7dKdAd@dLd>dMdNdOdP Z?de de fdQZ@dVdRedSe+de(fdTZAycc}}w)Zz* Functions to process nodes in HTML code. N)deepcopy)fix_relative_urls get_base_url)Element SubElementXPath_Element strip_tagstostring) HtmlElement)duplicate_test)CUT_EMPTY_ELEMSMANUALLY_CLEANEDMANUALLY_STRIPPEDDocument Extractor)is_image_element textfiltertrim)META_ATTRIBUTESdelete_element#iz#bz#uz#tz#subz#sup) emibstrongukbdsampttvarsubsup>figuresourcepicture){z("z('z treeoptionsreturnctjtj}}|js|j gdn|j dD] }d|_ |jr)|Dcgc] }|tvs |}}|jdt|||jdk(rX|jdGt|}|D]#}|j|D] }t|%|jd+|}n(|D]#}|j|D] }t|%t!||jScc}w)z/Prune the tree by discarding unwanted elements.)tabletdthtrz.//figure[descendant::table]divimgrecallz.//p)rcopyrtablesextendxpathtagimagesPRESERVE_IMG_CLEANINGremover focusfindriterr prune_html) r)r* cleaning_liststripping_listelemetcopy expressionelements R/root/tools/cai/cai_env/lib/python3.12/site-packages/trafilatura/htmlprocessing.py tree_cleaningrH/sH%5$9$9$;=N=S=S=U>M >>89JJ=> DDH ~~$1TqQ>S5ST Te$t^$}} TYYv%6%B' (J99Z0 (w' ( ( 99V  $D( (J99Z0 (w' ( ( dGMM **)Us 6 E E r<c||dk7}|jdD]"}|jtvst||$|S)zADelete selected empty elements to save space and processing time. precisionz-.//processing-instruction()|.//*[not(node())]) keep_tail)r7r8rr)r)r<tailsrFs rGr?r?RsB [ E::MN5 ;;/ ) 7e 45 Knodelist with_backupc|r$t|j}t|}|D]}||D] }t||r%t|j}|dz kDr|SS|S)z2Prune the HTML tree by removing unwanted sections.)len text_contentrr)r)rNrOold_lenbackuprEsubtreenew_lens rGprune_unwanted_nodesrX\sd'')*$$ !$' $G 7 #  $$d'')*1,t8&8 KrM links_xpathcd|DDcgc]}|s| }}ttt|}td|D}t|t|||fScc}w)zCollect heuristics on link textc3NK|]}t|jywN)rrS).0rBs rG z$collect_link_info..tsL$t0023Ls#%c3,K|] }|dks dyw) r N)r]lengths rGr^z$collect_link_info..ws<6 QRemove sections which are rich in links (probably boilerplate).//refFz .//graphicr r`drg?Tp<i,u8list link text/total: %s/%s – short elems/total: %s/%s皙?) findallr=rRrrSr8getnextriLOGGERdebug) rFrjrkrYrf len_threshold link_textlimitlenelemlenlinklenelemnumrhs rGlink_density_testr}{sL//(+K by||L!-byF ;1-3 Q4467 y>M )c)ns4y3.N8O{{c *22 ??  $HH$iG/@/M,*f a<<  F       Ws] "w{zG7Kc7Q<  &=rMc|jd}|sytt|j}|dkryt |\}}}}|dk(ryt j d|||dkr|d|zkDS|d |zkDS) z=Remove tables which are rich in links (probably boilerplate).rmFrTztable link text: %s / total: %sirrg?)rsrRrrSrirurv)rFrYrzr{r|_s rGlink_density_test_tablesrs//(+K $w++-./G}.{;GWa!| LL2GWE&-n7S7] "Q'C'M:QQrMrVtagname backtrackingcjg}|rdnd}|rdnd}|j|D]k}t|j}t|||\} } | s-|s1| s4dt |cxkr|ksInLt ||k\s[|j |mt j|D] }t||S)z{Determine the link density of elements with respect to their length, and remove the elements identified as boilerplate.rrnr r) r>rrSr}rRappenddictfromkeysr) rVrrrk deletionsrwdepth_thresholdrBelemtextresulttemplists rGdelete_by_link_densityrsI*CM*aO W%#))+,,T8_M lxAH 4U 4UZ]^bZcgvZv   T " # i(t NrMrB comments_fixpreserve_spacesc|jdk(r t|r|S|jdk(s&t|dk(r|js |jsy|s1|jdk(r"|st |jxsd|_|S|js?t|dk(r1|jdc|_|_|r|jdk(rd|_|sHt |jxsd|_|jrt |jxsd|_|js t |s|jr t||ry|S)z3Convert, format, and probe potential text elements.graphicdonerNlbro) r8rrRrjtailrrdedupr)rBr*rrs rGhandle_textnoders xx9!1$!7  xx6c$i1nTYYtyy DHH,TYY/4DI 99Ta $yy" 49 DHH,DH O+t 99TYY/4DI 99D)gmmtU\@] KrMc|jdk(s&t|dk(r|js |jsyt |jxsdt |jxsdc|_|_|jdk7r1|js%|jr|jdc|_|_|js |jr$t |s|j r t||ry|S)zBConvert, format, and probe potential text elements (light format).rrNr)r8rRrjrrrrr)rBr*s rG process_noders xx6c$i1nTYYtyy  ?2dDO4KtDIty xx4 dii#yy$ 49 yyDII d  .w2O KrMc$|jd|jd|_d}|jdddD]U}|jdvr>|jdt|jd||jdk(r|dz }d |_Wy ) zGConvert
    and
      to and underlying
    1. elements to .rendrcr dddtli)rr-itemN)setr8r>str)rBrsubelems rG convert_listsrsHHVTXXDH A99T4. ;;, & KK3w{{#3"4AaS 9 :{{d"Q rMc&d}|jdk(rot|dk(r|djdk(rd}|jd}|r#d}|D]}|jj t |j rd}|rd|_y d |_y ) z?Convert quoted elements while accounting for nested structures.Fprer rspanTz#.//span[starts-with(@class,'hljs')]codequoteN)r8rRr7attribclear_is_code_blockrj)rB code_flag code_elemsrs rGconvert_quotesrsI xx5 t9>d1gkkV3IZZ EF I% '$$& ' $)) $I"vDHDHrMc,|sytD]}||vsyy)z2Check if the element text is part of a code block.FT)CODE_INDICATORS)rj indicators rGrr2s' $   rMc~|jj|jd|jd|_y)z$Add head tags and delete attributes.rheadN)rrrr8rBs rGconvert_headingsr<s+KKHHVTXXDHrMcd|_y)zConvert
      and
      to rN)r8rs rGconvert_line_breaksrCs DHrMc6d|_|jddy)z7Convert , , to delr overstrikeN)r8rrs rGconvert_deletionsrHsDHHHV\"rMcLd|_|jdD] }d|_ y)zHandle details and summary.r1summaryrN)r8r>)rBrs rGconvert_detailsrNs(DH99Y' rMdlolulh1h2h3h4h5h6brhr blockquoterqrsstrikedetailsbase_urlcd|_|jd}|jj|r!|r t ||}|j d|yy)z7Replace link tags and href attributes, delete the rest.refhreftargetN)r8getrrrr)rBrrs rG convert_linkrlsLDH XXf FKK &x8F 6" rMurlc"|js=d}|jr|dz }|j|D] }d|_ t |dn2|xr t |}|j ddD]}t|||jrm|j tjD]F}|jj|jdt|jd|_Hnt |gtj|j tjD]}t|j||jr|j dD] }d|_ |jrt!|j dD]j}t!|j d}t#|D]}|j%||sA|j'j)r`t+|l|S) zBSimplify markup and convert relevant HTML tags to an XML standard.z).//*[self::div or self::li or self::p]//az |.//table//ararhir2r)linksr5r7r8r rr>r formattingREND_TAG_MAPPINGkeysrrr CONVERSIONSr9rcreversedaddnextrSstripr) r)r*r xpath_exprrBrrgraphicsrs rG convert_tagsrxs ==@ >> . (JJJz* DDH  4,<,IIc5) )D x ( )II.3356 D KK    HHV-dhh7 8DH  42*//12 +**,-$DHHd#$~~IIe$ !D DH ! ==DIIe,- ( 34'1)GKK() C$4$4$6$<$<$>"3' ( KrMrcDdt|jddddS)Nhrrr )intrrs rGrs%1S&$!7!;<=>rMrrc4t|jddS)Nrr)HTML_TAG_MAPPINGrrs rGrrs'(>?rM) rcrrrrrr2rrc|jtjD]}tt|j}t |r|||_n||_|jdk(r-|j d|jjdd{|jjd|_td}|j||S)zConvert XML to simplified HTML.rrrrbodyhtml) r>HTML_CONVERSIONSrrr8callablerrpoprrr)r)rB conversionroots rGconvert_to_htmlrs *//12  %c$((m4 J !$'DH!DH 88s? HHVT[[__Xr: ; KK     DH 6?DKK KrMdocument with_metadatact|j}|rFtd}tD] }t ||x}st |d||"|j d|t|ddjS)z1Convert the document to HTML and return a string.rmeta)namecontentrTunicode) pretty_printencoding) rrrrgetattrrinsertr r)rr html_treerrvalues rGbuild_html_outputrsw .Iv# CD$//u/4dEB C D! ID9 E K K MMrM)balanced)F)FF)TFr\)B__doc__loggingr4rcourlan.urlutilsrr lxml.etreerrrr r r lxml.htmlr deduplicationrsettingsrrrrrutilsrrrxmlrr getLogger__name__ruritemsrr:rrHrr?rcboolrXtuplerrir}rrrrrrrrrrrrrrrrr)kvs00rGrs<QQ!)650   8 $         &6%;%;%=>TQAqD>7- + +i +K +F[k{d5kPTal(9k"9 3S$s) #$9&{&#&&Y^_ceijmen_nYo&RRkRdR,!      :! ' ' '' ' _ 'Tx)4(  T  00d0&t8h4 #H## (t-- -             . >   !" #$% . #{ #cDj #T #.{.Y.S4Z.S^.d     >    ? (x( N N N# NE ?s G