AHjdZddlZddlmZddlmZmZmZddlm Z ddl m Z ddl m Z mZdd lmZd e d e fd Zd ed eeeeffdZddeded efdZy)z< Module regrouping baseline and basic extraction functions. N)Any)Element SubElement_Element) HtmlElement)BASIC_CLEAN_XPATH) load_htmltrim)delete_elementtreereturnc<t|D] }t||S)z-Remove a few section types from the document.)r r )r elems L/root/tools/cai/cai_env/lib/python3.12/site-packages/trafilatura/baseline.pybasic_cleaningrs#!$'t K filecontentct|}td}||ddfSd}|jdD]}|jsd|jvs t j |jj dd}|sRd|vr)t|}|t|jnd}n t|}|t|d_||rd|zn|z }t|d kDr||t|fSt|}d}|jd D]G}t|j}t|d kDs+|t|d_||rd|zn|z }It|dkDr||t|fSt} d}|jd d dd ddD]N} t| j} | | vs!| t|d_||rd| zn| z }| j| Pt|d kDr||t|fStd}|j!d} | {t|d} | j#Dcgc] }t|}}dj%|Dcgc]}|s| c}| _|| jt| jfSt'|d}|t|d_||t|fS#t$rd}Y\wxYwcc}wcc}w)a)Use baseline extraction function targeting text paragraphs and/or JSON metadata. Args: filecontent: HTML code as binary string or string. Returns: A LXML element containing the extracted paragraphs, the main text as string, and its length as integer. bodyrz&.//script[@type="application/ld+json"] articleBodyz

p dz .//article blockquotecodepreqquote.//body F)clean)r riterfindtextjsonloadsget Exceptionr text_contentrlenrsetiteraddfinditertextjoinhtml2txt)rr postbody temp_textr json_bodyparsedr% article_elemresultselemententry body_elemp_eleme text_elemss rbaseliner?s [ !DvH |QI FG ? 99$))3  JJtyy155mRH I%&y1F:@:L4 3 3 56RTD ?D15 8S).9S4Z$>  ? 9~C N22 $ DI l3; L--/0 t9s?-1Jx % * ytd :I ;  8}qC N22eGI99\63sGLW))+,  -2Jx % *  uu rPst  44!'"L%#L%%#s(:";L%^9c9$9#9r