AHjH t d Z ddlZddlZddlmZ ddlmZ ddlmZ ddl m
Z
ddl m Z m
Z
dd lmZmZ ej" e Z ej( d
Zd e d efd
Zh dZddhZh dZh dZh dZ h dZ ej( dej< ej( dej< ej( dej< ej( dej< ej( dej< ej( dej< dZ d d hZ d d hZ!d e d e"fd!Z# G d" d# Z$ G d$ d% Z% ej( d&ej< ej( d'ej< d(Z& ej( d)ej< Z'd*e d e(fd+Z)d.d e d,e*eef dz d e(fd-Z+y)/a Minimalistic fork of readability-lxml code
This is a python port of a ruby port of arc90's readability project
http://lab.arc90.com/experiments/readability/
Given a html document, it pulls out the main body text and cleans it up.
Ruby port by starrhorne and iterationlabs
Python port by gfxmonk
For list of contributors see
https://github.com/timbertson/python-readability
https://github.com/buriy/python-readability
License of forked code: Apache-2.0.
N)sqrt)
attrgetter)Any)tostring) HtmlElementfragment_fromstring ) load_htmltrimz\.( |$)stringreturnc & t | t d S )Nxml)encodingmethod)r str)r s T/root/tools/cai/cai_env/lib/python3.12/site-packages/trafilatura/readability_lxml.py _tostringr % s FS 77 >
apdloluldivimgpretable
blockquoter article> tdr r > ddr dtlir r formasideaddress>
h1h2h3h4h5h6thnavfooterheader> r r r$ r embedinputzcombx|comment|community|disqus|extra|foot|header|menu|remark|rss|shoutbox|sidebar|sponsor|ad-break|agegate|pagination|pager|popup|tweet|twitterz#and|article|body|column|main|shadowzKarticle|body|content|entry|hentry|main|page|pagination|post|text|blog|storyzbutton|combx|comment|com-|contact|figure|foot|footer|footnote|form|input|masthead|media|meta|outbrain|promo|related|scroll|shoutbox|sidebar|sponsor|shopping|tags|tool|widgetz.<(?:a|blockquote|dl|div|img|ol|p|pre|table|ul)z+https?:\/\/(?:www\.)?(?:youtube|vimeo)\.com)unlikelyCandidatesReokMaybeItsACandidateRe
positiveRe
negativeRedivToPElementsRevideoRebodyhtmlr r elemc F t t | j S )z7Return the length of the element with all its contents.)lenr text_content)r< s r text_lengthr@ T s tD%%'( ))r c , e Zd ZdZddgZdededdfdZy) Candidatez,Defines a class to score candidate elements.scorer< r
Nc || _ || _ y N)rC r< )selfrC r< s r __init__zCandidate.__init__^ s !
!% r )__name__
__module__ __qualname____doc__ __slots__floatr rG r r rB rB Y s) 2&!I&e &; &4 &r rB c e Zd ZdZg dZddedededdfdZdefd Z d
e
ee f d e defd Z d
e
ee f de dz fd
Z
dedefdZde
ee f fdZdedefdZdede fdZddZddZded
e
ee f defdZy)Documentz,Class to build a etree document out of html.docmin_text_length retry_lengthrR rS rT r
Nc . || _ || _ || _ y)a Generate the document
:param doc: string of the html content.
:param min_text_length: Set to a higher value for more precise detection of longer texts.
:param retry_length: Set to a lower value for better detection of very small texts.
The Document class is not re-enterable.
It is designed to create a new Document() for each HTML file to process it.
API method:
.summary() -- cleaned up content
NrQ )rF rR rS rT s r rG zDocument.__init__h s . (r c B | j j ddd D ] }|j d} |r| j | j | j }| j
| }|r| j || }n\|du rd}t j d ut j d | j j d }||n | j }| j || }t |xs d }|r|| j k rd}|S )
z
Given a HTML file, extracts the text of the article.
Warning: It mutates internal DOM representation of the HTML document,
so it is better to call other API methods before this one.
scriptstyle fencedframeTFz5Ended up stripping too much - going for a safer parsez=Ruthless and lenient parsing did not work. Returning raw htmlr: )rR iter drop_treeremove_unlikely_candidates&transform_misused_divs_into_paragraphsscore_paragraphsselect_best_candidate get_articleLOGGERdebugfindsanitizer> rT ) rF r< ruthless
candidatesbest_candidater r: cleaned_articlearticle_lengths r summaryzDocument.summaryy s HHMM(G]C D NN //1 7 7 9 ..0J ! 7 7
CN **:~Ft# $HLL!XY \]xx}}V,"&"2$ "mmGZ @O !6B 7NNT->-> > " "r rg rh c t d|j dz }t d }|j j }| t | n |j g}|D ] }d}||j k( s||v r|| j |k\ rd}nf|j dk( rW| j | } |j xs d}
t |
} | dkD r| d k s | dk r | d
k( rt j |
rd}|s|j | |S ) N
皙?zFTr rZ P g ?r )
maxrC r r< getparentlisttagget_link_densitytextr> DOT_SPACEsearchappend) rF rg rh sibling_score_thresholdoutputparentsiblingssiblingrx link_density node_content node_lengths r ra zDocument.get_article s #&b.*>*>*D"E $X. $$..0#)#54 |sy t |j t d d }t j t
j r?|d d D ]7 }t j d|j j |j 9 t t | S )NrC T)keyreverse z Top 5: %s %s)
sortedvaluesr rb isEnabledForloggingDEBUGrc r< rs rC nextr[ )rF rg sorted_candidates candidates r r` z Document.select_best_candidate s ":#4#4#6Jwz,Document.get_link_density.. s M+d+Ms z.//a)r@ sumfindall)rF r< total_length link_lengths r rt zDocument.get_link_density s5 "4(-A M V8LMM \))r c i }| j j ddd D ] }|j }||j }t |j }t | }|| j k rZ||fD ] }|||vs | j | ||< dt |j d z t |dz d z }|| xj |z
c_
||| xj |dz z
c_
|j D ], \ }} | xj d| j | z
z c_
. |S ) Nr r r! r ,d )
rR r[ rq r r? r> rS
score_nodesplitminrC itemsrt )
rF rg r< parent_nodegrand_parent_node elem_text
elem_text_lennoderC r s
r r_ zDocument.score_paragraphs sY
HHMM#ud3 AD ..*K" + 5 5 7 T . . 01I NM t 3 33$&78
=#J(>'+t' > >O ? r c d}t d |j d |j d f D ]= }t d j | r|dz }t d j | s9|dz
}? |S )Nr classidr7 r6 )filtergetREGEXESrw )rF r< weight attributes r class_weightzDocument.class_weight sr txx'8$((4.&IJ I|$++I6" |$++I6"
r c | j | }t |j }|j }|t v r|dz
}n)|t
v r|dz
}n|t v r|dz }n
|t v r|dz }t || S )Nr r ) r r rs lower
DIV_SCORES BLOCK_SCORESBAD_ELEM_SCORESSTRUCTURE_SCORESrB )rF r< rC rs names r r zDocument.score_node s| !!$'$((myy{ : QJE
\
! QJE
_
$ QJE
%
% QJE%%r c
| j j d D ] }dj t d |j d |j d f }t | dk rM|j t vs`t d j | syt d j | r|j y )Nz.//* r r r r4 r5 ) rR r joinr r r> rs
FRAME_TAGSr rw r\ )rF r< attrss r r] z#Document.remove_unlikely_candidates s HH$$V,
!DHHVD488G+r T)r r br)rR r r rw r mapr rr rs ru stripr insertr enumeratetailr\ )rF r< p_elemposchilds r r^ z/Document.transform_misused_divs_into_paragraphs s, HH$$X.
D -.55bggc)TRVZ>X6YZ
HH$$X. &DyyTYY__ .,V4)-D& TY Av& $Yt_d C
&
U::%**"2"2"4 0 8F.3jj$+FKKKa099$OO%
&
&r r c X |j dddddd D ]; }| j | dk s| j | dkD s,|j = |j d d
D ] }|j |j d D ]M }d |j v r-t
d
j
|j d rd|_ >|j O t }t |j d D ] }||v r | j | }||v r|| j nd}||z dk r3t j d|j || |j j|j j! d dk sd}t" D ci c] } | t% |j' d| " }
} |
dxx dz cc< |
dxx t% |j' d z cc< t) | } | j | } |j+ }
|
|
|v r||
j nd}|
d r |
d d|
d d z z kD r d |
d d }nx|
d |
d kD r|j t, vrd }nW|
d |
d d z kD rd!}nE| | j. k r|
d dk( rd"| d#}n&| | j. k r|
d d$kD rd"| d%}n|d&k r| d'kD r
d(| d)d*| }n|d&k\ r| d+kD r
d(| d)d*| }n|
d, dk( r| d-k s|
d, dkD rd.}n| sd/}g }|j1 D ]" }t) | }|s|j3 | n t% | dz }|j1 d0 D ]1 }t) | }|s|j3 | t% | |k\ s1 n |r6t5 | d1kD r(d2}|j7 |j d3d4d5d6 nd2}|s|j t j d7||j |xs d8 || _ t; | j8 S c c} w )9Nr( r) r* r+ r, r- r g Q ?r% textareaiframesrcr9 VIDEOz6//table|//ul|//div|//aside|//header|//footer|//sectionz+Removed %s with score %6.3f and weight %-3sr rm Tz.//r$ r r3 z.//input[@type="hidden"]r r r g?ztoo many images ()zmore
s than
sr z less than 3x
s than sztoo short content length z without a single imager z and too many imagesr rn ztoo many links z.3fz for its weight g ?r2 K z<