AHj# d Z ddlZddlZddlZddlmZ ddlmZ ddlm Z m
Z
m Z m Z m
Z
mZ ddlmZ ddlmZ dd lmZ dd
lmZ ej. e Zh d Z ej6 d Z ej6 d
Z ej6 d Z ej6 dej> Z ej6 d Z! ej6 d Z" G d d Z#de$de%fdZ&de'e$ de#de'e$ fdZ(de$de#de'e$ fdZ)de$de#de'e$ fdZ*d e$de#de'e$ fd Z+de#d e$dz de'e$ fd Z, d'd e$d!e$dz d"e%d#e-de'e$ f
d$Z.d%e$d!e$dz de'e$ fd&Z/y)(z>
Examining feeds and extracting links for further processing.
N)islice)sleep) check_url clean_url filter_urlsfix_relative_urls get_hostinfo is_valid_url )is_similar_domain) fetch_url) MAX_LINKS) load_html> text/rdftext/rsstext/xml text/atom
text/plain text/rdf+xml text/rss+xml
text/atom+xmlapplication/rdfapplication/rssapplication/xmlapplication/atomapplication/jsonapplication/rdf+xmlapplication/rss+xmlapplication/atom+xmlapplication/feed+jsonapplication/x-atom+xmlapplication/x.atom+xmlz<(feed|rss|\?xml)z(?:\s*)(?:)?(?:\s*)z \bcomments\bzn\.(?:atom|rdf|rss|xml)$|\b(?:atom|rss)\b|\?type=100$|feeds/posts/default/?$|\?feed=(?:atom|rdf|rss|rss2)|feed$c
D e Zd ZdZg dZ d dedededededz d df d
Zy) FeedParametersz.Store necessary information to proceed a feed.basedomainextlangrefNbaseurlr' referenceexternal target_langreturnc J || _ || _ || _ || _ || _ y )Nr% )selfr+ r' r, r- r. s I/root/tools/cai/cai_env/lib/python3.12/site-packages/trafilatura/feeds.py__init__zFeedParameters.__init__J s' ! ! ! + ! )FN)__name__
__module__ __qualname____doc__ __slots__strboolr3 r4 r2 r$ r$ E sU 48I "&
" " " "
" 4Z
"
"r4 r$ feed_stringr/ c L t j | ry| dd }d|v xs d|v S )z$Check if the string could be a feed.TNd zt |j |d s%t j d|j |d |j |d d|v sd|v s|j | |S )zGExamine links to determine if they are valid and
lead to a web page)languagefeedr z'Rejected, diverging domain names: %s %sr
feedburner feedproxy) sortedsetr r& r r) r( r r' LOGGERwarningappend)rD rE output_linksitemlinkcheckeds r2 handle_link_listrT a s Ls8}% & d3D6;;7 ::&"4=Nv}}^efg^h=i H&--Y`abYcd ##GAJ/
T
![D%8 % & r4 c B t | s| j d rh t j | j dg D cg c]& }|j d xs |j d ( }}|D cg c] }|| c}S t j d|j g S d| v rVd t t j | t D D cg c]" }d |vr d
|vrt j | d $ c}S d | v rBt t j | t D cg c] }|d j# c}S g S c c}w c c}w # t j
j $ r$ t j d|j Y g S w xY wc c}w c c}w )
zz find_links.. s [!1[s zatom+xmlz
rel="self"r z)rC
startswithjsonloadsgetdecoderJSONDecodeErrorrM debugr' r
LINK_ATTRSfinditerr LINK_HREFsearch
LINK_ELEMENTSstrip)r= rE rQ
candidatescrR r\ s r2
find_linksrm v su [ ) ! !# &
GLPJJWbLcLgLghoqsLt uDdhhuo ?$ ? u
u#-?a ??
LL4fmm D ; \vj.A.A+.NPY'Z[
%,d*B
T "1 %
; &,]-C-CK-PR[&\]!
]]
I- v?<<//
G 6
F
G
^sA 'E +E 0E 6E>EE 'F*F
E =FFc | s"t j d|j g S t | j | }t || D cg c]) }||j k7 s|j d dkD s(|+ }}|r+t j dt | t | |S t j d|j |S c c}w )z7Extract and refine links from Atom, RSS and JSON feeds.zEmpty feed: %s/ z!Links found: %s of which %s validzInvalid feed for %s) rM rd r' rm rj rT r* countlen)r= rE
feed_linksrR rP s r2
extract_linksrt s %v}}5 K - - /8J%5j&%IxTTU[U_U_M_dhdndnordsvwdwDxLx 8#j/3|K\] *FMM: ys
C "C 7C
htmlstringc P t | }|"t j d|j g S |j d D cg c]P }|j d t v s%t j |j dd r|j dd R }}|sT|j d D cg c]: }t j |j dd s)|j dd < }}g }t j | D ]i }t |j | }t | }|s'||j k7 s7t | sCt j | rY|j! | k t j dt# | t# | |S c c}w c c}w )zxParse the HTML and try to extract feed URLs from the home page.
Adapted from http://www.aaronsw.com/2002/feedfinder/zInvalid HTML/Feed page: %sz //link[@rel="alternate"][@href]typehref z
//a[@href]z%Feed URLs found: %s of which %s valid)r rM rd r& xpathra
FEED_TYPESLINK_VALIDATION_RErh dictfromkeysr r r* r
BLACKLISTrO rr )ru rE treerR feed_urls output_urlss r2 determine_feedr sq Z D | 16;;?
JJ @A 88F z )-?-F-FtxxPVXZG[-\
I -1ZZ -E
%)I[IbIbcgckcklrtvcwIxDHHVR
K
i( % d3 DFJJ&<+=iFVFVW[F\ t $ % LL8#i.#kJZ[ -
s AF /*F#F# urlfilterc | j rlt d| j d| j d }|rDt || }t || }t
j
dt | | j |S g S )z2Alternative way to gather feed links: Google News.z*https://news.google.com/rss/search?q=site:z&hl=z&scoring=n&num=100z!%s Google news links found for %s)r) r
r' rt r rM rd rr )rE r
downloadedrs s r2 probe_gnewsr sp
{{ !KFMM?Z^_e_j_j^kk} ~
&z6:J$Z;J LLW $ * 5 5 vy ))r4 r+ c F t j d| t | | S )zhShift into reverse and try the homepage instead of the particular feed
page that was given as input.z&Probing homepage for feeds instead: %s)rM rd r )r+ r. s r2 r r s LL97C '; //r4 )NFg @)0r8 r_ loggingre itertoolsr timer courlanr r r r r r
deduplicationr downloadsr
settingsr utilsr getLoggerr5 rM r{ compiler@ re rg DOTALLri r r| r$ r: r; rC listrT rm rt r r floatr r r< r4 r2 r s - 8 $
, rzz./
RZZ.
/
BJJ '
XZ\ZcZcd
BJJ ' RZZ
" "(73 74 7tCy . T#Y * C DI @s N tCy $ s N tCy D 3: $s) # 6* 6*t6* 6* 6*
#Y 6*r0# 0C$J 049 0r4