AHj{"tdZddlZddlZddlZddlmZddlmZddlm Z m Z m Z m Z m Z mZmZmZmZddlmZmZej*j,dk7r/eej*d rej*j1d ej2j,dk7r/eej2d rej2j1d d ej4d ej4fdZdeed ej<fdZdej<d ej<fdZ ddZ!dej<d dfdZ"e#dk(re!yy)z. Implementing a basic command-line interface. N)version)python_version) cli_crawler cli_discoveryexaminefile_processing_pipelineload_blacklistload_input_dictprobe_homepageurl_processing_pipeline write_result)PARALLEL_CORESSUPPORTED_FMT_CLIzUTF-8 reconfigurezutf-8)encodingparserreturnc |jdd}|j}|jdd}|jdd}|j}|jdd}|jd d }|j}|jd d d t|jddt|jdddt|jddtt |jdddt|jddd|jddd t|jd!d"t|jd#d$d|jd%d&d'd(d)*|jd+d,d'd(d)*|jd-d.d'd(d)*|jd/d0d'd(d)*|jd1d2d'd(d)*|jd3d4d|jd5d6d7t8|jd9d:d;d|jdd?d|jd@dAd|jdBdCdD|jdEdFdD|jdGdHd|jdIdJd|jdKdLt|jdMdNd|jdOdPt|jdQdRd|jdSdTd|jdUdVt dWX|jdYdZd|jd[d\d|jd]d^d|jd_d`d|jdadbd|jdcddd|jdedfd|jdgdhdidjdkl|jdmdndodptdqdrts|S)tz,Add argument groups and arguments to parser.Inputz%URLs, files or directories to processOutputz+Determines if and how files will be written NavigationzLink discovery and web crawling Extractionz-Customization of text and metadata processingFormatzSelection of the output formatz-iz --input-filez'name of input file for batch processing)helptypez --input-dirz5read files from a specified directory (relative path)z-uz--URLzcustom URL downloadz --parallelzAspecify a number of cores/threads for downloads and/or processing)rrdefaultz-bz --blacklistz:file containing unwanted URLs to discard during processingz--listz/display a list of URLs without downloading them store_true)ractionz-oz --output-dirz6write results in a specified directory (relative path)z --backup-dirz9preserve a copy of downloaded files in a backup directoryz --keep-dirsz-keep input directory structure and file namesz--feedz.look for feeds and/or pass a feed URL as input?TF)rnargsconstrz --sitemapzBlook for sitemaps for the given website and/or enter a sitemap URLz--crawlzJcrawl a fixed number of pages within a website starting from the given URLz --explorez=explore the given websites (combination of sitemap and crawl)z--probez?probe for extractable content (works best with target language)z --archivedz=try to fetch URLs from the Internet Archive if downloads failz --url-filterzLonly process/output URLs containing these patterns (space-separated strings)+)rr!rz-fz--fastz!fast (without fallback detection)z --formattingz,include text formatting (bold, italic, etc.)z--linksz5include links along with their targets (experimental)z--imagesz.include image sources in output (experimental)z --no-commentszdon't output any comments store_falsez --no-tableszdon't output any table elementsz--only-with-metadataz4only output those documents with title, URL and datez--with-metadataz&extract and add metadata to the outputz--target-languagez*select a target language (ISO 639-1 codes)z --deduplicatez+filter out duplicate documents and sectionsz --config-filezAoverride standard extraction parameters with a custom config filez --precisionz;favor extraction precision (less noise, possibly less text)z--recallz8favor extraction recall (more text, possibly more noise)z--output-formatzdetermine output formattxt)rchoicesrz--csvzshorthand for CSV outputz--htmlzshorthand for HTML outputz--jsonzshorthand for JSON outputz --markdownzshorthand for MD outputz--xmlzshorthand for XML outputz--xmlteizshorthand for XML TEI outputz--validate-teizvalidate XML TEI outputz-vz --verbosecountrz&increase logging verbosity (-v or -vv))rrrz --versionz!show version information and exitrz Trafilatura trafilaturaz - Python )rrr) add_argument_groupadd_mutually_exclusive_group add_argumentstrintrrrr) rgroup1 group1_exgroup2group3 group3_exgroup4group5 group5_exs G/root/tools/cai/cai_env/lib/python3.12/site-packages/trafilatura/cli.pyadd_argsr7s & &w0W XF335I  & &x1^ _F  & &|5V WF335I  & &|5d eF  & &x1Q RF335I 46_fij =/fmpq 4/D3O  P    m2nuxy 'Xamn n3kruv -hors  ,[dpq GsZ^hm Q  Y YadlpzYadlpz Zco  [   h-PYef -[dpq  (_htu  )Ybno .IR_`  ,MVcd %[dp )0Xamn +2^ehi .[dpq .qx{|  ,ir~  )clxy,3LVgqvw 7)CLY 8*El[ 8*El[ <.GP\] 7)CLY :,JS_` (/HQ]^   5   0w}56jAQ@RS  Margscvtjd}t|}t|j S)z(Define parser for command-line argumentsz&Command-line interface for Trafilatura) description)argparseArgumentParserr7map_args parse_args)r9rs r6r?r?s1  $ $1Y ZF f F F%%' ((r8c@dD]}t||s||_|S|S)z2Map existing options to format and output choices.)csvhtmljsonmarkdownxmlxmltei)getattr output_format)r9otypes r6r>r>s5F 4 !&D   K  Kr8cRttjdd}t|y)zRun as a command-line utility.rN)r?sysargv process_args)r9s r6mainrNs chhqrl #Dr8cxd}|jdk(r4tjtjtj nB|jdk\r3tjtjtj |jrt|j|_|js|js |jr t|}n|jr t|n|jr t!|n|j"r t%|nz|j&s |j(rt+|}t-||}nJt/tj0j2j5||j(}t7|||dk7rtj8|yy)z8Perform the actual processing according to the argumentsrr)streamlevel)urlN)verboselogging basicConfigrKstdoutWARNINGDEBUG blacklistr explorefeedsitemaprcrawlrprober input_dirr input_fileURLr r rstdinbufferreadrexit)r9 exit_code url_storeresults r6rMrMs(I ||q3::W__E  3::W]]C ~~'7  ||tyyDLL!$'  D t  & DHH#D) +D)< ))..0$DHHEVT"A~ r8__main__)rN)$__doc__r<rUrKimportlib.metadatarplatformr cli_utilsrrrr r r r r rsettingsrrrWrhasattrrstderrr=r7listr, Namespacer?r>rNrM__name__r8r6rvs+ &#   8::'!gcjj-&HJJG,::'!gcjj-&HJJG,iX,,i1H1HiX)T#Y)8#5#5)8%%(*<*< *x))*d*Z zFr8