AHj;ddlZddlmZmZmZmZmZddlZddlm Z ddl m Z ddl m Z mZmZddlmZ d"deeded d eed eed eeeeeff d Z d"de ded d eedeed eeeeeff dZ d#deed eed eefdZ d$deeded d eed eed ef dZ d#deed eed eeeffdZ d%de ded deed eefdZdeed efdZded eefdZ d%deeded d eed efdZ d%deeded d eed efdZ d%deededd eed efdZ ded efdZ!ded efd Z"ded e#fd!Z$y)&N)AnyListLiteralOptionalTuple)verbose_logger)Batch) CallTypes ModelInfoUsage) token_counterfile_content_dictionarycustom_llm_provider)openaiazure vertex_ai hosted_vllm anthropic model_name model_inforeturncfKt||||}t|||}t||}|||fSw)aO Calculate the cost and usage of a batch. Args: model_info: Optional deployment-level model info with custom batch pricing. Threaded through to batch_cost_calculator so that deployment-specific pricing (e.g. input_cost_per_token_batches) is used instead of the global cost map. )rrrrrrr)_batch_cost_calculator,_get_batch_job_total_usage_from_file_content#_get_batch_models_from_file_content)rrrr batch_cost batch_usage batch_modelss S/root/tools/cai/cai_env/lib/python3.12/site-packages/litellm/batches/batch_utils.pycalculate_batch_cost_and_usager! sS"(/ 7 J ? 7/K 7L {L 00s/1batchlitellm_paramscKt|||d{}t|||}t|||}t||}|||fS71w)a*Helper function to process a completed batch and handle logging Args: batch: The batch object custom_llm_provider: The LLM provider model_name: Optional model name litellm_params: Optional litellm parameters containing credentials (api_key, api_base, etc.) )r#N)rrrr),_get_batch_output_file_content_as_dictionaryrrr)r"rrr#rrrrs r _handle_completed_batchr&.sp"%Q ">% (/ 7J ? 7/K 7L {L 00)sAA2Ac|r|gSg}|D]>}t|st|}|jd}|s.|j|@|S)z. Get the models from the file content model)_batch_response_was_successful(_get_response_from_batch_job_output_filegetappend)rrr_item_response_body_models r rrVs^|L(, )% 0EeLN#''0F##F+ , c|dk(r)|r't||\}}tjd||St|||}tjd||S)zC Calculate the cost of a batch based on the output file id rzvertex_ai_total_cost=%s)rrr total_cost=%s)(calculate_vertex_ai_batch_cost_and_usagerdebug%_get_batch_job_cost_from_file_content)rrrrr_ total_costs r rrisgk)j@ #Z A 6 C7 7/J *5 r0vertex_ai_batch_responsesc2ddlm}d}d}d}d}|xsd}|D]}|jd} | | jdi} | jddxsd} | jd dxsd} | jd dxs| | z} t| | | } |||d \}}|||zz }|| z }|| z }|| z }t jd|||||t|||fS#t$r)}t j dt|Yd}~gd}~wwxYw)a+ Calculate both cost and usage from Vertex AI batch responses. Vertex AI batch output lines have format: {"request": ..., "status": "", "response": {"candidates": [...], "usageMetadata": {...}}} usageMetadata contains promptTokenCount, candidatesTokenCount, totalTokenCount. rbatch_cost_calculatorzgemini-2.0-flash-001responseN usageMetadatapromptTokenCountcandidatesTokenCounttotalTokenCount) prompt_tokenscompletion_tokens total_tokensr)usager(rz3vertex_ai batch cost calculation error for line: %szAvertex_ai batch cost: cost=%s, prompt=%d, completion=%d, total=%drDrBrC) litellm.cost_calculatorr;r+r Exceptionrr4strinfo)r8rr;r7rDrBrCactual_model_namer= response_bodyusage_metadata_prompt _completion_total line_usagep_costc_costes r r3r3sz>JLM"<&<- Z0  &**?B? $$%7;@q$(()?CHq ##$5q9Tg >S!) 2 '$/NFF &6/ )J  [( =@K  u!#+ #   Es1v   sC$$ D-DDc,Kddlm}ddlm}|dk(r t d|j t d|j }||}|r? |j dd j d d}tjd |||d}t|} |j| |di|d{} t| jS#ttf$r/}tjd |j d |Yd}~d}~wwxYw7Zw)aZ Get the batch output file content as a list of dictionaries Args: batch: The batch object custom_llm_provider: The LLM provider litellm_params: Optional litellm parameters containing credentials (api_key, api_base, etc.) Required for Azure and other providers that need authentication r) afile_content)"_is_base64_encoded_unified_file_idrz1Vertex AI does not support file content retrievalNz3Output file id is None cannot retrieve file contentzllm_output_file_id,;z3Extracted LLM output file ID from unified file ID: z;Failed to extract LLM output file ID from unified file ID: z , error: )file_idr)litellm.files.mainrV1litellm.proxy.openai_files_endpoints.common_utilsrW ValueErroroutput_file_idsplitrr4 IndexErrorAttributeErrorerror _extract_file_access_credentialsupdate_get_file_content_as_dictionarycontent) r"rr#rVrWrZis_base64_unified_file_idrTfile_content_kwargs credentials _file_contents r r%r%s7 1k)LMM #NOO""G B7 K  /556KLQOUUG  EgYO 2 3>BK{+'>*=>>M *=+@+@ AA!N+   MeNbNbMcclmnlop   ?s<A D >C ,D7D8DD %D D DDc>i}|rgd}|D]}||vs||||<|S)a Extract credentials from litellm_params for file access operations. This method extracts relevant authentication and configuration parameters needed for accessing files across different providers (Azure, Vertex AI, etc.). Args: litellm_params: Dictionary containing litellm parameters with credentials Returns: Dictionary containing only the credentials needed for file access ) api_keyapi_base api_version organizationazure_ad_tokenazure_ad_token_providervertex_projectvertex_locationvertex_credentialstimeout max_retriesr[)r#rjcredential_keyskeys r rdrdsCK  # 7Cn$#1##6 C  7 r0 file_contentc@ |jd}g}|jjdD])}|s|jt j |+t jdt j|d|S#t$r}|d}~wwxYw)zO Get the file content as a list of dictionaries from JSON Lines format zutf-8 zjson_objects=%sindentN) decodestripr`r,jsonloadsrr4dumpsrH)rz_file_content_str json_objectslinerTs r rfrf&s (//8 %++-33D9 6D##DJJt$45 6 .   ($**5LUV*W - BE-e4!I%!P)CNSE*..w;E3H##,?#- 40K +"??J'"9"9,:,?"+";";"A"A#J $$_jA' B( MqQs=CBC C;C66C;c|dk(r|rt||\}}|Sd}d}d}|D]Q}t|st|} t| } || jz }|| j z }|| j z }St|||S)z= Get the tokens of a batch job from the file content rrrF)r3r)r*rrDrBrCr ) rrrr6rrDrBrCr-r.rEs r rrcsk)jA #Z ;LM(9 )% 0EeLNB>RE E.. .L U00 0M !8!8 8  9 !#+ r0)rrrcd}d}|D]Q}|jdi}|jd|xsd}|jdg}|s@t||} || z }St||z||S)z Count the number of tokens in the input file Used for batch rate limiting to count the number of tokens in the input file rbodyr(rmessages)r(rrF)r+r r ) rrrrBrCr-rr(r item_tokenss r _get_batch_job_input_file_usagersM()yy$*"2388J+ 'ehGK [ (M) "%66#+ r0rLcH|jddxsi}tdi|}|S)z> Get the tokens of a batch job from the response body rENr[)r+r )rL _usage_dictrEs r rrs- ##GT28bK';'E Lr0batch_job_output_filec^|jddxsi}|jddxsi}|S)z9 Get the response from the batch job output file r=Nrr+)r _responser.s r r*r*s6,// DAGRI]]6406BN r0cX|jddxsi}|jdddk(S)z7 Check if the batch job response status == 200 r=N status_coder)rrs r r)r)s1,// DAGRI == - 44r0)NN)N)rNN)rN)%rtypingrrrrrrlitellm._loggingrlitellm.types.llms.openair litellm.types.utilsr r r litellm.utilsr dictrIfloatr!r&rrr3r%rdbytesrfr5rrrr*boolr)r[r0r rs 66++;;'!%&* 1!$Z 1 B 1  1 # 1 5%c "# 1P!%%) %1 %1 B%1  %1 TN %1 5%c "#%1T!%!$Z  #Y.  $&* !$Z B   #  >!%@#Dz@ @ 5%<@N %) 5B 5B B5B TN 5B  $Z 5Bp"Xd^""J%DJ* &* )!$Z) B) # )  )`  $ !$Z  B     JDL $!$Z !?@  :4EDS5$545r0