AHj^UddlZddlZddlZddlZddlZddlZddlZddlZddlZddl Z ddl Z ddl Z ddl Z ddl Z ddlZddlZddlmZmZmZddlmZmZmZmZmZmZmZmZmZmZmZmZm Z m!Z!ddl"Z"ddl#Z#ddl$Z#ddl%m&Z&m'Z'ddl(m)Z)ddl*m+Z+m,Z,m-Z-m.Z.m/Z/m0Z0m1Z1m2Z2m3Z3m4Z4m5Z5m6Z6m7Z7m8Z8m9Z9ddl:m;Z;ddlm?Z?m@Z@mAZAmBZBmCZCmDZDmEZEmFZFmGZGmHZHmIZImJZJmKZKmLZLmMZMmNZNmOZOmPZPmQZQmRZRmSZSmTZTmUZUmVZVmWZWmXZXmYZYmZZZm[Z[m\Z\m]Z]dd l^m_Z_m`Z`dd lambZbdd lcmdZdmeZemfZfmgZgdd lhmiZimjZjerddlkmlZlddlmmnZoddlpmqZqeeoefZnneZneZqddddZreesefetd<dodZueue_uejdewgZxeyetd<e jjdejjd ddl}Z}ddl~Z~ddlZddlZddlZddlmZgdZdZdd lmZdd!lmZdd"lmZddlZdd#lmZdd$lmZmZdd%lmZmZdd&lmZdd'l*mZmZmZmZmZmZmZmZmZmZmZmZmZmZmZdd(lmZdd)lmZdd*lmZdd+lmZdd,lmZmZdd-lmZdd.l:mZdd/lmZdd0lmZmZdd1lmZdd2lmZdd2lmZdd2lmZdd3lmZdd4lmZdd5l>dd2lmZdd6lmZdd2lmZdd7lmZmZdd2lmZdd8lmZdd2lmZdd2lmZdd9lmZmZmZdd:lmZdd;lmZddlmZmZmZdd2lmZdd2lmZdd?lmZmZmZdd@l^mZddAlmZdd2lmZddBlmZmZddClmZddDlmZmZmZmZddElmZmZddFlmZddGl m Z ddHl m Z ddIl mZdd2lmZdd2lmZddJlmZddKlmZddLlmZdd2lmZddMlmZdd2lmZdd2lmZddNl m!Z!m"Z"ddOl#m$Z$dd2l%mZ&ddPl'm(Z(ddQl)m*Z*ddRl+m,Z,dd2l-mZ.ddSl/m0Z0dd2l1mZ2dd2l3mZ4dd2l5mZ6dd2l7mZ8ddTl9m:Z:m;Z;dd2lmZ?dd2l@mZAdd2lBmZCdd2lDmZEdd2lFmZGddUlFmHZHdd2lImZJddVlKmLZLmMZMmNZNdd2lKmZOdd2lPmZQdd2lRmZSddWlTmUZUmVZVmWZWdd2lTmZXdd2lYmZZdd2l[mZ\dd2l]mZ^dd2l_mZ`ddXlambZbdd2lcmZddd2lemZfdd2lgmZhddYlgmiZimjZjdd2lkmZlddZlmmnZndd2lmmZodd2lpmZqdd2lrmZsdd[ltmuZudd\lvmwZwdd]lxmyZydd2lzmZ{dd2l|mZ}dd2l~mZdd^l~mZdd_lmZdd2lmZdd`lmZddalmZdd2lmZdd2lmZdd2lmZdd2lmZdd2lmZdd2lmZdd2lmZdd2lmZdd2lmZddblmZdd2lmZdd2lmZdd2lmZdd2lmZddclmZdd2lmZdddlmZdd2lmZddelmZmZmZmZmZmZmZmZmZmZmZmZmZmZmZmZmZmZdd2lmZdd2lmZdd2lmZdd2lmZdd2lmZddflǐmȐZȐmɐZɐmʐZʐmːZddgl̐m͐Z͐mΐZddhlϐmАZddilѐmҐZddjlӐmԐZԐmՐZՐm֐Z֐mאZddklؐmِZddllڐmېZېmܐZܐmݐZݐmސZddmlߐmZddnlmZddolmZddplmZmZddqlmZddrlmZddslmZddtlmZmZmZddulmZddvlmZmZddwlcmZmZddslcmZddxlcmZmZddylhmZ ddzlmZd|e_ddlZdd}lmZdd~lmZmZmZmZmZmZmZmZm Z m Z m Z m Z m Z mZmZddlmZddlmZddlmZddlmZddlmZmZmZmZmZddlmZddl m!Z!ddl"m#Z#ddl$m%Z%ddl&m'Z'eZ( ddl)m*cm+Z+ dd2l,mZ-ddl.m/Z/e-Z(e/Z0ee/etd<eZ1ea2tdjga4e5etd<tdjlZ7edetd<ejpdZ9eesetd<e Z:eZ;e9 dZ<e=etd<n e=e9Z<ejpdZ>eesetd<e> dZ?e@etd<n e@e>Z?e1dZAe1dZBe1dZCdeAdeBdZDeDdz ZDeDdeCdz ZDe1dZEeDdeEdz ZDdZFthrejpddndZGthrejpddeFdeDn deFdeDZHdZIdZJdZKedefdZLeeeeGeHee1eLZeZMddlmNZNdZOdZPejpdddk(rthr ePeŐ_Qn eOeŐ_QGddejZSeŐjeVde deVfdZUeZdgZV ejjejjeXZYejjeYddZ[e[Z\dZ]desde5fdZ^desde5fd„Z_desde5fdĄZ`desdesdeae5esffdDŽZbejpddɫjdk(ZdedrdːZene[Zeejpd̐eeZfe_e[sejd͐e[dΝefe[k7Zhehre`efZie^efZjejreirejdϐefefZ\nejr eisejdАefdѝefZ\nqejdҐefdӝebe[ef\ZmZgemrefZ\n@ejdԐefdՐegd֐e[dםe[Z\nejdؐe[e[Z\e_e\sejdِe\dڝe1r)e1dk7r"eje\ejZpepsejdܐe\dݝneje\D]\ZrZsZtetD]ZuejjereuZveujdޫr8 exevdd5ZyeyjZ{ddde{je]e1Z}e}jde1dZ}exevdd5Zyeyje}dddeŐjde%ejje\ddeŐje]de%ejje\ddeŐjde%e\d|ddesddfdZ e`e\Zieje\ejZpeirejde\dn=epsejde\dn!ee\ejde\ejjejjeXZYeŐj eeVd|dgdge8eŐj eyeŐj ewdZeeZejpdZeesetd<edk7reeŐjdddZddlmZdadadadadadad|adZdadaeesetd<d|Zdadaeeetd<daeeyetd<iaeetd<daeeeesefetd<dZdZdaeesetd<dZeee'etd<dadaeeetd<daedetd<eeSjFjHaeeSjFjHZe(tJ ZejPjSedaeeetd <daeed eese5fetd <gaeesetd <dae=etd<dadadae5etd<dae5etd<dadadadadadadadaiaeesee=eeesefffetd<dadagZeetd<dZead aeedefetd<eaeaeaeadadaeZdaee*etd<daɐe5etd<daeeqetd<etJthadZdZdZdadadadaeeetd<de fdZӐdZdefdZՐdpde5fd Z֐d!Zאd"esd#e@de@fd$Zؐd%eesd&eesd'eesd(ee@fd)Zِd"esd*esd+e@fd,Z dqd%eesd'eesd-eesd&eesd(ee@d.eend/eeesfd0Zېd1Zdee@fd2Zdesfd3Zސd4eyd5ee5d6ee=d7efd8Zߐd4eyd9eyd:eyfd;Zd9eyd:eyddfd<Zd=ZGd>d?eZGd@dAZeadBZ drdCZdDZdEe]dFefdGZdFedesfdHZdIedJesdFedKe5deee5ff dLZdEe]dFefdMZdEe]dFefdNZifdOefdPZdQZdRZGdSdTZejdUeegdVgWejdXeegdVgWeedddddddfdEe]dYee5d&eesdZee5d[ee5d\ee5d]eesd^eesfd_Zejd`eegdVgWejdaeegdVgWeefdbesdEe]fdcZejddeegdegWejdfeegdegWejdgeegdegWejdheegdegdidjdkiieldeefde dme dOeesdEe]fdnZejdoeegdpgWejdqeegdpgWejdreegdpgWejdseegdpgWdeefde dme dOeesdEe]fdtZejdueegedvgwejdxeegedvgwejdyeegedvgwejdzeegedvgwdeefde dme dOeesdEe]fd{Zejd|eeged}gwejd~eeged}gweefde dme dEe]fdZdedeedffdZejdeegdgWejdeegdgWeefde dme dEe]fdZejdeegdgWejdeegdgWedeefde dme de dEe]fdZeŐjdejddejddejddeefde dOeesdeesdeesfdZeedOesdeesdeeesesfdffdZeŐjdeŐjdejddejddeefde dOesdesdeesfdZejdeegdgWejdeegdgWeefde dme dEe]fdZejdeegdgWejdeegdgWeefde dme dEe]fdZejdeegdgWejdeegdgWeefde desdme dEe]fdZejdeegdgWejdeegdgWeefde dme dEe]fdZejdeegdgWejdeegdgWeefde desdme dEe]fdZejdeegdgWejdeegdgWeefde desdme dEe]fdZejdeegdgWejdeegdgWeefde desdme dEe]fdZejdeegdgWejdeegdgWeefde desdme dEe]fdZ ddl m Z dedesdeee eeseesffdZ ejddgeegegdpde[de5fdZ ejddgeegdOesfdZejddgeegede\fdZdeedEe]deedeefd„ZdeedeRdeefdĄZdeededEe]deefdƄZdeeQdedeeseesffdȄZdeeesedfdededeeseesffdʄZdeRdedeesfd̄ZdEe]dededeedeef d̈́Z dsdOeesefdΐe5deedeeseffdτZ dqdeeesefdesdѐe=dҐe=deededeesdeeeesefee=ffdՄZdedeefdׄZ dtdeeesefdeesdesdeeeseffdڄZdeeesefdѐe=dҐe=dee=deesdeeseff d܄Zdeeesefd&esdededeeeseff d݄Zdbesdeesdeaeyee=ffdބZejdߐddVgeegdeeejddejddejdddedddedddejddejddejddejddejdؐdf dEe]dOeesdee5dee5dee5dѐe=dҐe=deesdeesdeesdeesdeesfdZejdddVgdeegeedddfdEe]deesdeedeefdZejdddVgdeegeedddddfdEe]deesdeedeedeesdeesf dZ ejdddVgdeegeedddddfdEe]deesdeedeedeesdeesf dZ!ejdddVgdeegeedddddfdEe]deesdeedeedeesdeesf dZ"dOedefd Z#ejd dVgeegejd dVgeegeedfdEe]d eesfd Z$dedeesdeesdeefdZ%ejddVgeegeedfdEe]deesfdZ&ejdddVgeegddZ'ejdddgeegdeefdEe]fdZ(ejddgeegddeefde dme dOeesdEe]fdZ)eŐjddgdde fdZ*ejd dde fd!Z+ejd"dde fd#Z,ejd$dde fd%Z-ejd&dde fd'Z.eŐjd(dd)esde fd*Z/eŐjd+dd,eJfd-Z0eŐjd.dd/Z1eŐjd0dd1Z2eŐjd2dd3Z3ejd4d5geegeLd6eefd,eMdEe]fd7Z4ejd8d5geegeLd6eefd9esdEe]fd:Z5ejd;d5geegeLd6eefd,eNdEe]fd<Z6ejd=d5geegeLd6eefd,eKdEe]fd>Z7ejd?d@geegdeefdAeGdEe]fdBZ8 ejdCd@geegdeefd,eDdEe]fdDZ9ejdEd@geegeCd6eefdFesdEe]fdGZ:ejdHd@geegdeefdIeddEe]deeFfdJZ;ejdKd@geegdeefd,eBdEe]fdLZejdRd@geegddAeGfdSZ?ejdTdVgeegdeefdEe]fdUZ@ejdVdVgeegdeefdWe=dEe]fdXZAejdVdVgeegdeefdEe]fdYZBejdZdVgeegdeefdEe]fd[ZCejd\dVgeegdeefdEe]fd]ZDejd^dVgeegdeefdEe]fd_ZEejd`dVgeegdeefdWe=dEe]fdaZFejd`dVgeegdeefdEe]fdbZGejdcdVgeegdeefdEe]fddZHejdeegede fdfZIejdgeegedhZJeŐjeeŐjeeŐjeeŐjeeŐjeeŐje{eŐjeeŐjeƫeŐjeeŐjeeŐje.eŐjeeŐjeeŐjeeŐje«eŐjeeŐjeeŐjeeŐjeQeŐjeeŐje׫eŐje٫eŐje}eŐjeիeŐjeeŐjeīeŐjeeŐje&eŐjeOeŐjeGeŐjeheŐjeoeŐjebeŐjeZeŐje^eŐjeCeŐjeeŐjeeŐjeeŐjeeŐjeӫeŐjeeŐje\eŐjeqeŐjeeŐjeeŐjeeŐjeeŐje8eŐjeeŐjeeŐjeeŐjefeŐjeJeŐje4eŐjeXeŐjeSeŐjedeŐjeleŐjeAeŐje`eŐjeEeŐje6eŐje?eŐjeseŐje(eŐjeeŐjeΫeŐje=eŐjeʫeŐje2eŐjdigdjkdlesde fdmZMeŐje1eƐneŐjeëeŐjey#e$rZededdZ[wwxYw#e$rd{ZY'wxYw#e$r ddl+Z+n #e$rYnwxYwY&cwxYw#e$rdZ0Y&JwxYw#1swY!|xYw#1swY!,xYw#eeef$rY!wxYw#e$r%Zejde\deYdZ[ ?dZ[we$r%Zejde\deYdZ[ hdZ[wwxYw#e$rY ywxYw(uN)datetime timedeltatimezone) TYPE_CHECKINGAnyAsyncGeneratorDictListLiteralOptionalSetTupleUnioncastget_args get_originget_type_hints) BaseModelJson)uuid)AIOHTTP_CONNECTOR_LIMIT AIOHTTP_CONNECTOR_LIMIT_PER_HOSTAIOHTTP_KEEPALIVE_TIMEOUTAIOHTTP_NEEDS_CLEANUP_CLOSEDAIOHTTP_TTL_DNS_CACHEAUDIO_SPEECH_CHUNK_SIZEBASE_MCP_ROUTEDEFAULT_MAX_RECURSE_DEPTH$DEFAULT_SHARED_HEALTH_CHECK_LOCK_TTLDEFAULT_SHARED_HEALTH_CHECK_TTL DEFAULT_SLACK_ALERTING_THRESHOLD z6("X->->,?r'"MN  >"default)ramessagesz../..)AsyncIOSchedulerzMissing dependency z$. Run `pip install 'litellm[proxy]'`)z%'The thing I wish you improved is...'z'A feature I really want is...'z*'The worst thing about this product is...'z$'This product would be better if...'z 'I don't like how this works...'z&'It would help me if you could add...'z/'This feature doesn't meet my needs because...'z&'I get frustrated when the product...'cd}tjt}ttdd|zzdztdd|zzdztddj |ztddj dztdd|zzdztdd|zzdzttd ttttd tty) N<z#-z# zz# {:^59} #z-https://github.com/BerriAI/litellm/issues/newz/ Thank you for using LiteLLM! - Krrish & IshaanzRGive Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new)randomchoicelist_of_messagesprintformat) box_widthr`s rggenerate_feedback_boxrwsImm,-G G sY . ;< sY . ;< ,,33G< <=   $ $%T U V sY . ;< sY . ;< G ;< G G G b G Gri) defaultdict)asynccontextmanager) lru_cache)Routerverbose_proxy_loggerverbose_router_logger) DualCache RedisCache)RedisClusterCache)_REALTIME_BODY_CACHE_SIZEAPSCHEDULER_COALESCEAPSCHEDULER_MAX_INSTANCESAPSCHEDULER_MISFIRE_GRACE_TIMEAPSCHEDULER_REPLACE_EXISTINGDAYS_IN_A_MONTHDEFAULT_HEALTH_CHECK_INTERVALDEFAULT_MODEL_CREATED_AT_TIMELITELLM_PROXY_ADMIN_NAME)PROMETHEUS_FALLBACK_STATS_SEND_TIME_HOURSPROXY_BATCH_POLLING_ENABLEDPROXY_BATCH_POLLING_INTERVALPROXY_BATCH_WRITE_AT!PROXY_BUDGET_RESCHEDULER_MAX_TIME!PROXY_BUDGET_RESCHEDULER_MIN_TIME)RejectedRequestError)ModifyResponseException) CustomLogger) SlackAlerting)!_get_parent_otel_span_from_kwargs get_litellm_metadata_from_kwargs)CredentialAccessor)Logging)SensitiveDataMasker)AsyncHTTPHandler HTTPHandler) VertexBaserouter)app)global_mcp_tool_registry)*global_agent_registry)append_agents_to_model_groupappend_agents_to_model_info)claude_code_marketplace_router)ExperimentalUIJWTTokenget_team_objectlog_db_metrics)check_response_size_is_safe) JWTHandler) LicenseCheck)get_all_fallbacksget_complete_model_listget_key_modelsget_mcp_server_idsget_team_models)+_fetch_global_spend_with_event_coordinationuser_api_key_authuser_api_key_auth_websocket)ProxyBaseLLMRequestProcessing_is_azure_model_router_requestcreate_response)initialize_callbacks_on_proxy)init_verbose_loggers)decrypt_value_helperencrypt_value_helper)build_ui_login_form)_read_request_body_safe_get_request_headerscheck_file_size_under_limit get_form_data)!get_config_file_contents_from_gcsget_file_contents_from_s3)%remove_sensitive_info_from_deployment) ProxyState)ResetBudgetJob)ERROR_RESPONSESSpendLogCleanup)PrismaDBExceptionHandler)ui_discovery_endpoints_router)set_fine_tuning_config)init_guardrails_v2initialize_guardrails)perform_health_check)&_PROXY_VirtualKeyModelMaxBudgetLimiter)"_OPTIONAL_PromptInjectionDetection)_ProxyDBLogger)add_litellm_data_to_request)_user_has_admin_privilegesadmin_can_invite_user) user_update)delete_verification_tokensduration_in_secondsgenerate_key_helper_fn)_add_model_to_db_add_team_model_to_db"_deduplicate_litellm_router_models) scim_router) update_teamvalidate_membership),get_disabled_non_admin_personal_key_creation)create_audit_log_for_update)InFlightRequestsMiddleware)PrometheusAuthMiddleware)set_files_config)passthrough_endpoint_router$vertex_ai_live_websocket_passthrough)!initialize_pass_through_endpoints) route_request)get_logging_payload)get_instance_fn) PrismaClient ProxyLoggingProxyUpdateSpend_cache_user_row _get_docs_url_get_projected_spend_over_limit_get_redoc_url_is_projected_spend_over_limit_is_valid_team_configsget_custom_urlget_error_message_strget_server_root_pathhandle_exception_on_proxy hash_password hash_token!migrate_passwords_to_scrypt_async model_dump_with_preserved_fields update_spend)AssistantsTypedDict DeploymentLiteLLM_ParamsModelGroupInfo)FlowItem Scheduler) load_aws_kms)load_google_kms) get_secretget_secret_boolget_secret_str str_to_bool)SlackAlertingArgs)AnthropicMessagesRequestAnthropicResponse!AnthropicResponseContentBlockTextAnthropicResponseUsageBlock)HttpxBinaryResponseContent)WorkerRegistryEntry)ModelGroupInfoProxy)DefaultTeamSSOParams#LiteLLM_UpperboundKeyGenerateParams)RealtimeQueryParams)DeploymentTypedDict) ModelInfo)RouterGeneralSettingsSearchToolTypedDictupdateDeployment)DefaultPriorities)KeyManagementSettingsKeyManagementSystem)CredentialItemCustomHuggingfaceTokenizer)RawRequestTypedDictStandardLoggingPayload)-_add_custom_logger_callback_to_specific_event)versionz0.0.0T)r)DependsFastAPIFileFormHeader HTTPExceptionPathQueryRequestResponse UploadFile WebSocketWebSocketDisconnect applicationsstatus)jsonable_encoder)CORSMiddlewareget_swagger_ui_html) get_openapi) FileResponse JSONResponseORJSONResponseRedirectResponseStreamingResponse) APIRouter)OAuth2PasswordBearer) APIKeyHeader) StaticFiles) AgentConfig)EnterpriseProxyConfigenterprise_proxy_config premium_userr1premium_user_data+LITELLM_GLOBAL_MAX_PARALLEL_REQUEST_RETRIES'global_max_parallel_request_retries_env#global_max_parallel_request_retries1LITELLM_GLOBAL_MAX_PARALLEL_REQUEST_RETRY_TIMEOUT-global_max_parallel_request_retry_timeout_envgN@)global_max_parallel_request_retry_timeoutz/uiz/fallback/loginz/ui/model_hub_tableu(👉 [```LiteLLM Admin Panel on /ui```](zI). Create, Edit Keys with SSO. Having issues? Try [```Fallback Login```]()uB 💸 [```LiteLLM Model Cost Map```](https://models.litellm.ai/).u! 🔎 [```LiteLLM Model Hub```](z[). See available models on the proxy. [**Docs**](https://docs.litellm.ai/docs/proxy/ai_hub)z/ui/chatu 💬 [```LiteLLM Chat UI```](zN). ChatGPT-like interface for your users to chat with AI models and MCP tools.zr[**Customize Swagger Docs**](https://docs.litellm.ai/docs/proxy/enterprise#swagger-docs---custom-routes--branding) DOCS_TITLEz LiteLLM APIDOCS_DESCRIPTIONzJEnterprise Edition Proxy Server to call 100+ LLMs in the OpenAI format.  z5Proxy Server to call 100+ LLMs in the OpenAI format. c8dadadadadadadadadada da da da yN) master_keyuser_config_file_path otel_logginguser_custom_authuser_custom_auth_pathuser_custom_key_generateuser_custom_sso"user_custom_ui_sso_sign_in_handleruse_background_health_checksuse_shared_health_checkhealth_check_intervalhealth_check_concurrency prisma_clientrirgcleanup_router_config_variablesr_sNJ L #O)-&#' " #MricbKtjdtr1tjdtj d{t j &t j j d{tjd{ttjd{dt jvr. ddl m }||jjt!yt!y777u7U#t$r Yt!ywxYww)Nz"Shutting down LiteLLM Proxy ServerzDisconnecting from PrismalangfuserlangFuseLogger)r}infor]debug disconnectlitellmcache jwt_handlerclosedb_writer_clientsuccess_callback litellm.utilsrcLangfuseflush Exceptionr_rbs rgproxy_shutdown_eventrqsBC""#>?&&(((}} mm&&(((    #$$&&&W---  4)''--/ $%#%/ ) ) '  $%  slAD/ D 5D/DD/D!D/>D?D/"D8D/D/D/D/ D, D/+D,,D/c xK ddlm}m}ttd}t rd|d<t dkDr t |d<tdkDr t|d<|di|}||}tjd t|d t d td |S#t$r#}tjd |dYd}~yd}~wwxYww)zNInitialize shared aiohttp session for connection reuse with connection limits.r)rQ TCPConnector)keepalive_timeout ttl_dns_cacheTenable_cleanup_closedlimitlimit_per_host) connectorzJSESSION REUSE: Created shared aiohttp session for connection pooling (ID: z, limit=z, limit_per_host=rKz)Failed to create shared aiohttp session: z#. Continuing without session reuse.Nr^) aiohttprQrsrrrrrr}rdidrpwarning)rQrsconnector_kwargsrysessiones rg"_initialize_shared_aiohttp_sessionrs7";2,  (8< 4 5 "Q &(? W % +a /1Q - . 4#34 )4!!XY[\cYdXef,-->?_>``a c  $$7s:] ^  s/B:BB B: B7B2-B:2B77B:rcnKddl}ttjj dd}|r|j dD]}|j }|s d|vrtd|d|jdd\}}tj|}t||}tj|r|d{n|tjd |tj$d j't(t(d urt*j-at/d at3d} t/d} tj$d| | `tj4j7| r`t8j;| rIt8j=t>| d{\aa a!n | tE| tFr]tj4j7| r>t8j;| r(t8j=t>| d{\aa a!ntjj d8tE| tFr(t8j=t>| d{\aa a!nYtE| tHrtKd.i| d{n5|jL| } tE| tHrtKd.i| d{tN4t3dd} tPjS| tTtVd{a'tNd} tYjZ| tPj]t>tTt^tPjatBt^ tj$dt8jc} | j di}tj$dte|jgtPjit>|d{tj$dtPjktBtNtV!tltljot>"tj$d#tNtNdtpjrdkDrQtPjutv$tYjZtPjytvtVtN%tNstPj{tBtNt|t~ttT&d{tPjd{tPjd{trtYjZttPjtPjtd{aHdt2 tjd{tjd'tNQttNd)rAttNjd*r' tNjjd{tN-ttNd,r tNjd{td{y7#t $r}tj"d ||d}~wwxYw777P7+777#t $r%}tj"d|d Yd}~d}~wwxYw7777Z75#t $r#}tj"d(|Yd}~Ed}~wwxYw7#t $r#}tj"d+|Yd}~d}~wwxYw7#t $r#}tj"d-|Yd}~d}~wwxYw7w)/NrLITELLM_WORKER_STARTUP_HOOKS,r[zInvalid hook spec 'z1': expected format is 'module.path:function_name'z.Worker startup hook '%s' executed successfullyz#Worker startup hook '%s' failed: %szElitellm.proxy.proxy_server.py::startup() - CHECKING PREMIUM USER - {}FLITELLM_MASTER_KEY WORKER_CONFIGCONFIG_FILE_PATHzworker_config: %sconfig_file_pathrrLITELLM_CONFIG_BUCKET_NAME DATABASE_URL) database_urlproxy_logging_objuser_api_key_cachecK ttd{}tjd|y7#t$r"}tj d|Yd}~yd}~wwxYww)NzPassword migration: zPassword migration skipped: )rr]r}rdrpr|)resultrs rg_run_pw_migrationz.proxy_startup_event.._run_pw_migrationls_ Q@OO$)),@*IJP Q$,,/KA3-OPP Qs6A$646A$6 A!AA$A!!A$) llm_routerrredis_usage_cache)general_settingsrz(About to initialize semantic tool filterlitellm_settingszlitellm_settings keys = )rrz)After semantic tool filter initializationzSemantic filter init failed: Texc_info)rr]rrzprisma_client: %s)litellm_proxy_budget_name)litellm_proxy_admin_namerr])rr]!proxy_budget_rescheduler_min_time!proxy_budget_rescheduler_max_timeproxy_batch_write_atrz,SESSION REUSE: Closed shared aiohttp sessionz&Error closing shared aiohttp session: dbstop_token_refresh_taskz#Error stopping token refresh task: stop_db_health_watchdog_taskz(Error stopping DB health watchdog task: r^)Ojsonrosenvirongetsplitstrip ValueErrorrsplit importlib import_modulegetattrinspectiscoroutinefunctionr}rdrperrorrerurB_license_check is_premiumrrQrrXisfile proxy_configis_yaml load_configrllm_model_listr isinstancestrdict initializeloadsr]ProxyStartupEvent_setup_prisma_clientrrasyncio create_task_initialize_startup_loggingr)_validate_redis_transaction_buffer_configget_config_statelistkeys _initialize_semantic_tool_filter_initialize_jwt_authprompt_injection_detection_objupdate_environmentrg max_budget_add_proxy_budget_to_dbr_warm_global_spend_cache$initialize_scheduled_background_jobsrrr"_update_default_team_member_budget%_sync_ui_settings_to_general_settingsrY_run_background_health_check_init_dd_tracer_init_pyroscopershared_aiohttp_sessionrjhasattrrrrrq)rr_startup_hooks_env _hook_spec _module_path _func_name_module_hook_fnr worker_configenv_config_yaml_db_urlr_config_litellm_settingss rgproxy_startup_eventrs(FK,2237 J#))+J j($-j\9jk,6+<+>/ *|/C/C,0D0 #..!O/     " }c *}-$$m$D #..!M/   ZZ^^8 9 E* 3K #..!M/    t ,-}- - -'DJJ}5M-. 1=111!+ND!A/DD /1E    Q -/011++2??)+@ W""#MN//1#KK(:B?""&t,=,B,B,D'E&F G  @@!.A    ""#NO **)#-+ &1&999L2MB W%7%7!%;11&> 2    6 6)A#5+ 7   DD-'.O.O!5/ E    BBDDD EEGGG$ ( * %%'%%'$F#GG ) U(..0 0 0 % %&T U ! M4 ( M$$&? @ R""::< < <  W5& W<<> > >    U%  $**9:q  < . 2  H W""%B1##FQUVVW8  E HH 1 U & &)OPQs'S T T U = R & &)LQC'P Q Q R ? W & &)QRSQT'U V V W!sA^5A(Z#Z "Z#&C^5:[ ;A0^5+[,A^5['^5*[+5^5 [!9^5[A*^5B[![[!(D^5,\-^5\ ^5$\%A^5\^5\!)\*\!0^54!]]]^51^^ ^ ^5^2^5 Z## [ ,[[  ^5^5^5^5^5^5[!! \*\ ^5 \^5^5^5^5\!! ] *]^5]  ^5] ]>]93^59]>>^5^ ^/ ^*$^5*^//^5)docs_url redoc_urltitle descriptionr! root_pathlifespan)APIWebSocketRoutec tjrtjSddlm}|ttj tj tjtj}tjDcgc]}t|ts|}}|D]}|jjddjd}g} t|drX|jLt!|jdg}|r3|D].}|j#|j$d|j&d d id 0d d |j$xs|dd|j$xs|j-dd|dddiidgdi|d|<ddlm}|j3|}t4rddt4j7dzig|d<|t_tjScc}w#t(t*f$rYwxYw)Nr)get_openapi_schema_with_compat)get_openapi_funcrr!rroutes{? dependant query_paramsqueryrVstring)nameinrequiredschemarz WebSocket: zWebSocket connection endpoint websocket_/_101rzWebSocket Protocol Switchedr-)summaryr operationId parameters responsestagspathsCustomOpenAPISpecurlservers)ropenapi_schema0litellm.proxy.common_utils.openapi_schema_compatrr5rr!rrrrrXrrstriprrrappendrrAttributeError TypeErrorreplace.litellm.proxy.common_utils.custom_openapi_specradd_llm_api_request_schema_bodyserver_root_pathr) rrroutewebsocket_routes base_pathrrparamrs rgget_openapi_schemars !!!4$ii OOzz N::E;L)M "# JJ$$S)!,33C8   uk*u/J&uK !- "))(- &-,1NN$*H+"    ()@y(AB>!+EJJ,U):K:KCQT:U+VW(#m5R%ST$  . w *5# LQ&FF~VN&+S3C3I3I#3N-N%O$Py!'C   i6 *   sGGA$GG.-G.cxtjrtjSt}tjj }i}|D]}||dvs |d|||<||d<ddlm}|j|}trddtjdzig|d<|t_tjS)Nrrrrrr) rrr LiteLLMRoutes openai_routesvaluer rr r r)rrpaths_to_includer rs rgcustom_openapirDs !!!')N"//55ME N7+ +&4W&=e&D U #E/N7Q&FF~VN&+S3C3I3I#3N-N%O$Py!'C   ri DOCS_FILTEREDFalseTrueceZdZdZy)UserAPIKeyCacheTTLEnumrnN)r^ __module__ __qualname__in_memory_cache_ttlr^rirgrresrirrequestexccK|j}|j}t|jrt |jnt j d|i|Sw)Nr) status_codecontentheaders)r%to_dictr7codeintr0HTTP_500_INTERNAL_SERVER_ERROR)r r!r% error_dicts rgopenai_exception_handlerr+isMkkGJ  XXCM6+P+P*%  sAAr _experimentaloutz/litellm-asset-prefixrXreturnc tjj|xrttj|S#t $rYywxYw)NF)rrXisdiranyscandirFileNotFoundError)rXs rg_dir_has_contentr4s? 77==&@3rzz$/?+@ @   s>A A  A ui_pathc@tjj|sytjjtjj |dsytjj |d}tjj|syy)a Verify UI directory has minimum required structure. Checks for: - Directory exists - Has index.html (main entry point) - Has _next directory (Next.js assets) Returns True if UI directory appears valid and servable. F index.html_nextT)rrXr0existsjoin)r5next_dirs rg_validate_ui_directoryr<sgww}}W%ww~~bggll7LAB77<<1ww}}X&riui_dircLtjj|sytjj|d}tjj |rt j d|ytjj tjj|dsy tj|D]}|js|jjdr0tjj|jd}tjj |szt j d|jdy y#ttf$r%}t j d |d |Yd }~yd }~wwxYw) ad Detect if UI directory is already pre-restructured and ready to serve. Returns True if: 1. Marker file .litellm_ui_ready exists (created by Dockerfile), OR 2. Restructuring pattern detected (subdirectories with index.html inside) This allows skipping copy/restructure operations on read-only filesystems. Fz.litellm_ui_readyzFound UI ready marker: Tr7rz,Detected restructured UI via pattern: found z /index.htmlzCould not scan z for restructuring detection: N) rrXr0r:r9r}rer2is_dirr startswithPermissionErrorOSError)r= marker_fileentry index_pathrs rg_is_ui_pre_restructuredrFs>ww}}V$ggll6+>? 77>>+ & & &)@ 'N O ww~~bggll6<@A F+ $<<>%***?*?*D!#ejj,!GJww~~j1,22J5::,Vab $ $  )  & &!&)GsK   s18(E/!E/=A E/$E/,E//F#>FF# source_path target_pathc tj|dt|s/index.html..rr>litellm-asset-prefixr8z.htmlr7TrJN) rwalkrXrelpathrsependswithr:splitextrMr r3) r] current_rootrfilesrel_root first_segmentrb file_path target_dirrHs rg_restructure_ui_html_filesrl|s')ggg&6  "L!Uww|W=H"*c/Bx~~bff7Ma7PM AA! ((1X5MGGLLx@ WW--i8;  ggll:|D  J6JJy+6  &)sD D*)D*zSkipping UI restructuring: z is already pre-restructuredzCannot restructure UI at z: path is not writable. UI may not work correctly for extensionless routes. Pre-build and restructure UI in Dockerfile for read-only deployments.zRestructured UI directory: z2Permission error while restructuring UI directory z'Error while restructuring UI directory ) allow_originsallow_credentials allow_methods allow_headersexpose_headersctjjtd}tdnt}|j ds|dz}|dzt jdt|dfd}|t_ y)Nswaggerrz/swaggerrWrZc8t|i|ddddS)Nz/swagger-ui-bundle.jsz/swagger-ui.cssz /favicon.png)swagger_js_urlswagger_css_urlswagger_favicon_urlr3)argskwargscustom_root_path_swagger_paths rgswagger_monkey_patchz.mount_swagger_ui..swagger_monkey_patchsD"   <<r/r4)swagger_directory swagger_pathr{rzs @rgmount_swagger_uirso [)<*238HL   %#c) $09$<!IIj+0ABIS ( ((==n>NO((KK   !ri counter_keyfallback_spendcTKtj7 tjj|d{}| t|S tjj|}| t|S|S7A#t$r!}t j d||Yd}~Vd}~wwxYww)a Read current spend from the cross-pod spend counter. Reads Redis FIRST (authoritative cross-pod value), not DualCache's async_get_cache which returns in-memory first. This is critical: DualCache.async_get_cache returns stale per-pod values because each pod's in-memory cache is only updated by that pod's own increments. Fallback chain: 1. Redis counter (cross-pod, authoritative) 2. In-memory counter (single-instance or Redis failure) 3. Cached object's .spend from DB (cold start, no counter yet) NkeyzJget_current_spend: Redis read failed for %s, falling back to in-memory: %s) spend_counter_cache redis_cacheasync_get_cachefloatrpr}rein_memory_cache get_cache)rrvalrs rgget_current_spendrs&&2 +77GGHCSz!  - - 7 7K 7 HC Sz %   & &\   s>B(#A;A9A;1B(9A;; B%B B( B%%B(tokenteam_iduser_id response_costcHK||dk(ry|Ht|tr|jdr t|n|}t d|||d{|t d|d||d{|&|#t d |d |d |d ||d{yyy7K7/7 w) a Atomically increment spend counters for budget enforcement. Uses spend_counter_cache (DualCache with Redis backend when available) so counters are shared across all pods. Budget check functions read from these counters via get_current_spend() (Redis-first). Awaited (not create_task) in the cost callback, so the counter is updated before the next request's auth check runs. Nrsk-rz spend:key:)rsource_cache_key incrementz spend:team:zteam_id:zspend:team_member:r[zteam_membership:)rrr@r!_init_and_increment_spend_counter)rrrr hashed_tokens rgincrement_spend_countersrs   2 %%%*:*:5*A U #  0$\N3)#   /%gY/'y1#   w2/,WIQwi@/y'C#    3   s6AB"BB".B/%B"B B"B" B"rrcKtj|d{}|~tj|d{}d}|8t|tr|j ddxsd}nt |ddxsd}|dkDrtj||d{tj||d{y777(7 w)u Initialize counter from cached object's DB-loaded spend if not yet set, then atomically increment in both in-memory and Redis. On first access per pod: 1. Check spend_counter_cache (in-memory -> Redis via DualCache for init check) 2. If not found anywhere, read base spend from user_api_key_cache (DB-loaded object) 3. Seed counter via async_increment_cache (not async_set_cache) to avoid a check-then-set race: if two pods cold-start simultaneously, both may see the counter as absent and seed it. Using increment instead of set means the worst case is over-counting (conservative — blocks slightly early) rather than under-counting (would allow overspend). 4. Increment atomically (both in-memory + Redis) rNspendrrr)rrrrrrrasync_increment_cache)rrrcurrentsource base_spends rgrr s&(77K7HHG)99>N9OO  &$'#ZZ5< $VWc:Ac >%;;z<    3 3 y 4 IO sDC CC CAC CC ;C<C C C C  end_user_idparent_otel_spanrc Kg dtdtf fd } fd} fd}  fd}  fd} ||| d{|d{| d{| d{| d{tjtj d | y7t7e7V7G78w) z` Use this to update the cache with new user spend. Put any alerting logic in here. rrc Kt|tr|jdr t|}n|}t j d|t j|d{}t j d||y|j}||z}|jdur|jt||jdd urt||jjdd\}}|jjdtd }t|j xsd ||j"||j$||t&j( } t+j,t.j1d | |'t3|dd|j4xsd} | |z|_|'t3|dd|j6xsd} | |z|_||_ j9||fy7w)Nrrz"_update_key_cache: hashed_token=%srz&_update_key_cache: existing_spend_obj=F soft_budget) current_spendsoft_budget_limitTinfr)rr key_aliasrrprojected_spendprojected_exceeded_date event_groupprojected_limit_exceeded)rV user_info team_spendrteam_member_spend)rrr@rr}rerrrsoft_budget_cooldownlitellm_budget_tablerrrrr)rrrr8KEYrrr budget_alertsrrrr) rrrexisting_spend_objexisting_spend new_spendrr soft_limit call_infoexisting_team_spendexisting_team_member_spendvalues_to_update_in_caches rg_update_key_cachez'update_cache.._update_key_cacheAs& eS !e&6&6u&=%E2L L""#GVBTBdBdiuBdBv}tjd t|tjYd}~yd}~wwxYww)Nrz!_update_user_db: existing spend: ; response_cost: r{}:spendzSpend tracking - failed to update user spend in cache. Budget enforcement may use stale spend values. user_id=%s, response_cost=%s - %s %s)rrr}rerrrrrrurrpr|r traceback format_exc) user_ids_idrrrglobal_proxy_spendrrrrrs rg_update_user_cachez(update_cache.._update_user_caches90  W;+=+M+MRU+M+V%V"%-$**78J7KK\]j\kl0$7%7%@N%7%=%=N*]: 0$72;&w/-44c;M5NO/8&,-44c;M;R;R;T5UV3 W6(:'I'I%%&>?(J(" ")*/A/M.> )00&&'?@)L0N*;&W."  ( (8A$$&   sgE<!D2D-D2E<CD23D04D2:E<;/D2*E<-D20D22 E9;4E4/E<4E99E<c FKydj} tj|d{}|ytjd|d|d}n"t |t r|d}n |j}|z}t |t r||d<j||fy||_j||jfy7#t$r>}tjdt|tjYd}~yd}~wwxYww)Nzend_user_id:{}rz%_update_end_user_db: existing spend: rrrzSpend tracking - failed to update end user spend in cache. Budget enforcement may use stale spend values. end_user_id=%s, response_cost=%s - %s %s)rurrr}rerrrrrrpr|rrr)r rrrrrrrs rg_update_end_user_cachez,update_cache.._update_end_user_cachesB  -"7 %%k2$ '9'I'Ic'I'R!R !) & &78J7KK\]j\kl ")!"0$7%7%@N%7%=%=N&6I,d3.7"7+)00#7I1JK+4"()00#7I7N7N7P1QR1"S2  ( (<A$$&   sUD!CCCD!A/C+D!,(CD!C D 4DD!DD!c 2Kydj} tj|d{}|ytjd|d|d}n"t |t r|d}n |j}|d}|z}t |t r||d<j||fy||_j||fy7#t$r>}tjdt|tjYd}~yd}~wwxYww)Nz team_id:{}rz!_update_team_db: existing spend: rrrzSpend tracking - failed to update team spend in cache. Budget enforcement may use stale spend values. team_id=%s, response_cost=%s - %s %s)rurrr}rerrrrrpr|rrr)r rrrrrrrs rg_update_team_cachez(update_cache.._update_team_cachesG ?m3 !!'*( )88S8AA ") & &34F3GGXYfXgh ")250$7%7%@N%7%=%=N%!$&6I,d3.7"7+)00#7I1JK+4"()00#7I1JK5B6  ( (8A$$&   sUDC C C DA3C /D0C D C D4D DDDc nKy D]}|rt|tsd|}tj|d{}|}t jd t|tjYd}~yd}~wwxYww) z: Update the tag cache with the new spend. Nztag:rz*_update_tag_cache: existing spend for tag=r\rrrzSpend tracking - failed to update tag spend in cache. Budget enforcement may use stale spend values. tags=%s, response_cost=%s - %s %s)rrrrr}rerrrrrrpr|rr) tag_name cache_keyexisting_tag_objrrrrrrs rg_update_tag_cachez'update_cache.._update_tag_cache&sp <=0 *   Tz(C'@"8*- );)K)K!*L*$ $+$**@ "M]L^^op}o~.5%5%9%9'1%E%JN%,-=w%J%OaN+]: .509$W--44iAQ5RS-6$*-44iAQ5RS? T $4  ( (5A$$&   sAD56C+C)B'C+(D5)C++ D244D-(D5-D22D5N)rrrn) cache_listttllitellm_parent_otel_span)rrrrrasync_set_cache_pipeline) rrrrrrrrr rrrrs ```` ` @rg update_cacher/s8:PMsPM5PMf3l)X-`1f ]6e=III """$&&& """ !!! 330%5 4  J # ' # "sZAC  B? C CC ,C-C =C>C C1C C C C C c ddg}ttjd5}tj|||dddy#1swYyxYw#t $r#}t jd|dYd}~yd}~wwxYw)NollamaserverV)stdoutstderrzd LiteLLM Warning: proxy started with `ollama` model `ollama serve` failed with Exceptionz). Ensure you run `ollama serve` )openrdevnull subprocessPopenrpr}re)commandr!rs rgrun_ollama_server%qs  W% "**c " Fg   WWW E F F F  ""eefdgh     s3AAAA A A A:A55A:c ddl}|j|jj}tj dk(rt |dz St |dz S#t$rYywxYw)zj Get process RSS memory in MB. On Linux, ru_maxrss is in KB. On macOS, ru_maxrss is in bytes. rNdarwinii)resource getrusage RUSAGE_SELF ru_maxrsssysplatformrrp)r(r+s rg_get_process_rss_mbr.sd &&x';';<FF <<8 ##{3 3Y$&& sA A A A&%A&c$t}|y|dS)Nunknown.2f)r.)rss_mbs rg_rss_mb_for_logr3s "F ~S\ri model_listdetailsmax_concurrencyinstrumentation_contextcK t||||d{S7#t$r/}dt|vrt|||d{7cYd}~Sd}~wwxYww)N)r4r5r6r7r7r4r5r6)rrr)r4r5r6r7rs rg-_run_direct_health_check_with_instrumentationr:sp  )!+$;       $CF 2 *!+     sHAA AAAA A AAAhealthy_endpointsunhealthy_endpointsc |yddl}ddlm}| |jnd}|j}t j |||||||y)zKFire-and-forget: persist health check results to DB if prisma is available.Nr)$_save_background_health_checks_to_dbbackground_health_check) checked_by)r0litellm.proxy.health_endpoints._health_endpointsr>pod_idrr) r]shared_health_managerr4r;r< time_moduler>r@rs rg)_schedule_background_health_check_db_saverEsh ! , $$ & !!#J ,     !   ric ddlm} ttjsy|||}|rstjj |t jdtd|jDtd|jDyy#t$r)}t jdt|Yd}~yd}~wwxYw) z Write deployment health states to the router's health state cache for health-check-driven routing. No-op if the feature is disabled. r)build_deployment_health_statesN)r;r<z:health_check_routing_state_updated healthy=%d unhealthy=%dc3DK|]}|jdsdyw is_healthyrNr.0ss rg z6_write_health_state_to_router_cache..sF!!%% 2EAF  c3DK|]}|jdrdywrIrKrLs rgrOz6_write_health_state_to_router_cache..sJ!aeeL6IAJrPz0Failed to write health state to router cache: %s) litellm.proxy.health_checkrGrenable_health_check_routinghealth_state_cacheset_deployment_health_statesr}resumvaluesrpr|r)r;r<rGstatesrs rg#_write_health_state_to_router_cacherYsJ  Z%K%K // 3    ) ) F Fv N & &LFv}}FFJv}}JJ    $$ >A   sBA>B C(C  Cc FKttttr tdkrytr$t j dtt tdat jdtt tttjtd}tr8t2ddlm}|tt t"}t jd t$d z ad t$}t'j(}t+j,t.xsg}t1|}|Dcgc]&}|j3d ij3d d s|(}}t1|}|}tt trt dkDr|dkDrt5|t }t j6d|||tt ||dutjt dd|d} ttnd} |% |j9|| t d{\} } n!tA|tt | d{\} } | tBd<| tBd<t1| tBd<t1| tBd<t'j(|z dz}t j6d||t1| t1| |ttjt |tdzkDrt j d||ttEtF||| | tI| | tKjLtd{ccc}w76#t:$rL} t j<dt?| tA|tt | d{7\} } Yd} ~ _d} ~ wwxYw7o7iw)z Periodically run health checks in the background on the endpoints. Update health_check_results, based on this. Uses shared health check state when Redis is available to coordinate across pods. Nrzxbackground_health_check_loop_overlap_detected existing_loop_active=true interval_seconds=%s max_concurrency=%s shared=%sTzzbackground_health_check_loop_started interval_seconds=%s max_concurrency=%s shared=%s details=%s thread_count=%d rss_mb=%s)SharedHealthCheckManager)rhealth_check_ttllock_ttlz'Initialized shared health check managerrzbg- model_infodisable_background_health_checkFzbackground_health_check_cycle_start cycle_id=%s model_count_total=%d model_count_enabled=%d interval_seconds=%s max_concurrency=%s expected_peak_in_flight=%d shared=%s thread_count=%d rss_mb=%sproxy_background_loop)enabledrcycle_idr9zEError in shared health check, falling back to direct health check: %sr;r< healthy_countunhealthy_countizbackground_health_check_cycle_complete cycle_id=%s model_count_enabled=%d healthy_count=%d unhealthy_count=%d duration_ms=%.2f interval_seconds=%s thread_count=%d rss_mb=%szibackground_health_check_cycle_duration_exceeded_interval cycle_id=%s duration_ms=%.2f interval_seconds=%s)'r[rr(#background_health_check_loop_activer}r|r\rZrdhealth_check_details threading active_countr3r>+--7=20 % 55r*../PRWX   "/2"5 / 5(1,#a'&)*=?W&X #"" P    ! $ # ! -  " " $   - # %9$D $  ! , 0KK.($<L%'0D$('  !#5F016I23034E0F_-256I2J./!^^-0@@DH"" {   ! " # $  !  " " $     5 < = ( ({!%   2  !      ,,=?RSmm1222s  X  $**[FH#(,+ %' $X 3szD)N!++L<BN!0M M MN!-N.DN!5N6 N!M N 8NNNN!NN!N!c eZdZy)StreamingCallbackErrorN)r^rrr^rirgrzrz srirzc eZdZUdZdpdZdedefdZdedefdZ dqde edefd Z d ed edefd Z d efdZ defd edededefdZdedeedefdZdefdZdefdZdZdqde edefdZd efdZdZd edeefdZd ede eefdZhdZeee d <d efd!Z!d"e e"jFdefd#Z$d efd$Z%d e ed%e d&d'e d(fd)Z&d*efd+Z' dqd,e ede efd-Z(drde)fd.Z*d/e+defd0Z,d/e+defd1Z-d2e+de+fd3Z.d2e e/d4e0fd5Z1d6ed7ee2d8d9e+ddfd:Z3d;eddfd<Z4 dqd=ed>e edefd?Z5 drd=ed@edefdAZ6dBedefdCZ7e8dDe9de efdEZ: dqdFe dGd%e e;d4e dHde efdIZdLZ?dMe e/fdNZ@dOedPe2dQdRe9defdSZAd%e;d edTe efdUZBdVeCeeDfdefdWZEd%e;de+fdXZFd%e;d4e0fdYZGd%e;fdZZHd%e;fd[ZId%e;fd\ZJd%e;fd]ZKd%e;fd^ZLd%e;fd_ZMd`ZNd%e;fdaZOd%e;fdbZPd%e;fdcZQd%e;fddZRd%e;fdeZSd%e;fdfZTdgZUd%e;fdhZVd%e;fdiZWdjZXdkeCeeYfdefdlZZdmeefdnZ[d%e;fdoZ\y)s ProxyConfigz} Abstraction class on top of config loading/updating logic. Gives us one place to control all config updating logic. r.Nc<i|_d|_d|_g|_yrP)config_last_semantic_filter_config_last_hashicorp_vault_configworker_registry)selfs rg__init__zProxyConfig.__init__ s!&( FJ)FJ)<>rirctjj|sytjj|\}}|j dk(xs|j dk(S)NFz.yamlz.yml)rrXrrelower)rrrfile_extensions rgrzProxyConfig.is_yaml sWww~~./GG,,-=>>##%0TN4H4H4Jf4TTrirjc  t|d5}tj|xsicdddS#1swYyxYw#t$r}td|dt |d}~wwxYw)z, Load and parse a YAML file rTNzError loading yaml file r\)r yaml safe_loadrpr)rrjrdrs rg_load_yaml_filezProxyConfig._load_yaml_file sh Ni% 2~~d+1r 2 2 2 N6ykCF8LM M Ns) =1 =:== A%A  A%c K|xst}||atjj|r,t |d5}t j |}dddn|td|giiid} td|j|tjjtjj|xsd}|S#1swYixYww)z Given a config file path, load the config from the file. Args: config_file_path (str): path to the config file Returns: dict: config NrTzConfig file not found: )r4rrouter_settingsrzConfig cannot be None or Empty.r)r~base_dir) rRrrXr9r rrrp_process_includesdirnameabspath)rrrj config_filer~s rg_get_config_from_filez!ProxyConfig._get_config_from_file s%=(=  '$4 ! 77>>YK ) c* 5k 4 5 5  "5i[AB B!$&#%$& F >=> >''BGGOOBGGOOIOQS4T$U(   ) 5 5s>),'*CI;(OPP"229=O-335 ( UeT*sf}3K&&u-"'F3K  ( ( 9  ri new_configcK ttjdddustrv|j }d|vr0|dr+|j |dd}|j ||d<|jddtj|d d{yttd 5}tj||d dddy78#1swYyxYww) NrFTenvironment_variables)rreturn_original_valuerr4r~)data table_namerV)default_flow_style) r]rrrrl!_decrypt_and_set_db_env_variables_encrypt_env_variablespop insert_datar rRrdump)rrconfig_to_savedecrypted_env_varsrs rg save_configzProxyConfig.save_config s  $  !4e < D (__.N(>9"#:;&*%K%K*89P*Q*.&L&" ;?:U:U*<;V;67   |T 2++H+U U U./#6 M+ *keL M M V M Ms*BCCC/C CCCrdepth max_depthc||kDrtjd|d|S|jD]\}}t|tr|j ||dz|||<1t|t r0|D]*}t|ts|j ||dz|},qt|ts|jdst|||<|S)a Check for os.environ/ variables in the config and replace them with the actual values. Includes a depth limit to prevent infinite recursion. Args: config (dict): The configuration dictionary to process. depth (int): Current recursion depth. max_depth (int): Maximum allowed recursion depth. Returns: dict: Processed configuration dictionary. zMaximum recursion depth (z") reached while processing config.r)r~rr os.environ/) r}r|rrr_check_for_os_environ_varsrrr@r)rr~rrrritems rgrz&ProxyConfig._check_for_os_environ_vars8 s 9  ( (+I;6XY M ,,. 0JC%&"==  Y>s E4(!D!$-#>>#'uqyI ?  E3'E,<,<],K(/s  0 rirall_teams_configc i}|D]3}d|vr!tdtj|||dk(s1|}n|jD]6\}}t |t s|j ds)t|||<8|S)Nrzteam_id missing from team: r)rpSENSITIVE_DATA_MASKER mask_dictrrrr@r)rrr team_configteamkvs rg_get_team_configzProxyConfig._get_team_config] s $ D$12G2Q2QRV2W1XY$y/)"   %%' /DAq!S!all=&A!+A A /ric|j}|jdi}|jdd}|iS|j||}|S)z^ - for a given team id - return the relevant completion() call params rdefault_team_settingsN)rr)rrr)rrr~rrrs rgload_team_configzProxyConfig.load_team_configl sh&&("::&8"=+//0GN  #I++.>, ri cache_paramscLddlm}d|vr|dt_d|vr|dt_|di|t_tjYt tjjt tfr*tjjatt_ yyy)NrCacherdefault_redis_ttlr^) rgrrrrhrrrrrr)rrrs rg _init_cachezProxyConfig._init_cache s " "l 2,89P,QG ) , .(45H(IG %- - == $ MM  *.?!@* !( 3 3 .?  + * $ricddl}tA|j4tjdur!dt_t j dyyyy)z Enable caching on the router by setting cache_responses=True. This ensures caching works without needing caching=True in request body. Router passes caching=self.cache_responses to litellm.completion() rNTz8Set router.cache_responses=True after initializing cache)rgrrhcache_responsesr}re)rrgs rgswitch_on_llm_response_cachingz*ProxyConfig.switch_on_llm_response_caching sO   " )**$6)-J & & &J 7* #ricKtjjdtjjd}tjjd}tjjd}tjd|||dk(rt ||d{}n t ||}|%td|j| d{}t,td ur$|j|tt d{}tj|}|jd d|j| }|j!| |S777Uw)a  Load config file Supports reading from: - .yaml file paths - LiteLLM connected DB - GCS - S3 Args: config_file_path (str): path to the config file Returns: dict: config rN LITELLM_CONFIG_BUCKET_OBJECT_KEYLITELLM_CONFIG_BUCKET_TYPEzbucket_name: %s, object_key: %sgcs) bucket_name object_keyz(Unable to load config from given source.rT)r~r]rrr~)rrrr}rerrrprr]r_update_config_from_dbrlrmrrupdate_config_state)rrrr bucket_typer~ printed_yamls rg get_configzProxyConfig.get_config sT$ ::>>6 7 C**..)EFK(JKJ**..)EFK & &1;  e#@ +  3 + ~ JKK 55GW5XXF  $):d)B66+"37F}}V, 0$7000?    / ?Ys7B)E*+E$,4E* E&!1E*E(AE*&E*(E*c||_yrPr)rr~s rgrzProxyConfig.update_config_state s  ric tj|jS#t$r;}t j dj |j|icYd}~Sd}~wwxYw)z Returns a deep copy of the config, Do this, to avoid mutating the config state outside of allowed methods z^ProxyConfig:get_config_state(): Error returning copy of config state. self.config={} Error: {}N)rlrmr~rpr}reru)rrs rgrzProxyConfig.get_config_state sV  ==- -  & &qxxKK  I  s! A%0A A% A%cj|jd}g}|r|Dcgc] }tdi|}}|Scc}w)z< Load the credential list from the database credential_listr^)rr)rr~credential_list_dictrcreds rgload_credential_listz ProxyConfig.load_credential_list sC &zz*;< BVW$~55WOWXs0c |jdd}|s&|jdi}|r|jdd}|syg}td|D]}|jdd}|jdijdd}td |d |d |jdi}|rb|jD]J\} } t| ts| j d s)| j d d} t| } | || <L||d< tdi|} |j| |r|SdS#t$r/} tjd |dt | Yd} ~ d} ~ wwxYw)a/ Parse and validate search tools from config. Loads environment variables and casts to SearchToolTypedDict. Args: config: Config dictionary containing search_tools Returns: List of validated SearchToolTypedDict or None if not configured search_toolsNrz6LiteLLM: Proxy initialized with Search Tools:search_tool_namerlitellm_paramssearch_provider  z (z)rzError parsing search tool r\r^) rrtrrrr@r rrrrpr}r)rr~search_tools_rawrsearch_tools_parsed search_toolrrrrr_vsearch_tool_typedrs rgparse_search_toolszProxyConfig.parse_search_tools s"::nd;%zz*CGG!2O L!1 2"_4EXN O)__-=rBN*002.DAq!!S)all=.IYY}b9&rN,-q) . 1? ,- 9L9[{9[!#**+<=+ 8':"CtC  $**01A0B"SVHM  s D11 E):$E$$E)>HOMEPATHUSERSHELLLOGNAMENO_PROXYno_proxy LD_PRELOAD PYTHONHOME PYTHONPATH PYTHONSTARTUPLD_LIBRARY_PATHDYLD_LIBRARY_PATHDYLD_INSERT_LIBRARIES_BLOCKED_ENV_KEYSc|jdd}|r|jD]\}}||jvrtjd|+t |t r4|jdr#t|}|[|tj|<ot |tj|<d|vr3tjddt_ tjay)Nrz-Skipping blocked environment variable key: %sr) secret_nameLITELLM_LICENSE)rrrr}r|rrr@rrrrr license_strrrB)rr~rrrresolved_secret_strings rg_load_environment_variablesz'ProxyConfig._load_environment_variablesN s!' +BD I 399; 1 U$000(00GeS)e.>.>}.Ml0m1}|^tjd?|jet |d@tjZdA||dBk(rgt_3| D]O}d;|vr*tj\jitK|<1tj\ji|Qt |dCtjfdA| |dDk(rgt_5| D]O}d;|vr*tjjjmtK|<1tjjjm|Q|jdEtjn}|r#t |dFtjjdA|tjpdG|d k(r|dHk(rx| jdIi}|jdJda9|jdKga:|jdLdMa;tj|dNtrdOttdPtv|Q|dQk(rQty| D]\}} t{di|tj|dT|dU| |tt|| |dVk(r| ot| t~r_| j D]6\}} t| t s| j#ds)t| | |<8tdi| t_Ct}dW| |dXk(r?| d ur;d t_Dtjtj|dY|otj|dT|dU| |tt|| |jdZi}!|!i}!d}"d}#|!r|!jd[d}$|$tdi|$t_G|!jd\d}%|j|%|]|!jd^d}&t|&_|!jd`d}'t|'a|j|!b|!jdcd}(|(rG|(j#dr6tjddt|(}(tjde|(|!jdftdgdaLtr$tj#drttaLt$ttt rttaNntjdh|!jdid})|) tjtO|)djtjdkdljdmk(}*|*r#ttsd aVtjpdn|!jdo}+|+r#ttsd aWtjpdp|!jdqdaXtdaXt|!dq<|!jdrd},|, tK|,|1aY|!jdsd}-|- tK|-|1aZ|!jdtd}.|. tK|.|1a[|!jdud}/|/ tK|/|1a\ttj|!d{|!jdvd|!dva_t|!dvwd{|!jdxdyaa|!jdzd}0|0t*dur td{|!jd|tac|!jd}tad|!jd~tae|!jdtaf|!jdtag|!jddah|!jddai|!jdtak|!jddal|!jdd am|!jdd}"|!jdd}#tj2dttttt|"|!jdd}1|1|1D2cgc] }2tdi|2c}2|!d<|!jd.t*d ur&tdtjjzd|!vr"|!dt_stjadtj&dui}3|"rd |3d<|#|#|3d<|jdd}4|4r|4|3d<t d|4D]}5|5dj D]9\}6}7t|7t s|7j#ds)t|7|5d|6<;t d|5jddd|5dd}8|5djdd}9d|8vs|9td}:|jdd};|;rx|;dj D]W\}6}7t|7t s|7j#ds)|7jdd} tj| }7|7|;d|6<Ytdi|;}:|j|}<|jdd}=t|=|jdd}>t|>|jdd}?tj|?|jdd}@|@rt@t~rddh}AtjjDBcgc] }B|BAvsB }C}B@j D];\}6}7|6Cvr|7|3|6<|6dvrtd|6dtjpd|6d=tjdi|3|:|r[r\'zQ' is NOT a valid router_settings parameter. Please move it to 'general_settings'.zKey 'zC' is not a valid argument for Router.__init__(). Ignoring this key.async_only_mode)assistants_configrrouter_general_settingsignore_invalid_deployments)rh)all_guardrailsrr)r~r]rprompts) init_prompts) all_promptsrr^)rrrrgrrrtlitellm.caching.cachingrupdater}rernrrrrr@rrhrrBguardrail_name_config_maplitellm.integrations.dotpromptrrdlitellm.integrations.bitbucketrlitellm.integrations.gitlabrlitellm.types.utilsr r rlitellm.types.routerr r rrrrrrmrrrlradd_litellm_success_callbacklitellm.integrations.prometheusr_mount_metrics_endpointradd_litellm_failure_callbackrrrr|rrr enumeraterCrprrVsetattrrrr  _turn_on_jsonr_key_management_settingsinitialize_secret_managerrr_load_alerting_settingsrQrlitellm_master_key_hashcriticalrupdate_cache_ttlrrrrrrrrTrVrWrXrAload_enterprise_configrrrrrrr1rr2rYrZrr[r\rfr@r*not_premium_userrrrrr%r rrrrrset_default_vertex_configr{get_valid_argsrr_update_redis_cacher_init_policy_enginer]"litellm.proxy.prompts.init_promptsrGrr_init_non_llm_configsget_model_list)Hrrrr~rrblue_color_codereset_color_coderrrrcache_params_in_config cache_type cache_host cache_portcache_passwordrKrrrr r rrcallbackr_store_audit_logsridx team_setting_krr_enable_hc_routing _hc_stalenessr!r"r#rrr&use_pkcecp_urlr*r+r,r-r0rbac_role_permissionsrole_permission router_paramsr4r9rrlitellm_model_namelitellm_model_api_baserBr;rfinetuning_config files_configr>r exclude_argsxavailable_args guardrails_v2rFrGrsH rgrzProxyConfig.load_configv s"__>N_OO (((7#JJ':B? (9G %"::&8$?  #!  (O( .446_ 1 U'>etm_--EFG=#%L%)991A.1Q.$++,BC!-!1!1&'!BJ(../EzR#g-?O1Ol//12a7%/ d%C %/ d%C )-$++(2(2(2&&6=I-78H$-ON(//$.!"-220+,&  -220+,&  -220+,&  -224+,* '3&8&8&:B U%eS1e6F6F}6U0:50AL-B $$,$?}}0,22.//IJZI[\G^L(0E*/%1)9)9 1-9RG55506(--*++WXhWij5506(--*++WXhWij22T,U3(--*++TUeTfg;;O>?DU|G<77?#DD 811>%* 3! )-Z(8.=&*+;&<1A/ 3G/%&../1G,%*O(?#<<YY /h ? $<<YY ( ,x7!"$4#?$8$>$>(C%&%5$L$L$N)O**++LWMeMeLffghxgyz../1G,%* (?#<<YY /h ? $<<YY ( *++LWMeMeLffghxgyz1124G/$)I(?#77>> /h ?$77>>xH I)9(<(<*G,D,D)%).//RSZSnSnRooprBqCD-44}N*K'',ii0A2&FO0?0C0C+U1-.=-@-@0".*)8(;(;E4(H%(..*++STmSnoHI_H``fgxfyzJyKL33-6. )\ /?,? )..*++([\a[bcK'ETM(,G%))+(..*++KL\K]^)..*++ :"2"6"67Le"T  %:M N  ( (:J ( K+//EL 7 7 F$**+KL),7 $**+A<P)--j)=tDJj33MB' 3 %*Z*E*4Z*@'$--d&6%9%9($& "&1"33*/0F*G&*4yy!:GDJJLPVVH-55592(00I&))*=>F+33370(00U!1 4 45H% P  ($)!4E 0 1+..}dCK%8H$ #3"6"67Ld"S ".+:-@P,(*--lDAJ%"1$7G#-=,@,@/- )-85D7%562 '2-DDEUVVV ##$>? !$44-=>O-P*-88:  w}}  ;?M7 8  $@MM< =ZZ d3 *4M, ' T $ '!"2399;CDAq!!S)all=.I5?]./2C UYY|R%@$AIJ%*+;%?&'78;  @ !4 I6H I =A>88:a|>SN(--/ 1&'(M!$OO$A3op)00s"ef   /%$9 $%(,   (V\\-E-E-M  & &-> & ?/3  "JJ|T:M  ,!1! &&''   )-  jjD1G  G W?O P $888G"6(((777v,,.0@@@kPJ 3R )),=&/&Z[^Z__qr~sDsDsFrG%H'"!"#,"VWZV[[mnrsoAnB!C# JWJ8nR ( 8s-AE-AC;GAE-,AE->2AE-2F7AE-+*AE-AC>2A;AE-.AE-7H+AE-# AD.A1AE- AE-2N>AE-0AE1-AE-AEE4AE-AE%CAE-AE-AAE-+AE-.A AE-:AE- C9AE- AE$AE$C.AE-BAE)BA AE-C"AE+C#AE-C>AE-DAAEEAE-EAE-E AE-E+AE-cK|jdd}|rtj||jdd}|rtj||jdd}|rDddlm}|jdi}|jdd}|j||d{|jd d}|rEdd lm } tj| t_ tjj||jd d} | r| D cgc] } td i| c} |_yg|_y7cc} ww) zO Initialize non-LLM configs eg. MCP tools, vector stores, etc. mcp_toolsN agent_list mcp_serversrglobal_mcp_server_managerr mcp_aliasesvector_store_registryVectorStoreRegistryrr^)rrload_tools_from_configrload_agents_from_config9litellm.proxy._experimental.mcp_server.mcp_server_managerrload_servers_from_config+litellm.vector_stores.vector_store_registryrrgrload_vector_stores_from_configrr) rr~mcp_tools_config agent_configmcp_servers_configrrrvector_store_registry_configrworker_registry_configrs rgrez!ProxyConfig._init_non_llm_configss@ "::k48  $ ; ; & &'P Q  **Z6 & &'W X $*JJ/CT$J!!!#C$8#99L M +&?'%T1    sBBBBrc X|jdd}tjd||y|jdd}tjd|tj ||jdd|jdd|jdd|jd d|jd dt |D]^}|d k(r |t jvst|ddd |jd di }|@t jj|`y)z. Initialize alerting settings alertingNz_alerting_callbacks: z=_load_alerting_settings: Calling update_values with alerting=alerting_thresholdr alert_typesalert_to_webhook_url alerting_argsalert_type_config)rrrrrrrslack)logging_integrationinternal_usage_cachercustom_logger_init_args) rr}rer update_valuesrrg)_known_custom_logger_compatible_callbacksr&rr)rr_alerting_callbacks_alerting_value_alert_loggers rgrZz#ProxyConfig._load_alerting_settingsqsN /22:tD""%:;K:L#MN  & +..z4@""KOK\ ]  ''$/334H#N(,,]DA!1!5!56Ld!S*..E.223FM) ( * WF WNNNB,2-1#'+-=-A-A/SW-X1 G*88MMgV! W" rir"c|||tjjk(r tdy|tjjk(r t dy|tj jk(r(ddlm}|jdtjy|tjjk(r tdy|tjjk(rdd lm}|y|tj"jk(rdd lm}|y|tj(jk(rdd lm}|y|tj.jk(rdd lm}|| yt5dy)z_ Initialize the relevant secret manager if `key_management_system` is provided NTr%r$r)AWSSecretsManagerV2)use_aws_secret_managerr!) use_aws_kms)GoogleSecretManager)HashicorpSecretManager)CyberArkSecretManager)load_custom_secret_managerrz&Invalid Key Management System selected)rrrr GOOGLE_KMSrAWS_SECRET_MANAGER-litellm.secret_managers.aws_secret_manager_v2rload_aws_secret_managerrgrXAWS_KMSrGOOGLE_SECRET_MANAGER-litellm.secret_managers.google_secret_managerrHASHICORP_VAULT0litellm.secret_managers.hashicorp_secret_managerrCYBERARK/litellm.secret_managers.cyberark_secret_managerrCUSTOM4litellm.secret_managers.custom_secret_manager_loaderrr)rr"rrrrrrs rgrYz%ProxyConfig.initialize_secret_managers> ! ,$(;(K(K(Q(QQ)dC&*=*H*H*N*NNt4%&99??@$;;+/,3,L,L<'*=*E*E*K*KK.%)<)R)R)X)XX$%&*=*M*M*S*SS'(&*=*F*F*L*LL&'&*=*D*D*J*JJ+ flag for differentiating model stored in db vs. config -> used on UI Return model info w/ id model_idNr{Tdb_model created_at updated_at created_by updated_byF)r{rr^)rr^rBrrrRouterModelInfo)rr9rr  _model_infos rgget_model_info_with_idz"ProxyConfig.get_model_info_with_idsD%UJ= ?%(E  T "+/E  Z ( 4 -4UL$-OE  \ *-4UL$-OE  \ *-4UL$-OE  \ *-4UL$-OE  \ *    'Ju7G7G,N5+++).  &U---%2B2B:2NRW2W/7  ,)=E,<,<=K*U^^hOKri db_modelsc|Kg}tt|dk(ry|D]<}|j|}|j"|j |j> |j t d{}|jdd}|r|D]}|djD]9\} } t| ts| jds)t| |d| <;|jd ijd d} | tj!|d |d } n t| } |j | tj#} d} | D]%} | |vstj%| }|!| dz } '| S7#t$r)}tjdt|Yd}~yd}~wwxYww)a5 (Helper function of add deployment) -> combined to reduce prisma db calls - Create all up list of model id's (db + config) - Compare all up list to router model id's - Remove any that are missing Return: - int - returns number of deleted deployments Nrr9rzlFailed to load config in _delete_deployment: %s. Skipping deployment cleanup to avoid removing valid models.r4rrr^r{r8) model_grouprr{r)rrnrr{rrrRrpr}r|rrrrr@r_generate_model_id get_model_idsdelete_deployment)rrcombined_id_listrtr^r~rr4r9rrrrouter_model_idsdeleted_deployments is_deleteds rg_delete_deploymentzProxyConfig._delete_deployments  Y1!4 7A4414=J}}( '' 6 7  ??/40+1139DAq&:Qd'O*9OA& 9 #1"C?"C$**^_n^op55A5>K    ||#2*'t',  ' 6rirc^Ki}d} tjd{}|j|} t|tr|ng}ttt jdt||j|}t|dkDs|rt jd|tj|t!d|da t jd tn`t jdt||t |t_|j%| d{|j'| ttj+a|j/||j1|tt2 d{|j5|t6| y7#t$r*}t j dt |Yd}~d}~wwxYw7#t$r+}t j(d t |Yd}~d}~wwxYw7w)NziFailed to load config in _update_llm_router: %s. Proceeding with model loading using cached/empty config.zlen new_models: )rrz _model_list: Tr@)r4rCrrDzupdated llm_router: )rz+Error adding/deleting model to llm_router: ) config_datarr])rrr)rrrrpr}r|rrrrrQrernrrgr{rrrrrrfr_add_callbacks_from_db_config#_add_router_settings_from_db_configr]$_add_general_settings_from_db_configr)rrrrrr models_listrs rg_update_llm_routerzProxyConfig._update_llm_routers(    , 7 7 99K22;?L" .8T.J PRK!j&<$**-=c+>N=O+PQ$($C$C*%D% {#a'<(..{m/LM!(#.0E,01&237 "J)..1Ej\/RS$**-=c+>N=O+PQ+ 0F.:J+-- -DDD$${$;  !'668N **;766# -7   11#-/ 2 s:  ( (KA   BE   * *=c!fXF    sH-F<F9F<DG4:G2;G4A H-H+H-9F<< G/G*$H-*G//H-2G44 H(=!H#H-#H((H-rn event_types)successfailureexisting_callbacksc0|tjvr|D]}t||y||vrl|dgk(r tjj |y|dgk(r tjj |ytjj |yy)aQ Helper method to add a single callback to litellm for specified event types. Args: callback: The callback name to add event_types: List of event types (e.g., ["success"], ["failure"], or ["success", "failure"]) existing_callbacks: The existing callback list to check against rrN)rgrr rrQrTr)rrnrr event_types rg4_add_callback_from_db_to_in_memory_litellm_callbacksz@ProxyConfig._add_callback_from_db_to_in_memory_litellm_callbackss wHH H) T =h S T / /yk)00MMhW +00MMhW00EEhO 0rirc |jdixsi}|jdd}|jdd}|jdd}|:t|tr*|D]%}|j|dgtj '|:t|tr*|D]%}|j|dgtj '|=t|tr,|D]&}|j|ddgtj(yyy) z: Adds callbacks from DB config to litellm rrlNrr r)rnrrr)rrrrrgrlrr ) rrrsuccess_callbacksfailure_callbacksr rlrrns rgrz)ProxyConfig._add_callbacks_from_db_configs2'??+=rBHb,001CTJ,001CTJ$((d;  (Z8I4-P$5  II-!* '.'?'?J   (Z8I4-P$5  II-!* '.'?'?J   Z 4%@% II%!*I 6'.'8'8J &A rirnew_encryption_keyc^i}|jD]\}}t||}|||<|S)zR Encrypts a dictionary of environment variables and returns them. )rr)rr)rrrencrypted_env_varsrrencrypted_values rgrz"ProxyConfig._encrypt_env_variablessK  )//1 4DAq2,>O%4 q !  4 "!rirc i}|jD].\}} t|||}||tj|<|||<0|S#t$r*}t j d|t|Yd}~ad}~wwxYw)a! Decrypts a dictionary of environment variables and then sets them in the environment Args: environment_variables: dict - dictionary of environment variables to decrypt and set eg. `{"LANGFUSE_PUBLIC_KEY": "kFiKa1VZukMmD8RB6WXB9F......."}` rNz#Error setting env variable: %s - %s)rrrrrpr}rr)rrrrrrrrs rgrz-ProxyConfig._decrypt_and_set_db_env_variabless )//1 DAq "6:O##.$3BJJqM,;&q) "!  $**91c!f s(A A9 A44A9variables_dictc`i}|jD]\}}t||d}|||<|S)zF Decrypts a dictionary of variables and returns them. Tr)rr)rr decrypted_variablesrrrs rg_decrypt_db_variablesz!ProxyConfig._decrypt_db_variables5sL!"((* 5DAq2QdO&5  "  5 #"rircL|yd}t|tr|}n*t|trddl} t j |}t|tr|r|Sy#tj |jf$r, |j|}n#|j$rYnwxYwY_wxYw)z Parse a router_settings value that may be a dict or a JSON/YAML string. Returns a non-empty dict if valid, otherwise None. Nr) rrrrrr YAMLErrorJSONDecodeErrorr)rparsedrs rg_parse_router_settings_valuez(ProxyConfig._parse_router_settings_valueAs =!% eT "F s #  . fd #MNND$8$89 'TZZ.F++ s5AB#8B  B# BB#BB#"B#user_api_key_dictrFrcK| |jt|dd}||S|T|jH t|j|t|d{}|jt|dd}||S yy7'#t $rYywxYww)u Get router_settings in priority order: Key > Team Uses the already-cached key object and the cached team lookup (get_team_object) to avoid direct DB queries on the hot path. Global router_settings are NOT looked up here — they are already applied to the Router object at config-load / DB-sync time. Returns: dict: router_settings, or None if no settings found Nr)rr]rr)rrrrrrp)rrr]r key_settingsteam_obj team_settingss rg!_get_hierarchical_router_settingsz-ProxyConfig._get_hierarchical_router_settings]s*  (<<)+-F-F-R !0-55"/'9&7 " !% A AH&7>! !,((-    s@1B  A>A<#A>8B :B <A>> B B  B  B cK|||jjjddid{}|jdi}i}|Ft |t r6|4t |j t rddlm}|||j }n=|t |t r|}n(|&t |j t r |j }|r|jdi|yyyy7w)z Adds router settings from DB config to litellm proxy 1. Get router settings from DB 2. Get router settings from config 3. Combine both 4. Update router settings N param_namerwherer)_update_dictionaryr^) rlitellm_config find_firstrrr param_valuermrupdate_settings)rrrr]db_router_settingsconfig_router_settingscombined_router_settingsrs rgrz/ProxyConfig._add_router_settings_from_db_configs   !m&?'4'7'7'F'F'Q'Q#%67(R(" &1__5F%K "') $&25t<&21==tD<+=*,>,J,J,((3 &9,B(#/J"..5,>+I+I('* **F-EF(9'@ !"s0C*C(B6C*c|jdi}|>d|vr9||jddt|dtr|jddt|dtrt|d}t|d}t|j |}t|d|dDcgc] }||dvr| c}z}t j d|dd|dd|||d<|j|dnO| i}|d|d<|j|dn-t|tr|d|d<|j|d|:d|vr6|d|d<|d|_ |jj|dt |2d |vr-|d |d <|jj|d t yyycc}w) z Adds general settings from DB config to litellm proxy Args: config_data: dict general_settings: dict - global general_settings currently in use proxy_logging_obj: ProxyLogging rNrzMerging alerting values: YAML=z, DB=z , Merged=)rr)rrr)rr) rrrsetunionr}rerrrslack_alerting_instancer) rrrr_general_settings_yaml_alerting _db_alerting_merged_alertingrs rgrz0ProxyConfig._add_general_settings_from_db_configsw(OO,>C  (Z;L-L ,$((T:F/ ;TB%))*d;G0O-O.? .N ] +,<],K  )  5 5 C C,];  D   )&*;;7H&8 3 4  5 5 C C%56L%M% D  < *GIs1Gc dKty tjdtjdt j d}|ddlm}|}t j d}|r_ddl m } |j|}tj|j|tgdd t tjd |ydd lm}t j dd} ||} tj|jd| t)j*ddztgdd ttjd|yy#t$rY,wxYw#t $rtj"d |YywxYw#t $rtj"dYywxYww)z Reschedule the spend log cleanup job based on current general_settings. This is called when maximum_spend_logs_retention_period is updated dynamically. If the retention period is None, the job will be removed. Nspend_log_cleanup_jobz&Removed existing spend log cleanup job#maximum_spend_logs_retention_periodrrmaximum_spend_logs_cleanup_cron CronTriggerTrxr{replace_existingmisfire_grace_timez)Spend log cleanup rescheduled with cron: /Invalid maximum_spend_logs_cleanup_cron value: )r%maximum_spend_logs_retention_interval1dintervalrnsecondsrxr{r4r5z-Spend log cleanup rescheduled with interval: 3Invalid maximum_spend_logs_retention_interval value) scheduler remove_jobr}rdrprr7litellm.proxy.db.db_transaction_queue.spend_log_cleanuprapscheduler.triggers.cronr2 from_crontabadd_jobcleanup_old_spend_logsr]rrr*litellm.litellm_core_utils.duration_parserrrqrandint) rretention_periodrspend_log_cleanup cleanup_cronr2 cron_triggerrretention_intervalinterval_secondss rg!_reschedule_spend_log_cleanup_jobz-ProxyConfig._reschedule_spend_log_cleanup_jobs      !8 9 % %&N O ,//0UV  ' !0 1 +//0QRLA#.#;#;L#IL%%)@@$+_2)-+I &)--CL>R&6%9%9;T&"':;M'N$%%)@@" 06>>!R3H H+_2)-+I&)--GHZG[\a (    :"(..I,X6"(..MsrF0*EAF08AE%F0-A&F F0 E"F0!E""F0%!F F0F  F0 F-*F0,F--F0db_general_settingscpK|yt|}d|vr |dtd<d|vr |dtd<d|vr3|dtd<tjj tdd|vr'|dtd<t tdd{d|vr |dtd<d |vrh|d }| dtd <nWt |tr |td <n=t |tr|jd k(td <nt|td <d |vrW|d }|nBt |tr|a n/t |tr|jd k(a n t|a ttd <d |vrBtjd }|d }|td <||k7r|jd{yyy7#7 w) z; Pull from DB, read general settings value Nmax_parallel_requestsglobal_max_parallel_requestsr)rr.r/rstore_prompts_in_spend_logsrRrr/) rrrr(rrrboolrrrrrL)rrMr)r old_value new_values rg_update_general_settingsz$ProxyConfig._update_general_settingsOs*  &  !45 "&7 78I'9 4 5 *-> >?P.@ ; < / /0A/0R _ -  5 5 C C.? D  $'8 89J(: 5 64'78P'Q   0 01BCS1T - . ),= =%&CDE}BF !>?E4(BG !>?E3'KKMV+!!>? CGu+ !>? "3 3%&9:E}E4($)!E3'$)KKMV$;!$(K!4E 0 1 14E E(,,-RSI)*OPIFO B CI%<<>>>& FO Z?s%B F6F1DF6)F4*F64F6current_configrrrrrdb_param_valuec>dtdtddfd}|dk(r}|j|d}i}|jD]2\}}|||<|j} ||| <|tj | <4|j dij||S|d k(rBt|tr2|jD]\}}|tvstt||!||vr|||<|St||trt|tr|||||S|||<|S) am Updates the config fields with the new values from the DB Args: current_config (dict): Current configuration dictionary to update param_name (Literal): Name of the parameter to update db_param_value (Any): New value from the database Returns: dict: Updated configuration dictionary dstsrcr.NcJ||fg}|r|j\}}|jD]r\}}| t|trt |dk(r(t|t r6t|j |t r|j|||fn|||<t|ryy)z Deep-merge src into dst, skipping None values and empty lists from src. On conflicts, src (DB) wins, but empty lists are treated as "no value" and don't overwrite. Nr)rrrrrnrrr)rZr[stackdrNrrs rg_deep_merge_dictsz._deep_merge_dictss 3ZLEyy{1GGI !DAqy !!T*s1v{ !!T*z!%%(D/I adAY/ ! !rirT)rr) rrrupperrr setdefaultrJrr#rVrg) rrVrrXr_rmerged_env_varsrr upper_keys rg_update_config_fieldsz!ProxyConfig._update_config_fieldssa. !4 !d !t !( 0 0!%!G!Gd"H"  %'O0668 . U',$IIK -2 *(- 9%  .  % %&=r B I I " ! - -*^T2R,224 1 U==GS%0  1 ^ +)7N: &! ! nZ0$ 7J D=  nZ8. I *8N: &rircK|durtjd|Sg}gd}|D]'}|jd|d}|j|)t j |d{}|D]T}|t |dd} t |dd} tjd| d | | >| A|j|| | }V|S7_w) NTz4'store_model_in_db' is not True, skipping db updatesrWrr~)rrrr z param_name=z, param_value=)rVrrX) r}rdget_generic_datarrgatherrrerd) rr]r~r_tasksrrresponserrr s rgrz"ProxyConfig._update_config_from_dbs  D ( % %F M   $A$55 h6H MM( #  $ "..&11 ! H <>J!(M4@K & &j\ }E %+*A33#))#.4 " %2sA#C%C&AC*C-C object_typectjdd}|yt|ts#t j dt |dyt|tfd|DS)aD Check if an object type should be loaded from the database based on general_settings.supported_db_objects. Args: object_type: Type of object to check (e.g., SupportedDBObjectType.MODELS, "models", etc.) Returns: True if the object should be loaded, False otherwise supported_db_objectsNTz(supported_db_objects is not a list, got z. Loading all objects.c3:K|]}t|k(ywrP)r)rMobjobject_type_strs rgrOz5ProxyConfig._should_load_db_object..>sO33s8.Os) rrrrr}r|rVrr1)rrjrlros @rg_should_load_db_objectz"ProxyConfig._should_load_db_objectsx 0334JDQ '.5 ( (:4@T;U:VVlm k*O:NOOOricK |jjjd{}|S7#t$r:}t j dj t|g}Yd}~|Sd}~wwxYww)NzQlitellm.proxy_server.py::add_deployment() - Error getting new models from DB - {})rlitellm_proxymodeltable find_manyrpr}rrur)rr]rrs rg_get_models_from_dbzProxyConfig._get_models_from_db@su ,//GGQQSSJT  * *cjjF  J s6A:'424A:4 A7/A2,A:2A77A:cK |jdr5|j|d{}|j||d{|jjj ddid{}|$|j |j d{|j|d{y77w7I7%7 #t$r7}tjd jt|Yd}~yd}~wwxYww) z{ - Check db for new models - Check if model id's in router already - If not, add to router modelsrjr]N)rrrrr)rMz>litellm.proxy.proxy_server.py::ProxyConfig:add_deployment - {})rprtrrrrrUr _init_non_llm_objects_in_dbrpr}rrur)rr]rrrMrs rgadd_deploymentzProxyConfig.add_deploymentMs' **x*@#'#;#;-#;#XX --)=N.)6(8(8(G(G(R(R#%78)S)#  #.33(;(G(G4 222O O O%Y#  P  * *PWWF   sD 'CB<CB>/C6C7%CCC6C7C;D <C>CCCC D-D<D DD cK|jdr|j|d{|jdr|j|d{|jdr|j|d{|jdr|j |d{|jdr|j d{|jd r|j |d{|jd r|jd{|jd r|j|d{|jd r|j|d{|jd r|j|d{|jdr|j|d{|jdr|j|d{|jdr|j|d{|jdr!ddlm}|j!||d{|jdr|j#|d{|jdr|j%|d{yy77u7L7#77777Z7177777Y7/w)z Use this to read non-llm objects from the db and initialize them ex. Vector Stores, Guardrails, MCP tools, etc. rrwrxNr vector_storesvector_store_indexesmcpagentsr.rFrtoolsmodel_cost_mapanthropic_beta_headers sso_settingscache_settingsr)CacheSettingsManager)r]rsemantic_filter_settingsconfig_overrides)rp_init_guardrails_in_db_init_policies_in_db_init_vector_stores_in_db _init_vector_store_indexes_in_db_init_mcp_servers_in_db_init_agents_in_db"_init_pass_through_endpoints_in_db_init_prompts_in_db_init_search_tools_in_db_init_tool_policy_in_db _check_and_reload_model_cost_map(_check_and_reload_anthropic_beta_headers_init_sso_settings_in_db;litellm.proxy.management_endpoints.cache_settings_endpointsrinit_cache_settings_in_db$_init_semantic_filter_settings_in_db%_init_hashicorp_vault_config_override)rr]rs rgryz'ProxyConfig._init_non_llm_objects_in_dbws   & &< & @--M-J J J  & &: & >++-+H H H  & &? & C00}0M M M  & &3I & J77m7T T T  & &5 & 9..0 0 0  & &8 & <)) )F F F  & &3K & L99; ; ;  & &9 & =***G G G  & &> & B//m/L L L  & &7 & ;..].K K K  & &3C & D77m7T T T  & &3K & L??+@    & &> & B//m/L L L  & &3C & D '@@+$A    & &3M & N;;-;X X X  & &3E & F<<+=   Gg K I N U 1 G < H M L U  M   Y s(K3K+K3K +K3K+K3.K/)K3K+K3K)K3.K/+K3K+K3K +K32K#3+K3K&+K3 K) +K36K+72K3)K-*+K3K/+K3K1K3 K3K3K3K3K3K3K3 K3#K3&K3)K3+K3-K3/K31K3c2Kddl}ddl}ddlm} |jj j ddid{}| |jy|j}t|tr|j|}|jdd}|yt|dr|j|k(r|jj|}|rN|D]I} t| |s| j | j j"4t%j&d yt%j(d |jj+|j,||jd d rR|j/|t0 d{} | rF|jj3| t%j(dnt%j(d|j5|_ y77e#t6$r"} t%j8d| Yd} ~ yd} ~ wwxYww)z Initialize MCP semantic filter settings from database. Called periodically (approximately every 10 seconds) by background task to hot-reload settings across all pods. rNSemanticToolFilterHookrrrmcp_semantic_tool_filterrz=Semantic filter settings unchanged, skipping reinitializationzYSemantic filter settings unchanged, but hook is missing or uninitialized. Reinitializing.raFr~rz)MCP Semantic Filter reinitialized from DBzMCP Semantic Filter disabledz5Error initializing semantic filter settings from DB: )rrg'litellm.proxy.hooks.mcp_semantic_filterrrr find_uniquer rrrrrrrget_custom_loggers_for_typefilter tool_routerr}rerdremove_callbacks_by_typer initialize_from_configrrrlrpr) rr]rrgr config_recordrmcp_semantic_filter_config active_hooks active_hookhookrs rgrz0ProxyConfig._init_semantic_filter_settings_in_dbs& RF "/"2"2"A"A"M"M#%78#N#M$ (A(A(I,88 *C0#-4::.>#? )9)=)=*D* &*1t;<448RR 88TT2! $+7 +K)+7MN$/$6$6$B(3(:(:(F(F(R$8$>$>(g%&%+ +)--s  , , E E!!#9  *--i?3JJ5)K44II$O(--C%))*HI1K0O0O0QD -d  * *GsK   sH+G)G$G)HAG)HA G) G)+G)G)HA(G)G'A!G)#H$G)'G)) H2H HHHcLK |jjjddid{}||jj dd|jj dd|jj dd|jj Dcic]\}}|j |}}}|j|yy7cc}}w#t$r7}tjd jt|Yd}~yd}~wwxYww) zS Initialize SSO settings from database into the router on startup. r{ sso_configrN role_mappings team_mappingsrrzHlitellm.proxy.proxy_server.py::ProxyConfig:_init_sso_settings_in_db - {}) rlitellm_ssoconfigrrrrr`rrpr}rrur)rr]rrruppercase_sso_settingsrs rgrz$ProxyConfig._init_sso_settings_in_dbs#  !.!1!1!C!C!O!O\*"P"L'))--otD))--otD))--.>E'3&?&?&E&E&G*"UIIK&*&*66*@7(*  * *ZaaF   sQD$+C!CA7C!'CC!D$C!C!! D!*-DD$D!!D$cjKddlm}m}m}m}m} |j jjddid{}| |j|j||y||j}|j|k(ry|j|} ||} ||  |jd|j|_ tj dy7#t$r || wxYw#t$r)} tj"d t%| Yd} ~ yd} ~ wwxYww) z Load Hashicorp Vault config override from DB. Decrypts sensitive fields, sets HCP_VAULT_* env vars, and reinitializes the secret manager. Called periodically via _init_non_llm_objects_in_db to sync config across pods. r)HASHICORP_ENV_VAR_MAPPING_clear_hashicorp_vault_state_get_current_env_values_parse_config_value _set_env_vars config_typehashicorp_vaultrN)r"z.Hashicorp Vault config override loaded from DBz9Error loading Hashicorp Vault config override from DB: %s)C&&C>$D3%!C>D3!C>)C(;*C>%D3&C>(C;;C>> D0D+&D3+D00D3c K |jjjddid{}| |jy|j}|j d}|j dd}||duryt j }d}|rd}tjd nu|stV t jt}||z } | jd z } | |k\rd}tjd | d d |nd}tjd|rddlm} t j"} | | }|t _t't!j(||j+a |jjj-ddidt/|ddddt/|ddidd{tjd|r t1|ndyy7#t$r$} tjd| d}Yd} ~ d} ~ wwxYw7^#t$r+} tj2dt5| Yd} ~ yd} ~ wwxYww)z Check if model cost map needs to be reloaded based on database configuration. This function runs every 10 seconds as part of _init_non_llm_objects_in_db. rmodel_cost_map_reload_configrNinterval_hours force_reloadFTz4Model cost map reload triggered by force reload flagrzFModel cost map reload triggered by interval. Hours since last reload: r1 , Interval: Error parsing last reload time: zBModel cost map reload triggered - no previous reload time recordedrget_model_cost_maprrrrrr r createrJrrz4Model cost map reloaded successfully. Models count: z+Error in _check_and_reload_model_cost_map: )rrrr rrutcnowr}rdlast_model_cost_map_reload fromisoformat total_secondsrpr|-litellm.litellm_core_utils.get_model_cost_maprrgmodel_cost_map_url model_costrOadd_known_models isoformatupsertr'rnrr)rr]rr~rr current_time should_reloadlast_reload_timetime_since_last_reloadhours_since_last_reloadrrrnew_model_cost_maps rgrz,ProxyConfig._check_and_reload_model_cost_mapXs j "/"2"2"A"A"M"M#%CD#N#M$ (A(A(I"..F#ZZ(89N!::ne@P1P.2@@BTI03nD,0M055"hjABEiFFRSaRb!c%)M(--\&-%?%?"%7qcB)- -@6  * *=c!fXF   sI+H'G2H'I5H'8I98H'2AG5CH'H%)H'0I2H'5 H">HH'H""H'' I0!IIIIc lK |jjjddid{}| |jy|j}|j d}|j dd}||duryt j }d}|rd}tjd nu|stV t jt}||z } | jd z } | |k\rd}tjd | d d |nd}tjd|rddlm} | } |j!a |jjj#ddidt%|ddddt%|ddidd{t'd| j)D}tjd|yy7#t$r$} tjd| d}Yd} ~ d} ~ wwxYw7q#t$r+} tj*dt-| Yd} ~ yd} ~ wwxYww)z Check if anthropic beta headers config needs to be reloaded based on database configuration. This function runs every 10 seconds as part of _init_non_llm_objects_in_db. r$anthropic_beta_headers_reload_configrNrrFTz.-s(%..1 3E%sz@Anthropic beta headers config reloaded successfully. Providers: z3Error in _check_and_reload_anthropic_beta_headers: )rrrr rrrr}rd"last_anthropic_beta_headers_reloadrrrpr|&litellm.anthropic_beta_headers_managerrrrr'rVrrr)rr]rr~rrrrrrrrrrprovider_counts rgrz4ProxyConfig._check_and_reload_anthropic_beta_headerssi "/"2"2"A"A"M"M#%KL#N#M$ (A(A(I"..F#ZZ(89N!::ne,(2>@P1P.2@@BTI03nD,0M055"prIJMqNNZ[iZj!k%)M(--d89 6B5K5K5M2$&&55<<')OP+Q+56D49!",#*:6D49!",#=0"%%'__.%" %))VWeVfgSoT%-,44>qcB)- -2B  * *Ec!fXN   sH4+G=GG=H45G=8H498G=2AG BG= G; B-'A*A(/A*'B-(A** B*3-B% B-%B**B-cdKddlm}m}m} |j |d{}t j dt||D]}|jt|| y7G#t$r7}t jdjt|Yd}~yd}~wwxYww)Nr)IN_MEMORY_GUARDRAIL_HANDLER GuardrailGuardrailRegistryrxzguardrails from the DB %s) guardrailzFlitellm.proxy.proxy_server.py::ProxyConfig:_init_guardrails_in_db - {}) +litellm.proxy.guardrails.guardrail_registryrrrget_all_guardrails_from_dbr}rersync_guardrail_from_dbrrprru)rr]rrrguardrails_in_dbrrs rgrz"ProxyConfig._init_guardrails_in_db\s   (BB+C  ! & &+S1A-B .  +BB"9i8C    * *X__F   s? B0A-A+AA-*B0+A-- B-6-B(#B0(B--B0cbKddlm}ddlm} |}|}|j |d{|j |d{t jdy747#t$r7}t jdjt|Yd}~yd}~wwxYww)zi Initialize policies and policy attachments from database into the in-memory registries. r)get_attachment_registry)get_policy_registryrxNz4Successfully synced policies and attachments from DBzDlitellm.proxy.proxy_server.py::ProxyConfig:_init_policies_in_db - {}) /litellm.proxy.policy_engine.attachment_registryr+litellm.proxy.policy_engine.policy_registryrsync_policies_from_dbsync_attachments_from_dbr}rerprrur)rr]rrpolicy_registryattachment_registryrs rgrz ProxyConfig._init_policies_in_dbws  T 13O"9"; "77m7T T T&>>+?   ! & &F  U   * *V]]F   sP B/#A,A(A, A*A,'B/(A,*A,, B,5-B'"B/'B,,B/cKddlm} |}|j|d{tjdy7#t $r7}tj djt|Yd}~yd}~wwxYww)z Initialize tool policy from database into the in-memory registry. Synced periodically by add_deployment -> _init_non_llm_objects_in_db. r)get_tool_policy_registryrxNz'Successfully synced tool policy from DBzGlitellm.proxy.proxy_server.py::ProxyConfig:_init_tool_policy_in_db - {}) %litellm.proxy.db.tool_registry_writerr sync_tool_policy_from_dbr}rerprrur)rr]r registryrs rgrz#ProxyConfig._init_tool_policy_in_dbs| S /1H33-3P P P & &'P Q Q  * *Y``F   s>BAAABA B -A>9B>BBcKddlm} |j|d{}t|dkrytj ||t_y|D]"}tj j |$y7^#t$r7}tjdjt|Yd}~yd}~wwxYww)Nrrrx)r|) vector_storeIlitellm.proxy.proxy_server.py::ProxyConfig:_init_vector_stores_in_db - {}) rr_get_vector_stores_from_dbrnrgradd_vector_store_to_registryrpr}rrur)rr]rr|rrs rgrz%ProxyConfig._init_vector_stores_in_dbsS "5"P"P+#Q#M=!Q&,,40C"/1-%2L11NN%1O  * *[bbF   TCBA>BC"BC'B=C>B C -B;6C;CCcKddlm} |j|d{}t|dkrytj ||t_y|D]"}tj j |$y7^#t$r7}tjdjt|Yd}~yd}~wwxYww)Nr)VectorStoreIndexRegistryrx)r})vector_store_indexr) rr!_get_vector_store_indexes_from_dbrnrgvector_store_index_registryupsert_vector_store_indexrpr}rrur)rr]rr}rrs rgrz,ProxyConfig._init_vector_store_indexes_in_dbsX /PP"/Q ! '(A-22:6N)=73+?&77QQ+=R"  * *[bbF   rcKddlm}|stjdyddlm} |j d{y7#t$r7}tjdjt|Yd}~yd}~wwxYww)Nr)is_mcp_availablezB BB cKddlm} |j|d{}|j|ty7#t $r7}t jdjt|Yd}~yd}~wwxYww)Nrrrx) db_agentsrzBlitellm.proxy.proxy_server.py::ProxyConfig:_init_agents_in_db - {}) ,litellm.proxy.agent_endpoints.agent_registryrget_all_agents_from_dbload_agents_from_db_and_configrrpr}rrur)rr]AGENT_REGISTRYrrs rgrzProxyConfig._init_agents_in_dbs  ,CC+DI  9 9#- :    * *T[[F   s7B><>B> A>-A94B9A>>Bc.Kddlm}ddlm} |j |d{}t j dt|dt|dkDr^tB|jt|d{t j d t|d yt jd yt jd y77U#t$r7}t jd jt|Yd}~yd}~wwxYww)z Initialize search tools from database into the router on startup. Only updates router if there are tools in the database, otherwise preserves config-loaded tools. r)SearchToolRegistry)SearchAPIRouterrxNzLoading z) search tool(s) from database into router)router_instancerzSuccessfully loaded z search tool(s) into routerzMRouter not initialized yet, search tools will be added when router is createdzNNo search tools found in database, keeping config-loaded search tools (if any)zHlitellm.proxy.proxy_server.py::ProxyConfig:_init_search_tools_in_db - {})3litellm.proxy.search_endpoints.search_tool_registryr$&litellm.router_utils.search_api_routerr%get_all_search_tools_from_dbr}rdrnrupdate_router_search_toolsrerprrur)rr]r$r%rrs rgrz$ProxyConfig._init_search_tools_in_db s  K" !3!P!P+"Q"L ! % %3|,--VW  < 1$))DD(2E)--.s2)-$$2$$S)  * -6 -C  # # ' ' , -O2s)C CA C (C;C AC C cK |jjjd{}|Dcgc]}|j|}}|j |d{t j |y7Qcc}w7!#t$r9}tjdjt|gcYd}~Sd}~wwxYww)NzSlitellm.proxy_server.py::get_credentials() - Error getting credentials from DB - {}) rlitellm_credentialstablersr3r;rupsert_credentialsrpr}rrur)rr] credentialsrrs rgget_credentialszProxyConfig.get_credentialsfs  - 0 0 I I S S UUKFQRd433D9RKR))    1 1  VR    * *ellF  I  saC 'BA=BA? B"B#B<C =B?B C.C=C>C CC )r.NrPF)]r^rr__doc__rrrRrrrr rrrrr(rr r rrrrrrrrrrrrr __annotations__rrgr{rrercrZrYrrrrrrrrrr rrrrr  staticmethodrrrrrrrLrUrdrrrBrprtrzryrrrrrrrrrrrrrrrrrr3r;r@r^rirgr|r| s? UUUNNN15& ( & &P%%%%N(MD(MV*+=V###&#7:# #J  tDz d &@@0&9#9$9v$ 4D4H7D7D(4@S;T2U7Dv#s3x"&$&P] Aw~~.] ABE] A~0&$0&d( (  /( X& ( T* * ^+/9K'}9K#3-9Kv@@#$@#3@#D//#/bTd>H TNH (H TPP'"678P! P  P2$DPT "%) "?G} "  "JO"%)"BF" "2 #D #T #CHTN>7; 0#$450 -0$N3 0 $ 0d+G+GV$+G - +G  +GZFF37FLXF FPIVL?(4.L?\RR % RR Rh+#++$D> +Z P &;!;< P  PD |  (#(((T=|=~P PdL633joLobp)pdA |",6 @<$\6L<,l&.L.`8 !eD)O.D ! !-tN7K-.<rir|c Rddl}|j|tjd<y)Nrr)rdumpsrr)rrs rgsave_worker_configrG|s",$**T"2BJJric:Kddlm}|tjddj dk7r t |a|a|durdddl}ddl m }m }m }|j|j|j|j|j|j|dureddl}ddl m }m }m }|j|j|j|j|j|jn|d ur|d urtj j#d d}||j%d k(rGddl}dd l m }m }|j|j|j|jnw|j%d k(rdddl}ddl m }m }m }|j|j|j|j|j|jdit ii}|r't&j)t*|d{\aaa| r| a| |t d<|r|a||t d<|r|tj d<|r ||t d<|r|a||t d<|r|a||t d<|r ||t d<| durdt8_d|dd<| durdt8_d|dd<| r*t?| t8_ t8j@|dd<tBr | a"y7ݭw)Nr) show_bannerLITELLM_DONT_SHOW_FEEDBACK_BOXrrRT)verbose_loggerr}r~)levelF LITELLM_LOGINFOr|DEBUGgeneralrr%r:AZURE_API_VERSION max_tokens temperaturerequest_timeoutalias drop_paramsadd_function_to_promptr)#!litellm.proxy.common_utils.bannerrIrrrrw user_model user_debuglogginglitellm._loggingrKr}r~setLevelrNrOrrr`rrrrr user_headers user_api_baseuser_temperatureuser_request_timeoutrgrVrWrr experimentaluser_telemetry)r9rUr: api_versionredetailed_debugrSrRrTr telemetryrVrWr%save use_queuer~rIr[rKr}r~litellm_log_settingdynamic_configs rgrrs(>M yy126<<>&HJJ }   gll3&&&W\\:%%%GLL9  gmm4&&&W]];%%%GMM: %Ne3 jjnn]B?  *"((*f4X/%..!,,.$--!,,%**,7 ''gmm'<.%..!--.$--!-- Z4N ***v*V V     07z"9- 19z":.   3=z"<0&4?z"=1.8Gz"#45 .3z"7+d"37y!-0%)-&>By!":;":.292D2Dy!,/ NA WsH9L;L?FBFrirnrequested_model_from_clientmodel_mismatch_loggedc @|rt|ttfs||fSt|r||fS|j ddr||fSt|tr|j dn t |dd}|s.||k7r)t jd|j d||d}t|tr ||d<||fS t|d|||fS#t$rI}t jd|j d|t|t|dYd}~||fSd}~wwxYw) Nfastest_responseFr9zolitellm_call_id=%s: streaming chunk model mismatch - requested=%r downstream=%r. Overriding model to requested.litellm_call_idTzPlitellm_call_id=%s: failed to override chunk.model=%r on chunk_type=%s. error=%sr) rrrrrrr}rerVrprrVr)rnrrprdownstream_modelrs rg_restamp_streaming_chunk_modelrSsS 'jD@Q.R+++&&AB+++*E2+++)5 '75'SW;X !%59T%T"" }   . / '   !%%4g+++  w ;< ' ''  "" ^   . / ' K F    ' '' s: C D$r!} tjd| Yd} ~ d} ~ wwxYw#1swYyxYw#t7j8d 5t;|drG |j=d{7n,#t>$r } tjd| Yd} ~ nd} ~ wwxYwdddw#1swYwxYwxYww)Nrl)rpFrrrirp)rrir str_so_far) response_obj)rnrrprT)r exclude_unsetrmrNrrzIlitellm.proxy.proxy_server.async_data_generator(): Exception occured - {}rsrurvr`rVrwrr#rxryr)shieldaclosez7async_data_generator: error closing response stream: %s) r}rerr'async_post_call_streaming_iterator_hookrzrrKrLrgget_response_stringrrr{rr@rprrur|r'rzr?rrrFr&anyio CancelScoperr BaseException)rirrp error_messagerr _str_so_farrn response_strrr~rrrs rgasync_data_generatorrs12`'+ &O%' #!& ,TT/%U ! ,! ,% ,JJ"3!*5;4 KE%-1D!EF&::N |+ +I,G)&; , (E(%+--4t-TE3'E,<, E$FM/0 K >J3J0J3 M/F "F$ F -F?FF  F J-AJ(!G$"CJ("K,(J--K,0J33 K<KK KK  K)%M/,M, M L+$L' %L+*M + M 4M M M M  M, M) %M,,M/ct|||S)Nrirrp)rrs rgselect_data_generatorrs +! rir9c |jdi}|jdijdd} d|vs|jdr|jdd}tj|}|S#t$ricYSwxYw)Nr^rr9azure base_model)rrgget_model_inforp)r9r^model_to_lookuplitellm_model_infos rgget_litellm_model_infors<,Jii 0"599'4HO o % )E(nn\4@O$33OD!!  s=A44 BBc6tjd|dy)Nz$Backing off... this was attempt # %stries)r}re)r5s rg on_backoffr sEwwGWXric>t|txr;t|ddduxr*t|jtxrd|jv }t j ddury|r4tjtjdt |d|S)Nr`z"Max parallel request limit reached1disable_retry_on_max_parallel_request_limit_errorTgiveup)eventr) rr?rr`rrrr}rerrF)rrs rgrrs1n% > Ay$ 't 3 > qyy# & > 1AII = F PQ   ""4::sSTv.V#WX Mric eZdZedeededeefdZe de deefdZ edeede e effdZede d eed efd Zed e fd Zede d ed eddfdZedZedZede d ededededef dZedefdZedede ded efdZedee ded edeefdZedZedZy)rrrrc>t|j||y)z*Initialize logging and alerting on startup)rrN)r startup_event)clsrrrs rgrz-ProxyStartupEvent._initialize_startup_logging#s" ''!5F ( rircddlm}|jdd}t|tr||}|r| t dyy)z Validates that when use_redis_transaction_buffer is enabled, a Redis cache is properly configured in litellm_settings. r)ruse_redis_transaction_bufferFNa`use_redis_transaction_buffer` is enabled in general_settings but no Redis cache is configured. This will cause spend updates to not be tracked. Add a Redis cache in litellm_settings: litellm_settings: cache: true cache_params: type: redis url: os.environ/REDIS_URL )litellm.secret_managers.mainrrrrr)rrr_use_redis_transaction_buffers rgrz;ProxyStartupEvent._validate_redis_transaction_buffer_config2s\ =  !? G & 3S 9,78U,V ) (->-F2  .G (rircKddlm}|jdd}|r|jddstjdytjd|dud ||j || d{}|r5tjd t jj|ytjd y7Qw) z1Initialize MCP semantic tool filter if configuredrrrNraFzKSemantic tool filter not configured or not enabled, skipping initializationz.Initializing semantic tool filter: llm_router=z , config=rz$Semantic tool filter hook registeredzASemantic tool filter hook was configured but failed to initialize) rrrr}rerrgrrr|)rrrrrrs rgrz2ProxyStartupEvent._initialize_semantic_tool_filterOs S%5%9%9 && " *1K1O1O u2  ! & &*  ""'>#  rirr.NcK dj|}t|||d{y7#t$r }tjd|Yd}~yd}~wwxYww)zSWarm global spend cache once at startup to reduce impact of first wave of requests.r)rrr]Nz;Global spend cache warm-up at startup skipped or failed: %s)rurrpr}re)rrrr]rrs rgrz*ProxyStartupEvent._warm_global_spend_caches` "))*BCI=##5+     & &Mq   s6A".,.A. AA AAAc<Ktjytjjdxsg}|r[td|DrHddlm}|Dcgc] }t di|}}||tttd{yyycc}w7 w) z%Update the default team member budgetNteamsc3<K|]}t|tywrP)rr)rMrs rgrOzGProxyStartupEvent._update_default_team_member_budget..sDT*T40Dsr)!update_default_team_member_budgetr)rrr^) rgdefault_internal_user_paramsrr7litellm.proxy.ui_crud_endpoints.proxy_setting_endpointsrNewUserRequestTeamrFrrQ)r_teamsrrteams_pydantic_objs rgrz4ProxyStartupEvent._update_default_team_member_budgets  / / 7 5599'BHb cDVDD JP!P"4"% G-"3+      --'%!%#A     *!13DE!!=      4e < E E   )"/%5&7  /1B C XGX  D (]-F &3&6&6&E&E&P&P');<'Q'! !,!--t2,77;;>!R3H H+_2)-+I&  !&A (6&7"/)($ !!(998nnQ+,.%)'E" %))*WX  !&A ,>&7"/),( !!,AA8nnQ+,2%)'E" %))E$ u%!!GGeFf h c! $**BCF (  L  R."(..I,X("(..M8 $**-RSTRU+VW$**i  B $**?sC%**m s'E X+S,:S);A9S,4AXT"A XT% X4T(5XT+8X AT.X3A U X A.U;; XA.V?63X)S,, T5TXTX%X(X+X.!UXUXU84X7U88X; V<-V71X7V<<X? X-W;5X;XXr=cKddlm}ddlm}ddlm}|d{r|j |d{|j|d{ddlm }ddl m }m }m }m} |d{r|sx| d{ri |d{} | rW|| jd | jd | jd  } t j"j%| |j-|d{t j.durddlm} | j5|ddlm}m}t=|}t)j>d||duri ddl m!}tDE|tD}t)j>d|d|jG|jHd|dnt)j*dyyt)j>dy77777l7_#t&$r!} t)j*d | Yd} ~ )d} ~ wwxYw7#t&$r"} t)j*d| Yd} ~ yd} ~ wwxYww)a+ Initialize the spend tracking and other background jobs 1. CloudZero Background Job 2. Focus Background Job 3. Prometheus Background Job 4. Key Rotation Background Job Args: scheduler: The scheduler to add the background jobs to r)CloudZeroLogger) FocusLogger)is_cloudzero_setupNr) VantageLogger)_get_vantage_settingsis_vantage_setupis_vantage_setup_in_configis_vantage_setup_in_dbapi_keyintegration_tokenbase_url)rrrz5Failed to register VantageLogger from DB settings: %sTr)+LITELLM_KEY_ROTATION_CHECK_INTERVAL_SECONDSLITELLM_KEY_ROTATION_ENABLEDzkey_rotation_enabled: )KeyRotationManagerz,Key rotation background job scheduled every z, seconds (LITELLM_KEY_ROTATION_ENABLED=true)r9key_rotation_job)r;r{z4Key rotation enabled but prisma_client not availablez"Failed to setup key rotation job: zGKey rotation disabled (set LITELLM_KEY_ROTATION_ENABLED=true to enable))%(litellm.integrations.cloudzero.cloudzeror'litellm.integrations.focus.focus_loggerr0litellm.proxy.spend_tracking.cloudzero_endpointsrinit_cloudzero_background_job init_focus_export_background_job+litellm.integrations.vantage.vantage_loggerr.litellm.proxy.spend_tracking.vantage_endpointsrrrrrrgrrrpr}r|init_vantage_background_job$prometheus_initialize_budget_metricsrRr"initialize_budget_metrics_cron_joblitellm.constantsrrrre/litellm.proxy.common_utils.key_rotation_managerr r]rBprocess_rotations)rr=rrrrrrrr db_settingsvantage_loggerrrrrkey_rotation_enabledr key_rotation_managers rgrz.9oo>Q.R%0__Z%@*  88MM* ;;i;P P P  7 74 ? H  ? ?) ? T 0;;W/X""%;>" K- &)00N ! & &Y k & T P $5S"?!(00OQR QV W$,,/QRSQT-UVV WsIG*IG-IG0 I4G35I G6 I G<G9AG<8I H)AI,A&H,I-I0I3I6I9G<< H&H!I!H&&I, I5I IIIc K|s|jj[|Wtd|jddxsd}t |dd}|dj dk7r t d|j|jjd|tjtd tjd d z z|gd dt|j|jjddddt!j"dr_d dlm}|j|jj(dt*d |ddd|jj)d{yyyyy7 w)z'rstart_token_refresh_task&DISABLE_PRISMA_HEALTH_CHECK_ON_STARTUPFTstart_db_health_watchdog_task)rrpconnectrr}rerrr4rrcheck_view_exists(_set_spend_logs_row_count_in_proxy_stater health_checkr6rhandle_db_exception)rrrrr]rs rgrz&ProxyStartupEvent._setup_prisma_clients6 48M'$0%1EV%M '//111"=$/G!$$&@5(**CCEEE##!335##!JJL $$LeT (44666=*IJ'EEGGG Y!G2 #a&(Gs1v,=,22A-22B-22ZG &F 7H  $ 8 8 ; sF8F DDDD?F .F/A+F F #F >F ?F F8 DDDF D F#AE>>FF F F F5F0+F80F55F8cddlm}m}|rddl}|j dd|r3ddlm}|}|jtjdyy) z Initialize dd tracer - if `USE_DDTRACE=true` in .env DD tracer is used to trace Python applications. Doc: https://docs.datadoghq.com/tracing/trace_collection/automatic_instrumentation/dd_libraries/python/ r)_should_use_dd_profiler_should_use_dd_tracerNTF)r[openai)ProfilerzDatadog Profiler started......) %litellm.litellm_core_utils.dd_tracingr=r>ddtrace patch_allddtrace.profilingr@rr}re)rr=r>rBr@profs rgrz!ProxyStartupEvent._init_dd_tracersR ! "    d5  9 " $ 2:D JJL & &'G H %rictddstjdy ddl}t j d}|s t dt j d}|s t d i}t j d xst j d }|r||d <t j d }|||r|ndd}| tt||d<|jdi|d|d|d|d}d|vr |d|dz }tj|y#t tf$rt d|wxYw#t$rtjdYywxYw)a Optional continuous profiling via Grafana Pyroscope. Off by default. Enable with LITELLM_ENABLE_PYROSCOPE=true. Requires: pip install pyroscope-io (optional dependency). When enabled, PYROSCOPE_SERVER_ADDRESS and PYROSCOPE_APP_NAME are required (no defaults). Optional: PYROSCOPE_SAMPLE_RATE (parsed as integer) to set the sample rate. LITELLM_ENABLE_PYROSCOPEFzWLiteLLM: Pyroscope profiling is disabled (set LITELLM_ENABLE_PYROSCOPE=true to enable).NrPYROSCOPE_APP_NAMEzsLITELLM_ENABLE_PYROSCOPE is true but PYROSCOPE_APP_NAME is not set. Set PYROSCOPE_APP_NAME when enabling Pyroscope.PYROSCOPE_SERVER_ADDRESSzLITELLM_ENABLE_PYROSCOPE is true but PYROSCOPE_SERVER_ADDRESS is not set. Set PYROSCOPE_SERVER_ADDRESS when enabling Pyroscope.OTEL_ENVIRONMENT_NAMELITELLM_DEPLOYMENT_ENVIRONMENT environmentPYROSCOPE_SAMPLE_RATE)app_nameserver_addressr sample_ratez-PYROSCOPE_SAMPLE_RATE must be a number, got: z/LiteLLM: Pyroscope profiling started (app_name=z, server_address=zA). View CPU profiles at the Pyroscope UI and select application 'z'.z sample_rate=zLiteLLM: LITELLM_ENABLE_PYROSCOPE is set but the 'pyroscope-io' package is not installed. Pyroscope profiling will not run. Install with: pip install pyroscope-ior^)rr}re pyroscoperrrr(rr configurerd ImportErrorr|) rrQrNrOrenv_namesample_rate_envconfigure_kwargsmsgs rgrz!ProxyStartupEvent._init_pyroscopes95A & &i  0  yy!56H F YY'ABN! LDyy!89RYY0>H&.]# ii(?@O$"0 $$   *69%:P6Q$]3 I   3"2 3A(K\]k\lmQQYPZZ\^  00'7 'F&GHH % %c *#I.$G*-/  ( ([  s+BD>DAD>D;;D>>EE)r^rr classmethodr r{rrrrDrrr rrrrrrrrrrr(rrlrrrrrr^rirgrr"sv  V$  (  $J/    #J/8&V$&sCx.&&P   - &   (   8"%&$   & ""H] ] $] ,/ ] ,/ ] " ] (] ] ~k(kkZ8f#8f8f( 8f $ 8f8ft@sm@(@& @ ,  @@DII0>>rirz /v1/modelszmodel management) dependenciesrz/modelsreturn_wildcard_routesinclude_model_access_groupsonly_model_access_groupsinclude_metadata fallback_typescopec 4Kddlm}ddlm} m} ||dk7rt dd|d } |dk(r ||t tt d{} | rtg} i} n(tj} tj} |r.tt| t| jz} dd lm}|gg| dd |xsd t| |xsd |xsd }g}|D](}| |d |xsd |t}|j#|*t%|dS| |tt&t(t t||xsd |xsd |xsd t d{}g}|D](}| |d |xsd |t}|j#|*t%|dS7I7Cw)a Use `/model/info` - to get detailed model information, example - pricing, mode, etc. This is just for compatibility with openai projects like aider. Query Parameters: - include_metadata: Include additional metadata in the response with fallback information - fallback_type: Type of fallbacks to include ("general", "context_window", "content_policy") Defaults to "general" when include_metadata=true - scope: Optional scope parameter. Currently only accepts "expand". When scope=expand is passed, proxy admins, team admins, and org admins will receive all proxy models as if they are a proxy admin. r)r)create_model_info_responseget_available_models_for_userNexpandzIInvalid scope parameter. Only 'expand' is currently supported. Received: rFrr]rr)r) key_models team_modelsproxy_model_listrYinfer_model_from_keysrZrmodel_access_groupsr[r\r?rrr]r^rr)robject rrrrYr]rrr[r\rZr)/litellm.proxy.management_endpoints.common_utilsrrrarbr'r]rrrget_model_namesget_model_access_groupsrr&rlitellm.proxy.auth.model_checksrrrrrY)rrZrr[r\r]r^r_rrarbshould_expand_scoperhrjr all_models model_datar9r^s rgr4r4\s>  Uh.^_d^ef    $>/'1/ %    ! "$ )99; ","D"D"F  '#$t,?,D,D,F'GGH   L,-"'#9#BU! 3(C(Lu%=%F    *E3!!1!:U+% J   j ) *  5+)#+$?$H5!9!BU5>-  JJ&/-6'!   *%&   i t s&AF F DFFAFFz/v1/models/{model_id}z/models/{model_id}rc Kddlm}m}m}||tt t ttddddt d{}|||t tdd tj|}|td d |d  tj|jj \}}}}|||ddtS7w)ak Retrieve information about a specific model accessible to your API key. Returns model details only if the model is available to your API key/team. Returns 404 if the model doesn't exist or is not accessible. Follows OpenAI API specification for individual model retrieval. https://platform.openai.com/docs/api-reference/models/retrieve r)rarbvalidate_model_accessNFrm)ravailable_modelsrxzRouter not initializedrzModel 'z#' not found in router configurationrrk)rrarbrvrrrYr]rrr'"get_deployment_by_model_group_namergget_llm_providerrr9) rrrarbrvrsrrrs rgr^r^s25+)#+$)!&$-  J8jI4LMM>>xHJXJ&IJ   00z7P7P7V7VWAxA &  A s9CCB Cz/v1/chat/completionszchat/completionsz/chat/completionsz&/engines/{model:path}/chat/completionsz1/openai/deployments/{model:path}/chat/completionsrzSuccessful response)rYrrfastapi_responseclKt|d{}||jdi|d<t|dr|j|j|dd<t|dr|j|j|dd<t|dr|j |j |dd <t|d r|j |j |dd <t| } |j|||d ttttt|ttt t"t$t& d{}t)|t*r t-|dS|S7F7&#t.$rG}|j0}tj3|||d{7t5j6} |j8| _|j:| j<dj:_d| j<d_ |jdd||ddurut4jBjE| d} t5jF| |j8d|jdd} t| ||} tI| ddcYd}~St5jJddd} | | _&| cYd}~Sd}~wtN$rN}|j0}tj3|||d{7t5j6} |j:| j<dj:_|jdd|ddurt4jBjE| d} t5jF| |jddd|jdd} t| ||} tI| dt|dr |jPntRjTcYd}~St5jJddd} | | _&| cYd}~Sd}~wtV$r&}|jY||t d{7d}~wwxYww)!a Follows the exact same API spec as `OpenAI's Chat API https://platform.openai.com/docs/api-reference/chat` ```bash curl -X POST http://localhost:4000/v1/chat/completions -H "Content-Type: application/json" -H "Authorization: Bearer sk-1234" -d '{ "model": "gpt-4o", "messages": [ { "role": "user", "content": "Hello!" } ] }' ``` r Nmetadataruser_api_key_user_idruser_api_key_team_idorg_iduser_api_key_org_idagent_idr acompletionr r|r route_typerrrrrr9rYr`rauser_max_tokensr_r!T)rrsrcontent_filterstreammodel_responseconvert_to_deltacached_responselitellm_logging_obj)completion_streamr9custom_llm_provider logging_objrtext/event-streamr{ media_typer# prompt_tokenscompletion_tokens total_tokensr9rr#)rrr)-rrrrrrrrbase_process_llm_requestrrrrrrYr`rarr_r!rrrrrpr|rgrKr9r`choicesr$ finish_reasonutilsModelResponseIteratorCustomStreamWrapperr:Usageusagerr#r0HTTP_400_BAD_REQUESTrp_handle_llm_api_exception)r r|r9rrbase_llm_response_processorrr_data_chat_response _iterator_streaming_responseselected_data_generator_usages rgchat_completionr0s*h$G4 4D$ 88J  '!D  %y 1!))57H7P7PD 3 4 %y 1!))57H7P7PD 3 4 %x 0!((46G6N6ND 2 3 %z 2!**6+<+E+ED Z ("?T"Jg 2KK-/$/!-%"7!-!5+'!L  $ fi (3F$O OM_ 54 , #$66/ 7   !..0 ww45IIq!))12Bq!/ 88Hd # /DNd4J ;;-<I#*"="="+gg$5 HH%:DA #  '<,"3"' # %'.  Q!RST% %66/ 7   !..045IIq!))1 88Hd # /DNd4J ;;-<I#*"="="+hhw+$5 HH%:DA #  '<,"3"' # %'.q-0MM44  Q!RST%  /II//J     sP4ECP4$AE4E5 EP4EP4E P1''J)FC.J)=P1>P4 J)#P1$P4) P16'O?K C5O?P1P4 O?9P1:P4? P1 P,&P)'P,,P11P4z/v1/completions completionsz /completionsz!/engines/{model:path}/completionsz,/openai/deployments/{model:path}/completionsc Ki} t|d{}||jdi|d<t|dr|j|j|dd<t|dr|j|j|dd<t|dr|j |j |dd <t|d r|j |j |dd <t| }|j|||d ttttt|ttt t"t$t& d{S7&7#t($r}|j*}tj-|||d{7|jdd|ddurt/j0}t3|j4dd|j6|j8|_t/j:ddd} t3|d| t.j<j?|d} t/j@| |j8} t| ||} tC| ddcYd}~St/jD} |j6| j4d_#|j8| _t/j:ddd} | | _$| cYd}~Sd}~wtJ$ro}|j*}tj-|||d{7|jdd|ddurt/j0}t/j:ddd} | |_$|j6|j4dj6_&t.j<j?|d} t/j@| |jdd} t| ||} tC| dit|dr |jNntPjRcYd}~St/jD} |j6| j4d_#| cYd}~Sd}~wtT$r}tj-|||d{7tWjXdj[t]|t]|}t_ta|d |ta|d!d"ta|d#d"ta|d$dta|dd%&d}~wwxYww)'a Follows the exact same API spec as `OpenAI's Completions API https://platform.openai.com/docs/api-reference/completions` ```bash curl -X POST http://localhost:4000/v1/completions -H "Content-Type: application/json" -H "Authorization: Bearer sk-1234" -d '{ "model": "gpt-3.5-turbo-instruct", "prompt": "Once upon a time", "max_tokens": 50, "temperature": 0.7 }' ``` r~Nrrrrrrrrratext_completionrrsrTrtextrrr)rr9rrr{rr9rr#)rr%r#z?litellm.proxy.proxy_server.completion(): Exception occured - {}r`rVrwrr'rxr`rVr openai_coder')1rrrrrrrrrrrrrrrYr`rarr_r!rrpr|rgrKrVrr`r9rrrTextCompletionStreamWrapperr:rMrrrr$r#r0rrpr}rrurr?r)r r|r9rrrrr_text_responserrrr _responserrs rg completionrsR D[ '88  (xx #+#%Z )95%--9;L;T;TZ !78)95%--9;L;T;TZ !78)84%,,8:K:R:RZ !67):6%..:/@/I/IZ ,&C&N#0II-/)/!-%"7!-!5+'!J   392 $ #266/ 7   99Xt $ 0U8_5L$224N N**1-vqyy A#$77N ]]"#F NGV 4 ;;-<I#*"E"E"+gg#  '<,"3"' # %'.   668I() I  a %ggIO]]"#F %IO  +66/ 7   99Xt $ 0U8_5L$224N]]"#F $*N 89 N " "1 % - - 5 ;;-<I#*"E"E"+ii,#  '<,"3!' # %'.q-0MM44   668I() I  a %   66/ATX7    && M T TA  1vh Ay)4FF+!Wf-640M3/    sS6ED;D E6D>7E:S6;E>E S3 'K 1E42C)K S3S6!A#K S3S6 S3'Q>L?DQS3S63Q;S3<S6 S3 S.(Q+)BS..S33S6z/v1/embeddings embeddings)rYresponse_classrz /embeddingsz /engines/{model:path}/embeddingsz+/openai/deployments/{model:path}/embeddingsctKi} t|d{}ttjng}d|vrt|dtrt |ddkDrt|ddtrt|dddt rt|jd|vrtj|d}||jdixsi}|jddtjvxstfd tD}|s7g} |dD](} | jtjd | *| |d<||jd i|d <t|d r|j |j |d d<t|dr|j"|j"|d d<t|dr|j$|j$|d d<t|dr|j&|j&|d d<t)|} | j+|||dt,tt.t0t2|t4t6t8t:t<t>d{} | S7Z7 #t@$r7} t)|} | jC| |t,t>d{7d} ~ wwxYww)a Follows the exact same API spec as `OpenAI's Embeddings API https://platform.openai.com/docs/api-reference/embeddings` ```bash curl -X POST http://localhost:4000/v1/embeddings -H "Content-Type: application/json" -H "Authorization: Bearer sk-1234" -d '{ "model": "text-embedding-ada-002", "input": "The quick brown fox jumps over the lazy dog" }' ``` r~Ninputrr9)model_group_namerrc3@K|]}j|ywrP)r@)rMr litellm_models rgrOzembeddings..s&e$&00:esz gpt-3.5-turbo)r9tokensrrrrrrrrr aembeddingr)rrrr!)"rr model_namesrrrnr(rryrgopen_ai_embedding_modelsr1r"rdecoderrrrrrrrrrrrYr`rarr_r!rpr)r r|r9rrrouter_model_namesrrsupports_token_arrays input_listirrirrs @rgrrs`D] '888B7MZ33SU tO4=$/DM"Q&4=+T24=+A.4%$((7*;?Q*Q'JJ%)']K )%/^^4Db%I%ORN$2$6$6w$CM,9W=]=],]-ade(deb)1%' !%gA&-- '_Q O)3W  (xx #+#%Z )95%--9;L;T;TZ !78)95%--9;L;T;TZ !78)84%,,8:K:R:RZ !67):6%..:/@/I/IZ ,'D&N#5MM-/#/!-%"7!-!5+'!N  &e9~ (  &C&N#/II// J     sWJ8I5I0II5)I3*I5/J80I53I55 J5>,J0*J-+J00J55J8z/v1/moderations moderationsz /moderationscKi} |jd{}tj|}t||t|t t d{}tjddxstxs|jd|d<tr t|d<tj||dd{}tjt|dttd{}|d{}tjtj!|jd d d t#|d ixsi}|jddxsd }|jddxsd } |jddxsd } |j$j't)j*||| | t t#|dd |||S77|777#t,$r} tj/|| |d{7t1j2dj5t7| t9| t:rRt=t#| dt7| t#| ddt#| ddt#| dt>j@t7| } t=t#| d| t#| ddt#| ddt#| ddd} ~ wwxYww)a~ The moderations endpoint is a tool you can use to check whether content complies with an LLM Providers policies. Quick Start ``` curl --location 'http://0.0.0.0:4000/moderations' --header 'Content-Type: application/json' --header 'Authorization: Bearer sk-1234' --data '{"input": "Sample text goes here", "model": "text-moderation-stable"}' ``` Nrr rrr!rmoderation_modelr9 moderationrr call_type amoderationrrrrYrrrrr0_hidden_paramsrrr:allowed_model_regionrrrr:r! model_regionrp hidden_paramsrsz@litellm.proxy.proxy_server.moderations(): Exception occured - {}r`rVrwrr#ryrx)!bodyorjsonrrrr!rrrYr pre_call_hookrrrrrupdate_request_statusrr%rJrget_custom_headersrpr|r}rrurrr'r?r0r) r r|rrrllm_callrirrrr:rrs rgrr@s8DY\\^#||D!1-/%     !3T : ! !xx  W &DM'44/dl5   '$!!    ">   3 3 $):B ?  4   *:B?E2  $$Z6<"!%%k48>B  $$Z6<"  '' ) < <"3!#!$%68NPRS!+  $ $  ": 66/ATX7    && N U UA  a ' 9c!f5Q/a&1Q v/J/JK  q6(I 9i8Q/a&1Q s3  %sK!GG 9GGA"G6G72G)G* G3G4CG K! GGGGG K"K=H>CKKK!rctK|jtd{}|23d{}|77 6yw)N) chunk_size) aiter_bytesr)r _generatorrns rg_audio_speech_chunk_generatorrsB!,,8O,PPJ!e Qzs$828646868z/v1/audio/speechaudioz /audio/speechc RKi} |jd{}tj|}t||t|t t d{}|jdd|j|j|d<tr t|d<tj||dd{}t|dttd{}|d{}tjtj!|jdd d t#|d ixsi}|jd dxsd }|jddxsd } |jddxsd } |jddxsd } |jddxsd } t%j&||| | t | t#|dd d| || } d}|jdd }|r|j)}d|vr d|vsd|vrd}t+t-||| S777g7K7D#t.$r|}tj1|||d{7t3j4dj7t9|t3j:t=j>|d}~wwxYww)z` Same params as: https://platform.openai.com/docs/api-reference/audio/createSpeech Nrrr9aspeechrrrrrrrrrr:rr) rrrr:r!rr!fastest_response_batch_completioncall_idrprz audio/mpeggeminittsz preview-ttsz audio/wav)rr%rszAlitellm.proxy.proxy_server.audio_speech(): Exception occured - {}) rrrrrr!rrrrYrrrrrrrrrrrr:rrpr|r}rrurrerr)r r|rrrrrirrrr:rrcustom_headersr request_modelrequest_model_lowerrs rg audio_speechrs*D^\\^#||D!1-/%    88FD ! ).?.G.G.S,44DL &DM'44/di5   ' !!    ">   3 3 $):B ?  4   *:B?E2  $$Z6<"!%%k48>B  $$Z6<"%))/4@FB '++,=tDJ6II/' !24JBO.2#'  " "- "/"5"5"7 ..,, AT0T ! )( 3!"  U$   "b  66/ 7   "" O V VA  ""9#7#7#9: sJ'HH9HHAH/H0HH HHD6HJ'HHHHH J$(JIAJJ$$J'z/v1/audio/transcriptionsz/audio/transcriptions.rdc$ Ki} t|d{}|jDcic]\}}|dk7s ||}}}t||t|tt d{}|j dd|j|j|d<tj ddxstxs|j dd|d<tr t|d<ttjng}|jtdtjdd t||| |j!d{} t#j$| } |j| _| |d< t(j+||d d{}t-|d ttd{} | d{} | j1t3j4t(j7|j dddt9| dixsi}|j ddxsd}|j ddxsd}|j ddxsd}|j ddxsd}|j ddxsd}|j dixsi}|j:j=t?j@d'||||t|t9|dd|||d |t(jC||| tE|j:d{}|r|j:j=|| S7cc}}w77777#t.$r} | d} ~ wwxYw#| j1wxYw7c#t.$r} t(jG|| |d{7tIjJdjMtO| tQ| tRr\tt9| dtO| jTt9| dd t9| d!d t9| d"tj#tO| }tt9| d|t9| dd t9| d!d t9| d$dt9| d"d%&d} ~ wwxYww)(zq Same params as: https://platform.openai.com/docs/api-reference/audio/createTranscription?lang=curl Nrdrrrr9z.File name is None. Please check your file name bad_request)r`r'rVr)rprdr transcriptionratranscriptionrrrrrrrrr:radditional_headersr) rrrr:r!rrrrpr)rrrirequest_headersrszHlitellm.proxy.proxy_server.audio_transcription(): Exception occured - {}r`rVrwrr#ryr'rxrr^)+rrrrr!rrrrYrrrbr?r0rrreadioBytesIOrrrrrprjrrrrr%rJrrpost_call_response_headers_hookrr|r}rrurrr'r)r r|rdrr form_datarrr file_content file_objectrrirrrrr:rrrcallback_headersrs rgaudio_transcriptionsr<s,DH'00 -6__->PzsE#-U PP1-/%    88FD ! ).?.G.G.S,44DL  !3T : ' 'xx& W &DM7A7MZ33SU ==  H00"   $1 "YY[( jj. == "V  *88"3)9D++%% H &~H       3 3 $):B ?  4   *:B?E2  $$Z6<"!%%k48>B  $$Z6<"%))/4@FB '++,=tDJ#0#4#45I2#N#TRT  '' ) < < "3!#!+$%68NPRS'!+ %  ""3!R!R/ 1 "S"     $ $ + +,< =[1P J) & G     F  66/ATX7    && V ] ]A  a ' 9c!((m<Q/a&1Q v/J/JK  q6(I 9i8Q/a&1#Avt4Q s3  %sRM1L/M1 L2L2#M1"L8#CM1:L;;/M1+ML>M%M& M/M0M5EM1 M/ "M1.R/M12M1;M1>MMM MMMMM,,M11 R ;RNC1RR  Rz/vertex_ai/livezGOptional model name, used to determine Vertex region for global models.)rzCOverride the Vertex AI project id used for the upstream connection.z;Override the Vertex AI region (for example, 'us-central1'). websocketvertex_projectvertex_locationc>Kt|||||d{S7w)z Vertex AI Live API WebSocket Pass-through Endpoint This endpoint delegates to the WebSocket function defined in llm_passthrough_endpoints.py rr9rrrNrrs rg#vertex_ai_live_passthrough_endpointrs.,6%'+   s )maxsizeintentcLd|fg}||jd|ft|S)z Build a hashable representation of the realtime query params so we can cache the repetitive model/intent combinations. r9r)rtuple)r9rparamss rg_realtime_query_params_templater s2'.u%5$6F  x() =riz /realtimez'The intent of the websocket connection.zAComma-separated list of guardrail names to apply to this request.rcXK|jjdxsdjdDcgc]"}|jr|j$}}i}|r|d|d<|jdi|d{t t tt|}||d} |rA|jdD cgc]#} | js| j%c} | d<t|jjdxsg} tj} | | d<t| } |j| _fd }|| _t#| } |j%| t&|t(t*t,t.t0t2t4t6d d{\} } || d<tI| d tJt.d{}|d{ycc}w7jcc} w7D#t8$r}t;j<d |j?tAjBddtE|ddd{7n#t8$rYnwxYw|jGddd{7Yd}~yd}~wwxYw77#tLjNjP$rE}t;j<d|jG|jRdd{7Yd}~yd}~wt8$r4t;j<d|jGddd{7YywxYww)Nzsec-websocket-protocolrrr subprotocol)r9rrrr%)r_c"KtSwrPrIrsrg return_bodyz0realtime_websocket_endpoint..return_bodyJs%e,,s r _arealtime) r rrr!rrrYr`rarr_r9rzRealtime pre-call errorrguardrail_error)rVr`)rVrizPre-call error)r'reasonrrzInvalid status codezInternal server errorr^)*r%rrracceptrrrrrr_rYrlr*r_urlrr common_processing_pre_call_logicrr!rrrYr`rarr_rpr}r send_textrrFrrjrr websockets exceptionsInvalidStatusCoder#)rr9rrrprequested_protocols accept_kwargsrrg headers_listr_r r rrrrs ` rgrealtime_websocket_endpointrs ##''(@AGRNNsS 779   M':1'= m$ )   +] +++T"A%"PQL $D1;1A1A#1FTA!'')aggiT\ ++I6<"=L + 0 0 2E#E)E"G==GL-GL#@T"J $.NN-//%!-!5+'#O    HI$5 !&#!!    {,U6  &&'@A %% '$5'*1v"      oo40@oAAA#,   2 2P&&'<=oo1==9NoOOO I&&'>?oo40GoHHHIs1L*'G##L*>G(?AL*G+G+)A6II I I> II>II>3I64I>9L*>JL*J J L''5K'KK'"L*'7L'L!L'$L*&L''L*z/v1/assistants assistantsz /assistantscbKi} |jd{t||t|ttd{}t 't ddtjjit jdi|d{}tjtj|jddd t!|d ixsi}|jd dxsd}|jd dxsd}|jd dxsd}|j"j%t'j(||||tt!|dd|||S7H7&7#t*$r9} tj-|| |d{7t/j0dj3t5| t/j6t9j:t=| t r\t?t!| dt5| j@t!| ddt!| ddt!| dtBjDt5| } t?t!| d| t!| ddt!| ddt!| ddt!| ddd} ~ wwxYww)z Returns a list of assistants. API Reference docs - https://platform.openai.com/docs/api-reference/assistants/listAssistants Nrrxrrrrrrrrrr:rrrszClitellm.proxy.proxy_server.get_assistants(): Exception occured - {}r`rVrwrr#ryr'rr^)#rrrr!rrr'r* no_llm_routerraget_assistantsrrrrrrr%rJrrrpr|r}rrurrerrrr?rr0r r r|rrrirrrr:rrs rgget_assistantsr s*DIlln1-/%     2C2Q2Q2W2W(X $33;d;;   3 3 $):B ?  4   *:B?E2  $$Z6<"!%%k48>B  $$Z6<"  '' ) < <"3!#!$%68NPRS!+  [  <: 66/ATX7    "" Q X XA  ""9#7#7#9: a ' 9c!((m<Q/a&1Q v/J/JK  q6(I 9i8Q/a&1#Avt4Q s3  'siJ/E)E!$E)E$A E)E' CE) J/!E)$E)'E)) J,3J'FDJ''J,,J/cKi} |jd{}tj|}t||t|t t d{}t'tddtjjitjdi|d{}tjtj!|j#ddd t%|d ixsi}|j#d dxsd}|j#d dxsd}|j#d dxsd} |j&j)t+j,|||| t t%|dd|||S7]7&7#t.$r8} tj1|| |d{7t3j4dj7t9| t3j:t=j>tA| tr\tCt%| dt9| jDt%| ddt%| ddt%| dtFjHt9| } tCt%| d| t%| ddt%| ddt%| dt%| ddd} ~ wwxYww)z~ Create assistant API Reference docs - https://platform.openai.com/docs/api-reference/assistants/createAssistant Nrrxrrrrrrrrrr:rrrszElitellm.proxy.proxy_server.create_assistant(): Exception occured - {}r`rVrwrr#ryr'r^)%rrrrrr!rrr'r*rracreate_assistantsrrrrrrr%rJrrrpr|r}rrurrerrrr?rr0r) r r|rrrrirrrr:rrs rgcreate_assistantr#s* DI\\^#||D!1-/%     2C2Q2Q2W2W(X $66>>>   3 3 $):B ?  4   *:B?E2  $$Z6<"!%%k48>B  $$Z6<"  '' ) < <"3!#!$%68NPRS!+  ]$ ?: 66/ATX7    "" S Z ZA  ""9#7#7#9: a ' 9c!((m<Q/a&1Q v/J/JK  q6(I 9i8Q/a&1Q=#(FG  'skKE>E69E>E9A E>E<CE>5K6E>9E><E>> KJ;#F&$DJ;;KKz"/v1/assistants/{assistant_id:path}z/assistants/{assistant_id:path} assistant_idc.Ki} t||t|ttd{}t't ddt jjit jdd|i|d{}tjtj|jddd t|d ixsi}|jd dxsd}|jd dxsd}|jddxsd} |j j#t%j&|||| tt|dd|||S7%7#t($r8} tj+|| |d{7t-j.dj1t3| t-j4t7j8t;| t r\t=t| dt3| j>t| ddt| ddt| dt@jBt3| } t=t| d| t| ddt| ddt| dt| ddd} ~ wwxYww)z~ Delete assistant API Reference docs - https://platform.openai.com/docs/api-reference/assistants/createAssistant rNrxrrr$rrrrrrrr:rrrszElitellm.proxy.proxy_server.delete_assistant(): Exception occured - {}r`rVrwrr#ryr'r^)"rrr!rrr'r*rradelete_assistantrrrrrrr%rJrrrpr|r}rrurrerrrr?rr0r) r r$r|rrrirrrr:rrs rgdelete_assistantr'Y s,DG1-/%     2C2Q2Q2W2W(X $55X<XSWXX   3 3 $):B ?  4   *:B?E2  $$Z6<"!%%k48>B  $$Z6<"  '' ) < <"3!#!$%68NPRS!+  U Y: 66/ATX7    "" S Z ZA  ""9#7#7#9: a ' 9c!((m<Q/a&1Q v/J/JK  q6(I 9i8Q/a&1Q=#(FG  'YJ EE A E2E3CE J EE JJ 5E86DJ  JJz /v1/threadsz/threadsc`Ki} |jd{t||t|ttd{}t 't ddtjjit jdi|d{}tjtj|jddd t!|d ixsi}|jd dxsd}|jd dxsd}|jd dxsd}|j"j%t'j(||||tt!|dd|||S7H7&7#t*$r8} tj-|| |d{7t/j0dj3t5| t/j6t9j:t=| t r\t?t!| dt5| j@t!| ddt!| ddt!| dtBjDt5| } t?t!| d| t!| ddt!| ddt!| dt!| ddd} ~ wwxYww)zs Create a thread. API Reference - https://platform.openai.com/docs/api-reference/threads/createThread Nrrxrrrrrrrrrr:rrrszClitellm.proxy.proxy_server.create_threads(): Exception occured - {}r`rVrwrr#ryr'r^)#rrrr!rrr'r*rracreate_threadrrrrrrr%rJrrrpr|r}rrurrerrrr?rr0rrs rgcreate_threadsr+ s*DHlln1-/%     2C2Q2Q2W2W(X $22:T::   3 3 $):B ?  4   *:B?E2  $$Z6<"!%%k48>B  $$Z6<"  '' ) < <"3!#!$%68NPRS!+  [  ;: 66/ATX7    "" Q X XA  ""9#7#7#9: a ' 9c!((m<Q/a&1Q v/J/JK  q6(I 9i8Q/a&1Q=#(FG  'siJ.E)E!$E)E$A E)E' CE) J.!E)$E)'E)) J+3J&FDJ&&J++J.z/v1/threads/{thread_id}z/threads/{thread_id} thread_idc.Ki} t||t|ttd{}t't ddt jjit jdd|i|d{}tjtj|jddd t|d ixsi}|jd dxsd}|jd dxsd}|jddxsd} |j j#t%j&|||| tt|dd|||S7%7#t($r8} tj+|| |d{7t-j.dj1t3| t-j4t7j8t;| t r\t=t| dt3| j>t| ddt| ddt| dt@jBt3| } t=t| d| t| ddt| ddt| dt| ddd} ~ wwxYww)zs Retrieves a thread. API Reference - https://platform.openai.com/docs/api-reference/threads/getThread rNrxrrr,rrrrrrrr:rrrsz?litellm.proxy.proxy_server.get_thread(): Exception occured - {}r`rVrwrr#ryr'r^)"rrr!rrr'r*rr aget_threadrrrrrrr%rJrrrpr|r}rrurrerrrr?rr0r r r,r|rrrirrrr:rrs rg get_threadr0!s,DE0-/%     2C2Q2Q2W2W(X $//L)LtLL   3 3 $):B ?  4   *:B?E2  $$Z6<"!%%k48>B  $$Z6<"  '' ) < <"3!#!$%68NPRS!+  U M: 66/ATX7    "" M T TA  ""9#7#7#9: a ' 9c!((m<Q/a&1Q v/J/JK  q6(I 9i8Q/a&1Q=#(FG  'r(z /v1/threads/{thread_id}/messagesz/threads/{thread_id}/messagescKi} |jd{}tj|}t||t|t t d{}t'tddtjjitjdd|i|d{}tjtj!|j#ddd t%|d ixsi}|j#d dxsd}|j#d dxsd} |j#ddxsd} |j&j)t+j,||| | t t%|dd|||S7_7(7#t.$r8} tj1|| |d{7t3j4dj7t9| t3j:t=j>tA| tr\tCt%| dt9| jDt%| ddt%| ddt%| dtFjHt9| } tCt%| d| t%| ddt%| ddt%| dt%| ddd} ~ wwxYww)zv Create a message. API Reference - https://platform.openai.com/docs/api-reference/messages/createMessage Nrrxrrr,rrrrrrrr:rrrszAlitellm.proxy.proxy_server.add_messages(): Exception occured - {}r`rVrwrr#ryr'r^)%rrrrrr!rrr'r*rr a_add_messagerrrrrrr%rJrrrpr|r}rrurrerrrr?rr0r r r,r|rrrrirrrr:rrs rg add_messagesr4z!s,DI\\^#||D!1-/%     2C2Q2Q2W2W(X $11NINNN   3 3 $):B ?  4   *:B?E2  $$Z6<"!%%k48>B  $$Z6<"  '' ) < <"3!#!$%68NPRS!+  ]$ O: 66/ATX7    "" O V VA  ""9#7#7#9: a ' 9c!((m<Q/a&1Q v/J/JK  q6(I 9i8Q/a&1Q=#(FG  'skKFE89FE;A FE> CF7K8F;F>F K J=%F(&DJ==KKc.Ki} t||t|ttd{}t't ddt jjit jdd|i|d{}tjtj|jddd t|d ixsi}|jd dxsd}|jd dxsd}|jddxsd} |j j#t%j&|||| tt|dd|||S7%7#t($r8} tj+|| |d{7t-j.dj1t3| t-j4t7j8t;| t r\t=t| dt3| j>t| ddt| ddt| dt@jBt3| } t=t| d| t| ddt| ddt| dt| ddd} ~ wwxYww)z Returns a list of messages for a given thread. API Reference - https://platform.openai.com/docs/api-reference/messages/listMessages rNrxrrr,rrrrrrrr:rrrszAlitellm.proxy.proxy_server.get_messages(): Exception occured - {}r`rVrwrr#ryr'r^)"rrr!rrr'r*rr aget_messagesrrrrrrr%rJrrrpr|r}rrurrerrrr?rr0rr/s rg get_messagesr7!s,DE0-/%     2C2Q2Q2W2W(X $11NINNN   3 3 $):B ?  4   *:B?E2  $$Z6<"!%%k48>B  $$Z6<"  '' ) < <"3!#!$%68NPRS!+  U O: 66/ATX7    "" O V VA  ""9#7#7#9: a ' 9c!((m<Q/a&1Q v/J/JK  q6(I 9i8Q/a&1Q=#(FG  'r(z/v1/threads/{thread_id}/runsz/threads/{thread_id}/runscKi} |jd{}tj|}t||t|t t d{}t'tddtjjitjdd|i|d{}d|vr)|ddur"tt||| d i d{Stj t"j%|j'd d dt)|dixsi}|j'ddxsd }|j'ddxsd } |j'ddxsd } |j*j-t/j0||| | t t)|dd |||S77U7 7#t2$r8} t"j5|| |d{7t7j8dj;t=| t7j>tAjBtE| tr\tGt)| dt=| jHt)| ddt)| ddt)| dtJjLt=| } tGt)| d| t)| ddt)| ddt)| dt)| ddd} ~ wwxYww) zi Create a run. API Reference: https://platform.openai.com/docs/api-reference/runs/createRun Nrrxrrr,rTrr) generatorrr%rrrrrrrr:rrrsz?litellm.proxy.proxy_server.run_thread(): Exception occured - {}r`rVrwrr#ryr'r^)'rrrrrr!rrr'r*rr arun_threadrrrrrrrrr%rJrrrpr|r}rrurrerrrr?rr0rr3s rg run_threadr;<"s,DT\\^#||D!0-/%     2C2Q2Q2W2W(X $//L)LtLL  h4!7(9&7%!% /    3 3 $):B ?  4   *:B?E2  $$Z6<"!%%k48>B  $$Z6<"  '' ) < <"3!#!$%68NPRS!+  u$ M J 66/ATX7    "" M T TA  ""9#7#7#9: a ' 9c!((m<Q/a&1Q v/J/JK  q6(I 9i8Q/a&1Q=#(FG  'sK5F0F%9F0F(A F0F+ ,F0 F. F0K5CF0$K5%F0(F0+F0.F00 K2:K-GDK--K22K5)BaseTokenCounterr model_to_usec d|yddlm}|jdijdd}d}d} |||jdijd|jdijd|jdijd  \}}}}dd lm}dd lm} ||} | r(| j|| } | | j||fSy#t$rH|jdsd|jvr"ddl m } | } | j||fcYSYywxYw)z Auto-route to the correct provider's token counter based on model/deployment. Uses the existing get_provider_model_info infrastructure with switch-case pattern. Nr)rzrr9rrr:r)r9rr:r) LlmProviders)ProviderConfigManager)r9rz anthropic/ anthropic)AnthropicModelInfo)NNN)1litellm.litellm_core_utils.get_llm_provider_logicrzrrOr?rmr@get_provider_model_infoget_token_counterrpr@r#litellm.llms.anthropic.common_utilsrB)rr=rz full_modelr9rdynamic_api_keyr:r?r@llm_provider_enumprovider_model_inforBanthropic_model_infos rg_get_provider_token_counterrL"sjR 0"599'2FJE)-$X@P */? D H H%! ^^$4b9==jINN#3R8<>2BB#GH $ $!3!9!9#q!A!!D  +gmm 48)-!78S  9 5&"5  !L#  : :$7 ;   ,88)/R!!%%mm9F!flld&:00D8( & 4 4 O8O3%#//63 %,,6v7I7I6J#fNbNbMcd77# $$,U*   >B / 0 NN-t 4 mm55-=6O01N  ( L !mm%  A  * *h    FsJA"J6%$J J  JCJ6$J3%D&J6 JJ0,J6/J00J6z/utils/supported_openai_params)rrYcK tj|\}}}}dtj||iS#t$rt dddj |iwxYww)a* Returns supported openai params for a given litellm model name e.g. `gpt-4` vs `gpt-3.5-turbo` Example curl: ``` curl -X GET --location 'http://localhost:4000/utils/supported_openai_params?model=gpt-3.5-turbo-16k' --header 'Authorization: Bearer sk-1234' ``` rsupported_openai_paramsr9rrdrzCould not map model={}r)rgrzget_supported_openai_paramsrpr'ru)r9rrs rgrdrd}#sx"  +2+C+C%+P("Aq %w'J'J1D(   W.F.M.Me.T$U   sA#38A#(A  A#z/utils/transform_requestcRKddlm}||j|jSw)Nr)return_raw_request)endpointry)rmrhr request_body)r rhs rgtransform_requestrk#s$1 w'8'8AUAU VVs%'rvc|K|'tddtjjig}|D]}|j dij dd}|(|j j jd|id{}|[|j|jk(su|j||S76w) Check if model is in db Check if db model is 'created_by' == user_api_key_dict.user_id Only return models that match Nrxrrr^r{rr) r'r*db_not_connected_errorrrrrrrrrr)rvrr]filtered_modelsr9r{rs rg_check_if_model_is_user_addedrp#s.EEKKL  O . YY|R ( , ,T4 8 : &))AAMMr"N    ""&7&?&??&&u- .   sBB<B:B< B<%B<user_rowcg}|D]U}|jdijdd}|(||jvs7|jtt|W|S)zm Check if model is a team model Check if user is a member of the team that the model belongs to r^rN)rrrrr )rvrquser_team_modelsr9 model_team_ids rg_check_if_model_is_team_modelru#se$&; ,377 4H  $. ''T5(9: ; rirscK|'tddtjjit |||d{}|j r] |j jjd|j id{}|t|jxsg| z }t| }|S7{76#t$rtddd iwxYww) rmNrxrr)rvrr]rrrdzUser not found)rvrq)rv) r'r*rnrrprrlitellm_usertablerrprurfr)rsrrr]rq unique_modelss rgnon_admin_all_modelsry#s.EEKKL  5+#J    U*--??KK "3";";<LH 3,,.4"  7jIM - UCBR8ST T Us9;CB9C5B=B;B= 1C;B==CCteam_db_objects_typedci}|D]}t|jdk(s&tjj|jvr|j }|U|D]}|j dij dd}|(|j dij dd}d}|d}n||jvrd}|sd|j|tj|j|jD]} |j | |j} | #| D]Z}|j dij dd}|(|j|tj|j\|S) z' Add team models to all models rNr^r{rFTr8r) rnrvrAall_proxy_modelsrrfrrrar&add) rzrrg team_objectr4r9r team_model_id can_add_modelr8_modelss rg_add_team_models_to_all_modelsr$s(*K,   "" #q ( 1177;;M;MM#224J%' YE$yyr:>>tTJH' $)IIlB$?$C$CIt$TM$)M$,(, &+*=*==(, $#..x?CCKDWDWX Y*00  $33);3F3F4&!(#(99\2#>#B#B4#N#/'228SUCGG + 3 3 - B ri user_teamsc Kg}|dk(rS|jjjd{}|Dcgc]}tdi|j }}nX|jjjdd|iid{}|Dcgc]}tdi|j }}t ||}i}|j D]\}} t| ||<|S7cc}w7hcc}ww)z Get all models across all teams user is in. 1. Get all teams user is in 2. Get all models across all teams 3. Return {"model_id": ["team_id1", "team_id2"]} rNrrr)rzrr^)rlitellm_teamtablersr:rrrr) rr]rrzteam_db_objectsteam_db_objectrgreturned_team_modelsrteam_idss rgget_all_team_modelsr4$s.68S - 0 0 B B L L NN#2!   < 9 9 ; <! ! !. 0 0 B B L LtZ01!M!   #2!   < 9 9 ; <! ! 13K 24)//18()-hX&8 3O!  ! s>/C:C,C: C.0C: C3 C: C53:C:.C:5C:user_db_objectcg}|jD]T}|j|}||D]8}|jdijdd}|(|j|:V|S)z7 Get all models that user has direct access to r8Nr^r{)rvrfrr)rrdirect_access_modelsr9 deploymentsrrs rgget_direct_access_modelsr`$s')&&: //5/A  ") : %>>,;??dK'(//9 :: ricKd}g}|jtjk(rd}|jd}n~|jr|j j jd|jid{}|6tdi|j}|jxsg}t||}|t|||d{}|D]} | jd ijd d} | jd ijd d} | Jd } | |j| g} | r | | vrd} nd} | sn|j| g| d d <|D]4} | jd ijd d} | (| |vs-d| d d<6|D cgc]H} | jd ijdd s"| jd ijd gr| J}} |S7m7 cc} ww)z5 Get all models across all teams user is in. NrT)exclude_team_modelsrr)rr)rr]rr^r{rFaccess_via_team_ids direct_accessr^) user_roler< PROXY_ADMINrrrrwrr;rrrrr)rr]rrsrrr user_objectrgr`rteam_only_model_id can_use_modelrs rg%get_all_team_and_direct_access_modelsrs$sH<@J&(""&6&B&BB )77 $ 8   " " .,//AAMM/778 N    %+Jn.G.G.IJK$**0bJ#;*%$ /!'!  ! Fzz,377dCH!'L"!=!A!A)T!R # % %1*x? $9::lB/33D$?  H0D$D48F<  19!  ::lB ' + +OU C ::lB ' + +,A2 F J _  :sRA8G2:G';AG2 G* AG2#G2?AG2G2G2A G-#G2*G2-G2rec|jdi}|dur/d}||j|idxsd}nd}t|}||d<t| }|ik(r;|jd i}|jd d} t j | }|ik(re|jd i}|jd d}|r?|jd } t| d kDr| d} t j || d }|jD]\} } | |vs | || <||d<t|dh}|S#t $ri}YwxYw#t $ri}YUwxYw)a Enrich a model dictionary with litellm model info (pricing, context window, etc.) and remove sensitive information. Args: model: Model dictionary to enrich debug: Whether to include debug information like openai_client llm_router: Optional router instance for debug info Returns: Enriched model dictionary with sensitive info removed r^TrwNasync)rry client_typellm_router_is_None openai_clientrrr9rrrrelitellm_credential_name) excluded_keys) r _get_clientrrrgrrprrnrr) r9rerr^_openai_clientrrrr split_modelrrs rg$_enrich_model_info_with_litellm_datar$s <,J }  !&&%PW&X  2NN+ !.o0e<R#3R8&**7D9  $!(!7!7m!L R#3R8&**7D9 '--c2K;!# +B  (%,%;%;'[^&" #((*1 J JqM%E, 2 78 E L5 $!#  $ (%'" (s$<D3"E3 EE EEsearchpagesizersort_bycK|r|js|dfS|jj}|Dcgc]'}||jddjvs&|)} }g} t} | D]^}|jdi} | jdd} | jd}| r|r| j |N| j |`t | }t | }||z}g}d}||z}|) d|d d i}| rd d t| ii|d <|jjj|d{}||z}|rb|dkDr|jjj||d{}|D](}|j|g}|s|j|*nu||krpt||z |}|dkDr\|jjj||d{}|D](}|j|g}|s|j|*n|}| |z}||fScc}w777F#t$r-}t!j"dt%||}Yd}~Ed}~wwxYww)aC Apply search filter to models, querying database for additional matching models. Args: all_models: List of models to filter search: Search term (case-insensitive) page: Current page number size: Page size prisma_client: Prisma client for database queries proxy_config: Proxy config for decrypting models sort_by: Optional sort field - if provided, fetch all matching models instead of paginating at DB level Returns: Tuple of (filtered_models, total_count). total_count is None if not searching. Nr8rr^rFr{r insensitive)containsmodenotrrr)rtakez,Error querying database models with search: )rrrr&r~rrnrrrrcountrsrrrorpr}rr)rsrrrr]rr search_lowerrtfiltered_router_modelsfiltered_config_modelsdb_model_ids_in_routerr^ is_db_modelrconfig_models_countdb_models_in_router_countrouter_models_countrdb_models_total_countmodels_needed_for_pagedb_where_conditionsearch_total_count db_models_rawrdecrypted_modelsmodels_to_fetchrros rg_apply_search_filter_to_modelsr$s0 4<<>'')L|R1H1N1N1P!P   U #-UU<,  nnZ7 >>$' 8 " & &x 0 " ) )! ,-45 #$: ;-0III!D[ G 5 ,)2 &D$'=">?2":. $&&>>DD,E ""57L!L (1,+..FFPP"4!6Q"%2?+7+R+R%J,(,%,,-=> ?')??&).1DD-'O '*"/"2"2"J"J"T"T&8%4#U#&)6CH/;/V/V!) 0, 0 ) 0 01A B C1-y8O . ..WZ. 5 * *>s1vhG "5   5s9I7'H3#H3'BI7AH>H8:H>?H:H>AH>5H<6H>H>'I78H>:H><H>> I4#I/*I7/I44I7dtc|yt|tr |jdr|jddn|}t j |}|j "|jtj}|S|jtj}|St|trK|j |jtjS|jtjSy#ttf$rYywxYw)a Normalize a datetime value to a timezone-aware UTC datetime for sorting. This function handles: - None values: returns None - String values: parses ISO format strings and converts to UTC-aware datetime - Datetime objects: converts naive datetimes to UTC-aware, and aware datetimes to UTC Args: dt: Datetime value (None, str, or datetime object) Returns: UTC-aware datetime object, or None if input is None or cannot be parsed NZz+00:00tzinfo) rrrdr rrrrutc astimezonerr)rdt_str parsed_dts rg_normalize_datetime_for_sortingr%s z"c 24++c2BRZZX.F ..v6I'%--X\\-B  &00>  "h 99 ::X\\:2 2}}X\\** N+  sA'C:= C::D  D asc sort_orderc rdvr|S|jdk(dtttfdtffd } t ||}|S#t $r0}t jddt||cYd }~Sd }~wwxYw) a Sort models by the specified field and order. Args: all_models: List of models to sort sort_by: Field to sort by (model_name, created_at, updated_at, costs, status) sort_order: Sort order (asc or desc) Returns: Sorted list of models )r8rrcostsr0descr9r.c\|jdi} dk(r |jddjS dk(r~|jd}t|}|^ s.tjj t jStjj t jS|S dk(r~|jd}t|}|^ s.tjj t jStjj t jS|S dk(rP|jdd xsd }|jd d xsd }||z}|d k(r s td Std S|S d k(r|jdd}|Sy)Nr^r8rrrrrinput_cost_per_tokenroutput_cost_per_tokenrz-infr0rF) rrrrmaxr rrror) r9r^r normalized_dtr input_cost output_cost total_costrr6rs rg get_sort_keyz"_sort_models..get_sort_key%sYY|R0 l "99\2.446 6  $# 5J;JGM$#LL(( (="--X\\-B !  $# 5J;JGM$#LL(( (="--X\\-B !  #(>BGaJ$..)@!DIK#k1JQ+2uU|Ef E   !~~j%8HOri)rr6zError sorting models by r\N)rr rrr8rpr}r)rsrrr sorted_modelsrr6s ` @rg _sort_modelsr%s g& F*G+DcN+s+Zz|WM  &&)A'"SQRVH'UVsA B%B=BB total_countc | t|}|dz |z}|dkDr| |z nd}||||z}tjd|d|d|d|d| |||||dS) aD Paginate models and return response dictionary. Args: all_models: List of all models page: Current page number size: Page size total_count: Total count (if None, uses len(all_models)) search: Search term (for logging) Returns: Paginated response dictionary rrzPagination: skip=z, take=z, total_count=z, total_pages=z , search=rr current_page total_pagesr)rnr}re)rsrrrrskiprpaginated_modelss rg_paginate_models_responser&s(*o 1H D-81_[LD()!K!$5 D6n[MXcWddmntmuv !""  ric K |jjjd|id{}|tjd|dgSt di|j }t}t|jdk(s&tjj|jvr|r|j!ng}||D]k} | j#d ij#d d} | (| j#d ij#dd} d } | d } n| |k(rd } | s[|j%| mnh|jD]Y} |r|j!| | ng}||D]8} | j#d ij#d d} | (|j%| :[ |jrtjj|jvrc|jj&j)dd|jiid{}|D]"}|j*} | s|j%| $g}|D]}|j#d i}|j#d d} |j#dd r|j/|K|j#dg}t1|t2r||vr|j/|| s| |vs|j/||S7#t$r0}tjd|dt|gcYd}~Sd}~wwxYw7#t$r/}tj,d|dt|Yd}~d}~wwxYww)a Filter models by team ID. Returns models where: - direct_access is True, OR - team_id is in access_via_team_ids Also searches config and database for models accessible to the team. Args: all_models: List of models to filter team_id: Team ID to filter by prisma_client: Prisma client for database queries llm_router: Router instance for config queries Returns: Filtered list of models rrNzTeam z not found in databasezError fetching team r\rr^r{FTr|r8rz(Error querying database models for team rrr^)rrrr}r|r:rrprrr&rnrvrAr}rrfrr~rrrsrrerrr)rsrr]rrrrteam_accessible_model_idsr4r9rrrr8rrrror`r^rs rg_filter_models_by_team_idr*&s. ,//AAMMg& N    ! ( (5 9O)P QI'F.*C*C*EF +.% K  1$  - - 3 3{7I7I I5?Z..0B  !# < 99\26::4F# % , ; ? ? 4 P %  ($(M"g-$(M -11(; < &,, @J))Z)Q  "$@E$yyr:>>tTJH+155h?@ @   !2288 @R@RR,..FFPP#dK,>,>%?@QI& <#,,-11(; <O+ZZ b1 >>$- >>/5 1  " "6 * )nn-BBG )4 0W@S5S  " "6 *  $==  " "6 *%+( }  &&)=gYbQ'QR d  ""6wir#a& J   sM+KK KMK*B:M%//1L$r:J:#3#3#55" #."9+rJ(+J ) ))3  * *6xj3q6(K   sA0D(+C.C,C.>A.D(,C.. D%7$D D( D%%D(z/v2/model/infozv2 - returns models available to the user based on their API key permissions. Shows model info from config.yaml (except api key and api base). Filter to just user-added models with ?user_models_only=true)rrrYrz!Specify the model name (optional)z%Only return models added by this userz.Return all models across all teams user is in.rz Page number)rge2z Page sizez3Search model names (case-insensitive partial match)z,Search for a specific model by its unique IDzaFilter models by team ID. Returns models with direct_access=True or teamId in access_via_team_idszLField to sort by. Options: model_name, created_at, updated_at, costs, statuszSort order. Options: asc, descuser_models_onlyinclude_team_modelsmodelIdteamIdsortBy sortOrderc Kttjsgd|d|dSt'tddtj j itjd{|(t||tttd{\} } nvtjtj} t | tgz } || Dcgc] }|d|k(s |} }t| |xsd ||tt| d{\} } |rt| t|t d{} |rt|tt| d{} t!| D]\}}t#|||nd t| |< | H| j%r8t'| | j%ttd{} t)| } | t)| } | r7| r#| j+dvrtdd| dt-| | | xsd} t/j0d| t3| |d{} t)| } t5| ||| |S77cc}w7W767775w)zQ BETA ENDPOINT. Might change unexpectedly. Use `/v1/model/info` for now. Nrrrxrr)rrrr]rr8r)rsrrrr]rr)rsrrr])rr]rrsF)r9rer)rsrr]r)rrrdzInvalid sortOrder: z. Must be 'asc' or 'desc'r)rsrrall_models: %s)rvr)rsrrrr)rr4r]r'r*rnrrrrrlrmrYrryrrUrrrrnrrr}rerr)rr9rrrerrrrrrrrsrrtrr`s rg model_info_v2r&s\!6!6    .EEKKL   ! ! ###/@!'% 0 * & &]]:#8#89  ! :, &J  %/L1\?e3K!LJL0N!sA#I&%I&%I& I AI& II#I&I#I&&I' I&IA*I&2I"3A;I&.I$/ I&I&I&I&I&"I&$I&z/model/streaming_metricsz1View time to first token for models in spend logs)rrrrY_selected_model_group startTimeendTimecKt5ttjjddt j |xs!tjtdz }|xstj}|j|jk(}|rd}nd}t}tjj||||d{}i}||D]} | d} | d } | d } d } |r| d }|} ||vri||<n| d }|} | d } ||vri||<t| }d| vr t| }d|vr|jdd}|j!|| || |< g}t#t%|j'd}|D]@}dt|i}||j'D] \}}|||< |j)|B|t+|dSy7w)Ninternal_errorrwryr"a SELECT api_base, model_group, model, "startTime", request_id, EXTRACT(epoch FROM ("completionStartTime" - "startTime")) AS time_to_first_token FROM "LiteLLM_SpendLogs" WHERE "model_group" = $1 AND "cache_hit" != 'True' AND "completionStartTime" IS NOT NULL AND "completionStartTime" != "endTime" AND DATE("startTime") = DATE($2::timestamp) GROUP BY api_base, model_group, model, request_id ORDER BY time_to_first_token DESC; a SELECT api_base, model_group, model, DATE_TRUNC('day', "startTime")::DATE AS day, AVG(EXTRACT(epoch FROM ("completionStartTime" - "startTime"))) AS time_to_first_token FROM "LiteLLM_SpendLogs" WHERE "startTime" BETWEEN $2::timestamp AND $3::timestamp AND "model_group" = $1 AND "cache_hit" != 'True' AND "completionStartTime" IS NOT NULL AND "completionStartTime" != "endTime" GROUP BY api_base, model_group, model, day ORDER BY time_to_first_token DESC; r:r9time_to_first_tokenr request_idr&https:///openai/rc |dSNrr^rs rgz)model_streaming_metrics..' dSTgrirdater all_api_bases)r]r?r*rnrr0r)rr-rr r&r query_rawrrr~rr8rrr)rrrr is_same_day sql_query_all_api_bases db_response_daily_entriesrt _api_baser`r unique_key _request_id_day_combined_model_namerir&rD model_keylatencys rgmodel_streaming_metricsr |'sI%<<BB!66   ?X\\^iQ.??I'G.."glln4K 0 .UN%((22()WKN% SJ":.I(F",-B"C J(6 ( n424N;/!%(! &01F&G#~-+-N4(#&v; Y&'*9~$11';'A'A*'Ma'P$   3 4?RN: &'; <1 S4  "f^%9%9%;AUVW! #CSX&E&4S&9&?&?&A +" 7#*i  + OOE "  #!.1  _ sCG&G#DG&z/model/metricsz>View number of requests & avg latency per model on config.yamlz gpt-4-32krcustomercPKttdddtj|xs%t j t tz }|xst j }||dk(rd}||dk(rd}d}t}tjj||||||d{}i} ||D]n} | d } | d } | d } | d }| | vri| | <t| }| d | vr t| }|d|vr|jdd}|j||| | |<p g}tt| j!d} | D]@}dt|i}| |j!D] \}}|||< |j#|B|t%|dSy7w)N Prisma Client is not initializedrrwryr undefinednulla SELECT api_base, model_group, model, DATE_TRUNC('day', "startTime")::DATE AS day, AVG(EXTRACT(epoch FROM ("endTime" - "startTime")) / NULLIF("completion_tokens", 0)) AS avg_latency_per_token FROM "LiteLLM_SpendLogs" WHERE "startTime" >= $2::timestamp AND "startTime" <= $3::timestamp AND "model_group" = $1 AND "cache_hit" != 'True' AND ( CASE WHEN $4 != 'null' THEN "api_key" = $4 ELSE TRUE END ) AND ( CASE WHEN $5 != 'null' THEN "end_user" = $5 ELSE TRUE END ) GROUP BY api_base, model_group, model, day HAVING SUM(completion_tokens) > 0 ORDER BY avg_latency_per_token DESC; r:r9r&avg_latency_per_tokenrrrc |dSr r^r s rgr zmodel_metrics..f(r rirr r )r]r?r0r)rr-rrr&rr rrr~rr8rrr)rrrrrr r r r r rtr r`r _avg_latency_per_tokenr rir&rDr r s rg model_metricsr (s 6!66   MX\\^i_.MMI'G'[08{2!IDUN%((22()WgxKN% PJ":.I(Fe$D%/0G%H ">)')t$#&v; $y)@'*9~$#/JBV4V';'A'A*'Ma'P$   3 49ON4 !5 6 P   "f^%9%9%;AUVW! #CSX&E&4S&9&?&?&A +" 7#*i  + OOE "  #!.1  K sB*F&,F$-C8F&z/model/metrics/slow_responsesz/View number of hanging requests per model_groupc ,Kttdddtj||dk(rd}||dk(rd}|xs%t j t tz }|xst j }tjjxst}t|}d}tjj|||||||d{}|9|D]4} | jd xsd } d | vr| j!d d } | | d <6|S7Aw) Nr rrwryr r ra SELECT api_base, COUNT(*) AS total_count, SUM(CASE WHEN ("endTime" - "startTime") >= (INTERVAL '1 SECOND' * CAST($1 AS INTEGER)) THEN 1 ELSE 0 END) AS slow_count FROM "LiteLLM_SpendLogs" WHERE "model_group" = $2 AND "cache_hit" != 'True' AND "startTime" >= $3::timestamp AND "startTime" <= $4::timestamp AND ( CASE WHEN $5 != 'null' THEN "api_key" = $5 ELSE TRUE END ) AND ( CASE WHEN $6 != 'null' THEN "end_user" = $6 ELSE TRUE END ) GROUP BY api_base ORDER BY slow_count DESC; r:rrr)r]r?r0r)rr-rrrr(rr!r(rr rr) rrrrrr rr r rowr s rgmodel_metrics_slow_responsesr! s(s= 6!66   '[08{2MX\\^i_.MMI'G 11DD , +/0IB&((22K (C +1rIY&%OOJ7: 'C O  ( !sCDDADz/model/metrics/exceptionsz7View number of failed requests per model on config.yamlcKttdddtj|xs%t j t tz }|xst j }||dk(rd} d}tjj|||||d{}g}t} ||D]} | jd d } | jd d } | jd i} | | d}|j| |j|| jD]\}}| j||t!| dS7w)Nr rrwryrr r a? WITH cte AS ( SELECT CASE WHEN api_base = '' THEN litellm_model_name ELSE CONCAT(litellm_model_name, '-', api_base) END AS combined_model_api_base, exception_type, COUNT(*) AS num_rate_limit_exceptions FROM "LiteLLM_ErrorLogs" WHERE "startTime" >= $1::timestamp AND "endTime" <= $2::timestamp AND model_group = $3 GROUP BY combined_model_api_base, exception_type ) SELECT combined_model_api_base, COUNT(*) AS total_exceptions, json_object_agg(exception_type, num_rate_limit_exceptions) AS exception_counts FROM cte GROUP BY combined_model_api_base ORDER BY total_exceptions DESC LIMIT 200; combined_model_api_basertotal_exceptionsrexception_counts)r9r$ )rexception_types)r]r?r0r)rr-rrrr r&rrJrrr~r)rrrrrr r r rir& rtr9r$ r% curr_rowrrs rgmodel_metrics_exceptionsr( (sg 6!66   MX\\^i_.MMI'G'[0I,&((229g'-D> D  D  DDz /model/infoz/v1/model/infolitellm_model_idc :Ktg tttjt}t dtt|}|j}t|dh}d|iSttddd i ttddd i |Ltj| }|td dd|di t|jd}d|giSg}tt }tg}n(tj#}tj%}t'|||} t)|j*||} t-| | |tt.j1ddt} t3| dkDrQg} | D],} tj5| }|| j7|.tt9j:| }ng}|D]}t|}t=j>d|d|iS#t $ri}YwxYww)a Provides more info about each model in /models, including config.yaml descriptions (except api key and api base) Parameters: litellm_model_id: Optional[str] = None (this is the value of `x-litellm-model-id` returned in response headers) - When litellm_model_id is passed, it will return the info for that specific model - When litellm_model_id is not passed, it will return the info for all models Returns: Returns a dictionary containing information about each model. Example Response: ```json { "data": [ { "model_name": "fake-openai-endpoint", "litellm_params": { "api_base": "https://exampleopenaiendpoint-production.up.railway.app/", "model": "openai/fake" }, "model_info": { "id": "112f74fab24a7a5245d2ced3536dd8f5f9192c57ee6e332af0f0512e08bed5af", "db_model": false } } ] } ``` rrrrr* rrxrzLLM Model List not loaded in. Make sure you passed models in your config.yaml or on the LiteLLM Admin UI. - https://docs.litellm.ai/docs/proxy/configsrzLLM Router is not loaded in. Make sure you passed models in your config.yaml or on the LiteLLM Admin UI. - https://docs.litellm.ai/docs/proxy/configs)rrdz Model id = z not found on litellm proxyTr)rrhrj)rgrhrjriF)rfrgrhrYrirrrr) rYrr rgrrprrrrrr'rget_deploymentr, rxrrorprrrgrrrrnrfrrlrmr}re)rr- r^_deployment_info_deployment_info_dictdeployment_inforsrjrhrfrgall_models_str_relevant_modelsr9 router_modelsin_place_models rg model_info_v1r7 P)s` #D'*@*@z*RSJ&) "  !1 ; ; = E145! -..r  q  #$33=M3N  "{+;*<Q# 7E&555GM( '' 6 7  %'78JJ$E.^DE/< J s J s/H(H E8H+AH HHHHr3 rc g}g}|D]}||vs|j||D]e}|||k7r |j|}|)|jtdi|jHt|g}|j|gtj +|D]&}|j tj vs d|_(|S)N)r)r providersTr^)rget_model_group_inforrrgpublic_model_groupsris_public_model_group) rr3 r model_groupsrxr9_model_group_infomodel_group_infomgs rg_get_model_group_inforA )s/1LM(  %   '( 2  "{e'; &;;;N  (    3 U6G6R6R6T U V2!      0 1 2 "". 0B~~!|A9+)#+$)!&$-  N/Dn+/L 6!+L L !!/ $s$A B A=&B4A?5 B?Bz/model/settingszMReturns provider name, description, and required parameters for each providerc Kg}tjD]2}|jt|tj|4|Sw)a? Used by UI to generate 'model add' page { field_name=field_name, field_type=allowed_args[field_name]["type"], # string/int field_description=field_info.description or "", # human-friendly description field_value=general_settings.get(field_name, None), # example value } rR)rfields)rg provider_listr ProviderInfoget_provider_fields) returned_listrs rgmodel_settingsrI *sQ$M))  22xP    sA A z/alerting/settingszFReturn the configurable alerting param, description, and current valuerc K t'tddtjji|j t jk7rAtdddjtjj|j itjjjddid{}|E|j9t|j}|jdi}|jd }ni}d}d d id d id d id d id d id d id d id d id d id }t j"}|j$j'}g}d} t(jd r$t+t(d t,r dt(d vrd} t/d|dd d| |dnddd} |j1| t2j4j7D]m\} } | |vs d} | |vrd} nd} t/| || d | j8xsd|j| d| | j:| dk(rdnd} |j1| o|S7w)Nrdrr{}, your role={}rrrrrrVBooleanInteger) slack_alertingdaily_report_frequencyreport_check_intervalbudget_alert_ttloutage_alert_ttlregion_outage_alert_ttlminor_outage_alert_thresholdmajor_outage_alert_thresholdmax_outage_alert_list_sizeFrTrN zhEnable slack alerting for monitoring proxy in production: llm outages, budgets, spend tracking failures.) field_name field_typefield_description field_value stored_in_dbfield_default_value premium_fieldrrS )r]r'r*rnrrr<rrunot_allowed_accessrrrr rrrr(rrrrrr/rr  model_fieldsrrrj)rrMdb_general_settings_dictalerting_args_dictalerting_values allowed_args_slack_alerting_slack_alerting_args_dict return_valis_slack_enabled _response_objrW field_info _stored_in_dbs rgalerting_settingsrk *s.EEKKL  ""&6&B&BB+22%88>>%//  !. 0 0 ? ? J J/0!K!&+>+J+J+V#'(;(G(G#H #;#?#?QS#T*B*F*Fz*R"9-#)9"5"()!4#Y/#Y/$*I#6)/(;)/(;'-y&9 L&7%N%NO / = = H H JJJ'J$d- &z2 2# # 01&9E$,8Te Mm$"3"@"@"F"F"H- J  %,0M// $ % &%' 3F;","8"8">B599*dK*$.$6$6&*CCD M   m ,'-( IsBI?D)I)A(Iz/queue/chat/completionsrb)rrYrc K i} |jd{}t|j|jt |j t j |d|d<t jd|tjddxstxs|xs|jdd|d<|jdd|j|j|d<d|vri|d<|j|dd<|j|dd <t |j }|jd d||dd <t|d d|dd <|j|dd<t|dd|dd<t|dd|dd<t|dd|dd<t|j|dd<t r t |d<t"r t"|d<t$r t$|d<t&r t&|d<t('t+ddt,j.j0it)j2d-i|d{}d|vr |ddurt5t7|||d !S|j8j;d"t|d#i|S77V#t<$r}t>jA|||$d{7tC|t*rYtEt|d%d&t|d'tFjHt|d(d)t|d*tJjL+tC|tDr|tEd,t|ztFjHt|d(d)tJjL+d}~wwxYww).N)rrWr%rproxy_server_requestzreceiving data: %scompletion_modelr9rr user_api_keyuser_api_key_metadata authorizationr%ruser_api_key_aliasrrrobject_permission_id!user_api_key_object_permission_idteam_object_permission_id&user_api_key_team_object_permission_idrirSrTrRr:rxrrrTrrrzx-litellm-prioritypriorityrsrAuthentication Error(rKrrwr#ryAuthentication Error, r^)'rrrrWrrlr}rerrrYrrrrrr`rarr_rr'r*rrschedule_acompletionr:rr%rJrprr|rr?r> auth_errorr0r)r r|r9rr_headersrirs rgasync_queue_requestr~ S+s Db \\^#w{{#nn09>>@IIdO (  #$ ""#7>  !3T : ' ' 'xx& W  88FD ! ).?.G.G.S,44DL T !!D +<+D+DZ(4E4N4NZ01,W5::< T '/Z#18 {D2 Z-.4E3L3LZ/03: y$4 Z/0AH 5tA Z<=FM :DF ZAB(+7;;'7Z$ "2D  &:D" # !0D  ,D   2C2Q2Q2W2W(X $88@4@@  h4!7$$&7%!% /     '')=s4 CS?T(UV]$~A  66/ATX7    a ' 8/DSVHA-NO$//a&1Q v/J/JK   > *G,s1v5 ++!Wf-,,    sdNJJH+JJ'J.N/*JNJJ N (NKCNN  N)rrc Kddlm}tt|j}t j d}|jdr|dz }n|dz }|ddlm }|td d Sddlm }|td d Sw) z Create Proxy API Keys using Google Workspace SSO. Requires setting PROXY_BASE_URL in .env PROXY_BASE_URL should be the your deployed proxy endpoint, e.g. PROXY_BASE_URL="https://litellm-production-7002.up.railway.app/" Example: r)ui_link UI_USERNAMErz sso/callbackz /sso/callback) HTMLResponseF)show_deprecation_bannerr{r$r#) litellm.proxy.proxy_serverr rrrrrrdfastapi.responsesr r)r r redirect_url ui_usernamer s rgfallback_loginr +s3"#g&6&6"78L))M*KS!& '  3'FTW   3'FTW  sB Bz/logincPKddlm}m}ddlm}|j d{}t |jd}t |jd}|||ttd{}||tt}ddl } | jtt|tt td } |t |j } | j#d r| d z } n| d z } | d z } t%| d} | j'd| | S77w)Nrauthenticate_usercreate_ui_token_objectrusernamerr rrQr] login_resultrrBHS256 algorithmrui//ui/?login=successi/)rr#rr)litellm.proxy.auth.login_utilsr r rrformrrrQr]rrBjwtencoderrrrdr9 set_cookie) r r r rr r rr returned_ui_token_objectr jwt_tokenlitellm_dashboard_uiredirect_responses rgloginr +s# Y2 D488J'(H488J'(H+# L 6!)!   T+, S*I*#g.>.>*?@$$S)%&,,)-AsS  WI > O s#"D&D"AD&4D$5B.D&$D&z /v2/loginc LKddlm}m}ddlm} |j d{}t |jd}t |jd}|||ttd{}||tt}ddl } | jtt|tt td } |t |j } | j#d r| d z } n| d z } | d z } t%d| it&j(} | j+d| | S77#t,$r} t/j0dj3t | t5| t6r| t5| t8rUt7t;| dt | t<j>t;| ddt;| dt&j@t | }t7|t<j>dt&j@d} ~ wwxYww)Nrr r r rr r r r rr r r r r rrz>litellm.proxy.proxy_server.login_v2(): Exception occurred - {}rrrwr#ry)!r r r rrrrrrQr]rrBr r rrrrdr7r0 HTTP_200_OKr rpr}rrurr?r'rr>r| r))r r r rrr rr r r r r json_responserrs rglogin_v2r ,s Y2=\\^#txx +,txx +,.!'   $:%-%$  JJ / 0 j ! .c'2B2B.CD  ( ( - E ) F *  00$#%9:**    WI >K$ D && L S SA  a (G = ) 8SV4$//a&1Q v/T/TU  q6(I !$//::  #sMH$D8D3AD85D66Blitellm.proxy.proxy_server.login_v3(): Exception occurred - {}rrrwr#)'r r r rrrrr?r>not_found_errorr0HTTP_404_NOT_FOUNDrrrQr]rBr r rrrrdsecrets token_urlsaferasync_set_cacherr7r rpr}rrurr'rr| r))r r r rrr rr r r r r r'r cache_valuerrs rglogin_v3r h,s Y2P##$78 b$44)..  \\^#txx +,txx +,.!'   $:%-%$  JJ / 0 j ! .c'2B2B.CD  ( ( - E ) F *  00$$R(!$( );OP  (#33[b4  %44[b5  !4**  [$ B   && L S SA  a (G = ) 8SV4$//a&1Q v/T/TU  q6(I !$//::  #stJ;AG%G&AG5G6C GG  G"G #!GJ;GG G G J8CJ33J88J;z/v3/login/exchangecLK tjds+tdtjdt j |jd{}|jd}|s+tdtjdt jd|}ttj|d{}ntj|d{}|rt|ts+tdtjdt jttj!|d{ntj!|d{t#|d |d d t j$ }|j'd |d  |S7G777f7I#t$rt($rf}t+j,dj/t1|tt1|tjdt j2d}~wwxYww)Nr)zQ/v3/login/exchange is only available on workers with control_plane_url configuredryr'zMissing 'code' parameterr rzInvalid or expired login coderr r r rzGlitellm.proxy.proxy_server.login_v3_exchange(): Exception occurred - {}rw)rrr?r>r r0r rr| rrrrrrHTTP_401_UNAUTHORIZEDasync_delete_cacher7r r rpr}rrurr))r rr'r cached_datar rs rglogin_v3_exchanger ,s= ##$78 k$44)..  \\^#xx 2$//00  "$(  ( 1 A Ai A PPK 2 B By B QQK*[$"? 7$//11    (#6696E E E$77I7F F F$$W- +N ;**     WK4H IM$QQ F F    && U \ \A  F ++66    sH$AF)FA&F)>F!?F)F#A F)>F%?F)F'?F)H$F)!F)#F)%F)'F))H!;A!HH!!H$z/onboarding/get_token invite_linkcTKddlm}t+tdtj dt jt'tddtjji tjjjd |i d{}|td dd i t j"j%j'}|j(j'}||krtd ddi |j*durtd ddi tjj,jd|j.i d{}|td ddi |j0}t3d#ddi|j4t6t j8giid|j.dd d{}|d} t;t=|j>} | jAdr| dz } n| dz } ddl!} tE} ||j.| |j0|j4dtFtHjKdd| tM } | jOtQtR| td }| d!jU||z } | ||d"S7 7G7w)$z - Get the invite link - Validate it's still 'valid' - Invalidate the link (prevents abuse) - Get user from db - Pass in user_email if set r)ReturnedUITokenObjectNzMaster Key not set for Proxy. Please set Master Key to use Admin UI. Set `LITELLM_MASTER_KEY` in .env or set general_settings:master_key in config.yaml. https://docs.litellm.ai/docs/proxy/virtual_keys. If set, use `--detailed_debug` to debug issue.rQryrxrrr{r%Invitation link does not exist in db.Invitation link has expired.T&Invitation link has already been used.rUser does not exist in db.rrzlitellm-dashboard) rrkey_max_budgetrvrr~rrrrrz ui/onboardingz/ui/onboardingusername_passwordlitellm_key_header_name Authorization) rr user_emailr login_methodrBauth_header_name(disabled_non_admin_personal_key_creationr r r z?token={}&user_email={}) login_urlrr r^)+litellm.types.proxy.ui_ssor rQr?r>r| r0r)r]r'r*rnrrlitellm_invitationlinkrrgrget_utc_datetimer expires_at is_acceptedrwrr rrr%max_ui_session_budgetrrrrdr rrBrrrr rrru)r r r invite_obj utc_now_dateexpires_at_dateuser_objr rirr r r r r s rg onboardingr -sAP ++66   .EEKKL  %''>>JJ[!KJW.U$V  ==11388:L ++002O%W.L$M  %EF  #%%77CC*,,-DHW.J$K  $$J+  "++3%;;''*   H 7 C)#g.>.>*?@$$S)/ 00 56- 5  &&$$(!)-- % 2Z-/   T+,I 5<J&?C"J(#J(&J(z/onboarding/claim_tokenrcKt'tddtjjitj j jd|jid{}|tddditjjj}|jj}||krtddd i|jd urtddd i|j|jk7r3tddd j!|j|jit#|j$}tj j&j)d |jid|id{}|tddditjj}tj j j)d|ji||d |jdd{|r(t+|dr|j,j/dd|S7775w)a& Special route. Allows UI link share user to update their password. - Get the invite link - Validate it's still 'valid' - Check if user within initial session (prevents abuse) - Get user from db - Update user password This route can only update user password. Nrxrrr{rr r r Tr zwInvalid invitation link. The user id submitted does not match the user id this link is attached to. Got={}, Expected={}rrrr ) accepted_atrr r__dict__)r]r'r*rnrrr rinvitation_linkrgrr r r r rrurrrwrJrr r)rr r r hashed_pwr rs rgclaim_onboarding_linkr z-s_.EEKKL  %''>>JJT))*KJW.U$V  ==11388:L ++002O%W.L$M  %EF  T\\)SZZLL*"4"4  dmm,I"%%77>>*,,-Z4K?HW.J$K  ==113L    1 1 8 8T))*'&$,,   9 GHj1j$/ OuFs8A'I)I*D)IIA>* % % KK T 2 & &)Ej\'R S  $  Z,  [  )E(  ii BGG< +Ii):;J .,7I;YGL )=)=>U)V 77>>) $ lC C$$YK'U V !  ww~~j!J<@@34 G U K1144!3'L*--i88H##s**d+.qGGH,,-.$J<HH$L\JJI,??A ) % ( (4ZL1#FQR %J  %^9.. G & &)Ei[PRSTRU'V W F F GsCM9/K25EM96L;L-L;'L/L;M9 L;$M92L*L%M9%L**M9-L;/L84L;; M6'M1+M6,M91M66M9z /get_faviconcKddlm}tjj tjj t }tjj|ddd}tjdd}|s9tjj|r t|d Std d |jd r ddl m}ddlm}||j"ddi}|j%|d{}|j&dk(r0|j(j%dd}||j*|St-j.d||j&tjj|r t|d Std d tjj|r t|d Stjj|r t|d Std d 7#t$rt0$rZ} t-j2d|| tjj|rt|d cYd} ~ Std d d} ~ wwxYww)z$Get custom favicon for the admin UI.rr+r,r-z favicon.icoLITELLM_FAVICON_URLrz image/x-iconrw rxzDefault favicon not foundrr r r r r r Nr{ content-type)r$rz*Failed to fetch favicon from %s: status %szFavicon not foundz%Error downloading favicon from %s: %s)r r+rrXrrr r:rr9r6r'r@r r r r r rr#r%r$r}r|rpre) r+r|default_favicon favicon_urlr r r ri content_typers rg get_faviconr *.s+''//"''//(";>/ *NK K4OPP56 M U K1144!3'L*--k::H##s*'//33NNS $,,+ %,,@(( 77>>/2'NSS#>+ & G G 77>>/ *NK K4GHH=;    M & &7a ww~~o.#OOOC8KL L  MsjC I96H HAH I9A H I9 H "A%I9H I6AI1I6I9$ I11I66I9z/invitation/newz Invite Links)rrYrMrcK ddlm}t'tddtj j i|jtjk(xs#t|tttd{}|sAtdddjtjj |ji|jtjk7r?t|j |ttt d{}|stddd i||| d{}|S77+7 #t"$r}t%|d}~wwxYww) % Allow admin to create invite links, to onboard new users to Admin UI. ``` curl -X POST 'http://localhost:4000/invitation/new' -H 'Content-Type: application/json' -d '{ "user_id": "1234" // 👈 id of user in 'LiteLLM_UserTable' }' ``` r)create_invitation_for_userNrdrrrerK )target_user_idrr]rrzGYou can only create invitations for users in your organization or team.)rr)0litellm.proxy.management_helpers.user_invitationr r]r'r*rnrrr<rrrrrur^ rrrpr)rrr has_access can_inviterirs rgnew_invitationr f.s_*9+  !2!I!I!O!OP   ' '+;+G+G G /"3+#5"3  /66)<<BB)33   & &*:*F*F F4#||"3+#5"3  J# #!j4/  O( +'**+s`EA0D24D,5B D2D."D2%D0&D2+E,D2.D20D22 E ; EE  Ez/invitation/info invitation_idcKt'tddtjji|j t jk7rAtdddjtjj|j itjjjd|id{}|tdddi|S7w) r NrdrrrK r{r-Invitation id does not exist in the database.) r]r'r*rnrrr<rrur^ rr r)r rris rginvitation_infor .s..EEKKL  ""&6&B&BB+22%88>>%//  #%%<<HH]#IHLM   OsB;C=C>Cz/invitation/updatec Kt'tddtjji|j (tdddj |j itjj}tjjjd|ji|j|j|||j d d{}|tddd i|S7w) uV Update when invitation is accepted ``` curl -X POST 'http://localhost:4000/invitation/update' -H 'Content-Type: application/json' -d '{ "invitation_id": "1234" // 👈 id of invitation in 'LiteLLM_InvitationTable' "is_accepted": True // when invitation is accepted }' ``` Nrdrrrxz'Unable to identify user id. Received={}r{)r{r r rrrr )r]r'r*rnrrrurgrr rr rJr r )rrrris rginvitation_updater .s 2.EEKKL    (BII%--  ==113L"%%<<CCT''($$++'&+33  D  HLM   O! sCC: C8!C:z/invitation/deletecKt'tddtjji|j t jk(}t|tttd{}|sC|sAtdddjtjj|j i|ry|swtjjjd|j id{}|tddd i|j"|j$k7rtd dd itjjj'd|j id{}|tddd i|S777w) u  Delete invitation link ``` curl -X POST 'http://localhost:4000/invitation/delete' -H 'Content-Type: application/json' -d '{ "invitation_id": "1234" // 👈 id of invitation in 'LiteLLM_InvitationTable' }' ``` NrdrrrerK r{rr z=Organization admins can only delete invitations they created.)r]r'r*rnrrr<rrrrrur^ rr rr rrdelete)rris_proxy_adminis_other_admin invitationris rginvitation_deleter (/s0.EEKKL  '004D4P4PPN5+#-+ N .+22%88>>%//  n(++BBNN++,O    !PQ   $5$=$= =\ #%%<<CCT''(DHLM   O[( s8A)F+F,BF2F3A6F)F*FFFz/config/updatez config.yaml config_infoc xK |jtjk7r tddddl} t t dtdurtdd d i|jd }t j|}|jD]H\}}|d k(s t jjjd |i||dd|idd{Jtjd{}t!j"d||j$|j'di|j$j)d }|d}|jD]G\}}|dk(r8d|vrddgi}n.t+|dt,rd|dvr|dj/d|||<I||d<|j0l|j'di|j0} | jD]\}}t3|} | | |<|d} | jD] \}}| || |<|j4|j'di|j4} i| |d|d<d| vred|dvr^t+|ddt,rHt+| dt,r5t7| d} |dd| z}t-t9|}||dd<tj;|d{tj=t t>d{ddiS777757#t $r}t!j@djCtE|t!j"tGjHt+|trYtKtM|d d!tE|d"tNjPtM|d#d$tM|d%tRjT&t+|tJr|tKd'tE|ztNjPtM|d#d$tRjT&d}~wwxYww)(z For Admin UI - allows admin to update config via UI Currently supports modifying General Settings + LiteLLM settings r z#Only proxy admins can update configrrNzNo DB ConnectedTrxrCSet `'STORE_MODEL_IN_DB='True'` in your env to enable this feature.rrrrr rrzLoaded config: %srrrrrrrrlrrr`zConfig updated successfullyzBlitellm.proxy.proxy_server.update_config(): Exception occured - {}rry rKrrwr#ryrz )+rr<rr'rr]rprrjsonify_objectrrrrrrr}rerrarrrrrrrrGr&rrzrrrurrrr?rr>r| r0r)r rrupdated_settingsrrr~updated_general_settings_existing_settings_updated_environment_variablesr_existing_env_variablesupdated_litellm_settings$updated_success_callbacks_normalizedcombined_success_callbackrs rg update_configr% y/sD  & &*:*F*F FC8]^ ^   -. . D (b '+++>(778HI$**, DAq%%#&&55<<'+121"E#0!"4= $..00""#6?  ' ' 3   0" 5'2'C'C'H'H!(I( $"((:!; 0668 *1..!);;.87)-D*#$6z$BDI"* D1"6Q"?4C.q1 D'--D&E #6<<> O1-KA-N'* O  ' ' 3   0" 5'2'C'C $***+,*F% &#&>>&&1C*DD-./ABD !9:L!MtT;S01CD<8123EF>?.15S9R5S0T-2-.* &&&&999))';L*   899m1J :  "" P W WA  ""9#7#7#9: a ' 8/DSVHA-NO$//a&1Q v/J/JK   > *G,s1v5 ++!Wf-,,   ! syP:BL8LL L4L5G(LL&LLL P: LLLL P7!DP22P77P:z/config/field/updatec K t'tddtjji|j t jk7r'tddtjji|jtjvr(tdddj|ji tdi|j|jitj j"j%ddi d{}| |j&i}nt)|j&}|j||j<tj j"j+ddidt-j.|d d t-j.|id  d{}|S#t$r2tdddjt|jiwxYw77Fw)z= Update a specific field in litellm general settings NrdrrInvalid field={} passed in.z)Invalid type of field value={} passed in.rrrrr rrr^)r]r'r*rnrrr<rr^ rW r.r_ rurZ rprVrrrr rrrrFrrrMrris rgupdate_config_general_settingsr) 0s.EEKKL  ""&6&B&BB.AAGGH   3@@@:AA$//RS   D$2B2B CD!. 0 0 ? ? J J/0!K! "&9&E&E&M 3 ? ?@)-(8(8T__%"%%44;;/0%7 ScHde$djj1A&BC <H OC  DKK))*   s=B7G5:!F3/G5 G1 B!G5,G3-G53;G..G53G5z/config/field/inforW cK t'tddtjji|j t jk7r'tddtjji|tjvrtdddj|itjjjddid{}| |jtdddj|it!|j}||vrt#||| Stdddj|i7ww) Nrdrrr' rrrzField name={} not in DB)rW rZ )r]r'r*rnrrr<rr^ r.r_ rurrrr rr,)rW rrMrs rgget_config_general_settingsr+ d0st .EEKKL  ""&6&B&BB.AAGGH  .;;;:AA*MN  !. 0 0 ? ? J J/0!K! "&9&E&E&M6==jIJ   3 ? ?@ ) )"%3CJ3O  !:!A!A*!MN %sCEE A8Ez /config/listrcK t'tddtjji|j t jk7rAtdddjtjj|j itjjjddid{}|"|jt|j}ni}d d id d id d id d id d id d id d id d id did did d id d id did }g}tj j#D]\}}||vs ||d }|d k(r/|dk(rt$g} ng} | D]} | j&j#D cgc]3\} } t)| | j*dt,j/| dd5} } } d}| j j#D]6\} }t1|dr |j2|j2| |_|dz }8d}||vrd}n |t,vrd}t7|||d |j2xsdt,j/|d||j8| }|j;|Id} d}||vrd}n |t,vrd}t,j/|d}| ||vr||}t7|||d |j2xsd|||j8| }|j;||S7:cc} } ww)z List the available fields + current values for a given type of setting (currently just 'general_settings'user_api_key_dict: UserAPIKeyAuth = Depends(user_api_key_auth),) NrdrrrK rrrrVrM PydanticModelrL Stringr ) rOrPmax_request_size_mbmax_response_size_mbr.rrQr/mcp_internal_ip_rangesmcp_trusted_proxy_rangesalways_include_stream_usage!forward_client_headers_to_llm_apimcp_required_fieldsr.r)rW rX rY r\ r[ rrrTF)rW rX rY rZ r[ r\ nested_fields)r]r'r*rnrrr<rrur^ rrrr rr.r_ rr=rCr2r^rrrrrY r/rjr)rrrMr` rc rf rW ri typed_dict_typepydantic_class_listpydantic_class sub_fieldsub_field_typer6 rpsub_field_inforj rh _field_values rgget_config_listr> 0s  .EEKKL  ""&6&B&BB+22%88>>%//  !. 0 0 ? ? J J/0!K!&+>+J+J+V#'(;(G(G#H #% #))!4)/(; & 2!' 3#)?";$i0(. ':06/A#)6"2%+V$4(. ':.4i-@ &/L J"7"D"D"J"J"LN1 J  %+:6v>O/1!99+E*F'*,'&9*5N:H9W9W9]9]9_ %6I~$'0'5'>'>.00@0D0DYPT0U)-  %M %C(44::< !!&$NMB . : : F3A2L2L* #/q !%)M!%==(, #'77(- $.#-#/ #;F#C*4*@*@*FB$4$8$8T$J%2,6,>,>&3%M%%m4U*5Z!% $ !99$(M#33$)M/33JE 'J:R,R#;J#GL *)+J7?&0&<&<&B ,!.(2(:(:"/! !!-0]N1` WR %s-BK8?A?L?AL?8K; 7EL;Lz/config/field/deletecK t'tddtjji|j t jk7rAtdddjtjj|j i|jtjvr(tdddj|jitjjjddi d{}| |j (tddd j|jit#|j }|j%|jdtjjj'ddidt)j*|d d t)j*|id d{}|S77w)z| Delete the db value of this field in litellm general settings. Resets it to it's initial default value on litellm. NrdrrrK r' rrrzField name={} not in configrr rr)r]r'r*rnrrr<rrur^ rW r.r_ rrrr rrrrrFr( s rgdelete_config_general_settingsr@ 71s .EEKKL  ""&6&B&BB+22%88>>%//   3@@@:AA$//RS  !. 0 0 ? ? J J/0!K! "&9&E&E&M:AA$//RS   3 ? ?@$/"%%44;;/0%7 ScHde$djj1A&BC <H O3"s%DGGC G G GGz/config/callback/deletecKt'tddtjji|j t jk7rAtdddjtjj|j itdurtdddi tjd{}|jj}|jd i}|jd g}||vrtd dd |d i|j!|||j#d id <tj%|d{tj'tt(d{d|||t+j,j/dS77W72#t$rt0$r}t3j4dt7|t3j8t;j<t?dt7|zt@jBdtDjFd}~wwxYww)z> Delete specific logging callback from configuration. NrdrrrK Trxr rrlrxz Callback 'z#' not found in active configurationr rzSuccessfully deleted callback: )r`removed_callbackremaining_callbacks deleted_atzClitellm.proxy.proxy_server.delete_callback(): Exception occurred - zError deleting callback: callback_namery)$r]r'r*rnrrr<rrur^ rrrrE rrremoverarrzrrr-rrpr}rrrerrr?r>internal_server_errorr0r))rrr~rE rrrs rgdelete_callbackrH 1s\.EEKKL  ""&6&B&BB+22%88>>%//  $^  2 #..00**002 "::&8"=,001CRH  1 1z-8[\    /  ,b1  &&&&999))';L*   9H -#4",,.224   ;1, :     ""QRUVWRXQY Z  ""9#7#7#9:/#a&8 66!66    sbB#I &F2=F,>BF2F.&F2=F0>-F2+I ,F2.F20F22IA?III z/get/config/callbacks)rrrYc K ddl}t}tjd{}|j di}|j di}|j di}|j dg}|j dg}|j dg}d } | |}| |}| |}g} |D]} | j t | d | |D]} | j t | d | |D]} | j t | d | |j d g} g} d| vrdg}i}|D]D}|j |d}|tjdd}|||<3t||}|||<Ftjj}tjj}tjj}| j d|||dgd}i}|D].}|j |d}|d||<t||}|||<0| j d|dti}ntj!}d | | ||dS78#t"$r}t%j&dj)t+|t-|t.rYt1t3|ddt+|dt4j6t3|ddt3|dt8j:t-|t0r|t1dt+|zt4j6t3|ddt8j:d}~wwxYww)z~ For Admin UI - allows admin to view config via UI # return the callbacks and the env variables for the callback rNrrrrlrr c4t|tr|gS|gS|SrP)rr)rns rgnormalize_callbackz&get_config..normalize_callback1s$(C( z!! Orirrsuccess_and_failurerrSLACK_WEBHOOK_URLr/)r variables active_alertsalerts_to_webhook) SMTP_HOST SMTP_PORT SMTP_USERNAME SMTP_PASSWORDSMTP_SENDER_EMAILTEST_EMAIL_ADDRESSEMAIL_LOGO_URLEMAIL_SUPPORT_CONTACTemail)rrN )r0r alertsravailable_callbacksz?litellm.proxy.proxy_server.get_config(): Exception occured - {}rry rKrrwr#ryrz )r AllCallbacksrrrrrHrrrrr(r_all_possible_alert_typesrr get_settingsrpr}rrurrr'r?rr>r| r0r)rall_available_callbacksrrr)r_success_callbacks_failure_callbacks_success_and_failure_callbacksrK _data_to_return _callback _alerting alerting_data _slack_vars_slack_env_vars_var env_variabler_decrypted_value_alerting_types_all_alert_types_alerts_to_webhook _email_vars_email_env_vars_router_settingsrs rgrr1s\ "..(3355 'OO,>C'OO,>C +0G L.223ErJ.223ErJ):)>)>{B)O& 00BC/0BC); ** & , I  " " I7LM   , I  " " I7LM   8 I  " " 46K  &))*b9  i #K!O# =488tD 'YY':DAF,2OD)(<*($-=OD) =0GGSSO!99SSU "99NN   #!0%4);     9D044T4@L#(,%$8lPT#U (8% 9 ,    ! )668  (#/#:   y6F  && M T TA  a ' 8/DSVHA-NO$//a&1Q v/J/JK   > *G,s1v5 ++!Wf-,,    s:M%I$I!H6I$ M!I$$ M-C*MMMz /config/yamlcKddiSw)a This is a mock endpoint, to show what you can set in config.yaml details in the Swagger UI. Parameters: The config.yaml object has the following attributes: - **model_list**: *Optional[List[ModelParams]]* - A list of supported models on the server, along with model-specific configurations. ModelParams includes "model_name" (name of the model), "litellm_params" (litellm-specific parameters for the model), and "model_info" (additional info about the model such as id, mode, cost per token, etc). - **litellm_settings**: *Optional[dict]*: Settings for the litellm module. You can specify multiple properties like "drop_params", "set_verbose", "api_base", "cache". - **general_settings**: *Optional[ConfigGeneralSettings]*: General settings for the server like "completion_model" (default model for chat completion calls), "use_azure_key_vault" (option to load keys from azure key vault), "master_key" (key required for all calls to proxy), and others. Please, refer to each class's description for a better understanding of the specific attributes within them. Note: This is a mock endpoint primarily meant for demonstration purposes, and does not actually provide or change any configurations. helloworldr^)r s rgconfig_yaml_endpointru 2s. W sz/reload/model_cost_mapc K|jtjk7rtdd|j t tddddlm}tj}|| }|t_ ttj| tj}|jatj j"j%d d i d{}d}|r'|j&r|j&j)d}tj j"j+d d id t-dddddt-|ddidd{|r t/|nd}t1j2d|d|dd||jdS77F#t4$r?}t1j6dt9|tddt9|d}~wwxYww)z ADMIN ONLY / MASTER KEY Only Endpoint Manually reload the model cost map from the remote source. This will fetch fresh pricing data from the model_prices_and_context_window.json file. r 2Access denied. Admin role required. Current role: rNrx!Database connection not availablerrrrrrrrTrrr rrzCModel cost map reloaded successfully in current pod. Models count: z"Price data reloaded successfully! z models updated.r)r`r0 models_count timestampz!Failed to reload model cost map: )rr<rr'r]rrrgrrrOrrrrrrrrr rrr'rnr}rdrprr) rrrrrexisting_configexisting_intervalry rs rgreload_model_cost_mapr} 2s+ ""&6&B&BBGHYHcHcGde  @  (K  U$77/4FG/,.   0BC ( %1%;%;%="!. 0 0 ? ? K K!?@!L!  ! :: / ; ; ? ?@P Q --44!?@#A#-+/F$":+IYZ(%//1   ;  8  &&)J3q6('ST&GAx$P   sO8G?B0F4+F0,A?F4+F2,AF4/G?0F42F44 G<=:G77G<<G?z/schedule/model_cost_map_reloadhoursc K|jtjk7rtdd|j|dkr tdd t tdd tj j jd d id t|d d ddt|d d idd{tjd|dd|dd|tjjdS7H#t$r?}tjdt!|tddt!|d}~wwxYww)z ADMIN ONLY / MASTER KEY Only Endpoint Schedule periodic reload of the model cost map. This will create a background job that reloads the model cost map every specified hours. r rw rrrdHours must be greater than 0Nrxrx rrFrrr rrz*Model cost map reload scheduled for every  hoursrr`r0rrz z*Failed to schedule model cost map reload: rr<rr'r]rrrr'r}rdrrrrprrr~ rrs rgschedule_model_cost_map_reloadr 2s"""&6&B&BBGHYHcHcGde   z4RSS*  (K  --44!?@#A#-+0%H$":+0%H$ 5   " !!8v F DE7&Q#!*446   + 6  &&8Q A ?AxH   >A E A#C;0C91AC;8E9C;; E:D>>EEcK|jtjk7rtdd|j t tddtj j jddi d{tjd d d tjjd S7?#t$r?}tjd t|tdd t|d}~wwxYww)zp ADMIN ONLY / MASTER KEY Only Endpoint Cancel the scheduled periodic reload of the model cost map. r rw rNrxrx rrrz(Model cost map reload schedule cancelledrr`r0rz z(Failed to cancel model cost map reload: rr<rr'r]rrr r}rdrrrrprrrrs rgcancel_model_cost_map_reloadr B3s""&6&B&BBGHYHcHcGde    (K  --44!?@5    !!"LMB!*446     &&6s1vh ? &NsSTvh$W   ;8D AB?=B=>>B?<D =B?? D:DDD z&/schedule/model_cost_map_reload/statuscK|jtjk7rtdd|j t j dt tt j ddddddStjjjd d i d{}| |jt j d dddddS|j}|jd }|t j ddddddStj}d}t T tjt }||z }|j!dz }||kr|t#|zj%}d|t |dS7#t&$r"} t j(d| Yd} ~ 3d} ~ wwxYw#t&$r?} t j*dt-| tddt-| d} ~ wwxYww)zo ADMIN ONLY / MASTER KEY Only Endpoint Get the status of the scheduled model cost map reload job. r rw rz4Checking model cost map reload status. Last reload: N/No database connection, returning not scheduledF scheduledrlast_runnext_runrrrz,No model cost map reload configuration foundr/No interval configured, returning not scheduledrr~ rTz,Failed to get model cost map reload status: rx)rr<rr'r}rdrr]rrrr rrrrrrrrpr|rr rrr~rrr rrrrs rg get_model_cost_map_reload_statusr s3sM""&6&B&BBGHYHcHcGde  H  !!BC]B^ _   % %&W X""&   ,..==II!?@J    M$=$=$E % %&T U""&   **$45  ! % %&W X""&    (  & 1 U#+#9#9:T#U )58H)H&*@*N*N*PSW*W'*^;(9>+JJik,2   U N U$,,/OPQs-STT U  &&:3q6( C A#a&J   8H=F=8H9/F=(F )-F=H:F=HF=/AF F= H F= F:F50F=5F::F== H:HHHz/model/cost_map/sourcecK|jtjk7rtdd|j ddlm}|}t jrtt jnd}i|d|iS#t$r?}tjdt|tddt|d }~wwxYww) a ADMIN ONLY / MASTER KEY Only Endpoint Returns information about where the current model cost/pricing data was loaded from. Response fields: - source: "local" (bundled backup) or "remote" (fetched from URL) - url: the remote URL that was attempted (null when env-forced local) - is_env_forced: true if LITELLM_LOCAL_MODEL_COST_MAP=True forced local usage - fallback_reason: human-readable reason why remote failed (null on success) - model_count: number of models in the currently loaded cost map r rw rr)get_model_cost_map_source_info model_countz*Failed to get model cost map source info: rxN) rr<rr'rr rgrrnrpr}rr)rr source_infor rs rgget_model_cost_map_sourcer 3s*""&6&B&BBGHYHcHcGde    56 181C1Cc',,-   ;    &&8Q A ?AxH   s(8C?A;:C; C:B>>CCz/reload/anthropic_beta_headersc K|jtjk7rtdd|j t tddddlm}|}tj}|ja tjjjd d i d{}d}|r'|jr|jjd }tjjj!d d id t#dd dddt#|d didd{t%d|j'D}t)j*d|d|dd||jdS77W#t,$r?}t)j.dt1|tddt1|d}~wwxYww)z ADMIN ONLY / MASTER KEY Only Endpoint Manually reload the Anthropic beta headers configuration from the remote source. This will fetch fresh configuration from the anthropic_beta_headers_config.json file. r rw rNrxrx rrrrrrTrrr rrc3*K|] }|dvsd yw))rrrNr^rs rgrOz0reload_anthropic_beta_headers..I4s A5X,XA s zOAnthropic beta headers config reloaded successfully in current pod. Providers: zs1vh G Ec!fXN   r z./schedule/anthropic_beta_headers_reload/statuscK|jtjk7rtdd|j t j dt tt j ddddddStjjjd d i d{}| |jt j d dddddS|j}|jd }|t j ddddddStj}d}t T tjt }||z }|j!dz }||kr|t#|zj%}d|t |dS7#t&$r"} t j(d| Yd} ~ 3d} ~ wwxYw#t&$r?} t j*dt-| tddt-| d} ~ wwxYww)zw ADMIN ONLY / MASTER KEY Only Endpoint Get the status of the scheduled Anthropic beta headers reload job. r rw rz+JJik,:   ] V U$,,/OPQs-STT U  &&B3q6( K I#a&R   r )rYc Kyw)NzLiteLLM: RUNNINGr^r~s rghomer <5s sz/routesc&Kddlm}g}tjD]o}t |dd}|#|j |j ||5t|dsBt|dsO|j |j|qd |iSw) zD Get a list of available routes in the FastAPI application. r) GetRoutesriN)r endpoint_routerrX)r r) %litellm.proxy.common_utils.get_routesr rrrrget_app_routesrget_routes_for_mounted_app)r rr r s rg get_routesr A5s @ F M  D9  % MM((#1) UE "wuf'= MM)>>U>K L M f sAB B*'Bz/{mcp_server_name}/mcp)GETrUPUTDELETEPATCHOPTIONSHEAD)methodsmcp_server_namec K ddlm}ddlm}ddlm}|j |}|j||}|tdd|d  t|j}d ||d <dd l m }dddgfd} |||j| d{ddlm} D cic]$\} } | j!| j!&} } } | | | j#ddS7Wcc} } w#t$r}|d}~wt$$rB}t'j(d|dt+|tddt+| d}~wwxYww)z5Handle dynamic MCP server routes like /github_mcp/mcprr)IPAddressUtils)MCPAuth) client_ipNrxz MCP server 'z ' not foundrz/mcp/rX)handle_streamable_http_mcpFrir{cK|ddk(rd|d|jdgy|ddk(r|jddz yyw) NrVzhttp.response.startTr0r%zhttp.response.bodyrrirK)r` response_bodyresponse_headersresponse_startedresponse_statuss rg custom_sendz&dynamic_mcp_route..custom_send5s\v"77#' ")("3#*;;y"#= $88VS!99 9sAA)receivesendr r zapplication/json)r$r#r%rz%Error handling dynamic MCP route for r\rxzInternal server error: )rr#litellm.proxy.auth.ip_address_utilsr litellm.types.mcpr get_mcp_client_ipget_mcp_server_by_namer'rr_-litellm.proxy._experimental.mcp_server.serverr r starlette.responsesr+rrrpr}rr)r r rr r r mcp_serverr_r r r+rr headers_dictrr r r r s @@@@rgdynamic_mcp_router 5s @X  G-"44W= .EE yF   ,6G{(S  W]]#01f  !  :) 7??   1;KL41a AHHJ.L L!' #''8JK    M  X""3O3DBs1vh O 6McRSfX4VWW XsYEBC<C4C<,)C6C<3E4C<6C<< ED E=EEE)rXr)NNrArP)NNNNFFNNrNFTTNFFN)FN)r(Nrrlenumrrrrrqr rNr"r,rgrrwarningsrrrtypingrrrr r r r r rrrrrrrrwebsockets.exceptionspydanticrr litellm._uuidrrrrrrrrrrrr r!r"r#r$r%*litellm.litellm_core_utils.litellm_loggingr&*litellm.litellm_core_utils.safe_json_dumpsr'litellm.proxy._typesr(r)r*r+r,r-r.r/r0r1r2r3r4r5r6r7r8r9r:r;r<r=r>r?r@rArBrCrDrErF)litellm.proxy.common_utils.callback_utilsrGrH)litellm.proxy.common_utils.realtime_utilsrJrOrKrLrMrNrmrOrPrzrQopentelemetry.tracerR_Span"litellm.integrations.opentelemetryrSrYrrCrhfilterwarnings UserWarningrkrrXinsertrr[backofffastapirrapscheduler.schedulers.asynciorlrSrrsrw collectionsrx contextlibry functoolsrzrgr{r\r}r~rIrr#litellm.caching.redis_cluster_cacherrrrrrrrrrrrrrrrlitellm.exceptionsr%litellm.integrations.custom_guardrailr"litellm.integrations.custom_loggerr1litellm.integrations.SlackAlerting.slack_alertingr'litellm.litellm_core_utils.core_helpersrr.litellm.litellm_core_utils.credential_accessorrrLiteLLMLoggingObj0litellm.litellm_core_utils.sensitive_data_maskerrr rr&litellm.llms.vertex_ai.vertex_llm_baser;litellm.proxy._experimental.mcp_server.byok_oauth_endpointsrmcp_byok_oauth_router=litellm.proxy._experimental.mcp_server.discoverable_endpoints!mcp_discoverable_endpoints_router5litellm.proxy._experimental.mcp_server.rest_endpointsmcp_rest_endpoints_routerr rmcp_app4litellm.proxy._experimental.mcp_server.tool_registryr+litellm.proxy.agent_endpoints.a2a_endpoints a2a_routerrr'litellm.proxy.agent_endpoints.endpointsagent_endpoints_router0litellm.proxy.agent_endpoints.model_list_helpersrr5litellm.proxy.analytics_endpoints.analytics_endpointsanalytics_router7litellm.proxy.anthropic_endpoints.claude_code_endpointsr+litellm.proxy.anthropic_endpoints.endpointsanthropic_router2litellm.proxy.anthropic_endpoints.skills_endpointsanthropic_skills_routerlitellm.proxy.auth.auth_checksrrrlitellm.proxy.auth.auth_utilsrlitellm.proxy.auth.handle_jwtr"litellm.proxy.auth.litellm_licenserrqrrrrr$litellm.proxy.auth.user_api_key_authrrr)litellm.proxy.batches_endpoints.endpointsbatches_routerlitellm.proxy.caching_routescaching_router'litellm.proxy.common_request_processingrrrr&litellm.proxy.common_utils.debug_utilsrdebugging_endpoints_router0litellm.proxy.common_utils.encrypt_decrypt_utilsrr.litellm.proxy.common_utils.html_forms.ui_loginr-litellm.proxy.common_utils.http_parsing_utilsrrrr,litellm.proxy.common_utils.load_config_utilsrr0litellm.proxy.common_utils.openai_endpoint_utilsr&litellm.proxy.common_utils.proxy_stater+litellm.proxy.common_utils.reset_budget_jobr(litellm.proxy.common_utils.swagger_utilsr+litellm.proxy.container_endpoints.endpointscontainer_router,litellm.proxy.credential_endpoints.endpointscredential_routerr?r"litellm.proxy.db.exception_handlerr!litellm.proxy.discovery_endpointsr-litellm.proxy.fine_tuning_endpoints.endpointsfine_tuning_routerr(litellm.proxy.google_endpoints.endpoints google_router,litellm.proxy.guardrails.guardrail_endpointsguardrails_router(litellm.proxy.guardrails.init_guardrailsrrrRrrA health_router,litellm.proxy.hooks.model_max_budget_limiterr.litellm.proxy.hooks.prompt_injection_detectionr-litellm.proxy.hooks.proxy_track_cost_callbackr'litellm.proxy.image_endpoints.endpoints image_router$litellm.proxy.litellm_pre_call_utilsr9litellm.proxy.management_endpoints.access_group_endpointsaccess_group_router>litellm.proxy.management_endpoints.budget_management_endpointsbudget_management_routerrcache_settings_router@litellm.proxy.management_endpoints.callback_management_endpoints$callback_management_endpoints_routerrnrr7litellm.proxy.management_endpoints.compliance_endpointscompliance_routerrconfig_override_router9litellm.proxy.management_endpoints.cost_tracking_settingscost_tracking_settings_router5litellm.proxy.management_endpoints.customer_endpointscustomer_router@litellm.proxy.management_endpoints.fallback_management_endpointsfallback_management_router:litellm.proxy.management_endpoints.internal_user_endpointsinternal_user_routerrlitellm.proxy.pass_through_endpoints.llm_passthrough_endpointsrllm_passthrough_routerrr-rpass_through_router,litellm.proxy.policy_engine.policy_endpointspolicy_crud_router4litellm.proxy.policy_engine.policy_resolve_endpointspolicy_resolve_routerrprompts_routerlitellm.proxy.public_endpointspublic_endpoints_router%litellm.proxy.rag_endpoints.endpoints rag_router*litellm.proxy.realtime_endpoints.endpoints webrtc_router(litellm.proxy.rerank_endpoints.endpoints rerank_router.litellm.proxy.response_api_endpoints.endpointsresponse_routerlitellm.proxy.route_llm_requestr(litellm.proxy.search_endpoints.endpoints search_router5litellm.proxy.search_endpoints.search_tool_managementsearch_tool_management_routerr cloudzero_router7litellm.proxy.spend_tracking.spend_management_endpointsspend_management_router1litellm.proxy.spend_tracking.spend_tracking_utilsrrvantage_routerlitellm.proxy.types_utils.utilsrrui_crud_endpoints_routerrrrrrrrrrrrrrrrrrrr.litellm.proxy.vector_store_endpoints.endpointsvector_store_router9litellm.proxy.vector_store_endpoints.management_endpointsvector_store_management_router4litellm.proxy.vector_store_files_endpoints.endpointsvector_store_files_router4litellm.proxy.vertex_ai_endpoints.langfuse_endpointslangfuse_router'litellm.proxy.video_endpoints.endpoints video_routerlitellm.routerrrrrlitellm.schedulerrr*litellm.secret_managers.aws_secret_managerr"litellm.secret_managers.google_kmsrrrrrr)litellm.types.integrations.slack_alertingr litellm.types.llms.anthropicr r r r litellm.types.llms.openair+litellm.types.proxy.control_plane_endpointsrClitellm.types.proxy.management_endpoints.model_management_endpointsr/litellm.types.proxy.management_endpoints.ui_ssorrlitellm.types.realtimerrPrrrrrrlitellm.types.schedulerr"litellm.types.secret_managers.mainrrrr ModelMapInforrr litellm._versionr!rpsuppress_debug_inforr"r#r$r%r&r'r(r)r*r+r,r-r.r/r0fastapi.encodersr1fastapi.middleware.corsr2fastapi.openapi.docsr4fastapi.openapi.utilsr5r r6r7r8r9r:fastapi.routingr;fastapi.securityr<fastapi.security.api_keyr=fastapi.staticfilesr>litellm.types.agentsr?enterprise_routerlitellm.proxy.enterpriseproxy enterprise*litellm_enterprise.proxy.enterprise_routes_enterprise_router%litellm_enterprise.proxy.proxy_serverr@rAr rrrBrRairgapped_license_datarCrrE proxy_staterrGr(rIrJrr fallback_login_linkmodel_hub_link ui_message chat_linkcustom_swagger_message_title _descriptionr_rqrr#vertex_live_passthrough_vertex_baserrropenapiEnumrexception_handlerr+originsrr r|r:packaged_ui_pathr5litellm_asset_prefixr4r<rFrrPrr default_runtime_ui_pathruntime_ui_pathrshould_use_runtime_pathis_pre_restructured has_contentrdr|rr r is_writablerarootdirsrgrbrjrdr r rr$r modified_contentr_UnicodeDecodeErrorrArBr}rlradd_middlewarerrrrrr_rYrZrrar`rc user_configr^rR local_loggingrbrrrrrlog_filerrQrrSr]rrrrrmodel_max_budget_limiterrrrrrrrTrVrrrWrXrYrZrhr[r\rfrrerjrrrrrrr1rr[r2rirrrr async_resultcelery_app_conn celery_fnr=rrrkrrrrrrrrrr%r.r3r:rErYrrzr|rrGrrorrrrrrrrrr4r^postrrrrbytesrrrrrrrr r#r r'r+r0r4r7r; litellm.llms.base_llm.base_utilsr<rLrPrdrkrpruryrrrrrrrrrrrrr r r! r( r, r7 rA r? rI rk r~ r r r r r r r r r r r r r r r% r) r+ r> r@ rH rru r} r r r r r r r r r r include_router api_router r^rirgr s;     22" $"B         BM  %1@  D DM  3c3h# # K8 $rwww U?  8$*H9A"4I;KNSPP=I#LNTS F4; OB TGWO >=FFDDRTSSGGKKVPPLT<;TIHJLL  XHGTNXTSYXFQVLLFNLT99LXRQS;;(YK 21CC>> HG A@KK76>= 65KJ9KKG((# ".-2244--&%1111++,,K 11#WKK*?T?VX&;<V ()#..0 d0))8*:C1:'#l +-*2/0'0*-.U*V'?Hryy7@-x}187;-u;0551- c ")*/:$%%897yAJK^J__`a TT 3N3CD_``   ) 1)=KLL N4@< /]  BII VWmVnnrs}r~ AAW@XX\]g\h i  &&>>`!7`!`!F _   '1l# .-FR4 299_g&&0\ CK%CKTYY~& G . '   %y ''//"''//(";TIJCD< 5g>bii1  % % %-gY6RS  ( ( (+G95XY  'w / % % %(CG9&M Nggoobggooh78 %%+ +,-.<* ?#,299-@#A8C=A s?(4WWSEW*7+7     '+x}+  # HV #!%%$?ChtDcN';<C   HSM -1 x[)*1 (, x %,  0DDJJ 0DDJJB! 556NO 8 EJ5cD!@AI3(-,%*d*%)"#  DFd3c4S#X+?&? @@AF&+#$%!t237<gn-t34<$E!$E!;+l OS)K LS415x .5 )      !&*"04(+,3w8 4" 4 > ##e##L1 C=1 c]1 c]1 E? 1 h####Z!% C= c]#c]  E?  tn  49 D   Xe_    d^ c] "  2   D     >l3^ Y S/S/l^} 3       #{~36,!/6,?C6,r GD GS G5( 5("%5( 5(  5(  39 5(pd!/d?CdN!/?C*, $ Y &v v t(9 :;CUBV  W%678@R?S )00A(B-2!27/4',#'B%B$TNBc]B"*$ B 'tn B tn BC=B C=B BJ+,-    +,-   )00A(B<<%<  <~+,-    +,-    ,+,-    7+,-  ]$9:NoN   (/0A(B _ _ _  C=_ & _    *_ DW->%?$@  '*;"=66$'6 8$ %x}hsm CD6r +,-%   F!2F4F  FR$ +,-      2 +,-&   W%9W  W  J%L) $Z > $ %1B $Z(*T **&*L) *Z* 12** #s3x-*Z) d3i-.) ) )  #tCy. ) X %   #Y &A%AAAT A  $Z AJPTB S>B"&B` ` ` SM`"c]#`( SM)`0 SM1`8}9``FC  +,-  )00A(B+/$("& z %z #C=z !z h  z z zP  +,-  )00A(B+6$("&!" i %i #C=i !i h  i c] i sm i i X#A  +,-  )00A(B+6$("&!" P%P#C=P!Ph  P c] P sm PPfI  +,-  )00A(B+/$("&!" NH%NH#C=NH!NHh  NH c] NH sm NHNHb(($(V  +,-    +,-  )00A(B&*C %C smC  C L(,S @H  D  +,-  )00A(B!%z"%z"#z"  z"z_  +,-  4X +,-  )00A(Bc%ccN  +,-   (/0A(B q q q  C=q & q  q h .!1UK ' L @  ,,,^5 BGBBJ5 UGUUpE > W> > B E:o#oo;od #u=NoN>NbE2#3# /S@0S@l515I25Iv  +,-"  >EEV=WG+ G+,:G+G+T  +,-"  =DDU > %>  > B%  +,- )00A(B( %(  ( V,  +,-  )00A(BW %W  W t  +,-  )00A(B( %(  ( \$  +,-  )00A(BO %O  O d-  +,-  )00A(B> > %>  > B-  +,- )00A(B) %)  ) X4  +,-  )00A(B[ %[  [ |Cw'89:;<IW->%?$@AB`6?#>"*+=!::< #$=!< %&&'12,-$%)*=!()()#$*+< 12=!?#&'=!()'(;=!;&'>"?#*+#$>">"#$$%=!?#%&()01>"78-.+,&''()*+,*+78())*01)*-.())*./$%01)*$%:&' HBXSBX7BX BXJ ~7 +,-45akU +A3.RS TTUL G\       #"#v//66.I! !D  &&&@ A3 O    &&&5gYb D     s`%By7< Bz Bz 'B{!JB}4 B{*B{AB{*B{3B{*;B"B}BB{<y7Bz y< BzzBz z BzzBzz Bz>z'Bz.z-Bz>z.Bz7z4Bz>z6Bz7z7Bz>z=Bz>{B{ { B{ {B{ {B{*{B{' {"B{*{* B{9{5B}{8B{9{9B}{<B}|B|"|B}|" B}|+B} }B}} B}}B}}B}}B}