AHjddlZddlZddlZddlZddlZddlZddlZddlZddlm Z ddl m Z ddl m Z de edefdZdefdZej&d Zd edefd Zdefd Zdefd ZGddZy)N)Path)Optional)loggervaluereturnc,|y|jdvS)NF)true1tyyes)lower)rs R/root/tools/cai/cai_env/lib/python3.12/site-packages/litellm_proxy_extras/utils.py str_to_boolrs } ;;=: ::ctjj}ttjdrd|d<tjdd|d<|S)zJGet environment variables for Prisma, handling offline mode if configured.PRISMA_OFFLINE_MODEr NPM_CONFIG_PREFER_OFFLINENPM_CONFIG_CACHEz/app/.cache/npm)osenvironcopyrgetenv) prisma_envs r_get_prisma_envrsQ"J29923428 ./)+  1* %& rz ^(\d{14})_namechtj|}|rt|jdSdS)uExtract the leading `YYYYMMDDHHMMSS` timestamp from a migration name. Returns 0 if the name doesn't match the Prisma pattern — unexpected-format entries sort as "oldest" and are treated as historical. r)_MIGRATION_TS_REmatchintgroup)rms r_migration_timestampr$$s- t$A3qwwqz?&Q&rc,|sytd|DS)zz+_max_migration_timestamp..2s61#A&6s)max)namess r_max_migration_timestampr,.s  66 66rcttjdrd}tjd}|r9tjj |rt j d||Stjj |rt j d||St jd|dy) zHGet the Prisma command to use, bypassing Python wrapper in offline mode.rz;/app/.cache/prisma-python/binaries/node_modules/.bin/prismaPRISMA_CLI_PATHzUsing custom Prisma CLI at zUsing cached Prisma CLI at zPrisma CLI not found at z8. Falling back to Python wrapper (may attempt downloads)prisma)rrrpathexistsrinfowarning)default_cli_pathcustom_cli_paths r_get_prisma_commandr65s299234X))$56 rww~~o> KK5o5FG H" " 77>>* + KK56F5GH I# # &'7&89E E rcHeZdZedefdZededefdZededefdZ edefdZ edefd Z ed edefd Z ed edefd Z e ddeded efdZededefdZededdfdZededefdZe ddededefdZy)ProxyExtrasDBManagerrc<tjd}tjjt}|rtjj |rtj |D]}tjj||}tjj||}tjj|rtj||d{tj|||Stj|||S|S)z Get the path to the migrations directory Set os.environ["LITELLM_MIGRATION_DIR"] to a custom migrations directory, to support baselining db in read-only fs. LITELLM_MIGRATION_DIRT) dirs_exist_ok) rrr0dirname__file__r1listdirjoinisdirshutilcopytreecopy2)custom_migrations_dirpkg_migrations_diritemsrc_pathdst_paths r_get_prisma_dirz$ProxyExtrasDBManager._get_prisma_dirQs!# *A BWW__X6 ww~~34JJ'9:9D!ww||,>EH!ww||,A4HHww}}X.($O Xx8 9) ( 24IJ( (!!r schema_pathc tj}t|}|dz dz }|jddt j d}|st jdyt} t jd|d z }tjtd d d d |dgt|ddd|t jdtjtd dddgdd|y#tj$rt jdYytj $r9}t jd|d|j"d|j$|d}~wwxYw)z4Create a baseline migration for an existing database migrations0_initT)parentsexist_ok DATABASE_URLDATABASE_URL not setFz Generating baseline migration... migration.sqlmigratediffz --from-emptyz--to-url--scriptw)stdoutchecktimeoutenvz(Marking baseline migration as applied...resolve --applied)rYrZr[z=Migration timed out - the database might be under heavy load.z#Error creating baseline migration: , N)r8rIrmkdirrrrerrorrr2 subprocessrunr6openTimeoutExpiredr3CalledProcessErrorstderrrX)rJ prisma_dirprisma_dir_pathinit_dir database_urlrmigration_filees r_create_baseline_migrationz/ProxyExtrasDBManager._create_baseline_migrationlsk*99; z*"\1H< td3yy0  LL/ 0$& -  KK: ;%7N NN')" NC0 " KKB C NN')  ((  NNO ,,  NN5aS188*BqxxjQ G  s3BC<<(E1&E184E,,E1migrations_dirctj|d}tjdt|d||Dcgc]!}t |j j #c}Scc}w)z}d,|jvr%tjd-| d|jYd}~6d}~wwxYw).z 1. Compare the current database state to schema.prisma and generate a migration for the diff. 2. Run prisma migrate deploy to apply any pending migrations. 3. Mark all existing migrations as applied. rPrQN DIRECT_URLlitellm_migration_diff_)prefixrRz9Generating migration diff between DB and schema.prisma...rVrSrTz --from-urlz--to-schema-datamodelrUTry)rYrZrXr[z#Failed to generate migration diff: z$Migration diff generation timed out.uMigration diff was not created (prisma migrate diff failed — likely a pooler URL). Falling back to direct SQL execution of each migration file.z*/migration.sqldbexecutez--filez--schemarZrYr{textr[zApplied migration: zFailed to apply migration : Migration z timed out.zMigration diff created at z8Running prisma db execute to apply the migration diff...zprisma db execute stdout: u'✅ Migration diff applied successfullyz Failed to apply migration diff: z%Migration diff application timed out.z Resolving z migrationszResolving migration: r\r]zResolved migration: z/is already recorded as applied in the database.Failed to resolve migration )rrrr`rtempfilemkdtempr2rcrarbr6rrer3rfrdr1sortedrpstrrrrrXr8rurqdebug)rnrJrrjdiff_urldiff_dir diff_sql_pathfrlmigration_filesmig_fileresultmigration_namesrvs r_resolve_all_migrationsz,ProxyExtrasDBManager._resolve_all_migrations s!yy0  LL/ 0 99\*:l((0IJK ?2  C KKS TmS) Q+-!$ /#" ') .##% NN' %T.%9%>%>?P%QRO+ SSNN/1 %$M&'!#"'+!+- KK"5hoo6J6J5K LM% S2  0@A D KKR S^^') &##%F KK4V]]OD E KKA B .CCNS j_!5 6kBC- N  3N3CDE+-!!#& #'')  3N3CDE# y  ",, M NN@ K L L(( C NNA B CB"44NN4X__5I5I4J"QXXJW"00SNNZ0D0D/E[#QRS4,, J NN=ahhZH I I(( D NNB C D400 DAHHTNN6~6Fb S s,!K 3KKA'L=BO A QK KL:#"L  +L:9L:=O9NAOOQ,"P+QQR3RRurlcddlm}m}m}m}||}|j s|Shd}||j Dcgc]\}}||vs ||f} }}||j || Scc}}w)zRemove Prisma-specific query params (connection_limit, pool_timeout, schema, etc.) from DATABASE_URL so psycopg can parse it.r)urlparse urlunparse parse_qsl urlencode> sslkeyoptionsservicesslcertsslmode gssencmode krbsrvname sslpassword sslrootcertclient_encodingconnect_timeoutapplication_nametarget_session_attrs)query) urllib.parserrrrr_replace) rrrrrparsed libpq_paramskvkepts r_strip_prisma_query_paramsz/ProxyExtrasDBManager._strip_prisma_query_paramsst LK#||J $-V\\#:P41aa<>OAPP&// $/@AAQs A0 A0Nc 6tjd}|sy ddl}tj |}t tj|} |j|dd5} |jdj} dddDchc]}|d }}||z } | syt|} | D chc]} t!| | kDs| } } | syt#| } t%j&dt)| d j+| dd t)| d kDr d zyd zy#t$rYywxYw#|jj$r YdddywxYw#1swYxYw#|j|jf$rYywxYwcc}wcc} w) u_ Log a warning if _prisma_migrations contains applied migrations with timestamps newer than every migration this build ships. This is informational only for the v2 resolver — it tells the operator the DB was likely migrated by a newer deployment, which is usually a signal that this (older) version shouldn't run against it. We do NOT block startup: many users have weird _prisma_migrations state from prior thrashing bugs, and blocking them would be a breaking change. Safe no-op if psycopg isn't installed or DB isn't reachable. rPNr T)r autocommitzfSELECT migration_name FROM _prisma_migrations WHERE finished_at IS NOT NULL AND rolled_back_at IS NULLaGDatabase has %d migration(s) applied that are NEWER than any migration this LiteLLM version ships. This usually means the database was migrated by a newer LiteLLM deployment. Some API endpoints may fail because this proxy's Prisma client does not know about those schema changes. Consider upgrading this deployment. Unknown: %sr^z ...)rrpsycopg ImportErrorr8rsetruconnectrfetchallerrorsUndefinedTableOperationalError DatabaseErrorr,r$rrr3rqr?)rnrjr cleaned_urlknownconnrowsrappliedunknownhead_newest_tsrhostilesorted_hostiles r_warn_if_db_ahead_of_headz.ProxyExtrasDBManager._warn_if_db_ahead_of_headsyy0    +EElS (==nMN  RD! <<Shj ""&&A1Q4&&E/ 1%8$ (7JQ7NV W UW W S   &~~44    (('*?*?@    '  sqD.E0&E$(D=E0 F9F F. D:9D:=E!E$E0 E!!E$$E-)E00F F use_migratec  tjdz}tj}|sntj}tj| t j tdddgddt tj|ytj|tj}tj| td D]i} t j td d gddddt }tjd|j tj|ytd##t jt jf$r}td||d }~wwxYw#tj|wxYw#t j$rItjd|dzdt#j$t'j(ddY&t j$r}|j*xsd}d|vr4d|vr0tjdtj-|Yd }~d|vrt/j0d|}|rtj3|r|j5d} tjd| d tj7| n)#t jt jf$rYnwxYw tj9| n=#t jt jf$r} td| d| | d } ~ wwxYwYd }~wtd||d|vr$tj;|rtd ||t/j0d!|}|rtj3|r|j5d} tjd| d" tj7| n)#t jt jf$rYnwxYw tj9| n=#t jt jf$r} td| d| | d } ~ wwxYwYd }~td||td||d }~wwxYw#tj|wxYw)$u v2 migration resolver (opt-in via --use_v2_migration_resolver). Runs `prisma migrate deploy` and handles standard recovery paths (P3005 baseline, P3009/P3018 idempotent errors). Critically, it does NOT call `_resolve_all_migrations` — the diff-and-force recovery that caused schema thrashing when two LiteLLM versions contended for the same DB during rolling deploys. Ahead-of-HEAD state (DB has migrations newer than this build ships) is logged as a warning, not a fatal error — users whose DBs got into weird shapes from the old thrashing should still be able to start. /schema.prismarpush--accept-data-lossryT)rZrYr[z prisma db push failed. Detail: NrSdeployrprisma migrate deploy stdout: zprisma migrate deploy attempt rz timed out, retryingrrP3005database schema is not emptyu<Schema exists but no migrations ledger — creating baselineP3009z `(\d+_\S+?)`ru5 failed idempotently — marking applied and retryingzFailed to mark migration zU as applied after idempotent recovery. Manual intervention may be required. Detail: zeDatabase migration failed and cannot be auto-recovered. Manual intervention required. Prisma error: P3018zzDatabase migration failed due to insufficient permissions. Please grant the required privileges and retry. Prisma error: zMigration name: (\d+_\S+)u: SQL hit idempotent error — marking applied and retryingzDatabase migration failed after 4 attempts (retry loop exhausted by timeouts or repeated idempotent-recovery continues). Check database connectivity, load, and _prisma_migrations ledger state.)r8rIrgetcwdchdirrarbr6rrerd RuntimeErrorrrangerr2rXtimesleeprandom randrangerfrmrrrr"r~rr) rrJrn original_dirrlattemptrrfmigration_matchr resolve_errs r_setup_database_v2z'ProxyExtrasDBManager._setup_database_v2s+::t=j>tAjBd?d@Ynutj$r`}dA|z }|dBkDrdC|dDndE}tjdF|dG|t=j>tAjBd?d@Yd}~nd}~wwxYwt j|#t j|wxYw)Ha> Set up the database using either prisma migrate or prisma db push Uses migrations from litellm-proxy-extras package Args: use_migrate: Whether to use prisma migrate instead of db push use_v2_resolver: Opt into the v2 migration resolver (safer during rolling deploys; does not run the diff-and-force recovery that causes schema thrashing). Defaults to False for backwards compatibility. Returns: bool: True if setup was successful, False otherwise z9Using v2 migration resolver (--use_v2_migration_resolver))rrrzRunning prisma migrate deployrSrryTrrzprisma migrate deploy completedzNo pending migrations to applyu>No pending migrations — skipping post-migration sanity checkz&Running post-migration sanity check...F)ru)✅ Post-migration sanity check completedzprisma db error: z, e: rz`(\d+_.*)` migrationrrzS failed due to idempotent error (e.g., column already exists), resolving as appliedzFailed to roll back migration rz+. It may already be in a rolled-back state.Nu✅ Migration z1 resolved, retrying to apply remaining migrationsrzFound failed migration: z, marking as rolled backr\rxz" marked as rolled back... retryingrraDatabase schema is not empty, creating baseline migration. In read-only file system, please set an environment variable `LITELLM_MIGRATION_DIR` to a writable directory to enable migrations. Learn more - https://docs.litellm.ai/docs/proxy/prod#read-only-file-systemz4Baseline migration created, resolving all migrationsu✅ All migrations resolved.rzMigration name: (\d+_.*)ru❌ Migration zW failed due to insufficient permissions. Please check database user privileges. Error: z marked as rolled backz)Failed to mark migration as rolled back: z4Migration failed due to permission error. Migration zH was NOT applied. Please grant necessary database permissions and retry.z\Migration failed due to idempotent error (e.g., column already exists), resolving as appliedzRolling back migration zResolving migration z+ that failed due to existing schema objectszYP3018 error encountered but could not classify as permission or idempotent error. Error: rrr)rZrYzAttempt z timed outrrrz Retrying... (z attempts left)rz(The process failed to execute. Details: .)"rr2r8rrIrrrrrarbr6rrXrrerfrrr"rr~rdr3rrmrr` Exceptionrrrrr)rrrJrrrnrrlrfailed_migration rollback_errrrvrollback_error attempts_left retry_msgs rsetup_databasez#ProxyExtrasDBManager.setup_databases$  KKS T'::{:S S*::RS ""  &cq d[&88u& &7zqxxj$QR"ahh..0ii 7/O /3B3H3H3K 0#7#L#LQXX#V$*KK*45E4FGZ)[%& %*(<(Q(Q,<)*)3(E(E(2(A(A,&%*)/.LM]L^^`am`noX-Y)*)* %* %*(<(X(X,<)*)/ .<=M%Q%& %+KK*BCSBTTl(m%&%/NN,?,A,5,5,;,< )*13.27;-1,;,= %&%+KK*89I8JJl(m%&$qxx/ >!(( J"KK!k1KKKX"KK V1HH . #KK(FG#'r&s%03HHR35))$?3" (7%4$9$9!$<)2!/ !' &4^4DEUUVU]U]T^%`!" $3 %*(<(Q(Q,:)*)/ .88HH^,_)*,5%*(..WXfWg,h)*)*%* '3&Z[iZjkn%o'"()!) "6!J!J188!T & %;!" 35))$?3"$35D5J5J15MN%*(. .EnEU,V)*)=(Q(Q,:)*)3(E(E(2(A(A,&%*)/.L^L\\^_k^lmX-Y)*)* %*%*(. .B>BRSM-N)*)=(X(X,:)*)/ .<^JZZ\]h\i,j)*)* %*%9$P$P(6(39>%Q%&!''../hhZ%9!" !&ku&|,, 4 hw{m:>? 6++Ar2300 4 !G %q(%]O?C  Fqc9+VW 6++Ar233 4&&sS]+BG# AG#%$]#]7B+]#J98]9#LK=8]=L].L54]5#M=M83]8M==D]]B ]*.U] V"U?:]?VA<]-X/.]/#Y8Y3.]3Y88]S>T>>  S  $  C    CD4CD8HLHH*-HAEHHTBBBB4E #E $E E Nl#l#l#l#\;@HH48H HHrr8)rprrrrArarrpathlibrtypingrlitellm_proxy_extras._loggingrrrrdictrcompilerr!r$r,r6r8rrrr s  0;x};;  2::m,'s's'7s7S6` ` r