Nj X8ddlmZddlZddlZddlZddlmZmZddlm Z ddl m Z ddl m Z ddlmZmZddlZddlmZdd lmZmZdd lmZdd lmZmZdd lmZe ezZe e Z!Gd dZ" d ddZ# ddZ$y)) annotationsN)IteratorSequence) getLogger)Path)TemporaryDirectory)Anyoverload)delayed)Encoding Tokenizer)tqdm)DTypequantize_and_reduce_dim)ProgressParallelcteZdZ d dfd ZeddZeddZejddZeddZeddZ dddZ dddZ e d d d Z e d! d"d Ze d# d$d Ze d# d%d Z d# d&d Zd'dZ d( d)dZd*dZd+dZed,dZ d- d.dZxZS)/ StaticModelc t |tt|j j d\} } | |_|;t|t| k7r$tdt| dt|d||_ ||_ ||_ ||_ |t|jjdrW|jjj7|j |jjj|_nd|_t#t%j&|j D cgc] } t| c} |_|xsi|_||_||_t|jdrd |_nd |_|||_y|j*j5d d |_ycc} w) aInitialize the StaticModel. :param vectors: The vectors to use. :param tokenizer: The Transformers tokenizer to use. :param config: Any metadata config. :param normalize: Whether to normalize the embeddings. :param base_model_name: The used base model name. Used for creating a model card. :param language: The language of the model. Used for creating a model card. :param weights: The weights to use for the embeddings. If None, no weights are used. We always assume the norm of the embeddings is an implicit weight anyway. This is only used for models that have undergone vocabulary quantization. :param token_mapping: A mapping from token ids to indices in the vectors. If None, we don't remap the tokens during inference. This is only used for models that have undergone vocabulary quantization. :raises ValueError: if the number of tokens does not match the number of vectors. c |dS)N)xs `/Users/ahmed/devFolder/Ultron/claude-voice/.venv/lib/python3.12/site-packages/model2vec/model.pyz&StaticModel.__init__..6sQqT)keyNzNumber of tokens (z$) does not match number of vectors (z_). Please provide a token mapping or ensure the number of tokens matches the number of vectors. unk_tokenencode_batch_fastTF normalize)super__init__zipsorted get_vocabitemstokenslen ValueError embeddingweights token_mapping tokenizerhasattrmodelr unk_token_idintnpmedianmedian_token_lengthconfigbase_model_namelanguage_can_encode_fastrget) selfvectorsr,r4rr5r6r*r+r&_token __class__s rr!zStaticModel.__init__s6  3 3 5 ; ; =>RS   S\S[%@$S[M1UVYZaVbUcdoo  ! 1>" 4>>'' 5$..:N:N:X:X:d ) 3 3 5dnn6J6J6T6T UD  $D #&ryy$++1V+#e*+1V'W#X l .  4>>#6 7$(D !$)D !  &DN![[__[%@DN2WsG c4|jjdS)zGet the dimension of the model.r)r)shaper9s rdimzStaticModel.dimZs~~##A&&rc|jS)zHGet the normalize value. :return: The normalize value. ) _normalizer@s rrzStaticModel.normalize_s rc|jjd}||_|!||k7rtj d|d|d||jd<y)z4Update the config if the value of normalize changes.rNzSet normalization to `z&`, which does not match config value `z`. Updating config.)r4r8rCloggerwarning)r9valueconfig_normalizes rrzStaticModel.normalizegs^ ;;??;7  'E5E,E NN(/UVfUggz{ $) K rchtj|jjjS)z2Get the dtype (precision) of the embedding matrix.)r1dtyper)namer@s rembedding_dtypezStaticModel.embedding_dtypers#xx,,-222rc|jduxs+t|jt|jk7}|r"t |jj dSdS)zKGet the number of clusters used for vocabulary quantization, if applicable.Nr)r+r'r)r&r0r?)r9 is_quantizeds rvocabulary_quantizationz#StaticModel.vocabulary_quantizationwsU**$6dC>''*+EErc ddlm}|t||j|j|j |j |j|||j|j y)zSave the pretrained model. :param path: The path to save to. :param model_name: The model name to use in the Model Card. :param subfolder: The subfolder to save to. r)save_pretrained) folder_path embeddingsr,r4r5r6 model_name subfolderr*mappingN) model2vec.persistencerQrr)r,r4r5r6r*r+)r9pathrTrUrQs rrQzStaticModel.save_pretrained}sQ :T ~~nn;; 00]]!LL&& rc |!||jz}|Dcgc]}|d| }}|jr|jj|d}n|jj |d}|Dcgc]}|j }}|j /|D cgc]"}|D cgc]} | |j k7s| c} $}}} ||Dcgc]}|d| }}|Scc}wcc}wcc} wcc} }wcc}w)a Tokenize a list of sentences. :param sentences: The sentences to tokenize. :param max_length: The maximum length of the sentences in tokens. If this is None, sequences are not truncated. :return: A list of list of tokens. NF)add_special_tokens)r3r7r,r encode_batchidsr/) r9 sentences max_lengthmsentence encodingsencoding encodings_ids token_idstoken_ids rtokenizezStaticModel.tokenizes  !T555A6?@i("1iI@  (,(H(Hgl(H(mI33IRW3XI6?@i(i @    (hugtZc)U)hx4CTCT7T)Ugt   !ERS] Y{ 3]MS#AA VTs/ C/C C$C 3C 7C$ C*C$c *t||||||||| S)a}Load a StaticModel from a local path or huggingface hub path. NOTE: if you load a private model from the huggingface hub, you need to pass a token. :param path: The path to load your static model from. :param token: The huggingface token to use. :param normalize: Whether to normalize the embeddings. :param subfolder: The subfolder to load from. :param quantize_to: The dtype to quantize the model to. If None, no quantization is done. If a string is passed, it is converted to a DType. :param dimensionality: The dimensionality of the model. If this is None, use the dimensionality of the model. This is useful if you want to load a model with a lower dimensionality. Note that this only applies if you have trained your model using mrl or PCA. :param vocabulary_quantization: The number of clusters to use for vocabulary quantization. :param force_download: Whether to force the download of the model. If False, the model is only downloaded if it is not already present in the cache. :return: A StaticModel. clsrXr<rO quantize_todimensionalityrrUforce_download)_loading_helper) rirXr<rrUrjrkrOrls rfrom_pretrainedzStaticModel.from_pretraineds,<$;#))  rc btjdtdt|||||||d| S)z Deprecated: use from_pretrained.zVStaticModel.from_sentence_transformers() is deprecated; use from_pretrained() instead.) stacklevelNrh)warningswarnDeprecationWarningrm)rirXr<rrjrkrOrls rfrom_sentence_transformersz&StaticModel.from_sentence_transformerssC  d  $;#))  rcyNrr9r]r^ batch_sizeshow_progress_baruse_multiprocessingmultiprocessing_thresholds rencode_as_sequencezStaticModel.encode_as_sequencesrcyrwrrxs rr}zStaticModel.encode_as_sequencesrcd}t|tr|g}d}tj||}t j t ||z } |rZt ||kDrLdtjd<td|| fd|D} g} | D]} | j| n7g} t|| | D]#} | jj| %|r| d S| S) aEncode a list of sentences as a list of numpy arrays of tokens. This is useful if you want to use the tokens for further processing, or if you want to do sequence modeling. Note that if you just want the mean, you should use the `encode` method. This is about twice as slow. Sentences that do not contain any tokens will be turned into an empty array. NOTE: the input type is currently underspecified. The actual input type is `Sequence[str] | str`, but this is not possible to implement in python typing currently. :param sentences: The list of sentences to encode. :param max_length: The maximum length of the sentences. Any tokens beyond this length will be truncated. If this is None, no truncation is done. :param batch_size: The batch size to use. :param show_progress_bar: Whether to show the progress bar. :param use_multiprocessing: Whether to use multiprocessing. By default, this is enabled for inputs > multiprocessing_threshold sentences and disabled otherwise. :param multiprocessing_threshold: The threshold in number of sentences for using multiprocessing. :return: The encoded sentences with an embedding per token. FTfalseTOKENIZERS_PARALLELISMn_jobsuse_tqdmtotalc3VK|] }tj|"ywrw)r _encode_batch_as_sequence.0batchr^r9s r z1StaticModel.encode_as_sequence..9s,cXhu7667zJXh&)rdisabler) isinstancestrlist_batchmathceilr'osenvironrextendrr)r9r]r^ryrzr{r| was_singlesentence_batches total_batchesresults out_arrayrrs` ` rr}zStaticModel.encode_as_sequence s < i %" IJ  Iz BC #i.:"=>  3y>4M#M3:BJJ/ 0b&b;LTabcXhcG+-I  #I #--   !?!?z!RS  Q< rc|j||}g}|D]U}|r!|j|j|&|jtjd|j fW|S)z*Encode a batch of sentences as a sequence.r]r^r)rfappend_encode_helperr1zerosrA)r9r]r^r\outid_lists rrz%StaticModel._encode_batch_as_sequenceLsammiJmG "G 4..w78 288QM23   rc  d}t|tr|g}d}tj||} t j t ||z } |rWt ||kDrIdtjd<td|| fd| D} tj| d } nNg} t| | | D]#}| jj|%tj| d } |r| dS| S) aEncode a list of sentences. This function encodes a list of sentences by averaging the word embeddings of the tokens in the sentence. For ease of use, we don't batch sentences together. NOTE: the return type is currently underspecified. In the case of a single string, this returns a 1D array, but in the case of a list of strings, this returns a 2D array. Not possible to implement in numpy currently. :param sentences: The list of sentences to encode. You can also pass a single sentence. :param show_progress_bar: Whether to show the progress bar. :param max_length: The maximum length of the sentences. Any tokens beyond this length will be truncated. If this is None, no truncation is done. :param batch_size: The batch size to use. :param use_multiprocessing: Whether to use multiprocessing. By default, this is enabled for inputs > multiprocessing_threshold sentences and disabled otherwise. :param multiprocessing_threshold: The threshold in number of sentences for using multiprocessing. :param **kwargs: Any additional arguments. These are ignored. :return: The encoded sentences. If a single sentence was passed, a vector is returned. FTrrrrc3VK|] }tj|"ywrw)r _encode_batchrs rrz%StaticModel.encode..s,cL\5+**+E:>L\rraxisr)rrrrrrr'rrrr1 concatenaterrr)r9r]rzr^ryr{r|kwargsrrrrr out_arraysrs` ` rencodezStaticModel.encodeXs : i %" IJ  Iz BC #i.:"=>  3y>4M#M3:BJJ/ 0b&b;LTabcL\cGwQ7I,.J #-- !!$"4"4UJ"GH  z:I Q< rc|j|}n|j|}|j|}|j||j|dddfz}|S)ahHelper function to encode a list of ids. This function is used to deduplicate the logic in `encode` and `encode_as_sequence`. It retrieves the embeddings for the given list of ids, applying weights if available. :param id_list: A list of token ids. :return: The embeddings for the given ids, as a sequence of vectors. N)r+r)r*)r9rid_list_remappedembs rrzStaticModel._encode_helpersa    %& #11': nn-. << # W-ag66C rc|j||}g}|D]e}|r3|j|}|j|jd8|jt j |j gt j|}|jr*tjj|dddz}||z }|S)zEncode a batch of sentences.rrrrT)rkeepdimsg3#I9) rfrrmeanr1rrAstackrlinalgnorm) r9r]r^r\rrrrrs rrzStaticModel._encode_batchsmmiJmG "G))'2 3888+, 288DHH-. HHSM >>99>>)!d>CeKD!D(IrcFfdtdtDS)z%Batch the sentences into equal-sized.c3.K|] }||zywrwr)riryr]s rrz%StaticModel._batch..s\7[! !a*n-7[sr)ranger')r]rys``rrzStaticModel._batchs]uQIPZ7[\\rcddlm}t5}|j|||t |||||dddy#1swYyxYw)aPush the model to the huggingface hub. NOTE: you need to pass a token if you are pushing a private model. :param repo_id: The repo id to push to. :param private: Whether the repo, if created is set to private. If the repo already exists, this doesn't change the visibility. :param token: The huggingface token to use. :param subfolder: The subfolder to push to. r)push_folder_to_hub)rT)rUrepo_idprivater<N)rWrrrQr)r9rrr<rUrtemp_dirs r push_to_hubzStaticModel.push_to_hubsD =  !X  g > tH~G]dlq r" ! !s *AA )NNNNNN)r: np.ndarrayr,r r4zdict[str, Any] | Noner bool | Noner5 str | Noner6zlist[str] | Noner*np.ndarray | Noner+rreturnNone)rr0)rbool)rGrrr)rr)r int | None)NN)rXPathLikerTrrUrrrrw)r] Sequence[str]r^rrzlist[list[int]])NNNNNNT)ritype[StaticModel]rXrr<rrrrUrrjstr | DType | NonerkrrOrrlrrr)NNNNNT)rirrXrr<rrrrjrrkrrOrrlrrr)NFT')r]rr^rryr0rzrr{rr|r0rr)r]z list[str]r^rryr0rzrr{rr|r0rlist[np.ndarray])r]zstr | list[str]r^rryr0rzrr{rr|r0rzlist[np.ndarray] | np.ndarray)r]rr^rrr)FirTr)r]rrzrr^rryr0r{rr|r0rr rr)rz list[int]rr)r]rr^rrr)r]rryr0rzIterator[Sequence[str]])FNN) rrrrr<rrUrrr)__name__ __module__ __qualname__r!propertyrArsetterrLrOrQrf classmethodrnrur r}rrrr staticmethodrr __classcell__)r=s@rrrs )-!%&*%)%)+/>A>A>A& >A  >A $ >A#>A#>A)>A >A@''))33FF  ,:!!% $*.%).2#' ' ' '  '  ' ( ' #' ",' '  ' ' R!!%*.%).2#      (  #  ",     8"&"'$()/    " $' "&"'$()/    " $' "&"'$()/="== =  = " =$'= '=~ #(!$$()/< < < <  < " <$'<< <|($]] fjss%)s:DsXbs srrc |`ddlm}t|jt|jk7r t d|||j |j\}}}n$|j}|j}|j }||t|||}t||jt|j|||j|j|jS)aQuantize the model to a lower precision and possibly lower dimensionality. :param model: The model to quantize. :param vocabulary_quantization: The number of clusters to use for quantization. :param quantize_to: The dtype to quantize the model to. :param dimensionality: The desired dimensionality of the model. This needs to be < than the current model dimensionality. :return: A new StaticModel with the quantized embeddings. :raises ValueError: if the model is already quantized. r)quantize_vocabularyzCModel already has been vocabulary quantized, cannot quantize again.) n_clustersr*rS)rSrjrk)r:r,r4r*r+rr5r6)!model2vec.vocabulary_quantizationrr'r&r)r(r*r+rrr,dictr4rr5r6)r.rOrjrkrrSr+r*s rquantize_modelrs$*I u|| EOO 4 4bc c-@. RWRaRa. * M7__ ++ --."<,!#) //ELL!#//--  rc ddlm} | ||||\} } } } }}|| | ||| || jd| jd}t|||gs|St ||||S)z1Helper function to load a model from a directory.r)load_pretrained)folder_or_repo_pathr<rUrl base_modelr6)r:r,r*r+r4rr5r6)r.rOrjrk)rWrr8anyr)rirXr<rOrjrkrrUrlrrSr,r4metadatar*rVr.s rrmrms6@O % A=J 68Wg  \2j) E 'nE F  7%  r)NNN) r.rrOrrjrrkrrr)rirrXrr<rrOrrjrrkrrrrUrrlrrr)% __future__rrrrrcollections.abcrrloggingrpathlibrtempfilertypingr r numpyr1joblibr tokenizersr r rmodel2vec.quantizationrrmodel2vec.utilsrrrrrErrrmrrrrs" .' *A, #: 8 ysys| +/&*!% / /'/$/ /  /d* * * *( * $ *  *****r