Nj- ddlmZddlZddlZddlZddlmZddlmZddl m Z ddl m Z ddl mZmZmZddlmZdd lmZmZmZmZmZdd lmZdd lmZdd lmZmZdd l m!Z!m"Z"ddl#m$Z$ejJe&Z'dddddejPdejRf ddZ* ddZ+ddddddejPdejRf ddZ,y)) annotationsN)cast) model_info)TokenizerModel)reshape_embeddings) AutoModel AutoTokenizerPreTrainedTokenizerFast)PreTrainedModel) PCADimType PoolingMode apply_pcacompute_weightscreate_embeddings)select_optimal_device) StaticModel)DTypequantize_embeddings)clean_and_create_vocabularyturn_tokens_into_ids)quantize_vocabularyg-C6?z \[unused\d+\]c ht|}t||\}} |g}t|}tj|} | j } | j } | jd| _t| || } | j} | s tdt| | }t|| }t|||| jxsd| }|5tt!||}t#|||\}}}t%||}n2tt!||}t%||}||dddfz}d}d}t'||}t)|d d }d d g||||j*d dd| d }t,j.j1|r"t,j.j3|}d}n6 t5|}|j6|j6j9ddnd}tA|||| jC|||dS#t:$r%}t<j?d|dd}Yd}~Jd}~wwxYw)aDistill a staticmodel from a sentence transformer. This function creates a set of embeddings from a sentence transformer. It does this by doing either a forward pass for all subword tokens in the tokenizer, or by doing a forward pass for all tokens in a passed vocabulary. If you pass through a vocabulary, we create a custom word tokenizer for that vocabulary. If you don't pass a vocabulary, we use the model's tokenizer directly. :param model: The model to use. :param tokenizer: The tokenizer to use. :param vocabulary: The vocabulary to use. If this is None, we use the model's vocabulary. :param device: The device to use. :param pca_dims: The number of components to use for PCA. If this is None, we don't apply PCA. If this is 'auto', we don't reduce dimensionality, but still apply PCA. :param sif_coefficient: The SIF coefficient to use. If this is None, no weighting is applied. Should be a value > 0 and < 1.0. A value of 1e-4 is a good default. :param token_remove_pattern: If this is set to a string, we compile this into a regex. Any tokens that conform to this regex pattern will be removed from the vocabulary. If the pattern is so general that it removes all tokens, we throw an error. If the pattern can't be compiled into a valid regex, we also throw an error. :param quantize_to: The data type to quantize to. Can be any of the DType enum members or their string equivalents. :param vocabulary_quantization: The number of clusters to use for vocabulary quantization. If this is None, no quantization is performed. :param pooling: The pooling mode to use for creating embeddings. Can be one of: 'mean' (default): mean over all tokens. Robust and works well in most cases. 'last': use the last token's hidden state (often the [EOS] token). Common for decoder-style models. 'first': use the first token's hidden state ([CLS] token in BERT-style models). 'pooler': use the pooler output (if available). This is often a non-linear projection of the [CLS] token. :return: A StaticModel. :raises ValueError: if the vocabulary is empty after preprocessing. NT)token_remove_regexzTThe vocabulary is empty after preprocessing. Please check your token_remove_pattern.r) tokenizedmodeldevice pad_token_idpooling)sif_coefficient) n_clustersweights embeddings name_or_path model2vecri@B) model_type architecturestokenizer_namerr hidden_dim seq_length normalizerlanguagez7Couldn't get the model info from the Hugging Face Hub: z. Setting language to None.)vectorsr" token_mapping tokenizerconfigbase_model_namer.r-)"r_validate_parametersrrfrom_transformers_tokenizerprune_added_tokens deep_copyadds_prefix_spacersorted_vocabulary ValueErrorrrrrrlenrrrgetattrshapeospathexistsbasenamercardDataget Exceptionloggerwarningr to_tokenizer)rr1 vocabularyrpca_dimsr token_remove_pattern quantize_tovocabulary_quantizationrroriginal_tokenizer_modeltokenizer_model all_tokens token_idsr#r"r0 model_namer2r.infoes o/Users/ahmed/devFolder/Ultron/claude-voice/.venv/lib/python3.12/site-packages/model2vec/distill/distillation.pydistill_from_modelrUsl\ $K*>Pd*e'O' "6 *F-II)T7JJL/88:O((4,0)2/:btuO!22J opp%Z1IJI u&> ?E#-::?a J*!#j/?S-@.J. * M7z84 "#j/?Sz84 '!T'"22  $Z=J3J"'$* &&q) F ww~~j!WW%%j1  j)D>Bmm>Wt}}((T:]aH #!..0"     NNTUVTWWrs tH s-5H H1 H,,H1c|"d|cxkrdkstdtdd}| tj|}||fS||fS#tj$r}td||d}~wwxYw)a Validate the parameters passed to the distillation function. :param sif_coefficient: The SIF coefficient to use. If this is None, no weighting is applied. Should be a value >= 0 and < 1.0. A value of 1e-4 is a good default. :param token_remove_pattern: If this is set to a string, we compile this into a regex. Any tokens that conform to this regex pattern will be removed from the vocabulary. :return: The SIF coefficient to use. :raises ValueError: if the regex can't be compiled. Nrg?z.SIF coefficient must be a value > 0 and < 1.0.z$Couldn't compile the regex pattern: )r:recompileerror)r rJrrSs rTr4r4s"?(S(MN N)MN N15' P!#,@!A  . ..?. ..xx PCA3GHa O PsAA.A))A.Fc tj||} ttt j||d} t | | ||||||||  S)aDistill a staticmodel from a sentence transformer. This function creates a set of embeddings from a sentence transformer. It does this by doing either a forward pass for all subword tokens in the tokenizer, or by doing a forward pass for all tokens in a passed vocabulary. If you pass through a vocabulary, we create a custom word tokenizer for that vocabulary. If you don't pass a vocabulary, we use the model's tokenizer directly. :param model_name: The model name to use. Any sentencetransformer compatible model works. :param vocabulary: The vocabulary to use. If this is None, we use the model's vocabulary. :param device: The device to use. :param pca_dims: The number of components to use for PCA. If this is None, we don't apply PCA. If this is 'auto', we don't reduce dimenionality, but still apply PCA. :param sif_coefficient: The SIF coefficient to use. If this is None, no weighting is applied. Should be a value >= 0 and < 1.0. A value of 1e-4 is a good default. :param token_remove_pattern: If this is set to a string, we compile this into a regex. Any tokens that conform to this regex pattern will be removed from the vocabulary. :param trust_remote_code: Whether to trust the remote code. If this is False, we will only load components coming from `transformers`. If this is True, we will load all components. :param quantize_to: The data type to quantize to. Can be any of the DType enum members or their string equivalents. :param vocabulary_quantization: The number of clusters to use for vocabulary quantization. If this is None, no quantization is performed. :param pooling: The pooling mode to use for creating embeddings. Can be one of: 'mean' (default): mean over all tokens. Robust and works well in most cases. 'last': use the last token's hidden state (often the [EOS] token). Common for decoder-style models. 'first': use the first token's hidden state ([CLS] token in BERT-style models). 'pooler': use the pooler output (if available). This is often a non-linear projection of the [CLS] token. :return: A StaticModel )trust_remote_codeT)r[use_fast) rr1rHrrIrJr rKrLr)rfrom_pretrainedrr r rU) rQrHrrIr rJr[rKrLrrr1s rTdistillr^sfX'66zUfgE%%jDU`deI 1' 7  )rr r1r rHlist[str] | Noner str | NonerIr r float | NonerJrarK DType | strrL int | NonerPoolingMode | strreturnr)r rbrJrarfz+tuple[float | None, re.Pattern[str] | None])rQstrrHr`rrarIr r rbrJrar[boolrKrcrLrdrrerfr)- __future__rloggingr>rWtypingrhuggingface_hub.hf_apir skeletokenr skeletoken.external.transformersr transformersrr r transformers.modeling_utilsr model2vec.distill.inferencer r rrrmodel2vec.distill.utilsrmodel2vec.modelrmodel2vec.quantizationrrmodel2vec.tokenizerrr!model2vec.vocabulary_quantizationr getLogger__name__rEFloat16MEANrUr4r^r_rTr|s" -%?JJ7nn9'=QA   8 $ $($('7$}}*.!,!1!1K K&K!K  K  K " K%KK(KKK\/!/$/1/<$($('7#$}}*.!,!1!1== = = = " = % ===(===r_