Nj'ddlmZddlZddlZddlmZddlmZddlm Z ddl Z ddl Z ddl mZddlmZddlmZdd lmZdd lmZej,eZeezZedzeze d zZd ZGd deeZej@f ddZ! ddZ"e jFddZ$e jFddZ%e jFddZ&e jFddZ'ddZ(ddZ)y)) annotationsN)Enum)Path)Literal)PCA) pad_sequence)tqdm),BaseModelOutputWithPoolingAndCrossAttentions)PreTrainedModelautoc eZdZdZdZdZdZdZy) PoolingModeaPooling modes for embedding creation. - MEAN: masked mean over all tokens. - LAST: last non-padding token (often EOS, common in decoder-style models). - FIRST: first token hidden state (position 0). In BERT-style encoders, this corresponds to the [CLS] token representation. - POOLER: use the model's `pooler_output`. In BERT-like models this is computed as the hidden state at [CLS], passed through a learned dense layer + activation. Not all models provide this. meanlastfirstpoolerN)__name__ __module__ __qualname____doc__MEANLASTFIRSTPOOLERl/Users/ahmed/devFolder/Ultron/claude-voice/.venv/lib/python3.12/site-packages/model2vec/distill/inference.pyrrs  D D E Frrc `|j|j}g}dtj|jj v}t j|Dcgc] }t|c}}t j|} | D cgc]} ||  } } tt| dd} tdt| tD]} | | | tz}|Dcgc]'}tj|tj)}}i}t!|d||d <|d j#d }tj|Dcgc] }t|c}|d j$ }tj&||d j$ }|j)d|j)d k}|jtj|d <|rtj*|d |d<|t,j.k(r t1||}nn|t,j2k(r t5||}nN|t,j6k(r t9||}n.|t,j:k(r t=||}nt?d ||jA|jC| jEt|t j| D cgc]} ||  }} t jF|}t jH|}|Scc}wcc} wcc}wcc}wcc} w)aCreate output embeddings for a bunch of tokens using a pretrained model. It does a forward pass for all tokens passed in `tokens`. :param model: The model to use. This should be a transformers model. :param tokenized: All tokenized tokens. :param device: The torch device to use. :param pad_token_id: The pad token id. Used to pad sequences. :param pooling: The pooling mode to use. :return: The output embeddings. :raises ValueError: If the pooling mode is unknown. token_type_idszEncoding tokensz tokens)totaldescunitr)dtypeT) batch_first padding_value input_idsdeviceattention_maskzUnknown pooling: )%toevalinspectgetfullargspecforwardargsnpasarraylenargsortr range_DEFAULT_BATCH_SIZEtorchtensorlongrsizer*arange unsqueeze zeros_likerr_encode_mean_with_modelr_encode_last_with_modelr_encode_first_with_modelr_encode_pooler_with_model ValueErrorextendnumpyupdatestack nan_to_num)model tokenizedr* pad_token_idpoolingintermediate_weightsadd_token_type_idssequencelengths sort_orderisorted_tokenizedpbar batch_idx batch_listxbatchencodedseq_len batch_lengthstoken_positionsr+out out_weightss rcreate_embeddingsr_+s( HHV  ! ! #E.0*W-C-CEMM-R-W-WWjj B H#h- BCGG$J.89j ! j9 c*+2C) TD1c"235HI %i)>Q2QR @ZZ =ST=S03=ST((/0K-- ,K [C: H&B2Us L L%,L! L&  L+c|jDcic]!\}}||j|j#}}}|di|}|j}|j }t |dd}||j }|||fScc}}w)atMove inputs to the model device, run a forward pass, and standardize dtypes. :param model: The model to use. :param encodings: The encoded tokens to turn into features. :return: a tuple consisting of: - hidden: last_hidden_state - pooler: pooler_output if present, else None - encodings_on_device: the device-moved encodings (for masks) pooler_outputNr)itemsr,r*last_hidden_statefloatgetattr)rI encodingskvencodings_on_deviceoutputshiddenrs r_encode_with_modelrlws>G__=NO=NTQ1add5<<00=NODhhq$h'2237G '>D 99TWWV]]+AtQJ7 @ H H K O O QQrc<t||\}}}|dj}|jddz jdj }t j |jd|j}|||ddfjS)zEncode a batch of tokens using last token pooling. :param model: The model to use. :param encodings: The encoded tokens to turn into features. :return: The last hidden state for each token. r+r()dimrr)N) rlboolrp clamp_minr:r8r<r;r*ro)rIrfrkrtrirulast_idx batch_indicess rr@r@s&8y%I"FA" / 0 5 5 7DQ!#..q1668HLLQ FM -1, - 1 1 33rcVt||\}}}|dddddfjS)zEncode a batch of tokens using first token (CLS) pooling. :param model: The model to use. :param encodings: The encoded tokens to turn into features. :return: The first token representation for each token. Nr)rlro)rIrfrkrts rrArAs/&eY7LFAq !Q'?    rc\t||\}}}| td|jS)a Encode a batch of tokens using pooler output. :param model: The model to use. :param encodings: The encoded tokens to turn into features. :return: The pooler output for each token. :raises ValueError: If the model does not return pooler_output. zAPOOLER pooling requested, but model did not return pooler_output.)rlrCro)rIrfrtrs rrBrBs3&eY7LAvq ~\]] ::<rc|tj|Stjddtjd|dzz }|tj |z }|||zz }|S)z>Compute the weights based on Zipf's law and a SIF coefficient.zDEstimating word frequencies using Zipf's law, and then applying SIF.r()r2onesloggerinfor<rp) n_embeddingssif_coefficientinv_rankprobaweights rcompute_weightsrseww|$$ KKVWBIIa!123H rvvh' 'E % 7 8F Mrc||dk(r|jd}||jdkDr8tjd|d|jdd|jd}||jdk\r+tjd|d|jdd|S||jdkrt|trtj d |d ntj d ||jd}t |d }|j|}|jd|krtj|j}tj|j}tj d|d|jddtj d|ddtj d|dd|S)zApply PCA to the embeddings.r r(zPCA dimension (z=) is larger than the number of dimensions in the embeddings (z). Applying PCA, but not reducing dimensionality. Is this is not desired, please set `pca_dims` to None. Applying PCA will probably improve performance, so consider just leaving it.rz9) is larger than the number of tokens in the vocabulary (z). Not applying PCA.zApplying PCA with z explained variance.zApplying PCA with n_components full) n_components svd_solverzReduced dimensionality from z to .zExplained variance ratio: z.3fzExplained variance: ) shaperwarning isinstancerdrr fit_transformr2rpexplained_variance_ratio_explained_variance_) embeddingspca_dims orig_dimspexplained_variance_ratioexplained_variances r apply_pcars v !''*H j&&q) ) NN!(+hisiyiyz{i|h}~__  "''*H z''* * NN!(+deoeueuvwexdyzNO ( #))!, ,(E* 0 :NOP =hZHI"((+If=A4J"Y.+-66!2M2M+N(%'VVA,A,A%B" :9+T*JZJZ[\J]I^^_`a 89QRU8VVWXY 23Ec2J!LM r) rIr rJzlist[list[int]]r*strrKintrLzPoolingMode | strreturn np.ndarray)rIr rfdict[str, torch.Tensor]rzAtuple[torch.Tensor, torch.Tensor | None, dict[str, torch.Tensor]])rIr rfrrz torch.Tensor)rrrz float | Nonerr)rrr PCADimTyperr)* __future__rr.loggingenumrpathlibrtypingrrEr2r8sklearn.decompositionrtorch.nn.utils.rnnrr transformers.modeling_outputsr transformers.modeling_utilsr getLoggerrrrPathLikerrdrr7rrr_rlinference_moder?r@rArBrrrrrrsg" %+V7   8 $ #: 4Z% '&/ 1 #t."-!1!1 I II I I  I  IX/ /'>/F/2 R R 4 4!!   !r