wi ddlZddlmZmZddlZddlmZddlmZmZddl m Z ddl m Z ddl mZdd lmZmZmZdd lmZdd lmZdd lmZejeZd ZddZGddejZGddejZ GddejZ!GddejZ"Gdde eZ#Gdde eZ$dS) N)OptionalUnion)nn) ConfigMixinregister_to_config) ModelMixin) FeedForward) Attention)TimestepEmbedding Timestepsget_2d_sincos_pos_embed)Transformer2DModelOutput) AdaLayerNorm)loggingc&d}||d|zz ks ||d|zzkrtdtj5|||z |z }|||z |z }|d|zdz d|zdz |||tjdz| || |||cdddS#1swxYwYdS)Nc`dtj|tjdz zdz S)N?@)matherfsqrt)xs }/root/.openclaw/workspace/chatterbox_venv_py311/lib/python3.11/site-packages/diffusers/pipelines/unidiffuser/modeling_uvit.pynorm_cdfz(_no_grad_trunc_normal_..norm_cdfs)dhq49S>>1222c99zjmean is more than 2 std from [a, b] in nn.init.trunc_normal_. The distribution of values may be incorrect.r)minmax) loggerwarningtorchno_graduniform_erfinv_mul_rradd_clamp_)tensormeanstdabrlus r_no_grad_trunc_normal_r1s}::: q1s7{q1s7{ 2 2 ;    Ha$h#% & & Ha$h#% & & A 1q519---   C$)C..())) D  ! ###+s B0DD  D rrc(t|||||S)aFills the input Tensor with values drawn from a truncated normal distribution. The values are effectively drawn from the normal distribution :math:`\mathcal{N}(\text{mean}, \text{std}^2)` with values outside :math:`[a, b]` redrawn until they are within the bounds. The method used for generating the random values works best when :math:`a \leq \text{mean} \leq b`. Args: tensor: an n-dimensional `torch.Tensor` mean: the mean of the normal distribution std: the standard deviation of the normal distribution a: the minimum cutoff value b: the maximum cutoff value Examples: >>> w = torch.empty(3, 5) >>> nn.init.trunc_normal_(w) )r1)r*r+r,r-r.s r trunc_normal_r59s "&$Q : ::rc<eZdZdZ d fd Zd ZxZS) PatchEmbedz2D Image to Patch EmbeddingrFTc t||z||zz} ||_||_t j||||f|||_|rt j|dd|_nd|_| |_ |j rqt|t| dz} | dtj| dddSdS) N) kernel_sizestridebiasFgư>)elementwise_affineeps? pos_embedr) persistent)super__init__flatten layer_normrConv2dproj LayerNormnorm use_pos_embedrintregister_bufferr# from_numpyfloat unsqueeze) selfheightwidth patch_size in_channels embed_dimrGrFr>rL num_patchesrB __class__s rrEzPatchEmbed.__init__Os +0CD  $I Z0HQ[bf       Y5dSSSDIIDI*   r/ 3{C?O;P;PQQI  e.>y.I.I.O.O.Q.Q.[.[\].^.^kp q q q q q r rrc||}|jr)|ddd}|jr||}|jr ||jzS|S)Nrr)rIrF transposerGrKrLrB)rRlatents rforwardzPatchEmbed.forwardnsv6"" < 7^^A&&00A66F ? 'YYv&&F   DN* *Mr) r8r8r9rr:FTTT)__name__ __module__ __qualname____doc__rEr] __classcell__rYs@rr7r7Lsu%% rrrrrr>       rr7c*eZdZdeffd ZdZxZS) SkipBlockdimcttjd|z||_tj||_dS)Nr)rDrErLinear skip_linearrJrK)rRrfrYs rrEzSkipBlock.__init__{sH 9QWc22L%% rc|tj||gd}||}|S)Nrf)rir#catrK)rRrskips rr]zSkipBlock.forwards<   UY4yb999 : : IIaLLr)r^r_r`rMrEr]rbrcs@rrerezsS&C&&&&&&rreceZdZdZ dded ed ed eed ed eededededededededeffd Z ddZ xZ S)UTransformerBlockaS A modification of BasicTransformerBlock which supports pre-LayerNorm and post-LayerNorm configurations. Parameters: dim (`int`): The number of channels in the input and output. num_attention_heads (`int`): The number of heads to use for multi-head attention. attention_head_dim (`int`): The number of channels in each head. dropout (`float`, *optional*, defaults to 0.0): The dropout probability to use. cross_attention_dim (`int`, *optional*): The size of the encoder_hidden_states vector for cross attention. activation_fn (`str`, *optional*, defaults to `"geglu"`): Activation function to be used in feed-forward. num_embeds_ada_norm (:obj: `int`, *optional*): The number of diffusion steps used during training. See `Transformer2DModel`. attention_bias (:obj: `bool`, *optional*, defaults to `False`): Configure if the attentions should contain a bias parameter. only_cross_attention (`bool`, *optional*): Whether to use only cross-attention layers. In this case two cross attention layers are used. double_self_attention (`bool`, *optional*): Whether to use two self-attention layers. In this case no cross attention layers are used. upcast_attention (`bool`, *optional*): Whether to upcast the query and key to float32 when performing the attention calculation. norm_elementwise_affine (`bool`, *optional*): Whether to use learnable per-element affine parameters during layer normalization. norm_type (`str`, defaults to `"layer_norm"`): The layer norm implementation to use. pre_layer_norm (`bool`, *optional*): Whether to perform layer normalization before the attention and feedforward operations ("pre-LayerNorm"), as opposed to after ("post-LayerNorm"). Note that `BasicTransformerBlock` uses pre-LayerNorm, e.g. `pre_layer_norm = True`. final_dropout (`bool`, *optional*): Whether to use a final Dropout layer after the feedforward network. r2NgegluFTrGrfnum_attention_headsattention_head_dimcross_attention_dim activation_fnnum_embeds_ada_normattention_biasonly_cross_attentiondouble_self_attentionupcast_attentionnorm_elementwise_affine norm_typepre_layer_norm final_dropoutc t| |_|duo| dk|_||_| dvr|t d| d| dt |||||| r|nd| |_|| r t || s|nd||||| |_nd|_|jrt|||_ ntj || |_ || r3|jrt||ntj || |_ nd|_ tj || |_t|||| |_dS Nada_norm)r ada_norm_zeroz`norm_type` is set to zw, but `num_embeds_ada_norm` is not defined. Please make sure to define `num_embeds_ada_norm` if setting `norm_type` to .) query_dimheadsdim_headdropoutr>rtrz)rrtrrrr>rz)r?)rrur~rDrErxuse_ada_layer_normr} ValueErrorr attn1attn2rnorm1rrJnorm2norm3r ffrRrfrrrsrrtrurvrwrxryrzr{r|r}r~rYs rrEzUTransformerBlock.__init__$ $8!#6d#B"_ U_H_, 5 5 5:M:UXXXKTXXX  %'7K U 3 3QU-      *.C *"?T$^$7$7Z^)+#!1DJJDJ  " W%c+>??DJJc>UVVVDJ  *.C * *S S"5666\#:QRRR JJ DJ\#:QRRR c7-_lmmmrc |jr4|jr|||}n||}n|}||ni}|j|f|jr|nd|d|} |js3|jr|| |} n|| } | |z}|j|jr3|jr|||n||}n|}|j|f||d|} |js2|jr|| |n|| } | |z}|jr||}n|}||} |js|| } | |z}|SN)encoder_hidden_statesattention_mask r}rrrrxrrrr) rR hidden_statesrrencoder_attention_masktimestepcross_attention_kwargs class_labelsnorm_hidden_states attn_output ff_outputs rr]zUTransformerBlock.forwards   /& ?%)ZZ x%H%H""%)ZZ %>%>""!.