wi} ddlmZddlmZmZmZddlZddlmZddl mcm Z ddl m Z ddlmZddlmZdd lmZmZmZmZmZdd lmZdd lmZmZmZmZmZm Z Gd d ej!Z"Gddej!Z#dej$dej$fdZ%Gddej!Z&Gddej!Z'Gddej!Z(Gddej!Z)Gddej!Z*Gddej!Z+dS))partial)OptionalTupleUnionN) deprecate)get_activation) SpatialNorm) Downsample1D Downsample2DFirDownsample2D KDownsample2D downsample_2d) AdaGroupNorm) FirUpsample2D KUpsample2D Upsample1D Upsample2Dupfirdn2d_native upsample_2dc"eZdZdZddddddddd d dddd dd d edeedededededeedededededeededededeef fdZ de j de j d e j fd!Z xZ S)"ResnetBlockCondNorm2Da) A Resnet block that use normalization layer that incorporate conditioning information. Parameters: in_channels (`int`): The number of channels in the input. out_channels (`int`, *optional*, default to be `None`): The number of output channels for the first conv2d layer. If None, same as `in_channels`. dropout (`float`, *optional*, defaults to `0.0`): The dropout probability to use. temb_channels (`int`, *optional*, default to `512`): the number of channels in timestep embedding. groups (`int`, *optional*, default to `32`): The number of groups to use for the first normalization layer. groups_out (`int`, *optional*, default to None): The number of groups to use for the second normalization layer. if set to None, same as `groups`. eps (`float`, *optional*, defaults to `1e-6`): The epsilon to use for the normalization. non_linearity (`str`, *optional*, default to `"swish"`): the activation function to use. time_embedding_norm (`str`, *optional*, default to `"ada_group"` ): The normalization layer for time embedding `temb`. Currently only support "ada_group" or "spatial". kernel (`torch.Tensor`, optional, default to None): FIR filter, see [`~models.resnet.FirUpsample2D`] and [`~models.resnet.FirDownsample2D`]. output_scale_factor (`float`, *optional*, default to be `1.0`): the scale factor to use for the output. use_in_shortcut (`bool`, *optional*, default to `True`): If `True`, add a 1x1 nn.conv2d layer for skip-connection. up (`bool`, *optional*, default to `False`): If `True`, add an upsample layer. down (`bool`, *optional*, default to `False`): If `True`, add a downsample layer. conv_shortcut_bias (`bool`, *optional*, default to `True`): If `True`, adds a learnable bias to the `conv_shortcut` output. conv_2d_out_channels (`int`, *optional*, default to `None`): the number of channels in the output. If None, same as `out_channels`. NF ư>swish ada_group?T) out_channels conv_shortcutdropout temb_channelsgroups groups_outeps non_linearitytime_embedding_normoutput_scale_factoruse_in_shortcutupdownconv_shortcut_biasconv_2d_out_channels in_channelsr!r"r#r$r%r&r'r(r)r*r+r,r-r.r/c:t||_||n|}||_||_| |_||_| |_| |_||}|jdkrt|||||_ n8|jdkrt|||_ ntd|jtj||ddd|_|jdkrt|||||_n8|jdkrt|||_ntd|jt"j ||_|p|}tj||ddd|_t+| |_dx|_|_|jrt3|d |_n|jrt5|ddd |_| |j|kn| |_d|_|jr!tj||ddd | |_dSdS)Nr)r'spatialz" unsupported time_embedding_norm: r  kernel_sizestridepaddingFuse_convopr9r7namerr5r6r7bias)super__init__r0r!use_conv_shortcutr,r-r*r)rnorm1r ValueErrornnConv2dconv1norm2torchDropoutr#conv2r nonlinearityupsample downsamplerr r+r")selfr0r!r"r#r$r%r&r'r(r)r*r+r,r-r.r/ __class__s g/root/.openclaw/workspace/chatterbox_venv_py311/lib/python3.11/site-packages/diffusers/models/resnet.pyr@zResnetBlockCondNorm2D.__init__Js^( &&2&:{{ (!. #6 #6  J  #{ 2 2%m[&cRRRDJJ  % 2 2$[-@@DJJ\$BZ\\]] ]Y{LaPQ[\]]]  #{ 2 2%m\:SVWWWDJJ  % 2 2$\=AADJJ\$BZ\\]] ]x''00 3C|Y|-AqYZdefff *=99*..  7 ^&{UCCCDMM Y ^*;PQX\]]]DOKZKbt/3GGGhw!   !#$' """D      input_tensortembreturnc6t|dks|ddd}tdd||}|||}||}|jd|jddkr(|}|}||}||}n1|j*||}||}| |}| ||}||}| |}| |}|j | |}||z|jz }|S)NrscaleThe `scale` argument is deprecated and will be ignored. Please remove it, as passing it will raise an error in the future. `scale` should directly be passed while calling the underlying pipeline component i.e., via `cross_attention_kwargs`.1.0.0@)lengetrrBrKrLshape contiguousrMrFrGr#rJr"r*)rNrRrSargskwargsdeprecation_message hidden_states output_tensors rPforwardzResnetBlockCondNorm2D.forwards t99q==FJJw55A#U  gw(; < < <$  =$77 ))-88 = $"1%+++6688 - 8 8 : : ==66L MM-88MM _ (??<88L OOM::M =11  =$77 ))-88  ]33  =11   )--l;;L% 59QQ rQ)__name__ __module__ __qualname____doc__intrboolfloatstrr@rHTensorrc __classcell__rOs@rPrr,s{B'+# $($#.%(*.#'.2%IIIIsm I  I  IIISMIII!I#I"$I I !I"!#I$'sm%IIIIIIV%EL% %Z_Zf%%%%%%%%rQrc(eZdZdZddddddddd dd dd dddddd d edeedededededeededededededee j dedeededededeef&fd Z d!e j d"e j d#e j fd$Z xZ S)% ResnetBlock2Da9 A Resnet block. Parameters: in_channels (`int`): The number of channels in the input. out_channels (`int`, *optional*, default to be `None`): The number of output channels for the first conv2d layer. If None, same as `in_channels`. dropout (`float`, *optional*, defaults to `0.0`): The dropout probability to use. temb_channels (`int`, *optional*, default to `512`): the number of channels in timestep embedding. groups (`int`, *optional*, default to `32`): The number of groups to use for the first normalization layer. groups_out (`int`, *optional*, default to None): The number of groups to use for the second normalization layer. if set to None, same as `groups`. eps (`float`, *optional*, defaults to `1e-6`): The epsilon to use for the normalization. non_linearity (`str`, *optional*, default to `"swish"`): the activation function to use. time_embedding_norm (`str`, *optional*, default to `"default"` ): Time scale shift config. By default, apply timestep embedding conditioning with a simple shift mechanism. Choose "scale_shift" for a stronger conditioning with scale and shift. kernel (`torch.Tensor`, optional, default to None): FIR filter, see [`~models.resnet.FirUpsample2D`] and [`~models.resnet.FirDownsample2D`]. output_scale_factor (`float`, *optional*, default to be `1.0`): the scale factor to use for the output. use_in_shortcut (`bool`, *optional*, default to `True`): If `True`, add a 1x1 nn.conv2d layer for skip-connection. up (`bool`, *optional*, default to `False`): If `True`, add an upsample layer. down (`bool`, *optional*, default to `False`): If `True`, add a downsample layer. conv_shortcut_bias (`bool`, *optional*, default to `True`): If `True`, adds a learnable bias to the `conv_shortcut` output. conv_2d_out_channels (`int`, *optional*, default to `None`): the number of channels in the output. If None, same as `out_channels`. NFrrrTrrdefaultr )r!r"r#r$r%r&pre_normr'r( skip_time_actr)kernelr*r+r,r-r.r/r0r!r"r#r$r%r&rrr'r(rsr)rtr*r+r,r-r.r/ct| dkrtd| dkrtdd|_||_||n|}||_||_||_||_||_ | |_ | |_ ||}tj ||| d|_tj||ddd |_|g|j d krtj|||_nH|j d krtj|d |z|_ntd |j dd|_tj ||| d|_tj ||_|p|}tj||ddd |_t1| |_dx|_|_|jrR| dkr dfd|_n| dkr"t9t:jdd|_nqt?|d|_nZ|jrS| dkr dfd|_n@| dkr"t9t:j d d |_ntC|ddd|_| |j|kn||_"d|_#|j"r!tj||ddd||_#dSdS)NrzkThis class cannot be used with `time_embedding_norm==ada_group`, please use `ResnetBlockCondNorm2D` insteadr2ziThis class cannot be used with `time_embedding_norm==spatial`, please use `ResnetBlockCondNorm2D` insteadT num_groups num_channelsr'affiner3r r4rq scale_shiftrzunknown time_embedding_norm :  fir)r r3r3r c&t|SN)rt)rx fir_kernels rPz(ResnetBlock2D.__init__..%s+a *K*K*KrQsde_vpg@nearest) scale_factormodeFr8c&t|Sr~)rrs rPrz(ResnetBlock2D.__init__..-sM!J,O,O,OrQ)r5r6r:r;rr=)$r?r@rCrrr0r!rAr,r-r*r)rsrHrD GroupNormrBrErFLinear time_emb_projrGrIr#rJr rKrLrMrF interpolater avg_pool2dr r+r")rNr0r!r"r#r$r%r&rrr'r(rsr)rtr*r+r,r-r.r/rrOs @rPr@zResnetBlock2D.__init__s3.  + - -}  ) + +{  &&2&:{{ (!. #6 #6 *  JX''6 Y\ei'jj Y{LaPQ[\]]]  $'944%'Y}l%K%K"")]::%'Y}a,>N%O%O"" !]$BZ!]!]!]^^^!%D X'':L^ajn'oo x''00 3C|Y|-AqYZdefff *=99*..  7 b) K K K K 8## ' Ci X X X *; G G G Y b) "O"O"O"O8##")!,Aa"P"P"P".{UTU\`"a"a"aKZKbt/3GGGhw!   !#$' """D     rQrRrSrTct|dks|ddd}tdd||}||}||}|jd|jddkr(|}|}||}||}n1|j*||}||}| |}|j ?|j s||}| |ddddddf}|j dkr|||z}| |}nt|j dkrT|td|j tj|d d \}}| |}|d |zz|z}n| |}||}||}||}|j||}||z|jz } | S) NrrVrWrXrYrqrzz9 `temb` should not be None when `time_embedding_norm` is rr )dim)rZr[rrBrKrLr\r]rMrFrrsr)rGrCrHchunkr#rJr"r*) rNrRrSr^r_r`ra time_scale time_shiftrbs rPrczResnetBlock2D.forward@sw t99q==FJJw55A#U  gw(; < < <$  =11 ))-88 = $"1%+++6688 - 8 8 : : ==66L MM-88MM _ (??<88L OOM::M =11   )% /((..%%d++AAAqqq$,<=D  #y 0 0 - 4  JJ}55MM  % 6 6| jPTPhjj&+[qa%@%@%@ "J  JJ}55M)Q^ GroupNorm --> Mish Parameters: inp_channels (`int`): Number of input channels. out_channels (`int`): Number of output channels. kernel_size (`int` or `tuple`): Size of the convolving kernel. n_groups (`int`, default `8`): Number of groups to separate the channels into. activation (`str`, defaults to `mish`): Name of the activation function. mish inp_channelsr!r5n_groups activationcttj||||dz|_tj|||_t||_dS)Nrr7) r?r@rDConv1dconv1dr group_normr r)rNrr!r5rrrOs rPr@zConv1dBlock.__init__sb i lKQ\`aQabbb ,x>>":.. rQinputsrTc||}t|}||}t|}||}|SN)rrrr)rNrintermediate_reproutputs rPrczConv1dBlock.forwards\ KK//*+<== OO,=>>*+<==,-- rQ)rr rdrerfrgrhrrrkr@rHrlrcrmrns@rPrrs    / / / /3c3h/0 /  /  / / / / / /elu|rQrc eZdZdZ ddedededeeeeeffdef fd Zd e j d e j d e j fd Z xZ S)ResidualTemporalBlock1Da Residual 1D block with temporal convolutions. Parameters: inp_channels (`int`): Number of input channels. out_channels (`int`): Number of output channels. embed_dim (`int`): Embedding dimension. kernel_size (`int` or `tuple`): Size of the convolving kernel. activation (`str`, defaults `mish`): It is possible to choose the right activation function. rrr! embed_dimr5rcdtt||||_t||||_t ||_tj|||_ ||krtj ||dntj |_ dS)Nr ) r?r@rconv_inconv_outr time_emb_actrDrtime_embrIdentity residual_conv)rNrr!rr5rrOs rPr@z ResidualTemporalBlock1D.__init__s "<{KK #L, LL *:66 )\:: 9E 8T8TBIlL! 4 4 4Z\ZeZgZg rQrtrTc||}||}||t|z}||}|||zS)z Args: inputs : [ batch_size x inp_channels x horizon ] t : [ batch_size x embed_dim ] returns: out : [ batch_size x out_channels x horizon ] )rrrrrr)rNrrouts rPrczResidualTemporalBlock1D.forwardsp   a  MM!  ll6""^A%6%66mmC  T''////rQ)rrrrns@rPrrs   45       3c3h/0         & 0el 0u| 0 0 0 0 0 0 0 0 0rQrc reZdZdZ ddedeededeffd Zdd ej d ed ej fdZ xZ S)TemporalConvLayera Temporal convolutional layer that can be used for video (sequence of images) input Code mostly copied from: https://github.com/modelscope/modelscope/blob/1509fdb973e5871f37148a4b5e5964cafd43e64d/modelscope/models/multi_modal/video_synthesis/unet_sd.py#L1016 Parameters: in_dim (`int`): Number of input channels. out_dim (`int`): Number of output channels. dropout (`float`, *optional*, defaults to `0.0`): The dropout probability to use. Nrrin_dimout_dimr#norm_num_groupsc Bt|p|}||_||_t jt j||t jt j||dd|_ t jt j||t jt j |t j||dd|_ t jt j||t jt j |t j||dd|_ t jt j||t jt j |t j||dd|_ tj|j djtj|j djdS)Nr3r r )r rrr)r?r@rrrD SequentialrSiLUConv3drFrIrJconv3conv4initzeros_weightr>)rNrrr#rrOs rPr@zTemporalConvLayer.__init__s #V  ] L& 1 1 GII Ifgy) D D D   ] L' 2 2 GII Jw   Igvy) D D D    ] L' 2 2 GII Jw   Igvy) D D D    ] L' 2 2 GII Jw   Igvy) D D D     tz"~,--- tz"~*+++++rQr ra num_framesrTc|dddfd|f|jddzddddd}|}||}||}||}||}||z}|ddddd|jd|jdzdf|jddz}|S)Nrr rrr3r)reshaper\permuterFrJrr)rNraridentitys rPrczTemporalConvLayer.forward s $' " * *B +;m>QRSRTRT>U+U V V ^ ^_`bcefhikl m m ! =11  =11  =11  =11  =0 %--aAq!<<DD   #m&9!&< z0TemporalResnetBlock.__init__..7s///a16///rQrTrvr r4rsilur)r?r@r0r!rHrDrrBrrFrrrGrIr#rJr rKr+r")rNr0r!r$r'r5r7rOs rPr@zTemporalResnetBlock.__init__*s &&2&:{{ ( //;///X''2KUXae'ff Y  #       $!#=,!G!GD  !%D X''2LVYbf'gg x'',, Y  #     +622#/<?!   !# """D     rQrRrSrTc2|}||}||}||}|jY||}||ddddddddf}|ddddd}||z}||}||}||}||}|j||}||z}|S)Nrrr r3r) rBrKrFrrrGr#rJr")rNrRrSrarbs rPrczTemporalResnetBlock.forward`s$  =11 ))-88  =11   )$$T**D%%d++AAAqqq!!!T4,?@D<<1aA..D)D0M =11 ))-88  ]33  =11   )--l;;L$}4 rQ)Nrrrrns@rPrrs  '+  444sm4 4  444444lEL rQrceZdZdZ dded eed ed ed eed edeffd Z dde j dee j dee j fdZ xZ S)SpatioTemporalResBlocka A SpatioTemporal Resnet block. Parameters: in_channels (`int`): The number of channels in the input. out_channels (`int`, *optional*, default to be `None`): The number of output channels for the first conv2d layer. If None, same as `in_channels`. temb_channels (`int`, *optional*, default to `512`): the number of channels in timestep embedding. eps (`float`, *optional*, defaults to `1e-6`): The epsilon to use for the spatial resenet. temporal_eps (`float`, *optional*, defaults to `eps`): The epsilon to use for the temporal resnet. merge_factor (`float`, *optional*, defaults to `0.5`): The merge factor to use for the temporal mixing. merge_strategy (`str`, *optional*, defaults to `learned_with_images`): The merge strategy to use for the temporal mixing. switch_spatial_to_temporal_mix (`bool`, *optional*, defaults to `False`): If `True`, switch the spatial and temporal mixing. Nrr?learned_with_imagesFr0r!r$r' temporal_eps merge_factorswitch_spatial_to_temporal_mixc tt|||||_t ||n|||n||||n||_t ||||_dS)N)r0r!r$r')alphamerge_strategyr)r?r@rpspatial_res_blockrtemporal_res_block AlphaBlender time_mixer) rNr0r!r$r'rrrrrOs rPr@zSpatioTemporalResBlock.__init__s !.#%' " " " #6(4(@ k)5)A{' , 8 c # # # ')+I   rQrarSimage_only_indicatorcB|jd}|||}|j\}}}}||z} |dddf| ||||ddddd} |dddf| ||||ddddd}||| |d}|||}|| ||}|ddddd||||}|S)Nrrrr r3r) x_spatial x_temporalr)r\rrrrr) rNrarSrr batch_frameschannelsheightwidth batch_sizehidden_states_mixs rPrczSpatioTemporalResBlock.forwards` */3 ..}dCC 0=0C- h!Z/  $' " * *:z8VUZ [ [ c cdeghjkmnpq r r  $' " * *:z8VUZ [ [ c cdeghjkmnpq r r   << J;;D// tDD '$!5(  &--aAq!<<DD\S[]cejkk rQ)NrrNrrF)NN) rdrerfrgrhrrjrir@rHrlrcrmrns@rPrr{s('+ (,!,/4   sm     uo   )-      H(,7; |u|$'u|4 rQrc eZdZdZgdZ ddededeffd Zd e j d e d e j fd Z dde j de j d e e j d e j fdZxZS)ra A module to blend spatial and temporal features. Parameters: alpha (`float`): The initial value of the blending factor. merge_strategy (`str`, *optional*, defaults to `learned_with_images`): The merge strategy to use for the temporal mixing. switch_spatial_to_temporal_mix (`bool`, *optional*, defaults to `False`): If `True`, switch the spatial and temporal mixing. )learnedfixedrrFrrrct||_||_||jvrt d|j|jdkr+|dtj|gdS|jdks |jdkrH| dtj tj|gdSt d|j)Nzmerge_strategy needs to be in r mix_factorrrzUnknown merge strategy ) r?r@rr strategiesrCregister_bufferrHrlregister_parameterrD Parameter)rNrrrrOs rPr@zAlphaBlender.__init__s ,.L+  0 0OdoOOPP P  ' ) )  u|UG/D/D E E E E E  I - -1DH]1]1]  # #L%(2D2DU\SXRYEZEZ2[2[ \ \ \ \ \Lt7JLLMM MrQrndimsrTc|jdkr|j}n|jdkrtj|j}n|jdkr|t dtj|tjdd|jtj|jd}|dkr|dddddddf}nA|d kr!| d ddddf}nt d |d t|S) NrrrzMPlease provide image_only_indicator to use learned_with_images merge strategyr )device).Nrr3rzUnexpected ndims z. Dimensions should be 3 or 5) rrrHsigmoidrCwhererionesrrNotImplementedError)rNrrrs rP get_alphazAlphaBlender.get_alphas&  ' ) )OEE  I - -M$/22EE  $9 9 9#+ !pqqqK$))++ 1a(<(CDDD do..y9Ezzaaaqqq$45! b))!!!T4-8 !YU!Y!Y!YZZZ& % rQNrrc|||j}||j}|jrd|z }||zd|z |zz}|S)Nr )rndimtodtyper)rNrrrrrs rPrczAlphaBlender.forwards^ 3Y^DD))  . %KE I u : :rQ)rFr)rdrerfrgrrjrkrir@rHrlrhrrrcrmrns@rPrrs  =<rs )))))))))) '''''',,,,,,('''''NNNNNBINNNbxxxxxBIxxxxO5<OELOOOO     ")   H,0,0,0,0,0bi,0,0,0^DDDDD DDDNYYYYY")YYYzQQQQQRYQQQhNNNNN29NNNNNrQ