wiSddlZddlZddlZddlmZmZmZmZmZm Z m Z ddl Z ddl Z ddlmZddlmZddlmZmZmZmZmZmZmZerddlZerddlmZerddlmZerddlZd e fd Z!d Z"d e#fd Z$ d"dedede j%de j%de j%de j%de j%de&de ee j%ee j%ffdZ'dedee#e j%ffdZ(e j)fde e j*j+ee j*j+ffdZ,dee#e j%fde#de j*j+fdZ-Gd d!Z.dS)#N)AnyDictIterableListOptionalTupleUnion)UNet2DConditionModel)SchedulerMixin)convert_state_dict_to_diffusersconvert_state_dict_to_peft deprecateis_peft_availableis_torch_npu_availableis_torchvision_availableis_transformers_available)set_peft_model_state_dict) transformsseedc0tj|tj|tj|t r!tj|dStj|dS)z Args: Helper function for reproducible behavior to set the seed in `random`, `numpy`, `torch`. seed (`int`): The seed to set. N) randomrnptorch manual_seedrnpumanual_seed_allcuda)rs h/root/.openclaw/workspace/chatterbox_venv_py311/lib/python3.11/site-packages/diffusers/training_utils.pyset_seedr #s  KINN4 d) !!$''''' ""4(((((c|j}|dz}d|z dz}||j|}t |jt |jkr2|d}t |jt |jk2||j}||j|}t |jt |jkr2|d}t |jt |jk2||j}||z dz}|S)z Computes SNR as per https://github.com/TiankaiHang/Min-SNR-Diffusion-Training/blob/521b624bd70c67cee4bdf49225915f5945a872e3/guided_diffusion/gaussian_diffusion.py#L847-L849 ??device).N)alphas_cumprodtor&floatlenshapeexpand)noise_scheduler timestepsr(sqrt_alphas_cumprodsqrt_one_minus_alphas_cumprodalphasigmasnrs r compute_snrr53s] %3N(#-%(>%9c$A!.00 8H0II)TZZ\\ !' ( (3y+?+? ? ?1)< !' ( (3y+?+? ? ?  & &y 7 7E$A$D$DIL\$D$]$]^g$h$n$n$p$p! +1 2 2S5I5I I I(Ei(P% +1 2 2S5I5I I I ) 0 0 A AE 5=Q C Jr!interpolation_typectstd|dkrtjj}n|dkrtjj}n|dkrtjj}ns|dkrtjj}n[|dkrtjj}nC|dkrtjj }n+|dkrtjj }ntd |d |S) a Maps a string describing an interpolation function to the corresponding torchvision `InterpolationMode` enum. The full list of supported enums is documented at https://pytorch.org/vision/0.9/transforms.html#torchvision.transforms.functional.InterpolationMode. Args: interpolation_type (`str`): A string describing an interpolation method. Currently, `bilinear`, `bicubic`, `box`, `nearest`, `nearest_exact`, `hamming`, and `lanczos` are supported, corresponding to the supported interpolation modes in torchvision. Returns: `torchvision.transforms.InterpolationMode`: an `InterpolationMode` enum used by torchvision's `resize` transform. zhPlease make sure to install `torchvision` to be able to use the `resolve_interpolation_mode()` function.bilinearbicubicboxnearest nearest_exacthamminglanczoszThe given interpolation mode z is not supported. Currently supported interpolation modes are `bilinear`, `bicubic`, `box`, `nearest`, `nearest_exact`, `hamming`, and `lanczos`.) r ImportErrorrInterpolationModeBILINEARBICUBICBOXNEAREST NEAREST_EXACTHAMMINGLANCZOS ValueError)r6interpolation_modes rresolve_interpolation_moderJMs $ % %  v   Z'''9B y ( ('9A u $ $'9= y ( ('9A  . .'9G y ( ('9A y ( ('9A n,> n n n   r!r$unetr.r/noise noisy_latentstargetencoder_hidden_statesdream_detail_preservationreturncL|j|j|dddf}d|z dz} | |z} d} tj5||||j} dddn #1swxYwYd\} } |jjdkr\| }||z }| | | | |z} | |} n;|jjdkrtdtd|jj| | fS) a Implements "DREAM (Diffusion Rectification and Estimation-Adaptive Models)" from http://arxiv.org/abs/2312.00210. DREAM helps align training with sampling to help training be more efficient and accurate at the cost of an extra forward step without gradients. Args: `unet`: The state unet to use to make a prediction. `noise_scheduler`: The noise scheduler used to add noise for the given timestep. `timesteps`: The timesteps for the noise_scheduler to user. `noise`: A tensor of noise in the shape of noisy_latents. `noisy_latents`: Previously noise latents from the training loop. `target`: The ground-truth tensor to predict after eps is removed. `encoder_hidden_states`: Text embeddings from the text model. `dream_detail_preservation`: A float value that indicates detail preservation level. See reference. Returns: `tuple[torch.Tensor, torch.Tensor]`: Adjusted noisy_latents and target. Nr$r#NNepsilon v_predictionz/DREAM has not been implemented for v-predictionzUnknown prediction type ) r(r)r&rno_gradsampleconfigprediction_typedetachmul_addNotImplementedErrorrH)rKr.r/rLrMrNrOrPr(r1 dream_lambdapred_noisy_latents_targetpredicted_noise delta_noises r compute_dream_and_update_latentsrdys:%366y7GHHTXZ^`dIdeN%(>%9c$A!12KKL D LLtM9.CDDKLLLLLLLLLLLLLLL+NG-::.6688 &&&&**+H;+VWW**[))   /> A A!"STTT\O4J4Z\\]]] 7 ""s A++A/2A/ci}|D]_\}}t|drJt|d}|8|}|D]\}}|||d|<`|S)zL Returns: A state dict containing just the LoRA parameters. set_lora_layer lora_layerNz.lora.) named_moduleshasattrgetattr state_dictitems)rKlora_state_dictnamemodulergcurrent_lora_layer_sdlora_layer_matrix_name lora_params runet_lora_state_dictrss O**,,ZZ f 6+ , , Z 66J%(2(=(=(?(?%:O:U:U:W:WZZ6*JOYOt$K$K3I$K$KLL r!modelct|ts|g}|D]:}|D]#}|jr|||_$;dSN) isinstancelist parameters requires_gradr)data)rtdtypemparams rcast_training_paramsrsm eT " " --\\^^ - -E" -"XXe__  ---r!rmprefix text_encodercfd|D}tt|}t||ddS)aD Sets the `lora_state_dict` into `text_encoder` coming from `transformers`. Args: lora_state_dict: The state dictionary to be set. prefix: String identifier to retrieve the portion of the state dict that belongs to `text_encoder`. text_encoder: Where the `lora_state_dict` is to be set. cpi|]2\}}||d|3S)) startswithreplace).0kvrs r z5_set_state_dict_into_text_encoder..sR*.!QQ\\Z`MaMa 99VR "Ar!default) adapter_nameN)rlrr r)rmrrtext_encoder_state_dicts ` r!_set_state_dict_into_text_encoderrso2A2G2G2I2I99XYp9q9qrrl,CR[\\\\\\r!ceZdZdZ d d eejjd ed ed e d e de ee fde ee fde e deee ffdZed!dZdZde defdZejd eejjfdZd eejjddfdZd"d#dZdefdZd eejjddfdZd eejjddfdZdeddfdZdS)$EMAModelz6 Exponential Moving Average of models weights H.?rFr$UUUUUU?Nrydecay min_decayupdate_after_stepuse_ema_warmup inv_gammapower model_cls model_configc  t|tjjr+d} t dd| d|}d}| ddd } t dd| d| d}| d dd } t d d| d| d }t|}d |D|_| d d1d} t d d| d| | d d|_ ||_ ||_ ||_ ||_||_||_d|_d|_||_| |_dS)a Args: parameters (Iterable[torch.nn.Parameter]): The parameters to track. decay (float): The decay factor for the exponential moving average. min_decay (float): The minimum decay factor for the exponential moving average. update_after_step (int): The number of steps to wait before starting to update the EMA weights. use_ema_warmup (bool): Whether to use EMA warmup. inv_gamma (float): Inverse multiplicative factor of EMA warmup. Default: 1. Only used if `use_ema_warmup` is True. power (float): Exponential factor of EMA warmup. Default: 2/3. Only used if `use_ema_warmup` is True. device (Optional[Union[str, torch.device]]): The device to store the EMA weights on. If None, the EMA weights will be stored on CPU. @crowsonkb's notes on EMA Warmup: If gamma=1 and power=1, implements a simple average. gamma=1, power=2/3 are good values for models you plan to train for a million or more steps (reaches decay factor 0.999 at 31.6K steps, 0.9999 at 1M steps), gamma=1, power=3/4 for models you plan to train for less (reaches decay factor 0.999 at 10K steps, 0.9999 at 215.4k steps). zzPassing a `torch.nn.Module` to `ExponentialMovingAverage` is deprecated. Please pass the parameters of the module instead.z9passing a `torch.nn.Module` to `ExponentialMovingAverage`1.0.0F standard_warnT max_valueNzCThe `max_value` argument is deprecated. Please use `decay` instead. min_valuezGThe `min_value` argument is deprecated. Please use `min_decay` instead.cZg|](}|)S)clonerZrps r z%EMAModel.__init__..!s,EEEQaggii..00EEEr!r&z=The `device` argument is deprecated. Please use `to` instead.r%r)rwrnnModulerrygetrx shadow_paramsr)temp_stored_paramsrrrrrroptimization_stepcur_decay_valuerr) selfryrrrrrrrrkwargsdeprecation_messages r__init__zEMAModel.__init__sB j%(/ 2 2 "D  K##      $..00J"N ::k4 ( ( 4"g  k7,?u U U U U;'E ::k4 ( ( 4"k  k7,?u U U U U{+I*%% EE*EEE ::h % % 1"a  h) !ijj j   $lmm m**4+<==__&& ---  ..:... U%%''((( d#####r!rctd||jz dz }|dkrdS|jrdd||jz z|j zz }n d|zd|zz }t ||j}t||j}|S)zN Compute the decay factor for the exponential moving average. rr r )maxrrrrminrr)rrsteprs r get_decayzEMAModel.get_decayOs1'$*@@1DEE 1993   71tdn'<#<$*"LLOO 4xBI6Ootz::ot~>>r!ct|tjjr)d}t dd|d|}t |}|xjdz c_||j}||_ d|z }tj }tr"tjrddl}t!|j|D]\}}tr:tjr|j|d}|5|jr||||z zn||dddn #1swxYwYdS) NzPassing a `torch.nn.Module` to `ExponentialMovingAverage.step` is deprecated. Please pass the parameters of the module instead.z>passing a `torch.nn.Module` to `ExponentialMovingAverage.step`rFrr r) modifier_rank)rwrrrrryrxrrr contextlib nullcontextr transformers deepspeedis_deepspeed_zero3_enabledziprzeroGatheredParametersrzsub_copy_) rryrrone_minus_decaycontext_managerrs_paramr~s rrz EMAModel.stepbs j%(/ 2 2 1D  P##      $..00J*%%  !#t566$e)$0 $ & & <+A+\+\+^+^     !$"4jAA ) )NGU(** _|/E/`/`/b/b _"+."C"CEY]"C"^"^ "" ) )&)LLGeO!DEEEEMM%(((  ) ) ) ) ) ) ) ) ) ) ) ) ) ) )  ) )s69E;;E? E? ct|}t|j|D]<\}}|j||jj=dS)aa Copy current averaged parameters into given collection of parameters. Args: parameters: Iterable of `torch.nn.Parameter`; the parameters to be updated with the stored moving averages. If `None`, the parameters with which this `ExponentialMovingAverage` was initialized will be used. N)rxrrr{rr)r&)rryrr~s rrzEMAModel.copy_tose*%% !$"4jAA < .s^   120C0C0E0E ^ADDeD , , ,144W]4K^K^   r!N)r)rr&r|s ``rr)z EMAModel.tos=     '   r!c h|j|j|j|j|j|j|j|jdS)z Returns the state of the ExponentialMovingAverage as a dict. This method is used by accelerate during checkpointing to save the ema state dict. rrrrrrrrr)rs rrkzEMAModel.state_dicts?Z!%!7!%!7"1Z!/   r!c(d|D|_dS)z Args: Save the current parameters for restoring later. parameters: Iterable of `torch.nn.Parameter`; the parameters to be temporarily stored. c~g|]:}|;Sr)rZcpur)rr~s rrz"EMAModel.store..s8"X"X"XE5<<>>#5#5#7#7#=#=#?#?"X"X"Xr!N)r)rrys rstorezEMAModel.stores!#Y"XZ"X"X"Xr!c|jtdt|j|D]$\}}|j|j%d|_dS)aF Args: Restore the parameters stored with the `store` method. Useful to validate the model with EMA parameters without: affecting the original optimization process. Store the parameters before the `copy_to()` method. After validation (or model saving), use this to restore the former parameters. parameters: Iterable of `torch.nn.Parameter`; the parameters to be updated with the stored parameters. If `None`, the parameters with which this `ExponentialMovingAverage` was initialized will be used. NzGThis ExponentialMovingAverage has no `store()`ed weights to `restore()`)r RuntimeErrorrr{r)rryc_paramr~s rrestorezEMAModel.restoresg  " *kll l!$"9:FF + +NGU J  W\ * * * *#'r!rkc.tj|}|d|j|_|jdks |jdkrt d|d|j|_t |jtst d|d|j|_t |jtst d|d |j |_ t |j tst d |d |j |_ t |j tst d |d |j |_ t |j ttfst d|d|j|_t |jttfst d|dd}|]||_t |jt st dt#d|jDst ddSdS)a Args: Loads the ExponentialMovingAverage state. This method is used by accelerate during checkpointing to save the ema state dict. state_dict (dict): EMA state. Should be an object returned from a call to :meth:`state_dict`. rrr$zDecay must be between 0 and 1rzInvalid min_decayrzInvalid optimization_steprzInvalid update_after_steprzInvalid use_ema_warmuprzInvalid inv_gammarz Invalid powerrNzshadow_params must be a listc3JK|]}t|tjVdSrv)rwrTensorrs r z+EMAModel.load_state_dict..s.OOqz!U\22OOOOOOr!z!shadow_params must all be Tensors)copydeepcopyrrrHrrwr*rintrrboolrrrrxall)rrkrs rrzEMAModel.load_state_dictsM]:.. ^^GTZ88 :  tzC//<== =# T^DD$.%00 2011 1!+0CTE[!\!\$0#66 :899 9!+0CTE[!\!\$0#66 :899 9(nn-=t?RSS$-t44 7566 6# T^DD$.5#,77 2011 1^^GTZ88 $*ucl33 ._-- -"==  $!.D d0$77 A !?@@@OOD'4''''$-F$-F4-F-F-F-F-F-Fr!r)r$)/rrrtypingrrrrrrr numpyrrmodelsr schedulersr utilsr rrrrrrrpeftr torchvisionr torch_npurr r5rrJrr*rdrsfloat32rrrrrrr!rrsD DDDDDDDDDDDDDDDDDD ((((((&&&&&&/......'&&&&&& )3 ) ) ) ) 4)3))))h(+3#3# 3##3#|3# < 3# < 3# L 3#!<3# %3# 8EL !8EL#9 9:3#3#3#3#l3S%,=N8O&V[Ub--ehotEHO7L&L M----]#u|+,]69]IN]]]](\F\F\F\F\F\F\F\F\F\Fr!