wi`ddlZddlmZmZmZmZddlZddlmZddl m Z m Z ddl m Z ddlmZe jeZGdd eZGd d ee ZGd d ejZGddejZGddejZGddejZGddejZGddejZGddejZdS)N)CallableListOptionalUnion) ConfigMixinregister_to_config)logging) ModelMixinc eZdZdZdedffd Zddejdeee deejfd Z dd e e e jfd ed ededee f dZedee e e jffdZxZS) MultiAdaptera MultiAdapter is a wrapper model that contains multiple adapter models and merges their outputs according to user-assigned weighting. This model inherits from [`ModelMixin`]. Check the superclass documentation for the generic methods the library implements for all the model (such as downloading or saving, etc.) Parameters: adapters (`List[T2IAdapter]`, *optional*, defaults to None): A list of `T2IAdapter` model instances. adapters T2IAdaptercztt|t||_t j||_t|dkrtdt|dkrtd|dj }|dj }tdt|D][}||j |ks||j |kr7td|d|d|d||j d|d ||j \||_ ||_ dS) NrzExpecting at least one adapterr zQFor a single adapter, please use the `T2IAdapter` class instead of `MultiAdapter`zjExpecting all adapters to have the same downscaling behavior, but got: adapters[0].total_downscale_factor=z adapters[0].downscale_factor=z adapter[`z`].total_downscale_factor=z`].downscale_factor=) superr__init__len num_adapternn ModuleListr ValueErrortotal_downscale_factordownscale_factorrange)selfr$first_adapter_total_downscale_factorfirst_adapter_downscale_factoridx __class__s h/root/.openclaw/workspace/chatterbox_venv_py311/lib/python3.11/site-packages/diffusers/models/adapter.pyrzMultiAdapter.__init__)s lD!!**,,,x== h// x==A  =>> > x==A  pqq q08{/Q,)1!)E&CMM**  C 48\\\C=15SSS Z:^ZZ4RZZ!$ZZ@H}?cZZ!$ ZZ:B#9W ZZT'K# >Nxsadapter_weightsreturnc|&tjd|jz g|jz}ntj|}d}t|||jD]{\}}}||}|0|}t t |D]}|||z||<Ct t |D]}||xx|||zz cc<||S)a Args: xs (`torch.Tensor`): (batch, channel, height, width) input images for multiple adapter models concated along dimension 1, `channel` should equal to `num_adapter` * "number of channel of image". adapter_weights (`List[float]`, *optional*, defaults to None): List of floats representing the weight which will be multiply to each adapter's output before adding them together. Nr )torchtensorrziprrr) rr#r$ accume_statexwadapterfeaturesis r!forwardzMultiAdapter.forwardLs  "#lA0@,@+ADDT+TUUOO#l?;;O  _dmDD 7 7MAq'wqzzH#' s<0011::A&',q/&9LOO:s8}}--77A OOOq8A;6OOOO7r"Tsave_directoryis_main_process save_functionsafe_serializationvariantcpd}|}|jD])}|||||||dz }|d|z}*dS)a3 Save a model and its configuration file to a directory, so that it can be re-loaded using the `[`~models.adapter.MultiAdapter.from_pretrained`]` class method. Arguments: save_directory (`str` or `os.PathLike`): Directory to which to save. Will be created if it doesn't exist. is_main_process (`bool`, *optional*, defaults to `True`): Whether the process calling this is the main process or not. Useful when in distributed training like TPUs and need to call this function on all processes. In this case, set `is_main_process=True` only on the main process to avoid race conditions. save_function (`Callable`): The function to use to save the state dictionary. Useful on distributed training like TPUs when one need to replace `torch.save` by another method. Can be configured with the environment variable `DIFFUSERS_SAVE_MODE`. safe_serialization (`bool`, *optional*, defaults to `True`): Whether to save the model using `safetensors` or the traditional PyTorch way (that uses `pickle`). variant (`str`, *optional*): If specified, weights are saved in the format pytorch_model..bin. r)r2r3r4r5r _N)rsave_pretrained) rr1r2r3r4r5rmodel_path_to_saver-s r!r8zMultiAdapter.save_pretrainedgsy8+} @ @G  # #" /+#5 $    1HC!3i#ii!?   @ @r"pretrained_model_pathc d}g}|}tj|rStj|fi|}|||dz }|d|z}tj|St t|d|dt|dkr6tdtj |d|dzd||S) a[ Instantiate a pretrained MultiAdapter model from multiple pre-trained adapter models. The model is set in evaluation mode by default using `model.eval()` (Dropout modules are deactivated). To train the model, you should first set it back in training mode with `model.train()`. The warning *Weights from XXX not initialized from pretrained model* means that the weights of XXX do not come pretrained with the rest of the model. It is up to you to train those weights with a downstream fine-tuning task. The warning *Weights from XXX not used in YYY* means that the layer XXX is not used by YYY, therefore those weights are discarded. Parameters: pretrained_model_path (`os.PathLike`): A path to a *directory* containing model weights saved using [`~diffusers.models.adapter.MultiAdapter.save_pretrained`], e.g., `./my_model_directory/adapter`. torch_dtype (`str` or `torch.dtype`, *optional*): Override the default `torch.dtype` and load the model under this dtype. If `"auto"` is passed the dtype will be automatically derived from the model's weights. output_loading_info(`bool`, *optional*, defaults to `False`): Whether or not to also return a dictionary containing missing keys, unexpected keys and error messages. device_map (`str` or `Dict[str, Union[int, str, torch.device]]`, *optional*): A map that specifies where each submodule should go. It doesn't need to be refined to each parameter/buffer name, once a given module name is inside, every submodule of it will be sent to the same device. To have Accelerate compute the most optimized `device_map` automatically, set `device_map="auto"`. For more information about each option see [designing a device map](https://hf.co/docs/accelerate/main/en/usage_guides/big_modeling#designing-a-device-map). max_memory (`Dict`, *optional*): A dictionary device identifier to maximum memory. Will default to the maximum memory available for each GPU and the available CPU RAM if unset. low_cpu_mem_usage (`bool`, *optional*, defaults to `True` if torch version >= 1.9.0 else `False`): Speed up model loading by not initializing the weights and only loading the pre-trained weights. This also tries to not use more than 1x model size in CPU memory (including peak memory) while loading the model. This is only supported when torch version >= 1.9.0. If you are using an older version of torch, setting this argument to `True` will raise an error. variant (`str`, *optional*): If specified load weights from `variant` filename, *e.g.* pytorch_model..bin. `variant` is ignored when using `from_flax`. use_safetensors (`bool`, *optional*, defaults to `None`): If set to `None`, the `safetensors` weights will be downloaded if they're available **and** if the `safetensors` library is installed. If set to `True`, the model will be forcibly loaded from `safetensors` weights. If set to `False`, loading will *not* use `safetensors`. rr r7z adapters loaded from .zNo T2IAdapters found under z. Expected at least _0) ospathisdirrfrom_pretrainedappendloggerinforrdirname)clsr:kwargsrrmodel_path_to_loadr-s r!rAzMultiAdapter.from_pretraineds3` 3gmm.// C 01CNNvNNG OOG $ $ $ 1HC!6S!B  gmm.// C  s8}}TTS.T.TJJjCGkGJJJ s8}}r"N)TNTN)__name__ __module__ __qualname____doc__rrr'Tensorrfloatr0rstrr>PathLikeboolrr8 classmethodrA __classcell__r s@r!rrsJ  !?l!3!?!?!?!?!?!?F%,$u+9NZ^_d_kZl<!%"&#'!% (@(@c2;./(@(@ (@ ! (@ # (@(@(@(@TDHU3 CS=T4UDDD[DDDDDr"rc eZdZdZedgddddfdedeed ed ed ef fd Zd e j dee j fdZ e dZ e dZxZS)ra A simple ResNet-like model that accepts images containing control signals such as keyposes and depth. The model generates multiple feature maps that are used as additional conditioning in [`UNet2DConditionModel`]. The model's architecture follows the original implementation of [Adapter](https://github.com/TencentARC/T2I-Adapter/blob/686de4681515662c0ac2ffa07bf5dda83af1038a/ldm/modules/encoders/adapter.py#L97) and [AdapterLight](https://github.com/TencentARC/T2I-Adapter/blob/686de4681515662c0ac2ffa07bf5dda83af1038a/ldm/modules/encoders/adapter.py#L235). This model inherits from [`ModelMixin`]. Check the superclass documentation for the generic methods the library implements for all the model (such as downloading or saving, etc.) Parameters: in_channels (`int`, *optional*, defaults to 3): Number of channels of Aapter's input(*control image*). Set this parameter to 1 if you're using gray scale image as *control image*. channels (`List[int]`, *optional*, defaults to `(320, 640, 1280, 1280)`): The number of channel of each downsample block's output hidden state. The `len(block_out_channels)` will also determine the number of downsample blocks in the Adapter. num_res_blocks (`int`, *optional*, defaults to 2): Number of ResNet blocks in each downsample block. downscale_factor (`int`, *optional*, defaults to 8): A factor that determines the total downscale factor of the Adapter. adapter_type (`str`, *optional*, defaults to `full_adapter`): The type of Adapter to use. Choose either `full_adapter` or `full_adapter_xl` or `light_adapter`. @r[r full_adapter in_channelschannelsnum_res_blocksr adapter_typec$t|dkrt|||||_dS|dkrt |||||_dS|dkrt |||||_dSt d|d)Nr]full_adapter_xl light_adapterzUnsupported adapter_type: 'zH'. Choose either 'full_adapter' or 'full_adapter_xl' or 'light_adapter'.)rr FullAdapterr- FullAdapterXL LightAdapterr)rr^r_r`rrar s r!rzT2IAdapter.__init__s  > ) )&{HnN^__DLLL . . .(hP`aaDLLL _ , ,' X~O_``DLLL8l888 r"r+r%c,||S)a This function processes the input tensor `x` through the adapter model and returns a list of feature tensors, each representing information extracted at a different scale from the input. The length of the list is determined by the number of downsample blocks in the Adapter, as specified by the `channels` and `num_res_blocks` parameters during initialization. )r-rr+s r!r0zT2IAdapter.forward s||Ar"c|jjSrI)r-rrs r!rz!T2IAdapter.total_downscale_factors |22r"c$|jjjS)zThe downscale factor applied in the T2I-Adapter's initial pixel unshuffle operation. If an input image's dimensions are not evenly divisible by the downscale_factor then an exception will be raised. )r- unshufflerrks r!rzT2IAdapter.downscale_factors |%66r")rJrKrLrMr intrrPrr'rNr0propertyrrrTrUs@r!rrs4444 !* s)     ,$u|*<33X377X77777r"rc ~eZdZdZdgdddfdedeeded effd Zd ejd eejfd Z xZ S)re2 See [`T2IAdapter`] for more information. rWrXrr\r^r_r`rc t||dzz}tj||_tj|ddd|_tjtddgfdtdtD|_ |dtdz zz|_ dS)NrrrWr  kernel_sizepaddingcTg|]$}t|dz |d%Sr Tdown) AdapterBlock.0r/r_r`s r! z(FullAdapter.__init__..9sG!!a%(1+~TXYYYr") rrrPixelUnshufflermConv2dconv_inrrzrrbodyrrr^r_r`rr s `` r!rzFullAdapter.__init__(s !$4a$77 *+;<<yhqkqRSTTT MXa[(1+~FF "1c(mm44    '7s8}}q?P9Q&Q###r"r+r%c||}||}g}|jD]"}||}||#|S)a This method processes the input tensor `x` through the FullAdapter model and performs operations including pixel unshuffling, convolution, and a stack of AdapterBlocks. It returns a list of feature tensors, each capturing information at a different stage of processing within the FullAdapter model. The number of feature tensors in the list is determined by the number of downsample blocks specified during initialization. rmrrrBrr+r.blocks r!r0zFullAdapter.forwardBsa NN1   LLOOY  EaA OOA    r" rJrKrLrMrnrrr'rNr0rTrUs@r!rere#s 444 ! RRRs)R R  RRRRRR4$u|*<r"rec ~eZdZdZdgdddfdedeeded effd Zd ejd eejfd Z xZ S)rfrqrWrXrr^r_r`rc t||dzz}tj||_tj||ddd|_g|_tt|D]}|dkr9|j t||dz |||A|dkr;|j t||dz |||d|j t|||||tj |j|_|dz|_ dS)NrrrWr rsTrx)rrrr~rmrrrrrrBrzrr)rr^r_r`rr/r s r!rzFullAdapterXL.__init__ZsP !$4a$77 *+;<<yhqkqRSTTT  s8}}%% Y YAAvv   hq1uox{N![![\\\\a   hq1uox{Nae!f!f!fgggg   hqk8A;!W!WXXXXM$),, &6&:###r"r+r%c||}||}g}|jD]"}||}||#|S)z This method takes the tensor x as input and processes it through FullAdapterXL model. It consists of operations including unshuffling pixels, applying convolution layer and appending each block into list of feature tensors. rrs r!r0zFullAdapterXL.forwardvsa NN1   LLOOY  EaA OOA    r"rrUs@r!rfrfUs 444 " ;;;s); ;  ;;;;;;8$u|*<r"rfc ZeZdZdZd dedededeffd Zdejd ejfd Z xZ S) rza) An AdapterBlock is a helper model that contains multiple ResNet-like blocks. It is used in the `FullAdapter` and `FullAdapterXL` models. Parameters: in_channels (`int`): Number of channels of AdapterBlock's input. out_channels (`int`): Number of channels of AdapterBlock's output. num_res_blocks (`int`): Number of ResNet blocks in the AdapterBlock. down (`bool`, *optional*, defaults to `False`): Whether to perform downsampling on AdapterBlock's input. Fr^ out_channelsr`ryc>td|_|rtjddd|_d|_|krtj|d|_tjfdt|D|_ dS)NrTrtstride ceil_moder rtc.g|]}tS)AdapterResnetBlock)r|r7rs r!r}z)AdapterBlock.__init__..s" N N N1 .. N N Nr") rr downsampler AvgPool2din_convr Sequentialrresnets)rr^rr`ryr s ` r!rzAdapterBlock.__init__s   T lqdSSSDO , & &9[,ANNNDL} N N N Nn8M8M N N N  r"r+r%c|j||}|j||}||}|S)a This method takes tensor x as input and performs operations downsampling and convolutional layers if the self.downsample and self.in_conv properties of AdapterBlock model are specified. Then it applies a series of residual blocks to the input tensor. )rrrris r!r0zAdapterBlock.forwardsH ? &""A < # QA LLOOr"F rJrKrLrMrnrRrr'rNr0rTrUs@r!rzrzs     C  s  C  W[       %,r"rzcLeZdZdZdeffd ZdejdejfdZxZ S)rz An `AdapterResnetBlock` is a helper model that implements a ResNet-like block. Parameters: channels (`int`): Number of channels of AdapterResnetBlock's input and output. r_cttj||dd|_tj|_tj||d|_dS)NrWr rsrrrrrblock1ReLUactblock2rr_r s r!rzAdapterResnetBlock.__init__s\ i(1MMM 799i(BBB r"r+r%c|||}||}||zS)z This method takes input tensor x and applies a convolutional layer, ReLU activation, and another convolutional layer on the input tensor. It returns addition with the input tensor. rrrrr+hs r!r0zAdapterResnetBlock.forward7 HHT[[^^ $ $ KKNN1u r" rJrKrLrMrnrr'rNr0rTrUs@r!rrs{CCCCCCC  %,        r"rc ~eZdZdZdgdddfdedeeded effd Zd ejd eejfd Z xZ S)rgrqrW)rYrZr[r\r^r_r`rc t||dzz}tj||_tjt |dgfdttdz Dt ddd|_ |dtzz|_ dS)NrrcTg|]$}t||dzd%Srw)LightAdapterBlockr{s r!r}z)LightAdapter.__init__..sG&hqk8AE?NY]^^^r"r Trx) rrrr~rmrrrrrrrs `` r!rzLightAdapter.__init__s !$4a$77 *+;<<M!+x{NKK "3x==1#455  "(2, nSWXXX      '7!s8}}:L&M###r"r+r%c||}g}|jD]"}||}||#|S)z This method takes the input tensor x and performs downscaling and appends it in list of feature tensors. Each feature tensor corresponds to a different level of processing within the LightAdapter. )rmrrBrs r!r0zLightAdapter.forwardsR NN1  Y  EaA OOA    r"rrUs@r!rgrgs ... ! NNNs)N N  NNNNNN4  $u|*<        r"rgc ZeZdZdZd dedededeffd Zdejd ejfd Z xZ S) raB A `LightAdapterBlock` is a helper model that contains multiple `LightAdapterResnetBlocks`. It is used in the `LightAdapter` model. Parameters: in_channels (`int`): Number of channels of LightAdapterBlock's input. out_channels (`int`): Number of channels of LightAdapterBlock's output. num_res_blocks (`int`): Number of LightAdapterResnetBlocks in the LightAdapterBlock. down (`bool`, *optional*, defaults to `False`): Whether to perform downsampling on LightAdapterBlock's input. Fr^rr`rycft|dzd|_|rtjddd|_tj|d|_tjfdt|D|_ tj|d|_ dS)NrrTrr rc.g|]}tSr)LightAdapterResnetBlock)r|r7 mid_channelss r!r}z.LightAdapterBlock.__init__..s#&l&l&lQR'>|'L'L&l&l&lr") rrrrrrrrrrout_conv)rr^rr`ryrr s @r!rzLightAdapterBlock.__init__s #q(   T lqdSSSDOylJJJ }&l&l&l&lV[\jVkVk&l&l&lm  , !LLL r"r+r%c|j||}||}||}||}|S)z This method takes tensor x as input and performs downsampling if required. Then it applies in convolution layer, a sequence of residual blocks, and out convolutional layer. )rrrrris r!r0zLightAdapterBlock.forward sQ ? &""A LLOO LLOO MM!  r"rrrUs@r!rrs   M MC Ms MC MW[ M M M M M M  %,        r"rcLeZdZdZdeffd ZdejdejfdZxZ S)ra A `LightAdapterResnetBlock` is a helper model that implements a ResNet-like block with a slightly different architecture than `AdapterResnetBlock`. Parameters: channels (`int`): Number of channels of LightAdapterResnetBlock's input and output. r_cttj||dd|_tj|_tj||dd|_dS)NrWr rsrrs r!rz LightAdapterResnetBlock.__init__9s^ i(1MMM 799i(1MMM r"r+r%c|||}||}||zS)z This function takes input tensor x and processes it through one convolutional layer, ReLU activation, and another convolutional layer and adds it to input tensor. rrs r!r0zLightAdapterResnetBlock.forward?rr"rrUs@r!rr/s{NNNNNNN  %,        r"r)r>typingrrrrr'torch.nnrconfiguration_utilsrr utilsr modeling_utilsr get_loggerrJrCrrModulererfrzrrgrrrr"r!rsJ 222222222222 AAAAAAAA&&&&&&  H % %zzzzz:zzzzD7D7D7D7D7[D7D7D7T/////")///d/////BI///d-----29---`<,,,,,29,,,^((((( (((Vbir"