wi.8ddlZddlZddlZddlmZddlmZmZmZm Z m Z ddl Z ddl Z ddl mZddlmZddlmZddl mZdd lmZdd lmZmZmZmZmZmZmZmZmZm Z m!Z!m"Z"m#Z#m$Z$m%Z%d d l&m'Z'm(Z(e rdd l)m*Z*ddl+m,Z,m-Z-er ddl.m/Z/m0Z0m1Z1e!j2e3Z4dZ5dZ6dZ7dZ8dZ9dZ:GddZ;Gdde;Z<GddZ=dS)N)Path)CallableDictListOptionalUnion) model_info)HF_HUB_OFFLINE)validate_hf_hub_args)nn)load_state_dict)USE_PEFT_BACKEND_get_model_fileconvert_state_dict_to_diffusersconvert_state_dict_to_peftdelete_adapter_layersget_adapter_nameget_peft_kwargsis_accelerate_availableis_peft_versionis_transformers_availableloggingrecurse_remove_peft_layersscale_lora_layersset_adapter_layers!set_weights_and_activate_adapters) _convert_kohya_lora_to_diffusers"_maybe_map_sgm_blocks_to_diffusers)PreTrainedModel)text_encoder_attn_modulestext_encoder_mlp_modules)AlignDevicesHook CpuOffloadremove_hook_from_module text_encoderunet transformerzpytorch_lora_weights.binz pytorch_lora_weights.safetensorszYou are using an old version of LoRA backend. This will be deprecated in the next releases in favor of PEFT make sure to install the latest PEFT and transformers packages in the future.ceZdZdZeZeZeZ dZ d7de e e e ejfffdZeede e e e ejfffdZe d8d Zed Zed9d Ze d:d Zed9dZedefdZdZe d;de e ejfde e e ejj ejffde e ejj fde e ejj fde!de de"de!fdZ#e$de e ejfde de!de de"de!f dZ%dZ& dLoraLoaderMixinz Load LoRA layers into [`UNet2DConditionModel`] and [`CLIPTextModel`](https://huggingface.co/docs/transformers/model_doc/clip#transformers.CLIPTextModel). rN%pretrained_model_name_or_path_or_dictc :tstdt|tr|}|j|fi|\}}t d|D}|std|||t|dst||j n|j ||| ||t|dst||jn|j|j||dS) a_ Load LoRA weights specified in `pretrained_model_name_or_path_or_dict` into `self.unet` and `self.text_encoder`. All kwargs are forwarded to `self.lora_state_dict`. See [`~loaders.LoraLoaderMixin.lora_state_dict`] for more details on how the state dict is loaded. See [`~loaders.LoraLoaderMixin.load_lora_into_unet`] for more details on how the state dict is loaded into `self.unet`. See [`~loaders.LoraLoaderMixin.load_lora_into_text_encoder`] for more details on how the state dict is loaded into `self.text_encoder`. Parameters: pretrained_model_name_or_path_or_dict (`str` or `os.PathLike` or `dict`): See [`~loaders.LoraLoaderMixin.lora_state_dict`]. kwargs (`dict`, *optional*): See [`~loaders.LoraLoaderMixin.lora_state_dict`]. adapter_name (`str`, *optional*): Adapter name to be used for referencing the loaded adapter model. If not specified, it will use `default_{i}` where i is the total number of adapters being loaded. )PEFT backend is required for this method.c3&K|] }d|vpd|vV dSlora dora_scaleN.0keys f/root/.openclaw/workspace/chatterbox_venv_py311/lib/python3.11/site-packages/diffusers/loaders/lora.py z4LoraLoaderMixin.load_lora_weights..s0bb# D1DbbbbbbInvalid LoRA checkpoint.r(network_alphasr( adapter_name _pipeliner')r=r' lora_scaler>r?N)r ValueError isinstancedictcopylora_state_dictallkeysload_lora_into_unethasattrgetattr unet_namer(load_lora_into_text_encodertext_encoder_namer'r@)selfr,r>kwargs state_dictr=is_correct_formats r7load_lora_weightsz!LoraLoaderMixin.load_lora_weightsOs\4  JHII I ;T B B a4Y4^4^4`4` 1&:T%9:_%j%jci%j%j" NbbPZP_P_PaPabbbbb  9788 8   )6=dF6K6KZt~...QUQZ% !    (( )400#t'=>>>"% ) r:c |dd}|dd}|dd}|dd}|dd}|dd}|d d} |d d} |d d} |d d} |d d} d}| d} d}ddd}d}t|ts| r| | | dr | ||d|} t || pt ||||||| | | }tj |d}n)#ttj f$r}|s|d}Yd}~nd}~wwxYw|J| ||d|} t || pt||||||| | | }t|}n|}d}td|Dr$| t!|| }t#|\}}||fS)a Return state dict for lora weights and the network alphas. We support loading A1111 formatted LoRA checkpoints in a limited capacity. This function is experimental and might change in the future. Parameters: pretrained_model_name_or_path_or_dict (`str` or `os.PathLike` or `dict`): Can be either: - A string, the *model id* (for example `google/ddpm-celebahq-256`) of a pretrained model hosted on the Hub. - A path to a *directory* (for example `./my_model_directory`) containing the model weights saved with [`ModelMixin.save_pretrained`]. - A [torch state dict](https://pytorch.org/tutorials/beginner/saving_loading_models.html#what-is-a-state-dict). cache_dir (`Union[str, os.PathLike]`, *optional*): Path to a directory where a downloaded pretrained model configuration is cached if the standard cache is not used. force_download (`bool`, *optional*, defaults to `False`): Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist. resume_download: Deprecated and ignored. All downloads are now resumed by default when possible. Will be removed in v1 of Diffusers. proxies (`Dict[str, str]`, *optional*): A dictionary of proxy servers to use by protocol or endpoint, for example, `{'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}`. The proxies are used on each request. local_files_only (`bool`, *optional*, defaults to `False`): Whether to only load local model weights and configuration files or not. If set to `True`, the model won't be downloaded from the Hub. token (`str` or *bool*, *optional*): The token to use as HTTP bearer authorization for remote files. If `True`, the token generated from `diffusers-cli login` (stored in `~/.huggingface`) is used. revision (`str`, *optional*, defaults to `"main"`): The specific model version to use. It can be a branch name, a tag name, a commit id, or any identifier allowed by Git. subfolder (`str`, *optional*, defaults to `""`): The subfolder location of a model file within a larger model repository on the Hub or locally. weight_name (`str`, *optional*, defaults to None): Name of the serialized state dict file. cache_dirNforce_downloadFresume_downloadproxieslocal_files_onlytokenrevision subfolder weight_name unet_configuse_safetensorsTattn_procs_weightspytorch file_type framework .safetensors)file_extensionrX weights_namerTrUrVrWrXrYrZr[ user_agentcpudevicez.binc3K|]X}|dp>|dp)|dp|dVYdS)lora_te_ lora_unet_ lora_te1_ lora_te2_N) startswith)r5ks r7r8z2LoraLoaderMixin.lora_state_dict..s    Z((-<< ---<< ,,-<< ,,       r:)poprBrCendswith_best_guess_weight_namerLORA_WEIGHT_NAME_SAFE safetensorstorch load_fileIOErrorSafetensorErrorLORA_WEIGHT_NAMErrFrGr r)clsr,rOrTrUrVrWrXrYrZr[r\r]r^ allow_picklerh model_filerPer=s r7rEzLoraLoaderMixin.lora_state_dicts rJJ{D11 $4e<< **%6==**Y--!::&8$?? 7D))::j$//JJ{D11 jj55 jj55  **%6==  ""OL."   ?FF8 ? K$7'K,@,@,P,P'#*&)&A&AA+9-='B'' "1=%0%I4I"+'5(7 ')9#!)"+#- " " "J"-!2!##K-9!,!@0@'#1$3#%5%')    -Z88 >J    __&&     V&? KXX )I*)U)U &J>))s3AFF6&F11F6rdFc|strtdg}tj|rdStj|r!fdtj|D}n"t|j}fd|D}t|dkrdShdttfd|}td|Drttd|}n7td |Drttd |}t|d krtd d |d|d}|S)Nz>When using the offline mode, you must specify a `weight_name`.c>g|]}||Sr3)rtr5fres r7 z;LoraLoaderMixin._best_guess_weight_name..2s>PQPZPZ[iPjPjr:cRg|]#}|j|j$Sr3) rfilenamertrs r7rz;LoraLoaderMixin._best_guess_weight_name..7s1iiiaAKDXDXYgDhDhiakiiir:r> optimizer scheduler checkpointc<tfdDS)Nc3 K|]}|vV dSNr3)r5 substringxs r7r8zLLoraLoaderMixin._best_guess_weight_name....@s( Z Z !!3 Z Z Z Z Z Zr:)rF)runallowed_substringss`r7z9LoraLoaderMixin._best_guess_weight_name..@s&S Z Z Z ZEY Z Z ZZZr:c3JK|]}|tVdSrrtr|r5rs r7r8z:LoraLoaderMixin._best_guess_weight_name..Cs/DDqzz*++DDDDDDr:c6|tSrrrs r7rz9LoraLoaderMixin._best_guess_weight_name..Ds1::>N3O3Or:c3JK|]}|tVdSrrtrvrs r7r8z:LoraLoaderMixin._best_guess_weight_name..Es/KKq122KKKKKKr:c6|tSrrrs r7rz9LoraLoaderMixin._best_guess_weight_name..Fs1::>S3T3Tr:rz9Provided path contains more than one weights file in the z~ format. Either specify `weight_name` in `load_lora_weights` or make sure there's only one `.safetensors` or `.bin` file in .) r rAospathisfileisdirlistdirr siblingslenlistfilterany)r}r,rerXtargeted_files files_in_repor\rs ` @r7ruz'LoraLoaderMixin._best_guess_weight_name&s  _~ _]^^ ^ 7>>? @ @ j F W]]@ A A j:&KLLNN''LMMVMiiii=iiiN ~  ! # # F HGG ZZZZ\j k k   DD^DDD D D g!&)O)OQ_"`"`aaNN KKNKKK K K g!&)T)TVd"e"effN ~   " "CNCC[@CCC %Q' r:cd}d}||j|jD]\}}t|tjrt |dr|st|jt}|sTt|jtp9t |jdo$t|jj dt}t dt||||fSar Optionally removes offloading in case the pipeline has been already sequentially offloaded to CPU. Args: _pipeline (`DiffusionPipeline`): The pipeline to disable offloading for. Returns: tuple: A tuple indicating if `is_model_cpu_offload` or `is_sequential_cpu_offload` is True. FN_hf_hookhooksrzAccelerate hooks detected. Since you have called `load_lora_weights()`, the previous hooks will be first removed. Then the LoRA parameters will be loaded and the hooks will be applied again.)recurse hf_device_map componentsitemsrBr ModulerIrr%r$rloggerinfor&r}r?is_model_cpu_offloadis_sequential_cpu_offload_ components r7_optionally_disable_offloadingz.LoraLoaderMixin._optionally_disable_offloadingOs %$)!  Y%<%D ) 4 : : < < Z Z 9i33 Z :8V8V Z/Z/9):Lj/Y/Y,4&y'9;KLLZ&y'97CC Z *9+=+CA+FHX Y Y2 KKY,I?XYYYY$&?@@r:cftstdt|}t fd|D}t fd|Dr@|s@t djd| ||||dSdSdS)a This will load the LoRA layers specified in `state_dict` into `unet`. Parameters: state_dict (`dict`): A standard state dict containing the lora layer parameters. The keys can either be indexed directly into the unet or prefixed with an additional `unet` which can be used to distinguish between text encoder lora layers. network_alphas (`Dict[str, float]`): The value of the network alpha used for stable learning and preventing underflow. This value has the same meaning as the `--network_alpha` option in the kohya-ss trainer script. Refer to [this link](https://github.com/darkstorm2150/sd-scripts/blob/main/docs/train_network_README-en.md#execute-learning). unet (`UNet2DConditionModel`): The UNet model to load the LoRA layers into. adapter_name (`str`, *optional*): Adapter name to be used for referencing the loaded adapter model. If not specified, it will use `default_{i}` where i is the total number of adapters being loaded. r.c3LK|]}|jVdSr)rqrMr5r6r}s r7r8z6LoraLoaderMixin.load_lora_into_unet..s2VV#s/D E EVVVVVVr:c3LK|]}|jVdSr)rqrKrs r7r8z6LoraLoaderMixin.load_lora_into_unet..s1==s~~cm,,======r:Loading r)r=r>r?N) rrArrGrFrrrrKload_attn_procs)r}rPr=r(r>r?rGonly_text_encoders` r7rHz#LoraLoaderMixin.load_lora_into_unetrs(  JHII I JOO%%&&VVVVQUVVVVV ======= = = FW  KK33=333 4 4 4  > `i !         r:?ctstdddlm}t |} jntfd| Dr{fd| Dfd|D} t| dkr:t dd i} t| } t| } t|D] \} } | d }| |jd | |<!td | D}|rKt!|D];\} } | d }| d}| |jd | |<| |jd | |<<|Afd|Dfd|D}t#| || d}d|vrM|dr t%ddrtdn%t%ddr|d|di|}|t)|}|\}}||| |t/||||j|j|r|dS|r|dSdSdSdS)a? This will load the LoRA layers specified in `state_dict` into `text_encoder` Parameters: state_dict (`dict`): A standard state dict containing the lora layer parameters. The key should be prefixed with an additional `text_encoder` to distinguish between unet lora layers. network_alphas (`Dict[str, float]`): See `LoRALinearLayer` for more details. text_encoder (`CLIPTextModel`): The text encoder model to load the LoRA layers into. prefix (`str`): Expected prefix of the `text_encoder` in the `state_dict`. lora_scale (`float`): How much to scale the output of the lora linear layer before it is added with the output of the regular lora layer. adapter_name (`str`, *optional*): Adapter name to be used for referencing the loaded adapter model. If not specified, it will use `default_{i}` where i is the total number of adapters being loaded. r.r) LoraConfigNc3*K|] }j|vVdSr)rMrs r7r8z>LoraLoaderMixin.load_lora_into_text_encoder..s+<<s$+<<<<<.sE g g gqALL4H4H gQWWUX\\Z[_`fMfMfMfMfMfr:cRi|]#\}}|v |dd|$Srreplace)r5rrvrtext_encoder_keyss r7 z?LoraLoaderMixin.load_lora_into_text_encoder..sF,,,371aQRVgQgQg V,,,++QQgQgQgr:rrz.out_proj.lora_B.weightrc3K|]}d|vV dS)z.mlp.Nr3r4s r7r8z>LoraLoaderMixin.load_lora_into_text_encoder..s&^^33^^^^^^r:z.fc1.lora_B.weightz.fc2.lora_B.weightc|g|]8}|r!|ddk6|9Srrrs r7rz?LoraLoaderMixin.load_lora_into_text_encoder..sT"""ALL.sF&&&;?1a]^bl]l]l V,,,33Q]l]l]lr:F)is_unetuse_dora<0.9.0eYou need `peft` 0.9.0 at least to use DoRA-enabled LoRAs. Please upgrade your installation of `peft`.)r>adapter_state_dict peft_config)weight)rkdtyper3)rrApeftrrrGrMrrrrrrrr"shaper#rrrsrr load_adapterrtorkrenable_model_cpu_offloadenable_sequential_cpu_offload)r}rPr=r'rr@r>r?rrGtext_encoder_lora_state_dictranknamerrank_key patch_mlp rank_key_fc1 rank_key_fc2lora_config_kwargs lora_configrrrrs` ` @@r7rLz+LoraLoaderMixin.load_lora_into_text_encoders>  JHII I###### JOO%%&&*0.&&f <<< g g g gD g g g ,,,,,;E;K;K;M;M,,, (/00144 0v000111/NOk/l/l,0JJf/g/g,8FFUUGD!"&???H%A(%K%QRS%TDNN^^:V:[:[:]:]^^^^^ a#;L#I#Iaaa*.'B'B'B *.'B'B'B -I,-W-]^_-`\*-I,-W-]^_-`\**!-""""#1#6#6#8#8"""J&&&&&CQCWCWCYCY&&&N&5T>Kgqv%w%w%w"!333)*5?*388",!H## +388?.22:>>>(j>>+=>>  '#3L#A#ALBEBdBdenBoBo?$&?))!-'C +*",zBBBB|':,BTUUU(>6688888.>;;=====OG >G >54~>>r:cBddlm}m}m}t |} fd| Dfd|D}|Afd|Dfd|D}t|dkrV|t|divrtd |d i} |D]\} } d | vr| j d | | <t| ||} d | vr<| d rtddrtd| d |di| }|t|}|\}}||||||||}|1t|dd}|rt d|d|r|dS|r|dSdSdS)ab This will load the LoRA layers specified in `state_dict` into `transformer`. Parameters: state_dict (`dict`): A standard state dict containing the lora layer parameters. The keys can either be indexed directly into the unet or prefixed with an additional `unet` which can be used to distinguish between text encoder lora layers. network_alphas (`Dict[str, float]`): See `LoRALinearLayer` for more details. unet (`UNet2DConditionModel`): The UNet model to load the LoRA layers into. adapter_name (`str`, *optional*): Adapter name to be used for referencing the loaded adapter model. If not specified, it will use `default_{i}` where i is the total number of adapters being loaded. rrinject_adapter_in_modelset_peft_model_state_dictcHg|]}|j|Sr3rqtransformer_namer5rrr}s r7rz>LoraLoaderMixin.load_lora_into_transformer..!-RRR!q||C

LoraLoaderMixin.load_lora_into_transformer.."L   =AQ[\`p[p[pAII-000" 5 5q[p[p[pr:NcHg|]}|j|Sr3rrs r7rz>LoraLoaderMixin.load_lora_into_transformer..'s-aaaall3K_>`>`a!aaar:c\i|](\}}|v |jdd|)Srr)r5rrrrr}s r7rz>LoraLoaderMixin.load_lora_into_transformer..(sLAEAcdhrcrcr S1444b991crcrcrr:r Adapter name F already in use in the transformer - please select a new adapter name.lora_Brrrrrr>unexpected_keysXLoading adapter weights from state_dict led to unexpected keys not found in the model: . r3rrrrrrGrrrJrArrrrsrrrwarningrr)r}rPr=r)r>r?rrrrGrr6valrrrrincompatible_keysrrrs` @@r7load_lora_into_transformerz*LoraLoaderMixin.load_lora_into_transformer s$ XWWWWWWWWWJOO%%&&RRRRtRRR     EOEUEUEWEW     %aaaa^%8%8%:%:aaaJIWI]I]I_I_N z  ! !A % %w{M2FFFF xLxxxD&,,.. - -Ss?? # ! DI!0~z!R!R ///%j17oc76S6S7$'**:666$*::'9::K#/ << ?B>`>`aj>k>k ; "; # #K< X X X X 9 9+zS_ ` `  ,")*;=NPT"U"U"NN0+000 $ :2244444* :7799999[ & %X : :r:returnc4t|dr|jndS)N _lora_scaler)rIr rNs r7r@zLoraLoaderMixin.lora_scale\s!$+4#?#?HtSHr:c@t}t|dr9||jt|jdd|j`d|j_t|dr;||jt|jdd|j`d|j_dSdSdS)Nr'rtext_encoder_2)rrIr'rJr_hf_peft_config_loadedr)rN remove_methods r7!_remove_text_encoder_monkey_patchz1LoraLoaderMixin._remove_text_encoder_monkey_patchbs2 4 ( ( @ M$+ , , ,t(->>J%1;?!8 4) * * B M$- . . .t*M4@@L'3=A#:::  B BLLr:Tsave_directoryunet_lora_layerstext_encoder_lora_layerstransformer_lora_layersis_main_processr\ save_functionsafe_serializationc Ji} d} |s|s|std|r$| | ||j|r$| | ||j|r| | |d|| |||||dS)u Save the LoRA parameters corresponding to the UNet and text encoder. Arguments: save_directory (`str` or `os.PathLike`): Directory to save LoRA parameters to. Will be created if it doesn't exist. unet_lora_layers (`Dict[str, torch.nn.Module]` or `Dict[str, torch.Tensor]`): State dict of the LoRA layers corresponding to the `unet`. text_encoder_lora_layers (`Dict[str, torch.nn.Module]` or `Dict[str, torch.Tensor]`): State dict of the LoRA layers corresponding to the `text_encoder`. Must explicitly pass the text encoder LoRA state dict because it comes from 🤗 Transformers. is_main_process (`bool`, *optional*, defaults to `True`): Whether the process calling this is the main process or not. Useful during distributed training and you need to call this function on all processes. In this case, set `is_main_process=True` only on the main process to avoid race conditions. save_function (`Callable`): The function to use to save the state dictionary. Useful during distributed training when you need to replace `torch.save` with another method. Can be configured with the environment variable `DIFFUSERS_SAVE_MODE`. safe_serialization (`bool`, *optional*, defaults to `True`): Whether to save the model using `safetensors` or the traditional PyTorch way with `pickle`. ct|tjjr|n|}fd|D}|S)Nc&i|] \}}d||Srr3r5 module_nameparamrs r7rzKLoraLoaderMixin.save_lora_weights..pack_weights../ s s sFXkSXF!:!:[!:!:E s s sr:rBrxr rrPrlayersrlayers_weightslayers_state_dicts ` r7 pack_weightsz7LoraLoaderMixin.save_lora_weights..pack_weightsZ4>vux4W4WcV..000]cN s s s s\j\p\p\r\r s s s $ $r:zkYou must pass at least one of `unet_lora_layers`, `text_encoder_lora_layers`, or `transformer_lora_layers`.r)rPrrr\rrN)rAupdaterKrMwrite_lora_layers) r}rrrrrr\rrrPr's r7save_lora_weightsz!LoraLoaderMixin.save_lora_weightsqsD  % % % ! $< @W }   M   ll+;S]KK L L L # ]   ll+CSEZ[[ \ \ \ " T   ll+BMRR S S S !)+#'1      r:rPctj|r td|ddS||rd}n t j}tj|d||rt}nt}t|| }|||t d|dS)NProvided path (#) should be a directory, not a filecJtj||ddiSNformatpt)metadatarwrx save_fileweightsfilenames r7rz8LoraLoaderMixin.write_lora_layers..save_function'&,66wT\^bSc6dddr:Texist_okModel weights saved in rrrrerrorrxsavemakedirsrvr|ras_posixrrPrrr\rr save_paths r7r+z!LoraLoaderMixin.write_lora_layers 7>>. ) )  LL^>^^^ _ _ _ F  ! +eeee!&  NT2222  ! /3 . 55>>@@  j),,, 9i99:::::r:ctstdt|dst||jn|j}||dS) Unloads the LoRA parameters. Examples: ```python >>> # Assuming `pipeline` is already loaded with the LoRA parameters. >>> pipeline.unload_lora_weights() >>> ... ``` r.r(N)rrArIrJrKr( unload_lorarrNr(s r7unload_lora_weightsz#LoraLoaderMixin.unload_lora_weightsso  JHII I4;D&4I4IXwtT^,,,ty  ..00000r: fuse_unetfuse_text_encoderr@ safe_fusing adapter_namescddlm|s|r5|xjdz c_|jdkrtd|rDt |dst ||jn|j}| |||dfd }|rJt |d r||j ||| t |d r||j ||| d Sd Sd S)a Fuses the LoRA parameters into the original parameters of the corresponding blocks. This is an experimental API. Args: fuse_unet (`bool`, defaults to `True`): Whether to fuse the UNet LoRA parameters. fuse_text_encoder (`bool`, defaults to `True`): Whether to fuse the text encoder LoRA parameters. If the text encoder wasn't monkey-patched with the LoRA parameters then it won't have any effect. lora_scale (`float`, defaults to 1.0): Controls how much to influence the outputs with the LoRA parameters. safe_fusing (`bool`, defaults to `False`): Whether to check fused weights for NaN values before fusing and if values are NaN not fusing them. adapter_names (`List[str]`, *optional*): Adapter names to be used for fusing. If nothing is passed, all active adapters will be fused. Example: ```py from diffusers import DiffusionPipeline import torch pipeline = DiffusionPipeline.from_pretrained( "stabilityai/stable-diffusion-xl-base-1.0", torch_dtype=torch.float16 ).to("cuda") pipeline.load_lora_weights("nerijs/pixel-art-xl", weight_name="pixel-art-xl.safetensors", adapter_name="pixel") pipeline.fuse_lora(lora_scale=0.7) ``` rBaseTunerLayerrzThe current API is supported for operating with a single LoRA file. You are trying to load and fuse more than one LoRA which is not well-supported.r()rMrNrFNcBd|i}|D]}t|rr|dkr||tt j|jj}d|vr||d<nd|vr|td|jdi|dS)N safe_mergerrNzThe `adapter_names` argument is not supported with your PEFT version. Please upgrade to the latest version of PEFT. `pip install -U peft`r3) modulesrB scale_layerrinspect signaturemerge parametersrA)r'r@rMrN merge_kwargsmodulesupported_merge_kwargsrQs r7fuse_text_encoder_loraz9LoraLoaderMixin.fuse_lora..fuse_text_encoder_lora s(+6L&..00 1 1fn551!S((**:666.2'2CFL2Q2Q2\-]-]*&*@@@8E _55(0FFF=Kd(b !FL00<000! 1 1r:r')rNr)rFN) peft.tuners.tuners_utilsrQnum_fused_lorasrrrIrJrKr( fuse_lorar'r) rNrKrLr@rMrNr(r]rQs @r7r`zLoraLoaderMixin.fuse_loras\T <;;;;;  )   A % #a''j  ]8?f8M8M\74000SWS\D NN:;mN \ \ \ 1 1 1 1 1 1*  rt^,, p&&t'8*kanoooot-.. r&&t':J cpqqqqqq  r r r rr: unfuse_unetunfuse_text_encodercddlmt|dst||jn|j}|r;|D]&}t|r|'fd}|r@t|dr||j t|dr||j |xj dzc_ dS) af Reverses the effect of [`pipe.fuse_lora()`](https://huggingface.co/docs/diffusers/main/en/api/loaders#diffusers.loaders.LoraLoaderMixin.fuse_lora). This is an experimental API. Args: unfuse_unet (`bool`, defaults to `True`): Whether to unfuse the UNet LoRA parameters. unfuse_text_encoder (`bool`, defaults to `True`): Whether to unfuse the text encoder LoRA parameters. If the text encoder wasn't monkey-patched with the LoRA parameters then it won't have any effect. rrPr(c~|D]&}t|r|'dSr)rTrBunmerge)r'r[rQs r7unfuse_text_encoder_loraz=LoraLoaderMixin.unfuse_lora..unfuse_text_encoder_loraTsK&..00 % %fn55%NN$$$ % %r:r'rrN) r^rQrIrJrKr(rTrBrer'rr_)rNrarbr(r[rfrQs @r7 unfuse_lorazLoraLoaderMixin.unfuse_lora;s " <;;;;;4;D&4I4IXwtT^,,,ty  %,,.. % %fn55%NN$$$ % % % % %  >t^,, <(():;;;t-.. >(()<=== !r:r'r!text_encoder_weightsctstdd}t|tr|gn|}|||}|pt |dd}|tdt |||dS)a4 Sets the adapter layers for the text encoder. Args: adapter_names (`List[str]` or `str`): The names of the adapters to use. text_encoder (`torch.nn.Module`, *optional*): The text encoder module to set the adapter layers for. If `None`, it will try to get the `text_encoder` attribute. text_encoder_weights (`List[float]`, *optional*): The weights to use for the text encoder. If `None`, the weights are set to `1.0` for all the adapters. r.c t|ts|gt|z}t|t|kr/tdt|dt|d|D}|S)NLength of adapter names + is not equal to the length of the weights cg|]}||nd S)Nrr3)r5ws r7rzZLoraLoaderMixin.set_adapters_for_text_encoder..process_weights..s DDDqAMqqsDDDr:)rBrrrA)rNr8s r7process_weightszFLoraLoaderMixin.set_adapters_for_text_encoder..process_weightsvsgt,, 9")c-&8&88=!!S\\11 |s=/A/A||nqrynznz|| EDGDDDGNr:r'NzrThe pipeline does not have a default `pipe.text_encoder` class. Please make sure to pass a `text_encoder` instead.)rrArBstrrJr)rNrNr'rhros r7set_adapters_for_text_encoderz-LoraLoaderMixin.set_adapters_for_text_encoderas$  JHII I   ",6mS+I+I\} .}>RSS#Jwt^T'J'J  E  *, G[\\\\\r:ctstd|pt|dd}|tdt|ddS)a Disables the LoRA layers for the text encoder. Args: text_encoder (`torch.nn.Module`, *optional*): The text encoder module to disable the LoRA layers for. If `None`, it will try to get the `text_encoder` attribute. r.r'NText Encoder not found.Fenabled)rrArJrrNr's r7disable_lora_for_text_encoderz-LoraLoaderMixin.disable_lora_for_text_encoders`  JHII I#Jwt^T'J'J  677 7<777777r:ctstd|pt|dd}|tdt|jddS)a Enables the LoRA layers for the text encoder. Args: text_encoder (`torch.nn.Module`, *optional*): The text encoder module to enable the LoRA layers for. If `None`, it will try to get the `text_encoder` attribute. r.r'NrsTrt)rrArJrr'rvs r7enable_lora_for_text_encoderz,LoraLoaderMixin.enable_lora_for_text_encodersc  JHII I#Jwt^T'J'J  677 74,d;;;;;;r:adapter_weightsct|tr|gn|}tj|}t|ts|gt |z}t |t |kr/t dt |dt |ggg}}}|dD}fd|D}t||D]e\}} t| tr| dd} | dd} | dd} t | dkr(t d | d |d | *t|dstd t| | | ggd D]D\} }| =|||vr3td|d|d|d|d|d||d En| } | } | } || || || gt|dst#||jn|j}|||t|dr|||j|t|dr|||j|dSdS)Nrkrlch|] }|D]}| Sr3r3)r5adaptersadapters r7 z/LoraLoaderMixin.set_adapters..s9    (  ?FG    r:cTi|]#fdD$S)c"g|] \}}|v | Sr3r3)r5partr}r~s r7rz;LoraLoaderMixin.set_adapters...s(^^^~tX'U]J]J]dJ]J]J]r:)r)r5r~ list_adapterss @r7rz0LoraLoaderMixin.set_adapters..sM    ^^^^1D1D1F1F^^^   r:r(r'rrzGot invalid key 'z"' in lora weight dict for adapter rzsLora weight dict contains text_encoder_2 weights but will be ignored because pipeline does not have text_encoder_2.)r(r'rzLora weight dict for adapter 'z ' contains z#, but this will be ignored because z does not contain weights for z. Valid parts for z are: )rBrprDdeepcopyrrrAget_list_adaptersvaluesziprCrsrGrIrrappendrJrKr( set_adaptersrqr'r)rNrNrzunet_lora_weightstext_encoder_lora_weightstext_encoder_2_lora_weights all_adaptersinvert_list_adaptersr>r8unet_lora_weighttext_encoder_lora_weighttext_encoder_2_lora_weight part_weight part_namer(rs @r7rzLoraLoaderMixin.set_adapterss ,6mS+I+I\} -88/400 E./#m2D2DDO }  _!5!5 5 5A3}+=+=AAjmn}j~j~AA  UWXZ\^6Q4..00   $1$8$8$:$:       '    &)%H%H! K! K !L''4(( 5#*;;vt#<#< +2;;~t+L+L(-4[[9I4-P-P*w<>>//*K#.9DXYeDf3f3fG\GGV_GGEQGGqzGGNZGGbvwCbDGGG $+ +2(-4*  $ $%5 6 6 6 % , ,-E F F F ' . ./I J J J J4;D&4I4IXwtT^,,,ty -):;;; 4 ( ( l  . .}d>OQj k k k 4) * * p  . .}d>QSn o o o o o p pr:c^tstdt|dst||jn|j}|t|dr||jt|dr||j dSdSNr.r(r'r) rrArIrJrKr( disable_lorarwr'rrIs r7rzLoraLoaderMixin.disable_loras JHII I5s r7rzLoraLoaderMixin.delete_adapterss  JHII I mS ) ) ,*OM5modules_to_processr'text_encoder_modules r7set_lora_devicezLoraLoaderMixin.set_lora_devicegsR  JHII I;;;;;;5> % %K+~66 %$1%%L&|477???&|477???{,CDD%IjIvJUJk(K"V**$9,G   4 ( ( 9  % %d&7 8 8 8 4) * * ;  % %d&9 : : :. c cL'3';';'='= c c#1>BB c(5 c c +2<@CCFKKK+2<@CCFKKK$$79PQQc 3 I U!4 I, W Z Z[a b b0E , c c cr:r)rdFNN)NrNNNNNTNNT)TTrFN)TT)7__name__ __module__ __qualname____doc__TEXT_ENCODER_NAMErM UNET_NAMErKTRANSFORMER_NAMErr_rrprrxTensorrR classmethodr rErurrHrLr propertyfloatr@rrPathLiker rboolrr, staticmethodr+rJrrr`rgrqrwryrrrrrrrkintrr3r:r7r+r+Ds *I'Ohl8 8 5:3S%,EV@W;W5X8 8 8 8 tY*/4S$sEL?P:Q5Q/RY*Y*Y*[Y*vdi&&&[&P A A[ AD!!![!F q>q>q>[q>hM:M:M:[M:`IEIIIXI B B BMQ?C>B $"&#'> > c2;./> sE%(/5<*G$HHI> #'sEHO';"< > "&c58?&:!; >  > >  > !> > > [> @;el*+;;; ;  ; ! ;;;\;B111."&!-1 PrPrPr Pr Pr  Pr  S * PrPrPrPrd$"$"t$"$"$"$"$"R59PT -]-]T#Y^,-]01-]'uUDKd-K'LM -]-]-]-]^88(CT:U8888"<<BS9T<<<<&RVIpIpT#Y^,Ip"%tT%[$t*(L"MNIpIpIpIpV D D D C C CIU49c>-BIIII0T#Y@4T#Y#7.3cT#Y3celTWY\F\@]3cbf3c3c3c3c3c3cr:r+c~eZdZdZ ddeeeeejffde efdZ e ddeee j fdeeeejjejffd eeeejjejffd eeeejjejffd ed ed edefdZdZdS) StableDiffusionXLLoraLoaderMixinz\This class overrides `LoraLoaderMixin` with LoRA loading/saving code that's specific to SDXLNr,r>c tstdt|tr|}|j|fd|jji|\}}td| D}|std| |||j||d| D}t|dkr&| |||jd|j|| d | D}t|dkr(| |||jd |j|| d Sd S) a_ Load LoRA weights specified in `pretrained_model_name_or_path_or_dict` into `self.unet` and `self.text_encoder`. All kwargs are forwarded to `self.lora_state_dict`. See [`~loaders.LoraLoaderMixin.lora_state_dict`] for more details on how the state dict is loaded. See [`~loaders.LoraLoaderMixin.load_lora_into_unet`] for more details on how the state dict is loaded into `self.unet`. See [`~loaders.LoraLoaderMixin.load_lora_into_text_encoder`] for more details on how the state dict is loaded into `self.text_encoder`. Parameters: pretrained_model_name_or_path_or_dict (`str` or `os.PathLike` or `dict`): See [`~loaders.LoraLoaderMixin.lora_state_dict`]. adapter_name (`str`, *optional*): Adapter name to be used for referencing the loaded adapter model. If not specified, it will use `default_{i}` where i is the total number of adapters being loaded. kwargs (`dict`, *optional*): See [`~loaders.LoraLoaderMixin.lora_state_dict`]. r.r]c3&K|] }d|vpd|vV dSr0r3r4s r7r8zEStableDiffusionXLLoraLoaderMixin.load_lora_weights..r9r:r;r<c"i|] \}}d|v || S)z text_encoder.r3r5rrrs r7rzFStableDiffusionXLLoraLoaderMixin.load_lora_weights..s)"_"_"_DAq/]^J^J^1aJ^J^J^r:rr')r=r'rr@r>r?c"i|] \}}d|v || S)ztext_encoder_2.r3rs r7rzFStableDiffusionXLLoraLoaderMixin.load_lora_weights..s*$c$c$cdaL]abLbLbQLbLbLbr:rN)rrArBrCrDrEr(configrFrGrHrrrLr'r@r) rNr,r>rOrPr=rQtext_encoder_state_dicttext_encoder_2_state_dicts r7rRz2StableDiffusionXLLoraLoaderMixin.load_lora_weightss:  JHII I ;T B B a4Y4^4^4`4` 1&:T%9 1& &  (& & & " N  bbPZP_P_PaPabbbbb  9788 8   ~DIT`lp !   #`"_J4D4D4F4F"_"_"_ & ' '! + +  , ,'-!.%?) -   %d$cj6F6F6H6H$c$c$c! ( ) )A - -  , ,)-!0'?) -      . -r:Trrrtext_encoder_2_lora_layersrr\rrc 6i} d} |s|s|std|r| | |d|r| | |d|r| | |d|| |||||dS)u6 Save the LoRA parameters corresponding to the UNet and text encoder. Arguments: save_directory (`str` or `os.PathLike`): Directory to save LoRA parameters to. Will be created if it doesn't exist. unet_lora_layers (`Dict[str, torch.nn.Module]` or `Dict[str, torch.Tensor]`): State dict of the LoRA layers corresponding to the `unet`. text_encoder_lora_layers (`Dict[str, torch.nn.Module]` or `Dict[str, torch.Tensor]`): State dict of the LoRA layers corresponding to the `text_encoder`. Must explicitly pass the text encoder LoRA state dict because it comes from 🤗 Transformers. text_encoder_2_lora_layers (`Dict[str, torch.nn.Module]` or `Dict[str, torch.Tensor]`): State dict of the LoRA layers corresponding to the `text_encoder_2`. Must explicitly pass the text encoder LoRA state dict because it comes from 🤗 Transformers. is_main_process (`bool`, *optional*, defaults to `True`): Whether the process calling this is the main process or not. Useful during distributed training and you need to call this function on all processes. In this case, set `is_main_process=True` only on the main process to avoid race conditions. save_function (`Callable`): The function to use to save the state dictionary. Useful during distributed training when you need to replace `torch.save` with another method. Can be configured with the environment variable `DIFFUSERS_SAVE_MODE`. safe_serialization (`bool`, *optional*, defaults to `True`): Whether to save the model using `safetensors` or the traditional PyTorch way with `pickle`. ct|tjjr|n|}fd|D}|S)Nc&i|] \}}d||Srr3rs r7rz\StableDiffusionXLLoraLoaderMixin.save_lora_weights..pack_weights..r!r:r"r#s ` r7r'zHStableDiffusionXLLoraLoaderMixin.save_lora_weights..pack_weightsr(r:zmYou must pass at least one of `unet_lora_layers`, `text_encoder_lora_layers` or `text_encoder_2_lora_layers`.r(r'rr)N)rAr*r+) r}rrrrrr\rrrPr's r7r,z2StableDiffusionXLLoraLoaderMixin.save_lora_weightssJ  % % % ! $< @Z    F   ll+;VDD E E E # V   ll+C^TT U U U % Z   ll+EGWXX Y Y Y !)+#'1      r:ct|jt|jdd|j`d|j_t|jt|jdd|j`d|j_dSdS)Nr)rr'rJrrrr s r7rzBStableDiffusionXLLoraLoaderMixin._remove_text_encoder_monkey_patch1s"4#4555 4$mT : : F!-7;D  4"4#6777 4& t < < H#/9=D  6 6 6 I Hr:rr)rrrrrrprrxrrrRrrrr rrrr,rr3r:r7rrsoff '+KK/4S$sEL?P:Q5Q/RKsmKKKKZMQTXVZ $"&#'@ @ c2;./@ sE%(/5<*G$HHI@ #'sE%(/5<2O,P'P"Q @ %)eEHOU\4Q.R)R$S @  @ @  @ !@ @ @ [@ D > > > > >r:rceZdZdZeZdZ ddeee ee j fffdZ e edeee ee j fffdZe ddZe dd eeejfd e ee jjfd ed ed edef dZede ee j fd ed ed ed edef dZdZe dZdS)SD3LoraLoaderMixinz: Load LoRA layers into [`SD3Transformer2DModel`]. rNr,c tstdt|tr|}|j|fi|}t d|D}|std||t|dst||j n|j ||dS)a Load LoRA weights specified in `pretrained_model_name_or_path_or_dict` into `self.unet` and `self.text_encoder`. All kwargs are forwarded to `self.lora_state_dict`. See [`~loaders.LoraLoaderMixin.lora_state_dict`] for more details on how the state dict is loaded. See [`~loaders.LoraLoaderMixin.load_lora_into_transformer`] for more details on how the state dict is loaded into `self.transformer`. Parameters: pretrained_model_name_or_path_or_dict (`str` or `os.PathLike` or `dict`): See [`~loaders.LoraLoaderMixin.lora_state_dict`]. kwargs (`dict`, *optional*): See [`~loaders.LoraLoaderMixin.lora_state_dict`]. adapter_name (`str`, *optional*): Adapter name to be used for referencing the loaded adapter model. If not specified, it will use `default_{i}` where i is the total number of adapters being loaded. r.c3&K|] }d|vpd|vV dSr0r3r4s r7r8z7SD3LoraLoaderMixin.load_lora_weights..gr9r:r;r))r)r>r?N) rrArBrCrDrErFrGr rIrJrr))rNr,r>rOrPrQs r7rRz$SD3LoraLoaderMixin.load_lora_weightsFs.  JHII I ;T B B a4Y4^4^4`4` 1*T)*OZZSYZZ bbPZP_P_PaPabbbbb  9788 8 '' DKDR_D`D`vd&;<< We support loading A1111 formatted LoRA checkpoints in a limited capacity. This function is experimental and might change in the future. Parameters: pretrained_model_name_or_path_or_dict (`str` or `os.PathLike` or `dict`): Can be either: - A string, the *model id* (for example `google/ddpm-celebahq-256`) of a pretrained model hosted on the Hub. - A path to a *directory* (for example `./my_model_directory`) containing the model weights saved with [`ModelMixin.save_pretrained`]. - A [torch state dict](https://pytorch.org/tutorials/beginner/saving_loading_models.html#what-is-a-state-dict). cache_dir (`Union[str, os.PathLike]`, *optional*): Path to a directory where a downloaded pretrained model configuration is cached if the standard cache is not used. force_download (`bool`, *optional*, defaults to `False`): Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist. resume_download (`bool`, *optional*, defaults to `False`): Whether or not to resume downloading the model weights and configuration files. If set to `False`, any incompletely downloaded files are deleted. proxies (`Dict[str, str]`, *optional*): A dictionary of proxy servers to use by protocol or endpoint, for example, `{'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}`. The proxies are used on each request. local_files_only (`bool`, *optional*, defaults to `False`): Whether to only load local model weights and configuration files or not. If set to `True`, the model won't be downloaded from the Hub. token (`str` or *bool*, *optional*): The token to use as HTTP bearer authorization for remote files. If `True`, the token generated from `diffusers-cli login` (stored in `~/.huggingface`) is used. revision (`str`, *optional*, defaults to `"main"`): The specific model version to use. It can be a branch name, a tag name, a commit id, or any identifier allowed by Git. subfolder (`str`, *optional*, defaults to `""`): The subfolder location of a model file within a larger model repository on the Hub or locally. rTNrUFrVrWrXrYrZr[r\r^Tr_r`rardrfrirj) rsrBrCrtrrvrwrxryrzr{r|r)r}r,rOrTrUrVrWrXrYrZr[r\r^r~rhrrPrs r7rEz"SD3LoraLoaderMixin.lora_state_dictrs3pJJ{D11 $4e<< **%6>>**Y--!::&8$?? 7D))::j$//JJ{D11 jj55  **%6==  ""OL."   ?FF+ ? K$7'K,@,@,P,P'!0=%0%I4I"+'5(7 ')9#!)"+#- " " "J"-!2!JsAE F6FFcddlm}m}m}t |}fd|Dfd|D}t|dkrW|t|divrtd|di} |D]\} } d| vr| j d | | <t| d | } d | vr<| d rtd drtd| d |di| } |t|}|\}}|| ||||||}|1t|dd }|rt d|d|r|d S|r|d Sd Sd S)a This will load the LoRA layers specified in `state_dict` into `transformer`. Parameters: state_dict (`dict`): A standard state dict containing the lora layer parameters. The keys can either be indexed directly into the unet or prefixed with an additional `unet` which can be used to distinguish between text encoder lora layers. transformer (`SD3Transformer2DModel`): The Transformer model to load the LoRA layers into. adapter_name (`str`, *optional*): Adapter name to be used for referencing the loaded adapter model. If not specified, it will use `default_{i}` where i is the total number of adapters being loaded. rrcHg|]}|j|Sr3rrs r7rzASD3LoraLoaderMixin.load_lora_into_transformer..rr:c\i|](\}}|v |jdd|)Srrrs r7rzASD3LoraLoaderMixin.load_lora_into_transformer..rr:rrrrrN)network_alpha_dictpeft_state_dictrrrrrrrrr3r)r}rPr)r>r?rrrrGrr6rrrrrrrrs` @r7r z-SD3LoraLoaderMixin.load_lora_into_transformers XWWWWWWWWWJOO%%&&RRRRtRRR     EOEUEUEWEW    z  ! !A % %w{M2FFFF xLxxxD&,,.. - -Ss?? # ! DI!0$`j!k!k!k ///%j17oc76S6S7$'**:666$*::'9::K#/ << ?B>`>`aj>k>k ; "; # #K< X X X X 9 9+zS_ ` `  ,")*;=NPT"U"U"NN0+000 $ :2244444* :7799999[ & %X : :r:Trrrr\rrci}d}|std|r$||||j|||||||dS)a Save the LoRA parameters corresponding to the UNet and text encoder. Arguments: save_directory (`str` or `os.PathLike`): Directory to save LoRA parameters to. Will be created if it doesn't exist. transformer_lora_layers (`Dict[str, torch.nn.Module]` or `Dict[str, torch.Tensor]`): State dict of the LoRA layers corresponding to the `transformer`. is_main_process (`bool`, *optional*, defaults to `True`): Whether the process calling this is the main process or not. Useful during distributed training and you need to call this function on all processes. In this case, set `is_main_process=True` only on the main process to avoid race conditions. save_function (`Callable`): The function to use to save the state dictionary. Useful during distributed training when you need to replace `torch.save` with another method. Can be configured with the environment variable `DIFFUSERS_SAVE_MODE`. safe_serialization (`bool`, *optional*, defaults to `True`): Whether to save the model using `safetensors` or the traditional PyTorch way with `pickle`. ct|tjjr|n|}fd|D}|S)Nc&i|] \}}d||Srr3rs r7rzNSD3LoraLoaderMixin.save_lora_weights..pack_weights..Yr!r:r"r#s ` r7r'z:SD3LoraLoaderMixin.save_lora_weights..pack_weightsWr(r:z(You must pass `transformer_lora_layers`.r)N)rAr*rr+) r}rrrr\rrrPr's r7r,z$SD3LoraLoaderMixin.save_lora_weights8s:  % % % ' IGHH H " [   ll+BCDXYY Z Z Z !)+#'1      r:rPctj|r td|ddS||rd}n t j}tj|d||rt}nt}t|| }|||t d|dS)Nr.r/cJtj||ddiSr1r5r7s r7rz;SD3LoraLoaderMixin.write_lora_layers..save_function|r:r:Tr;r=r>rCs r7r+z$SD3LoraLoaderMixin.write_lora_layerslrEr:ct|dst||jn|j}t |t|dr|`dSdS)rGr)rN)rIrJrr)rr)rNr)s r7rJz&SD3LoraLoaderMixin.unload_lora_weightssgCJ$P]B^B^tgdD$9:::dhdt ";/// ; . . (''' ( (r:cd}d}||j|jD]\}}t|tjrt |dr|st|jt}|sTt|jtp9t |jdo$t|jj dt}t dt||||fSrrrs r7rz1SD3LoraLoaderMixin._optionally_disable_offloadings %$)!  Y%<%D ) 4 : : < < Z Z 9i33 Z :8V8V Z/Z/9):Lj/Y/Y,4&y'9;KLLZ&y'97CC Z *9+=+CA+FHX Y Y2 KKY,I?XYYYY$&?@@r:rr)NTNNT)rrrrrrr_rrprrxrrRrr rEr rrr rrrr,rr+rJrr3r:r7rr>s(Ohl* * 5:3S%,EV@W;W5X* * * * Xy/4S$sEL?P:Q5Q/Ryyy[yvE:E:E:[E:P?C $"&#'1 1 c2;./1 "&c58?&:!;1  1  1  1 !1 1 1 [1 f;el*+;;; ;  ; ! ;;;\;B(((" A A[ A A Ar:r)>rDrVrpathlibrtypingrrrrrrwrxhuggingface_hubr huggingface_hub.constantsr huggingface_hub.utilsr r models.modeling_utilsrutilsrrrrrrrrrrrrrrrlora_conversion_utilsrr transformersr! models.lorar"r#accelerate.hooksr$r%r& get_loggerrrrrrr|rvLORA_DEPRECATION_MESSAGEr+rrr3r:r7rs  88888888888888 &&&&&&444444666666333333"hgggggggR,,,,,,QQQQQQQQWVVVVVVVVVV  H % %"   -:WVcVcVcVcVcVcVcVcr"^>^>^>^>^>^>^>^>BCACACACACACACACACACAr: