wieddlZddlmZmZmZmZmZmZddlZ ddl Z ddl m cm ZddlmZmZmZmZddlmZmZddlmZddlmZmZddlmZd d lmZm Z m!Z!ej"e#Z$d Z%Gd d e e!Z&dS)N)AnyCallableDictListOptionalUnion)ClapTextModelWithProjectionRobertaTokenizerRobertaTokenizerFastSpeechT5HifiGan) AutoencoderKLUNet2DConditionModel)KarrasDiffusionSchedulers)loggingreplace_example_docstring) randn_tensor)AudioPipelineOutputDiffusionPipelineStableDiffusionMixinaj Examples: ```py >>> from diffusers import AudioLDMPipeline >>> import torch >>> import scipy >>> repo_id = "cvssp/audioldm-s-full-v2" >>> pipe = AudioLDMPipeline.from_pretrained(repo_id, torch_dtype=torch.float16) >>> pipe = pipe.to("cuda") >>> prompt = "Techno music with a strong, upbeat tempo and high melodic riffs" >>> audio = pipe(prompt, num_inference_steps=10, audio_length_in_s=5.0).audios[0] >>> # save the audio sample as a .wav file >>> scipy.io.wavfile.write("techno.wav", rate=16000, data=audio) ``` c$eZdZdZdZdededeee fde de de f fd Z d(d eejd eejfd ZdZdZdZ d(dZd)dZejee d*deeeefdeedededeeeeefdeeded eeejeejfd!eejd eejd eejd"ed#ee eeejgd fd$eed%ee!ee"fd&eef d'Z#xZ$S)+AudioLDMPipelinea Pipeline for text-to-audio generation using AudioLDM. This model inherits from [`DiffusionPipeline`]. Check the superclass documentation for the generic methods implemented for all pipelines (downloading, saving, running on a particular device, etc.). Args: vae ([`AutoencoderKL`]): Variational Auto-Encoder (VAE) model to encode and decode images to and from latent representations. text_encoder ([`~transformers.ClapTextModelWithProjection`]): Frozen text-encoder (`ClapTextModelWithProjection`, specifically the [laion/clap-htsat-unfused](https://huggingface.co/laion/clap-htsat-unfused) variant. tokenizer ([`PreTrainedTokenizer`]): A [`~transformers.RobertaTokenizer`] to tokenize text. unet ([`UNet2DConditionModel`]): A `UNet2DConditionModel` to denoise the encoded audio latents. scheduler ([`SchedulerMixin`]): A scheduler to be used in combination with `unet` to denoise the encoded audio latents. Can be one of [`DDIMScheduler`], [`LMSDiscreteScheduler`], or [`PNDMScheduler`]. vocoder ([`~transformers.SpeechT5HifiGan`]): Vocoder of class `SpeechT5HifiGan`. ztext_encoder->unet->vaevae text_encoder tokenizerunet schedulervocoderct|||||||dt|jjjdz z|_dS)N)rrrrrrr)super__init__register_moduleslenrconfigblock_out_channelsvae_scale_factor)selfrrrrrr __class__s ~/root/.openclaw/workspace/chatterbox_venv_py311/lib/python3.11/site-packages/diffusers/pipelines/audioldm/pipeline_audioldm.pyr#zAudioLDMPipeline.__init__Nst  %    !"c$(/*L&M&MPQ&Q RN prompt_embedsnegative_prompt_embedsc :|t|trd}n4|%t|trt|}n |jd}|7||d|jjdd} | j} | j} ||dd j} | jd | jd krrtj | | s]|j | dd|jjdz d f} t d |jjd | || || | }|j}t#j|d }||jj|}|j\}}|d|}|||z|}|r@|=|dg|z}nt-|t-|ur0t/dt-|dt-|dt|tr|g}n>|t|kr)t1d|dt|d|d|d |}|jd}||d|dd}|j|}|j|} ||| }|j}t#j|d }|rs|jd}||jj|}|d|}|||z|}tj||g}|S)a` Encodes the prompt into text encoder hidden states. Args: prompt (`str` or `List[str]`, *optional*): prompt to be encoded device (`torch.device`): torch device num_waveforms_per_prompt (`int`): number of waveforms that should be generated per prompt do_classifier_free_guidance (`bool`): whether to use classifier free guidance or not negative_prompt (`str` or `List[str]`, *optional*): The prompt or prompts not to guide the audio generation. If not defined, one has to pass `negative_prompt_embeds` instead. Ignored when not using guidance (i.e., ignored if `guidance_scale` is less than `1`). prompt_embeds (`torch.Tensor`, *optional*): Pre-generated text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not provided, text embeddings will be generated from `prompt` input argument. negative_prompt_embeds (`torch.Tensor`, *optional*): Pre-generated negative text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not provided, negative_prompt_embeds will be generated from `negative_prompt` input argument. Nr!r max_lengthTpt)paddingr0 truncationreturn_tensorslongest)r2r4z\The following part of your input was truncated because CLAP can only handle sequences up to z tokens: )attention_mask)dim)dtypedevicez?`negative_prompt` should be the same type to `prompt`, but got z != .z`negative_prompt`: z has batch size z, but `prompt`: zT. Please make sure that passed `negative_prompt` matches the batch size of `prompt`.) isinstancestrlistr%shapermodel_max_length input_idsr7torchequal batch_decodeloggerwarningrto text_embedsF normalizer9repeatviewtype TypeError ValueErrorcat)r)promptr:num_waveforms_per_promptdo_classifier_free_guidancenegative_promptr-r. batch_size text_inputstext_input_idsr7untruncated_ids removed_textbs_embedseq_len uncond_tokensr0 uncond_inputuncond_input_idss r+_encode_promptzAudioLDMPipeline._encode_promptcsZD  *VS"9"9 JJ  Jvt$<$< VJJ&,Q/J  ..$>:# )K)2N(7N"nnVYW[n\\fO$R(N,@,DDDU[NND $~::#AAAt~'F'JR'O$OP   Q7QQBNQQ !--!!&))-0088.M*5MK 2>>>M%((t/@/Fv(VV    &,,Q0HII %**86N+NPWXX  '& Q+A+I&!#z 1 fT/%:%:::(VZ[jVkVk((V (((OS11 0!0 1 s?3333 3/333K_K_33330:333 !0 &,Q/J>>$%# *L ,588@@ )8;;FCCN%)%6%6 -&7&& "&<%G "%&[1GR%P%P%P " & O,215G%;%>%>TEVE\ek%>%l%l "%;%B%B1F^%_%_ "%;%@%@NfAfho%p%p " "I'=}&MNNMr,crd|jjjz |z}|j|j}|S)Nr!)rr&scaling_factordecodesample)r)latentsmel_spectrograms r+decode_latentszAudioLDMPipeline.decode_latentss4dho44w>(//'229r,c|dkr|d}||}|}|S)Nr!)r8squeezercpufloat)r)rfwaveforms r+mel_spectrogram_to_waveformz,AudioLDMPipeline.mel_spectrogram_to_waveforms[    A % %-55a88O<<00<<>>''))r,c6dttj|jjjv}i}|r||d<dttj|jjjv}|r||d<|S)Neta generator)setinspect signaturerstep parameterskeys)r)rqrp accepts_etaextra_step_kwargsaccepts_generators r+prepare_extra_step_kwargsz*AudioLDMPipeline.prepare_extra_step_kwargss s7#4T^5H#I#I#T#Y#Y#[#[\\\   +'* e $(3w/@AT/U/U/`/e/e/g/g+h+hh  7-6 k *  r,c||jz}||krtd|d|d|jjj|jzdkr*td|jjjd|jd||>t |t r|dkr#td|dt|d||td |d |d ||td |It |ts4t |tstd t|||td|d|d |2|2|j |j kr$td|j d|j ddSdSdS)NzH`audio_length_in_s` has to be a positive value greater than or equal to z , but is r<rzwThe number of frequency bins in the vocoder's log-mel spectrogram has to be divisible by the VAE scale factor, but got z bins and a scale factor of z5`callback_steps` has to be a positive integer but is z of type zCannot forward both `prompt`: z and `prompt_embeds`: z2. Please make sure to only forward one of the two.zeProvide either `prompt` or `prompt_embeds`. Cannot leave both `prompt` and `prompt_embeds` undefined.z2`prompt` has to be of type `str` or `list` but is z'Cannot forward both `negative_prompt`: z and `negative_prompt_embeds`: zu`prompt_embeds` and `negative_prompt_embeds` must have the same shape when passed directly, but got: `prompt_embeds` z != `negative_prompt_embeds` ) r(rPrr& model_in_dimr=intrNr>r?r@) r)rRaudio_length_in_svocoder_upsample_factorcallback_stepsrUr-r.min_audio_length_in_ss r+ check_inputszAudioLDMPipeline.check_inputssz!8$:O O 4 4 4+[p++'+++  <  +d.C Cq H H,-1\-@-M,,(,,,   "  & >30O0O &SaefSfSf,,,((,,,   -";000}000 ^ 5w  FC)@)@ TZ\`IaIa `RVW]R^R^``aa a  &+A+M_/__*___   $)?)K"&<&BBB 8-:-@88.4888 % $)K)KBBr,c||t||jzt|jjj|jzf}t |t r6t||kr#tdt|d|d|t||||}n| |}||j j z}|S)Nz/You have passed a list of generators of length z+, but requested an effective batch size of z@. Make sure the batch size matches the length of the generators.)rqr:r9) r~r(rr&r}r=r?r%rPrrHrinit_noise_sigma) r)rVnum_channels_latentsheightr9r:rqrer@s r+prepare_latentsz AudioLDMPipeline.prepare_latentsIs  KK40 0  #0 1 1T5J J   i & & 3y>>Z+G+Gi#i..ii&iii  ?"5IfTYZZZGGjj((GDN;;r, @r!TnprRrnum_inference_stepsguidance_scalerUrSrprqre return_dictcallbackrcross_attention_kwargs output_typec tj|jjj|jjjz }||jjj|jz|z}t||z }t||jjjz}||jzdkrXttj ||jz |jz}t d|d||zd|d| |||||| | |t|trd}n4|%t|t rt#|}n | jd}|j}|dk}||||||| | } |j|| |jj}|jjj}|||z||| j||| } |||}t#|||jjzz }|| 5}t=|D]\}}|rt?j | gd zn| }|j!||}|||d| | j"}|r#|#d \} }!| ||!| z zz}|jj$||| fi|j%} |t#|dz ks|dz|kr[|dz|jjzdkrE|&| /||zdkr&|tO|jddz}"| |"||  dddn #1swxYwY|(| }#|)|#}$|$ddd|f}$|dkr|$*}$| s|$fStW|$S)u The call function to the pipeline for generation. Args: prompt (`str` or `List[str]`, *optional*): The prompt or prompts to guide audio generation. If not defined, you need to pass `prompt_embeds`. audio_length_in_s (`int`, *optional*, defaults to 5.12): The length of the generated audio sample in seconds. num_inference_steps (`int`, *optional*, defaults to 10): The number of denoising steps. More denoising steps usually lead to a higher quality audio at the expense of slower inference. guidance_scale (`float`, *optional*, defaults to 2.5): A higher guidance scale value encourages the model to generate audio that is closely linked to the text `prompt` at the expense of lower sound quality. Guidance scale is enabled when `guidance_scale > 1`. negative_prompt (`str` or `List[str]`, *optional*): The prompt or prompts to guide what to not include in audio generation. If not defined, you need to pass `negative_prompt_embeds` instead. Ignored when not using guidance (`guidance_scale < 1`). num_waveforms_per_prompt (`int`, *optional*, defaults to 1): The number of waveforms to generate per prompt. eta (`float`, *optional*, defaults to 0.0): Corresponds to parameter eta (η) from the [DDIM](https://arxiv.org/abs/2010.02502) paper. Only applies to the [`~schedulers.DDIMScheduler`], and is ignored in other schedulers. generator (`torch.Generator` or `List[torch.Generator]`, *optional*): A [`torch.Generator`](https://pytorch.org/docs/stable/generated/torch.Generator.html) to make generation deterministic. latents (`torch.Tensor`, *optional*): Pre-generated noisy latents sampled from a Gaussian distribution, to be used as inputs for image generation. Can be used to tweak the same generation with different prompts. If not provided, a latents tensor is generated by sampling using the supplied random `generator`. prompt_embeds (`torch.Tensor`, *optional*): Pre-generated text embeddings. Can be used to easily tweak text inputs (prompt weighting). If not provided, text embeddings are generated from the `prompt` input argument. negative_prompt_embeds (`torch.Tensor`, *optional*): Pre-generated negative text embeddings. Can be used to easily tweak text inputs (prompt weighting). If not provided, `negative_prompt_embeds` are generated from the `negative_prompt` input argument. return_dict (`bool`, *optional*, defaults to `True`): Whether or not to return a [`~pipelines.AudioPipelineOutput`] instead of a plain tuple. callback (`Callable`, *optional*): A function that calls every `callback_steps` steps during inference. The function is called with the following arguments: `callback(step: int, timestep: int, latents: torch.Tensor)`. callback_steps (`int`, *optional*, defaults to 1): The frequency at which the `callback` function is called. If not specified, the callback is called at every step. cross_attention_kwargs (`dict`, *optional*): A kwargs dictionary that if specified is passed along to the [`AttentionProcessor`] as defined in [`self.processor`](https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/attention_processor.py). output_type (`str`, *optional*, defaults to `"np"`): The output format of the generated image. Choose between `"np"` to return a NumPy `np.ndarray` or `"pt"` to return a PyTorch `torch.Tensor` object. Examples: Returns: [`~pipelines.AudioPipelineOutput`] or `tuple`: If `return_dict` is `True`, [`~pipelines.AudioPipelineOutput`] is returned, otherwise a `tuple` is returned where the first element is a list with the generated audio. NrzAudio length in seconds z is increased to z; so that it can be handled by the model. It will be cut to z after the denoising process.r!g?)r-r.)r:)totalr)encoder_hidden_states class_labelsrorderr)audios),rprodrr&upsample_rates sampling_rater sample_sizer(r~ceilrFinforr=r>r?r%r@_execution_devicer`r set_timesteps timesteps in_channelsrr9r{r progress_bar enumeraterCrQscale_model_inputrdchunkru prev_sampleupdategetattrrgrnnumpyr)%r)rRrrrrUrSrprqrer-r.rrrrrrroriginal_waveform_lengthrVr:rTrrrynum_warmup_stepsritlatent_model_input noise_prednoise_pred_uncondnoise_pred_textstep_idxrfaudios% r+__call__zAudioLDMPipeline.__call___s^#%'$,*=*L"M"MPTP\PcPq"q  $ $ 0 >,?$.BV,VV   %8  9 9 7\!),, 7 71A\%iUYy1}%=%=%=bi"%)^%E%EFXZ[%\%\""YY&*.!.+A '  /l9C9I9I!9L9L6%!2^YjGj5k!kJ.$.-j!WZZHYZZfI***A9I/I/IqSTuX\XfXlNlpqNqNq '')))+N0Ba0G0G#$(K(K#K 1g6667 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7>--g6600AAaaa22223 $  KKMME 8O"%0000s&D'MM!M)NNN)N)NNrrNr!rNNNNTNr!Nr)%__name__ __module__ __qualname____doc__model_cpu_offload_seqrr rr r rrr r#rrCTensorr`rgrnr{rrno_gradrEXAMPLE_DOC_STRINGr>rrlr~ Generatorboolrrrr __classcell__)r*s@r+rr4s.6S S2S)+??@ S # S - S!SSSSSS6049=II  -I!) 6IIIIV !!!.#8888v,U]__122)--1#% #;?23MQ*.049= GK();?%)#A1A1c49n%A1$E?A1! A1  A1 "%T#Y"78 A1#+3-A1A1E%/43H"HIJA1%,'A1  -A1!) 6A1A18S#u|$rsd================ mmmmmmmmmmmm9999999933333377777777------YYYYYYYYYY  H % %(n1n1n1n1n1(*>n1n1n1n1n1r,