wiO8ddlZddlZddlmZddlZddlmZdde fdZ ej j j ddfde de fdZGdd ejZGd d ejZGd d ejZGddejZGddejZdS)Nkey_chunk_sizecjdd\}jdt|tjz t jt jdfdfd}t j |tj d| \}}} tj | dd } tj | | z } |tj | d z}|| z}|d } tj |dd } | | z S) zBMulti-head dot product attention with a limited number of queries.NF) prevent_csecZtjd||}tj|dd}tj|}tj||z }tjd||}tjd|}||d|fS) Nz...qhd,...khd->...qhk) precisionrTaxiskeepdimsz...vhf,...qhv->...qhfz ...qhk->...qhr )jnpeinsummaxjaxlax stop_gradientexpsum)querykeyvalue attn_weights max_score exp_weights exp_valuesr s o/root/.openclaw/workspace/chatterbox_venv_py311/lib/python3.11/site-packages/diffusers/models/attention_flax.pysummarize_chunkz/_query_chunk_attention..summarize_chunksz"95#QZ[[[ GLrDAAA G)))44 glY677 Z 7 W`aaa J :: KOOO44i@@c ntjdgjdz z|ddgzt jddgz}tj dg jdz z|ddgzt jdd gz}||S)Nrroperand start_indices slice_sizes)rr dynamic_slicendimlistshape) chunk_idx key_chunk value_chunk k_featuresrr num_headsrr v_featuresrs r chunk_scannerz-_query_chunk_attention..chunk_scanner+sG))#A.)Q1BBSYss^,, :/VV*  g++#a0Iq!3DDU["-...)Z1XX,  ui===r r)fxsTr r)r*minrsqrt functoolspartialr checkpointrmaparangerr expand_dimsr)rrrr rnum_kvr1 chunk_values chunk_weights chunk_max global_max max_diffs all_values all_weightsr.r/rr0s````` @@@@r_query_chunk_attentionrDs$'IbccN!FIzRJ00N CHZ(( (Es~5999 A A A A:9 A>>>>>>>>>>>>".1W[[=SZXY[acqMrMr[-s-s*L-T:::J J.//ICOIB7777LYM!!q!))J/-4488a8@@K  ##r iquery_chunk_sizec  jdd\   f d}tj|ddt j z \}}t j|dS)a Flax Memory-efficient multi-head dot product attention. https://arxiv.org/abs/2112.05682v2 https://github.com/AminRezaei0x443/memory-efficient-attention Args: query (`jnp.ndarray`): (batch..., query_length, head, query_key_depth_per_head) key (`jnp.ndarray`): (batch..., key_value_length, head, query_key_depth_per_head) value (`jnp.ndarray`): (batch..., key_value_length, head, value_depth_per_head) precision (`jax.lax.Precision`, *optional*, defaults to `jax.lax.Precision.HIGHEST`): numerical precision for computation query_chunk_size (`int`, *optional*, defaults to 1024): chunk size to divide query array value must divide query_length equally without remainder key_chunk_size (`int`, *optional*, defaults to 4096): chunk size to divide key and value array value must divide key_value_length equally without remainder Returns: (`jnp.ndarray`) with shape of (batch..., query_length, head, value_depth_per_head) rNc  tj dg jdz z|ddgzt jddt gz}| zt| fS)Nrr"rr#)rrrr r)rrr'r(r)r*r4rD) r+_ query_chunkrrr/num_qr q_featuresrrErs rr1z5jax_memory_efficient_attention..chunk_scannerasg++3%*q.1iA5FFU["-..#6F2N2NPY[e1ff,   ( ( "!s%9]k     r r)r2initr3lengthr)r*rrscanmathceilr concatenate) rrrr rErr1rHresr/rJrKs `````` @@@rjax_memory_efficient_attentionrSJs*$);rss#3 E9j              W\\   y!1122 FAs ?3R ( ( ((r ceZdZUdZeed<dZeed<dZeed<dZe ed<d Z e ed <d Z e ed <e jZe jed <d ZdZdZddZdS) FlaxAttentiona A Flax multi-head attention module as described in: https://arxiv.org/abs/1706.03762 Parameters: query_dim (:obj:`int`): Input hidden states dimension heads (:obj:`int`, *optional*, defaults to 8): Number of heads dim_head (:obj:`int`, *optional*, defaults to 64): Hidden states dimension inside each head dropout (:obj:`float`, *optional*, defaults to 0.0): Dropout rate use_memory_efficient_attention (`bool`, *optional*, defaults to `False`): enable memory efficient attention https://arxiv.org/abs/2112.05682 split_head_dim (`bool`, *optional*, defaults to `False`): Whether to split the head dimension into a new axis for the self-attention computation. In most cases, enabling this flag should speed up the computation for Stable Diffusion 2.x and Stable Diffusion XL. dtype (:obj:`jnp.dtype`, *optional*, defaults to jnp.float32): Parameters `dtype` query_dimheads@dim_headdropoutFuse_memory_efficient_attentionsplit_head_dimdtypec|j|jz}|jdz|_tj|d|jd|_tj|d|jd|_tj|d|jd|_tj|j |jd|_ tj |j |_ dS) NgFto_q)use_biasr_nameto_kto_vto_out_0)r_rcrate)rZrXscalennDenser_rrrrV proj_attnDropoutr\ dropout_layerself inner_dims rsetupzFlaxAttention.setupsMDJ. ]D( Xi%tzPVWWW 8ITZfUUUXi%tzPVWWW $. TTTZT\:::r c|j\}}}|j}||||||z}tj|d}|||z|||z}|SN)rr"r*rXreshaper transposerptensor batch_sizeseq_lendim head_sizes rreshape_heads_to_batch_dimz(FlaxAttention.reshape_heads_to_batch_dimsh#)< GSJ  GYy@PQQv|44 Y 6 AQRR r c|j\}}}|j}|||z|||}tj|d}|||z|||z}|Srtrwrzs rreshape_batch_dim_to_headsz(FlaxAttention.reshape_batch_dim_to_headssg#)< GSJ  i 7GSQQv|44 i 7# /RR r NTc||n|}||}||}||}|jrw|jd}t j||d|j|jf}t j||d|j|jf} t j||d|j|jf} n?| |}| |} | |} |j r| ddd}| ddd} | ddd} |jd} | dzdkrt| dz } nG| dzdkrt| dz } n+| dzdkrt| dz } nt| } t|| | | d }| ddd}n|jrt jd | |} nt jd || } | |jz} t!j| |jrdnd }|jrIt jd|| }|jd}t j||d|j|jzf}n+t jd|| }||}||}|||S)NrrrvrurrYi@)rErzb t n h, b f n h -> b n f tzb i d, b j d->b i jrzb n f t, b t n h -> b f n hzb i j, b j d -> b i d deterministic)rrrr^r*rrxrXrZrr]ryintrSrrirjsoftmaxrrlrn)rp hidden_statescontextr query_projkey_proj value_projb query_states key_states value_statesflatten_latent_dimrEattention_scoresattention_probss r__call__zFlaxAttention.__call__s$#*?--ZZ .. 88G$$ZZ((   G#A&A;zAr4:t}3UVVLX2tz4=/QRRJ;zAr4:t}3UVVLL:::FFL88BBJ:::FFL  .( O'11!Q::L#--aA66J'11!Q::L ".!3B!7 !B&!++#&'9B'>#?#?  #b(A--#&'9B'>#?#?  #a'1,,#&'9A'=#>#>  #&'9#:#: :j,IYjrM*33Aq!<