+ &jXRt^RIt^RIHt^RIHtHt^RIt]P!] 4t .R.Ot RRlt ]!^R7RR l4t !R R]4t]PP!R /R 7R/RRll4t]P$R/RRll4tRRRRRR0RR RRRRRR/RRllt]PP!RR0R 7R/RRll4t]P$R/RR ll4tRRRRRR0RR RRRRRR/R!R"lltR#R$lt]PP!R%/R 7R1R&R'll4t]P$R1R(R)ll4tR*R+lt]P9]]R,7]P:P=]P>P@PB4^R-I"H#t#H$t$H%t%H&t&]$]&]P>PNP"&]%]&]P>PNP*&]#]&]P>PNP2&R#)2z Variable-length attention implementation using Flash Attention. This module provides a high-level Python interface for variable-length attention that calls into the optimized Flash Attention kernels. N) lru_cache)Any NamedTuple AuxRequestcjV^8dQhR\\,R,R\\,/#) window_sizeNreturn)listint)formats"q/Users/jameslopez/projects/CWCArchive/cwc-podcast/.venv/lib/python3.14/site-packages/torch/nn/attention/varlen.py __annotate__rs'S D(8T#YcdVfRR.p\V4^8wd\R\V4 24hV#)Nz$window_size must have length 2, got )len ValueError)rs&r _normalize_window_sizers=2h  ;1?K@P?QRSS r)maxsizec0V^8dQhR\R\/#)r device_indexr )r bool)r s"r rrsCDrcR#)z;Cache device capability check to avoid repeated CUDA calls.F)rs&r _should_use_cudnnrs rc4a]tRt^#toRtRtV3RltRtVtR#)rz Request which auxiliary outputs to compute from varlen_attn. Each field is a boolean indicating whether that auxiliary output should be computed. Fc&<V^8dQh/S[;R&#)rlse)r)r __classdict__s"r rAuxRequest.__annotate__#s rrN) __name__ __module__ __qualname____firstlineno____doc__r__annotate_func____static_attributes____classdictcell__)rs@r rr#s Crztorch_attn::_varlen_attn) mutates_argsFc!*V^8dQhR\PR\PR\PR\PR\PR,R\R\R \R \R,R \ \,R,R \R \PR,R\PR,R\R,R\ \P\P\P3,/#rquerykeyvaluecu_seq_qcu_seq_kNmax_qmax_k is_causalscaler enable_gqa seqused_k block_table num_splitsr torchTensorr rfloatr tuple)r s"r rr.sV+V+ <<V+ V+ <<V+ll V+ llT! V+  V+ V+V+ 4<V+cT!V+V+||d"V+$V+d V+ 5<<u|| 34V+rcD\V 4p VP;'d \VPP4pV'd\ P R4V 'd \R4hV e \R4hV ^,R8wgV ^,R8wd \R4hV fV e \R4h\PPPVVVRVVVVRRVR VR 7 pV^,V^,V^,pppMb\ P R 4\PPPVVVVVVVRVR VV ^,V ^,V V V R 7wpppp\P!R\PVPR 7pVVV3#)z Private custom op for variable-length attention. This is the internal implementation. Users should use the public varlen_attn function instead. #Using cuDNN backend for varlen_attnz,GQA is not supported with the cuDNN backend.Nz3num_splits is not supported with the cuDNN backend.TcuDNN backend does not support window attention. Please use Flash Attention backend.zBseqused_k/block_table is not yet supported with the cuDNN backend.TFr4-Using Flash Attention backend for varlen_attn)return_debug_maskr4window_size_leftwindow_size_rightr6r7r8dtypedevicerr)ris_cudarrIindexloginfo RuntimeErrorr:opsaten_cudnn_attention_forward_flash_attention_forwardzerosuint64)r,r-r.r/r0r1r2r3r4rr5r6r7r8 use_cudnnresultoutput softmax_lse rng_state_ rng_state_s&&&&&&&&&&&&&& r _varlen_attnr]-s,)5K GG"3ELL4F4F"GI 67 MN N  !TU U q>R ;q>R#7f   K$;T 88            9  *0F1IvayY Y @A/4yy~~/V/V         #(^)!n#!!0W0 , Y1& ELLJ ; **rc!*V^8dQhR\PR\PR\PR\PR\PR,R\R\R \R \R,R \ \,R,R \R \PR,R\PR,R\R,R\ \P\P\P3,/#r+r9)r s"r rrs.(.( <<.( .( <<.(ll .( llT! .(  .( .(.( 4<.(cT!.(.(||d".($.(d .( 5<<u|| 34.(rc\V 4p \P!V4pVP^4pVP^4p\P!VV3\P VP R7p\PP'd\PP4pV\PPP8XdMVP^4^, p\P!VVV3\P VP R7p\P!R\PVP R7pVVV3#)z Fake implementation for meta tensor computation and tracing. Based on the 3D varlen path from meta__flash_attention_forward: - query shape: (total, num_heads, head_dim) - logsumexp shape: (num_heads, total_q) rGrJ)rr: empty_likesizeemptyr<rIversionhip_C_get_rocm_fa_preferred_backend_ROCmFABackendAOTritonrU)r,r-r.r/r0r1r2r3r4rr5r6r7r8rXtotal_q num_heads logsumexp preferred batch_sizerZs&&&&&&&&&&&&&& r _varlen_attn_fakerns0)5K  e $FjjmG 1 I GEKK I }}HH;;= //88 8!q)A-J Y.ekk%,,I D U\\JI 9i ''r return_auxr4rr5r6r7r8c!BV^8dQhR\PR\PR\PR\PR\PR,R\R\R \R,R \R,R \ \\3,R \ R \PR,R\PR,R\R,R\P\ \P\P3,,/#)rr,r-r.r/r0Nr1r2ror4rr5r6r7r8r r:r;r rr<r=r)r s"r rrs VV <<V V <<Vll V llT! V  V VT!V 4<VsCxVV||d"V$Vd V  \\E%,, 455!VrcVP^4pV eVP^4MVP^4pV 'gW8wd\RV RV R24hV 'd W,^8wd\RV RV R24hV R8Hp\PPP VVVVVVVVV\ V 4V V V V 4wpppVeVP'dVV3#V#)a-Compute variable-length attention using Flash Attention. This function is similar to scaled_dot_product_attention but optimized for variable-length sequences using cumulative sequence position tensors. Args: query (Tensor): Query tensor; shape :math:`(T_q, H_q, D)` key (Tensor): Key tensor; shape :math:`(T_k, H_{kv}, D)`, or :math:`(\text{total\_pages}, \text{page\_size}, H_{kv}, D)` when ``block_table`` is provided. value (Tensor): Value tensor; shape :math:`(T_k, H_{kv}, D)`, or :math:`(\text{total\_pages}, \text{page\_size}, H_{kv}, D)` when ``block_table`` is provided. cu_seq_q (Tensor): Cumulative sequence positions for queries; shape :math:`(N+1,)` cu_seq_k (Tensor): Cumulative sequence positions for keys/values; shape :math:`(N+1,)` max_q (int): Maximum query sequence length in the batch. max_k (int): Maximum key/value sequence length in the batch. return_aux (Optional[AuxRequest]): If not None and ``return_aux.lse`` is True, also returns the logsumexp tensor. scale (float, optional): Scaling factor for attention scores window_size (tuple[int, int], optional): Window size for sliding window attention as (left, right). Use (-1, -1) for full attention (default), (-1, 0) for causal attention, or (W, 0) for causal attention with sliding window of size W. enable_gqa (bool): If set to True, enables Grouped Query Attention (GQA) and allows key/value to have fewer heads than query. Each KV head is shared by a group of :math:`H_q / H_{kv}` query heads, so :math:`H_q` must be divisible by :math:`H_{kv}`. Default is False. seqused_k (Tensor, optional): Number of valid KV tokens per batch element; shape :math:`(N,)`. When set, only the first ``seqused_k[i]`` tokens in the key/value sequence for batch element *i* participate in attention. Useful for KV-cache decoding where the cache slot is larger than the actual sequence. Inference-only (not supported in backward). block_table (Tensor, optional): Block table for paged KV cache; shape :math:`(N, \text{max\_pages\_per\_seq})`, dtype ``int32``. Requires ``seqused_k``. Inference-only (not supported in backward). When ``block_table`` is provided, ``key`` and ``value`` are a "pool" of pages of tokens of KV data and the pages belong to any sequence/order. The ``block_table`` is what maps each sequence's logical chunks back to physical pages in this pool. ``seqused_k[i]`` tells the kernel how many tokens in sequence *i* are actually valid, since the last page is typically only partially filled. num_splits (int, optional): Number of splits for split-KV. Set to ``1`` to disable split-KV which enables batch invariance. Split-KV parallelizes the key/value sequence dimension across multiple thread blocks and combines partial results. The split decision depends on ``max_k`` (the longest sequence in the batch), so different batch compositions can change the reduction order and produce different floating-point results for the same sequence. When this is disabled, bitwise identical outputs are guaranteed for a given sequence regardless of what other sequences are in the batch, at the cost of lower GPU utilization when there are few queries. When ``None`` (default), the kernel chooses automatically. Returns: output (Tensor): Output tensor from attention computation; shape :math:`(T_q, H_q, D)`. If ``return_aux`` is not None and ``return_aux.lse`` is True: lse (Tensor): Log-sum-exp of attention scores; shape :math:`(T_q, H_q)`. Shape legend: - :math:`N`: Batch size - :math:`T_q`: Total number of query tokens in the batch (sum of all query sequence lengths) - :math:`T_k`: Total number of key/value tokens in the batch (sum of all key/value sequence lengths) - :math:`H_q`: Number of query attention heads - :math:`H_{kv}`: Number of key/value attention heads (equal to :math:`H_q` unless GQA is enabled) - :math:`D`: Head dimension Example:: >>> # xdoctest: +REQUIRES(env:TORCH_DOCTEST_CUDA) >>> batch_size, max_seq_len, embed_dim, num_heads = 2, 512, 1024, 16 >>> head_dim = embed_dim // num_heads >>> seq_lengths = [] >>> for _ in range(batch_size): ... length = torch.randint(1, max_seq_len // 64 + 1, (1,)).item() * 64 ... seq_lengths.append(min(length, max_seq_len)) >>> seq_lengths = torch.tensor(seq_lengths, device="cuda") >>> total_tokens = seq_lengths.sum().item() >>> >>> # Create packed query, key, value tensors >>> query = torch.randn( ... total_tokens, num_heads, head_dim, dtype=torch.float16, device="cuda" ... ) >>> key = torch.randn( ... total_tokens, num_heads, head_dim, dtype=torch.float16, device="cuda" ... ) >>> value = torch.randn( ... total_tokens, num_heads, head_dim, dtype=torch.float16, device="cuda" ... ) >>> >>> # Build cumulative sequence tensor >>> cu_seq = torch.zeros(batch_size + 1, device="cuda", dtype=torch.int32) >>> cu_seq[1:] = seq_lengths.cumsum(0) >>> max_len = seq_lengths.max().item() >>> >>> # Call varlen_attn >>> output = varlen_attn( ... query, key, value, cu_seq, cu_seq, max_len, max_len ... ) GExpect query and key/value to have the same number of heads but got Hq= and Hkv=&. Try setting enable_gqa=True for GQA.MExpect number of query heads to be a multiple of kv heads for GQA but got Hq=.r)rarr:rP torch_attnr]r r)r,r-r.r/r0r1r2ror4rr5r6r7r8 num_heads_q num_heads_kr3outrr[s&&&&&&&$$$$$$$ r varlen_attnr~s j**Q-K!,!8#((1+chhqkK +4%i }=3 4  k/14%i }A ?  w&I))&&33       [KCa *...Cx Jrztorch_attn::_varlen_attn_outr}c V^8dQhR\PR\PR\PR\PR\PR\PR,R\R \R \R \R,R \ \,R,R \R\PR,R\PR,R\R,R\P/#rr}r,r-r.r/r0Nr1r2r3r4rr5r6r7r8r r:r;r rr<r )r s"r rrSs22 2 <<2 2 << 2 ll 2 llT! 2 2 22 4<2cT!22||d"2$2d 2  \\!2rcb\V 4p VP;'d \VPP4pV'd \ R4h\ PR4\PPPVVVVVVVVRVRV V ^,V ^,V V VR7pV#)z Private custom op for variable-length attention with pre-allocated output. Same as _varlen_attn but writes the attention output into the provided out tensor. z+cuDNN backend does not support out variant.z1Using Flash Attention backend for varlen_attn_outrAF)r4rErFr6r7r8) rrKrrIrLrOrMrNr:rPrQ+_flash_attention_forward_no_dropout_inplace)r}r,r-r.r/r0r1r2r3r4rr5r6r7r8rVrYs&&&&&&&&&&&&&&& r _varlen_attn_outrRs,)5K GG"3ELL4F4F"GIHIIHH @A))..LL        $Q%a.#MK( rc V^8dQhR\PR\PR\PR\PR\PR\PR,R\R \R \R \R,R \ \,R,R \R\PR,R\PR,R\R,R\P/#rr)r s"r rrs"" " <<" " << " ll " llT! " " "" 4<"cT!""||d""$"d "  \\!"rcVP^4pVP^4p\P!VV3\PVPR7p\P P 'd\PP4pV\PPP8XdMVP^4^, p\P!VVV3\PVPR7pV#)> Fake implementation for meta tensor computation and tracing. rG) rar:rbr<rIrcrdrerfrgrh)r}r,r-r.r/r0r1r2r3r4rr5r6r7r8rirjrkrlrms&&&&&&&&&&&&&&& r _varlen_attn_out_fakers*jjmG 1 I GEKK I }}HH;;= //88 8!q)A-J Y.ekk%,,I rc#bV^8dQhR\PR\PR\PR\PR\PR\PR,R\R \R \R,R \R,R \ \\3,R \ R\PR,R\PR,R\R,R\P\ \P\P3,,/#)rr}r,r-r.r/r0Nr1r2ror4rr5r6r7r8r rq)r s"r rrs:: : <<: : << : ll : llT! : : :T!: 4<:sCx::||d":$: d !:" \\E%,, 455#:rcVP^4pV eVP^4MVP^4pV 'gVV8wd\RV RV R24hV 'd!VV,^8wd\RV RV R24hV R8Hp\PPP VVVVVVVVVV \ V 4V V V V4pVeVP'dVV3#V#)zCompute variable-length attention using Flash Attention with a pre-allocated output tensor. Same as :func:`varlen_attn` but writes the attention output into the provided ``out`` tensor instead of allocating a new one. rsrtrurvrwrx)rarr:rPrzrr r)r}r,r-r.r/r0r1r2ror4rr5r6r7r8r{r|r3rs&&&&&&&&$$$$$$$ r varlen_attn_outrs 0**Q-K!,!8#((1+chhqkK +4%i }=3 4  kK/14%i }A ?  w&I ))   / /        [ C"*...Cx JrcZV^8dQhR\R\\R3,R\RR/#)rctxinputs.rXr N)rr=)r s"r rrs0"""U38_"c"d"rcVwppppppp p p p p pppVwpppVe \R4hVe \R4hVPW4WVVVVV4WnWnWnWnWnR#)Nz)seqused_k is an inference-only parameter.z+block_table is an inference-only parameter.)rOsave_for_backwardr1r2r3r4r)rrrXr,r-r.r/r0r1r2r3r4rr5r6r7r8r}rrZs&&& r _setup_contextrs        CiFGGHII%exc9UIIMI!Orz!torch_attn::_varlen_attn_backwardc!V^8dQhR\PR\PR\PR\PR\PR\PR\PR\PR \R \R \R \PR \R,R\ \,R,R\ \P\P\P3,/#rgrad_outr,r-r.r}rr/r0r1r2r3rZr4Nrr r9)r s"r rr sAAllA <<A A << A  A  AllAllA A AA||A 4<AcT!A 5<<u|| 34Arc\V 4p \P!^VPR7pVP;'d \ VPP 4pV'dz\PR4V ^,R8wgV ^,R8wd \R4h\PPPVVVVVVVVVV RV V VV R7wpppMa\PR4\PPPVVVVVVVVVV RV V VV V ^,V ^,R7wpppVVV3#) ry)rIr?r@rArBrC)r4rErFr)rr:rbrIrKrrLrMrNrOrPrQ_cudnn_attention_backward_flash_attention_backward)rr,r-r.r}rr/r0r1r2r3rZr4runusedrVdqdkdvs&&&&&&&&&&&&&& r _varlen_attn_backwardr sG")5K [[5<< 0F GG"3ELL4F4F"GI 67 q>R ;q>R#7f YY^^==              > B$ @AYY^^==              (^)!n#> B& r2:rc!V^8dQhR\PR\PR\PR\PR\PR\PR\PR\PR \R \R \R \PR \R,R\ \,R,R\ \P\P\P3,/#rr9)r s"r rrRs,,ll, <<, , << ,  ,  ,ll,ll, , ,,||, 4<,cT!, 5<<u|| 34,rc\V 4p \P!V4p\P!V4p\P!V4pWV3#)r)rr:r`)rr,r-r.r}rr/r0r1r2r3rZr4r grad_querygrad_key grad_values&&&&&&&&&&&&&& r _varlen_attn_backward_fakerQsI()5K!!%(J$H!!%(J  ++rc V^8dQhR\R\PR\PR\PR\\PR,R3,/#)rrrgrad_lsegrad_rngr N.)rr:r;r=)r s"r rrnsS11 1 105 1HM 1 5<<$  #$1rc2VPwrErgrrVPp VPp VPpVPpVP p\ PPPVVVVV V VVV V VV VV4wppp^ pVVV.RV,O5#) )N) saved_tensorsr1r2r3r4rr:rPrzr)rrrrr,r-r.r/r0r}rrZr1r2r3r4rrrr num_paramss&&&& r _backwardrnsBEARAR>E IIE IIE I IIE//K%%;;        JBB$J B 0'J. 00r) setup_context)_varlen_attn_backward_flop_varlen_attn_forward_flop_varlen_attn_out_flop flop_registry)r~rr)FNNFNNN)rr)NN)(r%logging functoolsrtypingrrr: getLoggerr!rM__all__rrrlibrary custom_opr] register_fakernr~rrrrrrrregister_autograd_dynamodisallow_in_graphrPrQrtorch.utils.flop_counterrrrrrzrrr rso" ! : 1 3"EV+FV+r.(.(bV%)VV$,VV&*V(,V"Vr7ugN2O2j" "J:%)::$,::&*:(,: "!:z"B<2NAOAH$$,%,81B yG  IINN>>4M eii""//07L eii""334