+ &j5Rt^RIHtHt^RIHt^RIt^RIHuH t ^RI H t H t HtHt^RIHt^RIHtHtHtHt.ROt]P0P3]4]P0P3] 4]P0P3] 4]P0P3]4!R R]4t!R R]P64tR R ltR RltR#)zCDefines bias subclasses that work with scaled_dot_product_attention)autoIntEnum)warnN)can_use_efficient_attentioncan_use_flash_attentionis_flash_attention_available SDPAParams)_raise_kernel_warnings)_calculate_scale_input_requires_grad_postprocess_flash_output_validate_sdpa_input CausalVariant CausalBiasc6]tRt^!tRt]!4t]!4tRtR#)ra Enum for causal variants used in attention mechanisms. Defines two types of causal biases: ``UPPER_LEFT``: Represents upper-left triangular bias for standard causal attention. The equivalent pytorch code for constructing this bias is: .. code-block:: python torch.tril(torch.ones(size, dtype=torch.bool)) For instance, with ``shape=(3,4)``, the materialized bias tensor will be: .. code-block:: text [[1, 0, 0, 0], [1, 1, 0, 0], [1, 1, 1, 0]] ``LOWER_RIGHT``: Represents lower-right triangular bias, the include values are aligned to the lower right corner of the matrix. The equivalent pytorch code for constructing this bias is: .. code-block:: python diagonal_offset = size[1] - size[0] torch.tril( torch.ones(size, dtype=torch.bool), diagonal=diagonal_offset, ) For instance, with ``shape=(3,4)``, the materialized bias tensor will be: .. code-block:: text [[1, 1, 0, 0], [1, 1, 1, 0], [1, 1, 1, 1]] Note that these variants are equivalent to each other when the sequence lengths of the query and key/value tensors are equal since the triangular matrix is square. .. warning:: This enum is a prototype and subject to change. N) __name__ __module__ __qualname____firstlineno____doc__r UPPER_LEFT LOWER_RIGHT__static_attributes__ro/Users/jameslopez/projects/CWCArchive/cwc-podcast/.venv/lib/python3.14/site-packages/torch/nn/attention/bias.pyrr!s.`J&Krcaa]tRt^VtoRtV3RlV3RlltV3RlRltV3RlRltRV3RlR llt] RV3R lR ll4t ] RV3R ll4t V3R lRlt RtVtV;t#)ra A bias representing causal attention patterns. For an overview of the bias structure, see the :class:`CausalVariant` enum. This class is used for defining causal (triangular) attention biases. For construing the bias, there exist two factory functions: :func:`causal_upper_left` and :func:`causal_lower_right`. Example: .. code-block:: python from torch.nn.attention.bias import causal_lower_right bsz, num_heads, seqlen_q, seqlen_kv, head_dim = 32, 8, 4, 12, 8 # Create a lower-right causal bias attn_bias = causal_lower_right(seqlen_q, seqlen_kv) q = torch.randn( bsz, num_heads, seqlen_q, head_dim, device="cuda", dtype=torch.float16 ) k = torch.randn( bsz, num_heads, seqlen_kv, head_dim, device="cuda", dtype=torch.float16 ) v = torch.randn( bsz, num_heads, seqlen_kv, head_dim, device="cuda", dtype=torch.float16 ) out = F.scaled_dot_product_attention(q, k, v, attn_bias) .. warning:: This class is a prototype and subject to change. c0<V^8dQhRS[RS[RS[RR/#)variant seq_len_q seq_len_kvreturnN)rint)format __classdict__s"r __annotate__CausalBias.__annotate__ws* #3SWrc <\V\4'g"\R\V4P 24h\ SV`4WnW nW0n W#8d'V\P8Xd\R^R7R#R#R#)a Initializes the CausalBias instance with a specified variant and sequence lengths. Args: variant (CausalVariant): The type of causal bias to use (either UPPER_LEFT or LOWER_RIGHT). seq_len_q (int): The sequence length of the query tensor. seq_len_kv (int): The sequence length of the key/value tensor. Raises a warning if the LOWER_RIGHT variant is used with seq_len_q > seq_len_kv, as it may produce NaNs. z%variant must be a CausalVariant, got zTLower right causal bias will produce NaNs in the output when seq_len_q > seq_len_kv!) stacklevelN) isinstancerAssertionErrortypersuper__init__rr r!rr)selfrr r! __class__s&&&&rr.CausalBias.__init__wsz'=11 7W 8N8N7OP   "$  !g1J1J&J f 'K !rcN<V^8dQhRS[PRS[P/#rdevicer"torchr4Tensor)r$r%s"rr&r's#  %,, 5<< rc  \P!\P!VPVPV\P R74#)zUpper left causal biasr4dtype)r6trilonesr r!boolr/r4s&&r _upper_leftCausalBias._upper_lefts1zz JJt~~tvUZZ X  rcN<V^8dQhRS[PRS[P/#r3r5)r$r%s"rr&r's#  5<< ELL rc  VPVP, p\P!\P!VPVPV\P R7VR7#)zLower right causal biasr9)diagonal)r!r r6r;r<r=)r/r4diagonal_offsets&& r _lower_rightCausalBias._lower_rightsK//DNN:zz JJejj %   rc\<V^8dQhRS[PR,RS[P/#)rr4Nr"r5)r$r%s"rr&r's(--5<<$#6-%,,-rc Vf\P!R4pVP\P8XdVP V4#VP\P 8XdVPV4#R#)aX Materializes the causal bias into a tensor form. Depending on the variant, this method generates either an upper-left or lower-right triangular matrix to represent the causal bias. Args: device (Optional[torch.device]): The device on which to create the tensor. Defaults to CPU. Returns: torch.Tensor: The materialized bias tensor. Ncpu)r6r4rrrr?rrEr>s&&r _materializeCausalBias._materializesb >\\%(F <<=33 3##F+ + \\]66 6$$V, ,7rc<V^8dQhRS[PRS[PRS[PRRRS[RS[RS[R ,R S[R S[P/ #) rquerykeyvalue attn_maskr dropout_p is_causalscaleN enable_gqar")r6r7floatr=)r$r%s"rr&r'soo||o \\o||o o  o  ot|oo orc V'd \R4hVPVP8Xg VP\P 8Xd\ P!VVVRVRVVR7#VP\P8XEdM\WVRWEV4\WVRWEV4p\V4'Ed$VPPR8Xd^@M^p VPR 4p \W4p W,^8gp V 'dWV ,, p \ P"P$P'V^V 34p\ P"P$P'V^V 34p\ P"P$P'V^V 34p\ P(P*P-VVVVRRV R7^,p\/W4#\1V4'dRp\3WV4'dRp\ P(P*P5VP7^^4VP7^^4VP7^^4RRRRRV\9VP4VVRR7 ^,P7^^4#\;V4\ P!VVVVP=VP4VRVVR7#\R VP 24h) a Handles the logic for computing attention with the specified causal bias. Args: query (Tensor): Query tensor; shape :math:`(N, ..., L, E)`. key (Tensor): Key tensor; shape :math:`(N, ..., S, E)`. value (Tensor): Value tensor; shape :math:`(N, ..., S, Ev)`. attn_mask (CausalBias): The type of causal attention to apply. A boolean mask where a value of True indicates that the element *should* take part in attention. A float mask of the same type as query, key, value that is added to the attention score. dropout_p (float): Dropout probability; if greater than 0.0, dropout is applied is_causal (bool): If true, assumes upper left causal attention masking and errors if both attn_mask and is_causal are set. scale (optional float): Scaling factor applied prior to softmax. If None, the default value is set to :math:`\frac{1}{\sqrt{E}}`. enable_gqa (optional bool): If set to True, Grouped Query Attention (GQA) is enabled, by default it is set to False. Returns: output (Tensor): Attention output; shape :math:`(N, ..., L, Ev)`. Raises: ValueError: If the causal bias variant is not a CausalVariant type. z.CausalBias should not be used with causal=TrueNT)rPrQrRrSrTxpuF)rRreturn_debug_maskrS) bias cu_seqlens_q cu_seqlens_k max_seqlen_q max_seqlen_krQcustom_mask_typecompute_log_sumexprSseqlen_kzF 588&&CC C==$1&1 1w)$tDDrc <V^8dQhRS[/#rr")str)r$r%s"rr&r'0s..#.rc>VP4P4#N)rJ__repr__)r/s&rrCausalBias.__repr__0s  "++--r)r!r rr)gFNF)rN)rrrrrr.r?rErJ staticmethodru classmethodrxrr__classdictcell__ __classcell__)r0r%s@@rrrVsu@2    --(ooobEE...rc$V^8dQhR\/#rr)r$s"rr&r&4sGG Grct\V4^8wd \R4hVwr\\PW4#)a Creates an upper-left triangular causal bias. This function generates a upper-left triangular matrix to represent causal attention bias with a diagonal offset set so that the inclusive values are aligned to the upper left corner of the matrix. This equivalent to the `is_causal=True` argument in `scaled_dot_product_attention`. The equivalent pytorch code for constructing this bias is: .. code-block:: python torch.tril(torch.ones(size, dtype=torch.bool)) For instance, with `shape=(3,4)`, the materialized bias tensor will be: .. code-block:: text [[1, 0, 0, 0], [1, 1, 0, 0], [1, 1, 1, 0]] Args: size: The size of the bias matrix. Returns: CausalBias: The UPPER_LEFT triangular causal bias variant. z*causal_upper_left only supports 2D tensors)lenr+rrrrer r!s* rcausal_upper_leftr4s68 4yA~IJJ I m.. FFrc$V^8dQhR\/#rr)r$s"rr&r&Vs"H"H"Hrct\V4^8wd \R4hVwr\\PW4#)a Creates a lower-right triangular causal bias. This function generates a lower-right triangular matrix to represent causal attention bias with a diagonal offset set so that the inclusive values are aligned to the lower right corner of the matrix. The equivalent pytorch code for constructing this bias is: .. code-block:: python diagonal_offset = size[1] - size[0] torch.tril( torch.ones(size, dtype=torch.bool), diagonal=diagonal_offset, ) For instance, with `shape=(3,4)`, the materialized bias tensor will be: .. code-block:: text [[1, 1, 0, 0], [1, 1, 1, 0], [1, 1, 1, 1]] Args: size: The size of the bias matrix. Returns: CausalBias: The LOWER_RIGHT triangular causal bias variant. z+causal_lower_right only supports 2D tensors)rr+rrrrs* rcausal_lower_rightrVs6> 4yA~JKK I m// GGr)rrrr)renumrrwarningsrr6torch.nn.functionalrfrgrctorch.backends.cudarrrrtorch.nn.attentionr torch.nn.attention._utilsr r r r __all___dynamoallow_in_graphrr7rrrrrrrsI  6 U 9: 45 89 Z(2G2j[.[.|GD"Hr