+ &jhRt^RIt^RIt^RIHt^RIHtHt^RIH t ^RI t ^RI H t .R6Ot ]!R4t ] !R4t]R7,t]R8,tR9RR lltR9R R lltR9R R lltRRltRRltR9RRlltR:RRlltR:RRlltR;RRlltRRltRRltRRltR R!ltRR,R-llt#R>R.R/llt$R?R0R1llt%R@R2R3llt&R4R5lt']'!]4t(]'!]4t)]'!]4t*]'!]4t+]'!]4t,]'!] 4t-]'!]!4t.]'!]#4t/]'!]$4t0]'!]%4t1]'!]&4t2R#)AzHThis file contains utilities for initializing neural network parameters.N)Callable)LiteralTypeVar) ParamSpecTensor_R_Pc vV^8dQhR\R\R\R\PR,R\/#tensorab generatorNreturnrfloattorch Generator)formats"e/Users/jameslopez/projects/CWCArchive/cwc-podcast/.venv/lib/python3.14/site-packages/torch/nn/init.py __annotate__rEs=:: ::!&:38??T3I: :c\P!4;_uu_4VPWVR7uuRRR4# +'giR#;irN)rno_graduniform_r rrrs&&&&r_no_grad_uniform_r Es+ qy9  < A c vV^8dQhR\R\R\R\PR,R\/#r r meanstdrNrr)rs"rrrLsC>> > > >% >  >rc\P!4;_uu_4VPWVR7uuRRR4# +'giR#;ir)rrnormal_r r$r%rs&&&&r_no_grad_normal_r)Ls+ ~~d9~= r!cV^8dQhR\R\R\R\R\R\PR,R\/# r r r$r%rrrNrr)rs"rrrVs`JJ J J J J  J % J Jrc <VP'dV#RRlpW^V,, 8gW^V,,8d\P!R^R7\P!4;_uu_4V!WA, V, 4V!W1, V, 4, pVR8dVP VRR7P 4pVP VRR7P 4p VPWVR7p W8W8,p V P4'gM?\P!V \P!V 4PWVR7V 4p KdW JdVPV 4EM\V\W44p R W, V, ^,,p \P!V4p\P!V4pVPW4VR7VPV4VPV4P!V4P#^4P%R 4PV 4VPVR7P'4P)V4pVP4'gMTp VPW4VR7\P!VW4p VPV4P!V4P#^4P%R 4PV 4\P!VVPVR7P'4P)V4V4pVP4'dKTPT 4VuuRRR4# +'giR#;i) c0V^8dQhR\R\/#)r xr)r)rs"rr,_no_grad_trunc_normal_..__annotate__bs::E:e:rcR\P!V\P!R4, 4,R, #)?@)matherfsqrt)r.s&rnorm_cdf(_no_grad_trunc_normal_..norm_cdfbs(dhhq499S>122c99rzjmean is more than 2 std from [a, b] in nn.init.trunc_normal_. The distribution of values may be incorrect. stacklevelg333333?cpu)devicerNg)is_metawarningswarnrr new_tensoritemr'anywhere empty_likecopy_maxminrsub_div_pow_mul_log_gt)r r$r%rrrr6plohiresultmaskmodelog_peak candidates accept_bufpendings&&&&&& r_no_grad_trunc_normal_rWVs~~~ : 1s7{1s7{ 2  ;  ah#% &18s2B)C C s7""1U"388:B""1U"388:B^^D^CF  4xxzz$$V,44T)4T # V$q#d,'D s2q88H))&1J))&1J OOAIO 6   V $ OOD ! & &s + 0 0 3 8 8 > C CH M ))I)>CCEHHTG;;=='' 'B"[[*EFOOD)..s388;@@FKKHU#kk"++i+@EEGJJ:VG #;;== V$i s8K/N ,N  N c<V^8dQhR\R\R\/#r r valrrr)rs"rrrs!!!6!!&!rc\P!4;_uu_4VPV4uuRRR4# +'giR#;iN)rrfill_r rZs&&r_no_grad_fill_r`s% ||C  s : A c0V^8dQhR\R\/#r r rr)rs"rrrs6frc\P!4;_uu_4VP4uuRRR4# +'giR#;ir])rrzero_r s&r_no_grad_zero_rfs" ||~ s 9 A c`V^8dQhR\R\\,R,R\/#)r nonlinearityparamNr)_NonlinearityTypeintr)rs"rrrs7GEGE#GE,/%K$,>GE GErc.R OpW9gVR8Xd^#VR8XdR #VR8Xd\P!R4#VR8XdVfRpMT\V\4'g\V\4'g\V\ 4'dTpM\ RV R24h\P!R^V^,,, 4#VR 8XdR #\ R V 24h)aReturn the recommended gain value for the given nonlinearity function. The values are as follows: ================= ==================================================== nonlinearity gain ================= ==================================================== Linear / Identity :math:`1` Conv{1,2,3}D :math:`1` Sigmoid :math:`1` Tanh :math:`\frac{5}{3}` ReLU :math:`\sqrt{2}` Leaky Relu :math:`\sqrt{\frac{2}{1 + \text{negative\_slope}^2}}` SELU :math:`\frac{3}{4}` ================= ==================================================== .. warning:: In order to implement `Self-Normalizing Neural Networks`_ , you should use ``nonlinearity='linear'`` instead of ``nonlinearity='selu'``. This gives the initial weights a variance of ``1 / N``, which is necessary to induce a stable fixed point in the forward pass. In contrast, the default gain for ``SELU`` sacrifices the normalization effect for more stable gradient flow in rectangular layers. Args: nonlinearity: the non-linear function (`nn.functional` name) param: optional parameter for the non-linear function Examples: >>> gain = nn.init.calculate_gain( ... "leaky_relu", 0.2 ... ) # leaky_relu with negative_slope=0.2 .. _Self-Normalizing Neural Networks: https://papers.nips.cc/paper/2017/hash/5d44ee6f2c3f71b73125876103c8f6c4-Abstract.html sigmoidtanhrelur2 leaky_relu{Gz?znegative_slope z not a valid numberseluzUnsupported nonlinearity )linearconv1dconv2dconv3dconv_transpose1dconv_transpose2dconv_transpose3dg?g?)r3r5 isinstanceboolrkr ValueError)rhri linear_fnsnegative_slopes&& rcalculate_gainrsLJ!\Y%>    yy~  % =!N5$''5#&&%''#Nug5HIJ JyyNA$5 5677    4\NCDDrc vV^8dQhR\R\R\R\PR,R\/#r r)rs"rrrsC66 6 6 6% 6  6rc \PPV4'd)\PP\V3WW#R7#\ WW#4#)aFill the input Tensor with values drawn from the uniform distribution. :math:`\mathcal{U}(a, b)`. Args: tensor: an n-dimensional `torch.Tensor` a: the lower bound of the uniform distribution b: the upper bound of the uniform distribution generator: the torch Generator to sample from (default: None) Examples: >>> w = torch.empty(3, 5) >>> nn.init.uniform_(w) r)r overrideshas_torch_function_variadichandle_torch_functionrr rs&&&&rrrsO( 226::44 viq5   V 55rc vV^8dQhR\R\R\R\PR,R\/#r#r)rs"rrrsC:: : : :% :  :rc \PPV4'd)\PP\V3WW#R7#\ WW#4#)aFill the input Tensor with values drawn from the normal distribution. :math:`\mathcal{N}(\text{mean}, \text{std}^2)`. Args: tensor: an n-dimensional `torch.Tensor` mean: the mean of the normal distribution std: the standard deviation of the normal distribution generator: the torch Generator to sample from (default: None) Examples: >>> w = torch.empty(3, 5) >>> nn.init.normal_(w) r()rrrrr'r)r(s&&&&rr'r'sO( 226::44 fYvc5   F# 99rcV^8dQhR\R\R\R\R\R\PR,R\/#r+r)rs"rrr-s`!P!P !P !P !P !P  !P % !P !Prc \WW#WER7#)aFill the input Tensor with values drawn from a truncated normal distribution. The values are effectively drawn from the normal distribution :math:`\mathcal{N}(\text{mean}, \text{std}^2)` with values outside :math:`[a, b]` redrawn until they are within the bounds. The method used for generating the random values works best when :math:`a \leq \text{mean} \leq b`. For reduced-precision types (``torch.float16`` and ``torch.bfloat16``), sampling quality depends on the underlying ``normal_()`` and ``uniform_()`` implementations which operate at higher internal precision to avoid quantization artifacts. Args: tensor: an n-dimensional `torch.Tensor` mean: the mean of the normal distribution std: the standard deviation of the normal distribution a: the minimum cutoff value b: the maximum cutoff value generator: the torch Generator to sample from (default: None) Examples: >>> w = torch.empty(3, 5) >>> nn.init.trunc_normal_(w) r)rW)r r$r%rrrs&&&&&&r trunc_normal_r-sB "& OOrc<V^8dQhR\R\R\/#rYr[)rs"rrrQs!''f'5'V'rc\PPV4'd(\PP\V3WR7#\ W4#)zFill the input Tensor with the value :math:`\text{val}`. Args: tensor: an n-dimensional `torch.Tensor` val: the value to fill the tensor with Examples: >>> w = torch.empty(3, 5) >>> nn.init.constant_(w, 0.3) r_)rrrr constant_r`r_s&&rrrQsK 226::44 y5   & &&rc0V^8dQhR\R\/#rbr)rs"rrrcs ' '& 'V 'rc\VR4#)zFill the input Tensor with the scalar value `1`. Args: tensor: an n-dimensional `torch.Tensor` Examples: >>> w = torch.empty(3, 5) >>> nn.init.ones_(w) r1)r`res&rones_rcs &# &&rc0V^8dQhR\R\/#rbr)rs"rrrps " "6 "f "rc\V4#)zFill the input Tensor with the scalar value `0`. Args: tensor: an n-dimensional `torch.Tensor` Examples: >>> w = torch.empty(3, 5) >>> nn.init.zeros_(w) )rfres&rzeros_rps & !!rc0V^8dQhR\R\/#rbr)rs"rrr}sFrc VP4^8wd \R4h\P!4;_uu_4\P!VP RVRVP /RRR4V# +'giT#;i)aFill the 2-dimensional input `Tensor` with the identity matrix. Preserves the identity of the inputs in `Linear` layers, where as many inputs are preserved as possible. Args: tensor: a 2-dimensional `torch.Tensor` Examples: >>> w = torch.empty(3, 5) >>> nn.init.eye_(w) ,Only tensors with 2 dimensions are supportedout requires_gradN) ndimensionr|rreyeshaperres&reye_r}scaGHH  6<<PVP6;O;OP  M  Ms -A55 B c<V^8dQhR\R\R\/#)r r groupsr)rrk)rs"rrrs!556535v5rc VP4pVR9d \R4hVP4pV^,V,^8wd \R4hVP'dV#V^,V,p\ WC^,4p\ P !4;_uu_4VP4\V4Fp\V4FpV^8Xd-^WV,V,WpP^4^,3&K6V^8XdE^VWd,V,VVP^4^,VP^4^,3&K^VWd,V,VVP^4^,VP^4^,VP^4^,3&K K RRR4V# +'giT#;i)aFill the {3, 4, 5}-dimensional input `Tensor` with the Dirac delta function. Preserves the identity of the inputs in `Convolutional` layers, where as many input channels are preserved as possible. In case of groups>1, each group of channels preserves identity Args: tensor: a {3, 4, 5}-dimensional `torch.Tensor` groups (int, optional): number of groups in the conv layer (default: 1) Examples: >>> w = torch.empty(3, 16, 5, 5) >>> nn.init.dirac_(w) >>> w = torch.empty(3, 24, 5, 5) >>> nn.init.dirac_(w, 3) z5Only tensors with 3, 4, or 5 dimensions are supportedz!dim 0 must be divisible by groupsN)) rr|sizer<rFrrrdrange)r r dimensionssizesout_chans_per_grpmin_dimgds&& rdirac_rs ""$J"PQQ KKME Qx&A<== ~~~ aF*#1X.G  vA7^?PQF0014aQ19LLM1_  -1 A!+ A!+- -1 A!+ A!+ A!+ -$ , M- , Ms &D F<< G cRV^8dQhR\R\\\3,/#rb)rtuplerk)rs"rrrs"&U38_rc"VP4pV^8d \R4hVP^4pVP^4p^pVP4^8d#VPR,F pWE,pK W$,pW4,pWg3#)r zNFan in and fan out can not be computed for tensor with fewer than 2 dimensions:r NN)dimr|rr)r rnum_input_fmapsnum_output_fmapsreceptive_field_sizesfan_infan_outs& r_calculate_fan_in_and_fan_outrsJA~ \  kk!nO{{1~ zz|ab!!A % "  3F5G ?rcjV^8dQhR\R\R\PR,R\/#r r gainrNrr)rs"rrrs977 7 7%7 7rc\V4wr4V\P!R\W4,4, 4,p\P!R4V,p\ W)Wb4#)aFill the input `Tensor` with values using a Xavier uniform distribution. The method is described in `Understanding the difficulty of training deep feedforward neural networks` - Glorot, X. & Bengio, Y. (2010). The resulting tensor will have values sampled from :math:`\mathcal{U}(-a, a)` where .. math:: a = \text{gain} \times \sqrt{\frac{6}{\text{fan\_in} + \text{fan\_out}}} Also known as Glorot initialization. Args: tensor: an n-dimensional `torch.Tensor` gain: an optional scaling factor generator: the torch Generator to sample from (default: None) Examples: >>> w = torch.empty(3, 5) >>> nn.init.xavier_uniform_(w, gain=nn.init.calculate_gain("relu")) r2@)rr3r5rr )r rrrrr%rs&&& rxavier_uniform_rsQ44F;OF 3v'7!889 9C #A VR 66rcjV^8dQhR\R\R\PR,R\/#rr)rs"rrrs999 9 9%9 9rc\V4wr4V\P!R\W4,4, 4,p\ VRWR4#)aFill the input `Tensor` with values using a Xavier normal distribution. The method is described in `Understanding the difficulty of training deep feedforward neural networks` - Glorot, X. & Bengio, Y. (2010). The resulting tensor will have values sampled from :math:`\mathcal{N}(0, \text{std}^2)` where .. math:: \text{std} = \text{gain} \times \sqrt{\frac{2}{\text{fan\_in} + \text{fan\_out}}} Also known as Glorot initialization. Args: tensor: an n-dimensional `torch.Tensor` gain: an optional scaling factor generator: the torch Generator to sample from (default: None) Examples: >>> w = torch.empty(3, 5) >>> nn.init.xavier_normal_(w) r2)rr3r5rr))r rrrrr%s&&& rxavier_normal_rs?24F;OF 3v'7!889 9C FC 88rc<V^8dQhR\R\R\/#)r r rRr)r_FanModerk)rs"rrrs!33633c3rcVP4pRR.pW9d\RV RV 24h\V4wr4VR8XdV#T#)rrzMode z" not supported, please use one of )lowerr|r)r rR valid_modesrrs&& r_calculate_correct_fanrsT ::C>C >C >C >C$ >C % >C  >Crc 8\PPV4'd,\PP\V3VVVVVR7#^VP 9d\ P!R^R7V#\W4p\W14pV\P!V4, p\P!R4V,p\P!4;_uu_4VPV)WR7uuRRR4# +'giR#;i)aDFill the input `Tensor` with values using a Kaiming uniform distribution. The method is described in `Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification` - He, K. et al. (2015). The resulting tensor will have values sampled from :math:`\mathcal{U}(-\text{bound}, \text{bound})` where .. math:: \text{bound} = \text{gain} \times \sqrt{\frac{3}{\text{fan\_mode}}} Also known as He initialization. Args: tensor: an n-dimensional `torch.Tensor` a: the negative slope of the rectifier used after this layer (only used with ``'leaky_relu'``) mode: either ``'fan_in'`` (default) or ``'fan_out'``. Choosing ``'fan_in'`` preserves the magnitude of the variance of the weights in the forward pass. Choosing ``'fan_out'`` preserves the magnitudes in the backwards pass. nonlinearity: the non-linear function (`nn.functional` name), recommended to use only with ``'relu'`` or ``'leaky_relu'`` (default). generator: the torch Generator to sample from (default: None) Examples: >>> w = torch.empty(3, 5) >>> nn.init.kaiming_uniform_(w, mode="fan_in", nonlinearity="relu") Note: Be aware that ``fan_in`` and ``fan_out`` are calculated assuming that the weight matrix is used in a transposed manner, (i.e., ``x @ w.T`` in ``Linear`` layers, where ``w.shape = [fan_out, fan_in]``). This is important for correct initialization. If you plan to use ``x @ w``, where ``w.shape = [fan_in, fan_out]``, pass in a transposed weight matrix, i.e. ``nn.init.kaiming_uniform_(w.T, ...)``. )r rrRrhr,Initializing zero-element tensors is a no-opr8rrN)rrrrkaiming_uniform_rr=r>rrr3r5rr) r rrRrhrfanrr%bounds &&&&& rrr*sV 226::44  I%5   FLL DQRS  .C , *D 3 C IIcNS E vuB s )D D c V^8dQhR\R\R\R\R\P R,R\/#rr)rs"rrrksM2;2; 2; 2; 2;$ 2; % 2;  2;rc\^VP9d\P!R^R7V#\W4p\ W14pV\ P !V4, p\P!4;_uu_4VP^WtR7uuRRR4# +'giR#;i)a+Fill the input `Tensor` with values using a Kaiming normal distribution. The method is described in `Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification` - He, K. et al. (2015). The resulting tensor will have values sampled from :math:`\mathcal{N}(0, \text{std}^2)` where .. math:: \text{std} = \frac{\text{gain}}{\sqrt{\text{fan\_mode}}} Also known as He initialization. Args: tensor: an n-dimensional `torch.Tensor` a: the negative slope of the rectifier used after this layer (only used with ``'leaky_relu'``) mode: either ``'fan_in'`` (default) or ``'fan_out'``. Choosing ``'fan_in'`` preserves the magnitude of the variance of the weights in the forward pass. Choosing ``'fan_out'`` preserves the magnitudes in the backwards pass. nonlinearity: the non-linear function (`nn.functional` name), recommended to use only with ``'relu'`` or ``'leaky_relu'`` (default). generator: the torch Generator to sample from (default: None) Examples: >>> w = torch.empty(3, 5) >>> nn.init.kaiming_normal_(w, mode="fan_out", nonlinearity="relu") Note: Be aware that ``fan_in`` and ``fan_out`` are calculated assuming that the weight matrix is used in a transposed manner, (i.e., ``x @ w.T`` in ``Linear`` layers, where ``w.shape = [fan_out, fan_in]``). This is important for correct initialization. If you plan to use ``x @ w``, where ``w.shape = [fan_in, fan_out]``, pass in a transposed weight matrix, i.e. ``nn.init.kaiming_normal_(w.T, ...)``. rr8rN) rr=r>rrr3r5rrr')r rrRrhrrrr%s&&&&& rkaiming_normal_rkssV FLL DQRS  .C , *D 3 C ~~a~: s <B B+ cjV^8dQhR\R\R\PR,R\/#rr)rs"rrrs900 0 0%0 0rcVP4^8d \R4hVP4^8XgVP'dV#VP ^4pVP4V,pVP W434P ^^VR7pW48dVP4\PPV4wrg\P!V^4pVP4p Wi,pW48dVP4\P!4;_uu_4VPV4PV4VP!V4RRR4V# +'giT#;i)alFill the input `Tensor` with a (semi) orthogonal matrix. Described in `Exact solutions to the nonlinear dynamics of learning in deep linear neural networks` - Saxe, A. et al. (2013). The input tensor must have at least 2 dimensions, and for tensors with more than 2 dimensions the trailing dimensions are flattened. Args: tensor: an n-dimensional `torch.Tensor`, where :math:`n \geq 2` gain: optional scaling factor generator: the torch Generator to sample from (default: None) Examples: >>> # xdoctest: +REQUIRES(env:TORCH_DOCTEST_LAPACK) >>> w = torch.empty(3, 5) >>> nn.init.orthogonal_(w) z4Only tensors with 2 or more dimensions are supportedrN)rr|numelr<r new_emptyr't_rlinalgqrdiagsignrview_asrDrJ) r rrrowscols flattenedqrrphs &&& r orthogonal_rs,QOPP ||~fnnn ;;q>D <<>T !D  $.66q!y6QI {  <>> w = torch.empty(3, 5) >>> nn.init.sparse_(w, sparsity=0.1) rrN) rr|r<rr3ceilrrr'rrandperm) r rr%rrr num_zeroscol_idx row_indices zero_indicess &&&& rsparse_rs.aGHH ~~~ JD (/*I q#3T{G...K&z 2L,-F( )#  M  Ms .__annotate__s)%%rww%"))%%rc\<\P!RS RS R2\^R7S!V/VB#)z `nn.init.z)` is now deprecated in favor of `nn.init.z`.r8)r=r> FutureWarning)rrrnew_nameold_names*,rdeprecated_init(_make_deprecate..deprecated_inits; z!J8*TV W  T$V$$rz z_(...) .. warning:: This method is now deprecated in favor of :func:`torch.nn.init.z"`. See :func:`~torch.nn.init.z` for details.)__name____doc__)rrrrsf @@r_make_deprecatersa}}H}H%%$ JHIQzR'j :O (O r)rrr'rrrrrrrrrrrruniformnormalconstantrdiracxavier_uniform xavier_normalkaiming_uniformkaiming_normal orthogonalsparse) rsrtrurvrwrxryrmrnrorprr)rrr])rr1N)rr1gr2N))r1N)rrpN)rN)rqN)3rr3r=collections.abcrtypingrrtyping_extensionsrrr__all__rr rjrr r)rWr`rfrrr'rrrrrrrrrrrrrrrrrrrrrrrrrrrrrsGN $#'  > T]t_    & ':>JZ!  GET66:6!PH'$ ' "*5p*7B9>3>CB2;j0f&T. ( #  ! 9 %d 1/ !"23 1 [ )  !r