+ &jnP Rt^RIHt^RIt^RIHt^RIHtHtHtH t H t H t H t H t HtHtHtHtHtHtRR.t!RR]4tRR ] R ] R ] R ] R ] R 2 ,]nR RltRRlt] !]R7RRRll4tR#)z)Implementation for the RMSprop algorithm.)castN)Tensor)_capturable_doc_default_to_fused_or_foreach_differentiable_doc_disable_dynamo_if_unsupported _foreach_doc!_get_capturable_supported_devices_get_scalar_dtype _maximize_doc _params_doc _to_scalar_use_grad_for_differentiable _view_as_real OptimizerParamsTRMSproprmspropchaa]tRt^toRV3RlV3RllltV3RltRt]RRl4tRt Vt V;t #) rc~<V^8dQhRS[RS[S[,RS[RS[RS[RS[RS[RS[R S[R ,R S[R S[R R / #)paramslralphaeps weight_decaymomentumcentered capturableforeachNmaximizedifferentiablereturn)rfloatrbool)format __classdict__s"k/Users/jameslopez/projects/CWCArchive/cwc-podcast/.venv/lib/python3.14/site-packages/torch/optim/rmsprop.py __annotate__RMSprop.__annotate__s'+'+'+ FN'+ '+  '+  '+'+'+'+'+'+'+ '+c <\V\4'd!VP4^8wd \R4hRV8:g\RV 24hRV8:g\RV 24hRV8:g\RV 24hRV8:g\RV 24hRV8:g\RV 24hRVR VR VR VR VR VRVRV RV RV / p \S V`W4R#)zTensor lr must be 1-elementgzInvalid learning rate: zInvalid epsilon value: zInvalid momentum value: zInvalid weight_decay value: zInvalid alpha value: rrrrrrrrr r!N) isinstancernumel ValueErrorsuper__init__)selfrrrrrrrrrr r!defaults __class__s&&&&&&&&&&&& r'r1RMSprop.__init__s b& ! !bhhjAo:; ;by6rd;< <cz6se<= =h7zBC Cl";L>JK Ke|4UG<= = "  U 3  L * w  n   *r*c<\SV`V4VPEFBpVPR^4VPRR4VPRR4VPRR4VPRR4VPRR4VR,FpVPP V.4p\ V4^8wgK1\P!VR ,4'dKV\VR ,4pVR,'d,\P!V\4VPR 7M\P!V\4R 7VR &K EKE R#) rrFrNr r!rrstepdtypedevicer9) r0 __setstate__ param_groups setdefaultstategetlentorch is_tensorr#tensorr r:)r2r?grouppp_statestep_valr4s&& r'r<RMSprop.__setstate__Hs U#&&E   Z +   Z /   Y -   Z /   -u 5   \5 18__**..B/w<1$U__WV_-M-M$WV_5H !.. $,=,?#\\(:K:MN FO %'r*c^RpVR,EFp V PfKV\P!V 4,pVPV 4V PP'd \ R4hVPV P4VP V ,p \V 4^8XdVR,'d,\P!R \4V PR7M\P!R \4R7V R&\P!V \PR7V R&VR ,^8d*\P!V \PR7V R &VR ,'d*\P!V \PR7V R &VPV R,4VPV R,4VR ,^8dVPV R ,4VR ,'gEKVPV R ,4EK V#)Frz)RMSprop does not support sparse gradientsrr8r;r7) memory_format square_avgrmomentum_bufferrgrad_avg) gradrB is_complexappend is_sparse RuntimeErrorr?rAzerosr r: zeros_likepreserve_format) r2rEparams_with_gradgrads square_avgsmomentum_buffer_list grad_avgs state_steps has_complexrFr?s &&&&&&&& r' _init_groupRMSprop._init_group]s xAvv~ 5++A. .K  # #A &vv"#NOO LL JJqME5zQ\**KK*;*=ahhOR/@/BCf ',&6&6U%:%:'l#$q(/4/?/?)>)>0E+,$$(-(8(8)>)>)E*%   u\2 3   uV} -Z 1$$++E2C,DEZ    z!23I!Lr*c VP4RpVe.\P!4;_uu_4V!4pRRR4VPFp.p.p.p.p.p.p VP VVVVVVV 4p \ VVVVVV VR,VR,VR,VR,VR,VR,VR,VR ,VR ,VR ,V R 7K V# +'giL;i) zPerform a single optimization step. Args: closure (Callable, optional): A closure that reevaluates the model and returns the loss. Nrrrrrrrr r!r) rrrrrrrr r!rr^)'_accelerator_graph_capture_health_checkrB enable_gradr=r_r) r2closurelossrErXrYrZr\r[r]r^s && r'r7 RMSprop.steps  446  ""$$y%&&E-/ "$E(*K&(I13 (*K** $K  $;Gn%L">2z*z*i(z*$%56 .'# %'L S%$s C C. rO) g{Gz?gGz?g:0yE>rgFFNFFN) __name__ __module__ __qualname____firstlineno__r1r<r_rr7__static_attributes____classdictcell__ __classcell__)r4r&s@@r'rrs4'+'+R*1f"4"44r*aj Implements RMSprop algorithm. .. math:: \begin{aligned} &\rule{110mm}{0.4pt} \\ &\textbf{input} : \alpha \text{ (alpha)}, \: \gamma \text{ (lr)}, \: \theta_0 \text{ (params)}, \: f(\theta) \text{ (objective)} \\ &\hspace{13mm} \lambda \text{ (weight decay)},\: \mu \text{ (momentum)}, \: centered, \: \epsilon \text{ (epsilon)} \\ &\textbf{initialize} : v_0 \leftarrow 0 \text{ (square average)}, \: \textbf{b}_0 \leftarrow 0 \text{ (buffer)}, \: g^{ave}_0 \leftarrow 0 \\[-1.ex] &\rule{110mm}{0.4pt} \\ &\textbf{for} \: t=1 \: \textbf{to} \: \ldots \: \textbf{do} \\ &\hspace{5mm}g_t \leftarrow \nabla_{\theta} f_t (\theta_{t-1}) \\ &\hspace{5mm}if \: \lambda \neq 0 \\ &\hspace{10mm} g_t \leftarrow g_t + \lambda \theta_{t-1} \\ &\hspace{5mm}v_t \leftarrow \alpha v_{t-1} + (1 - \alpha) g^2_t \hspace{8mm} \\ &\hspace{5mm} \tilde{v_t} \leftarrow v_t \\ &\hspace{5mm}if \: centered \\ &\hspace{10mm} g^{ave}_t \leftarrow g^{ave}_{t-1} \alpha + (1-\alpha) g_t \\ &\hspace{10mm} \tilde{v_t} \leftarrow \tilde{v_t} - \big(g^{ave}_{t} \big)^2 \\ &\hspace{5mm}if \: \mu > 0 \\ &\hspace{10mm} \textbf{b}_t\leftarrow \mu \textbf{b}_{t-1} + g_t/ \big(\sqrt{\tilde{v_t}} + \epsilon \big) \\ &\hspace{10mm} \theta_t \leftarrow \theta_{t-1} - \gamma \textbf{b}_t \\ &\hspace{5mm} else \\ &\hspace{10mm}\theta_t \leftarrow \theta_{t-1} - \gamma g_t/ \big(\sqrt{\tilde{v_t}} + \epsilon \big) \hspace{3mm} \\ &\rule{110mm}{0.4pt} \\[-1.ex] &\bf{return} \: \theta_t \\[-1.ex] &\rule{110mm}{0.4pt} \\[-1.ex] \end{aligned} For further details regarding the algorithm we refer to `lecture notes `_ by G. Hinton. and centered version `Generating Sequences With Recurrent Neural Networks `_. The implementation here takes the square root of the gradient average before adding epsilon (note that TensorFlow interchanges these two operations). The effective learning rate is thus :math:`\gamma/(\sqrt{v} + \epsilon)` where :math:`\gamma` is the scheduled learning rate and :math:`v` is the weighted moving average of the squared gradient. z Args: a0 lr (float, Tensor, optional): learning rate (default: 1e-2) alpha (float, optional): smoothing constant (default: 0.99) eps (float, optional): term added to the denominator to improve numerical stability (default: 1e-8) weight_decay (float, optional): weight decay (L2 penalty) (default: 0) momentum (float, optional): momentum factor (default: 0) centered (bool, optional) : if ``True``, compute the centered RMSProp, the gradient is normalized by an estimation of its variance z z c"`V^8dQhR\\,R\\,R\\,R\\,R\\,R\\,R\R\R \R \R \R \R \R\R\R\RR/#rrrYrZr\r[r]rrrrrrr r!rr^r"Nlistrr#r$)r%s"r'r(r( sJ1J1 LJ1 <J1fJ1F| J1 v, J1 f J1 J1 J1 J1J1J1J1J1J1 !J1"#J1$ %J1r*c \PP4'g \V4p\ V4EFwppVV,p\P P 4'glV'dd\4pVPPVPP8XdVPPV9g\RV R24hVV,pV 'gTMV)pVV,pV^, pV ^8wdVPVV R7p\P!V4pV'dC\P!V4p\P!V4p\P!V4pVPV4PVV^V, R7V 'ddVV,pV'd\P!V4pVP!V^V, 4VP#VVRR7P%4pMVP'4pV 'dVPV4pMVP)V4pV ^8d`VV,pV'd\P!V4pVPV 4P+VV4VP)VV)R7EKwVP+VVV)R7EK R#)IIf capturable=True, params and state_steps must be on supported devices: .rvalueN)rBjit is_scriptingr enumeratecompiler is_compilingr r:typeAssertionErroraddrQ view_as_realmul_addcmul_lerp_addcmulsqrt_sqrtadd_addcdiv_)rrYrZr\r[r]rrrrrrr r!rr^iparamr7capturable_supported_devicesrPrLis_complex_paramrNavgbufs&&&&&&$$$$$$$$$$ r'_single_tensor_rmspropr s-& 99 ! ! # # ^f%51~~~**,,+L+N ( !!T[[%5%55LL%%)EE$_`|_}}~Qx#t$ ^    1 88E86D ++E2 &&u-E%%d+D++J7J''d!e)'D  |H --h7 NN4U +$$Xxr$BHHJC//#C ''#,C((3-C a<&q)C((- HHX  ' 'c 2 JJs2#J & NN4RCN 0i&r*c"`V^8dQhR\\,R\\,R\\,R\\,R\\,R\\,R\R\R \R \R \R \R \R\R\R\RR/#rqrr)r%s"r'r(r(VsCWCW LCW <CWfCWF| CW v, CW f CW CW CW CWCWCWCWCWCW !CW"#CW$ %CWr*c  a!\V4^8XdR#V 'd \R4h\PP 4'gxV'dp\ 4o!\ ;QJd*V!3Rl\WRR74F 'dK RM RM!V!3Rl\WRR744'g\RS! R24h\V4p\P!WW#WE.4pVP4EF wwppppppp\\\,V4p\\\,V4p\\\,V4p\\\,V4pV'dyVV.pV ^8d-\\\,V4pVPV4V 'd-\\\,V4pVPV4\!V.VO5!V 'd\P"!V4p\PP 4'gJV^,P$'d1\P&!V\P(!R R R 7R R 7M\P&!V^4V ^8wd<V 'd\P&!VVV R 7M\P*!VVV R 7p\P,!VV4\P.!VVV^V, R 7V 'd\\\,V4p\P0!VV^V, 4\P2!VVVRR 7p\P4!V4\P&!VV4M-\P6!V4p\P&!VV4V ^8d\\\,V4p\P,!VV 4\P8!VVV4V'dS\;V\P4'd3\P<!VV)4p \P&!VV 4EKx\P&!VVV)R 7EKV'dT\;V\P4'd4\P>!VV)4\P8!VVV4EK\P8!VVVV)R 7EK R#)rgNz#_foreach ops don't support autogradc3<"TFUwrVPPVPP8H;'dVPPS9xKW R#5irh)r:r).0rFr7rs& r' (_multi_tensor_rmsprop..rsT A HHMMT[[-- - > > !== >@s :A "A T)strictFrurvg?cpu)r:rwrxrz) rArrBr~rr allzipr r"_group_tensors_by_device_and_dtypevaluesrrsrrRr _foreach_negis_cpu _foreach_add_rD _foreach_add _foreach_mul__foreach_addcmul__foreach_lerp__foreach_addcmul_foreach_sqrt_ _foreach_sqrt_foreach_addcdiv_r- _foreach_mul _foreach_div_)"rrYrZr\r[r]rrrrrrr r!rr^grouped_tensorsgrouped_params_grouped_grads_grouped_square_avgs_grouped_grad_avgs_grouped_momentum_buffer_list_grouped_state_steps__grouped_params grouped_gradsgrouped_square_avgsgrouped_state_stepsstate_and_gradsgrouped_momentum_buffer_listgrouped_grad_avgsr momentum_lrrs"&&&&&&$$$$$$$$$$ @r'_multi_tensor_rmsproprVs& 6{aBCC >> & & ( (Z'H'J$s v4@ sss v4@   ![\x[yyz{  BBBB  0DRO  " " $     ) d6lO<T&\>: "4<1EF"4<1EF ,.ABO!|/3L"?0, &&'CD$(f7I$J!&&'89 . ;? ; !..}=M ~~**,,1DQ1G1N1N1N   #U\\#e%DC     3Q 7 1 ##M>V % 2 2!>!  /7   QY   $T&\3E F   !2M1u9 M((#%68IQSC   %   S )%%&9:C   S ) a<+/V ;, (    .s!5-8 1ell##[s')FTzPAPI has changed, `state_steps` argument must contain a list of singleton tensorsN) use_fusedz6torch.jit.script not supported with foreach optimizers) rrrrrrr rr!r^) rBr~rrrTrr{r|rr)rrYrZr\r[r]rr r!rr^rrrrrrrfuncs&&&&&&&&&&&$$$$$$ r'rrs: >> & & ( (5-855-8522 ^  1 e 7599))++STTuyy--//$%   !%!r*)NFFFF)__doc__typingrrBr optimizerrrrrrr r r r r rrrr__all__rrrrrOr*r'rs0 $ i gigV+X       Y<BJ1ZCWL 1GHAIAr*