+ &jb^RIHtHtHtHtHt^RIHt^RI H t ^RI H t H t HtHtHt!RR4t!RR]4t!R R ]4t!R R ]4t!R R]4t!RR]4t!RR]4t!RR]4t!RR]4t!RR]4t!RR]4t!RR]4tRtR#))CallableListOptionalTupleUnionN)Module) tree_flattentree_map tree_merge tree_reducetree_unflattenc*a]tRt^ toRtRRltV3RlRltV3RlRltV3RlR ltV3R lR lt V3R lR lt ] R4t ] PV3RlRl4t ] R4t] R4t]PV3RlRl4tV3RlRltRtVtR#) OptimizerzThe base class for all optimizers. It allows us to implement an optimizer on a per-parameter basis and apply it to a parameter tree. NcRVnR\P!^\P4/VnT;'g/P 4UUu/uFwr#W#bK uppVnR#uuppi)FstepN) _initializedmxarrayuint64_stateitems _schedulers)self schedulerskvs&& q/Users/jameslopez/projects/CWCArchive/cwc-podcast/.venv/lib/python3.14/site-packages/mlx/optimizers/optimizers.py__init__Optimizer.__init__sV!rxx29956 .8.>.>B-E-E-GH-GTQAD-GHHs A)c&<V^8dQhRS[RS[/#)model gradients)rdict)format __classdict__s"r __annotate__Optimizer.__annotate__s = =F =t =c FVPVPW!44R#)a Apply the gradients to the parameters of the model and update the model with the new parameters. Args: model (mlx.nn.Module): An mlx module to be updated. gradients (dict): A Python tree of gradients, most likely computed via :func:`mlx.nn.value_and_grad`. N)updateapply_gradients)rr"r#s&&&rr+Optimizer.updates  T)));>> optimizer = optim.SGD(learning_rate=1e-1, momentum=0.9) >>> model = nn.Linear(2, 2) >>> optimizer.init(model.trainable_parameters()) >>> optimizer.state.keys() dict_keys(['step', 'learning_rate', 'weight', 'bias']) c <\V\\34'd\V4p\\ V44FpS!W,W,4W&K \ V4\ V48wd)VP \ RV\ V4R44\V4!V4#\V\4'dDVP4F-wr4W19d\ RV4W&KS!WAV,4W&K/ V#V#)c/#N_s&r6Optimizer.init..update_state..;sBr)Nc/#r5r6r7s&rr9r:@sbr)) isinstancelisttuplerangelenextendr typer$r)paramsstateirr update_states&& rrF$Optimizer.init..update_state5s&4-00U s5z*A+FIux@EH+u:V,LL,s5z|8L!MNF|E**FD))"LLNDA~#+L!#<#/8#< +   r)c:<T;'gSPW4#r5) init_single)psrs&&rr9 Optimizer.init..Hsa994#3#3A#99r)TN)rr r)rr0rFsf&@rinitOptimizer.inits1, $ Z-9:t{{S r)c:<V^8dQhRS[PRS[/#r! parameterrDrrr$)r%r&s"rr'r(Ks$$RXX$d$r)c \4h)zTo be extended by the children classes to implement each optimizer's state initialization. Args: parameter (mx.array): A single parameter that will be optimized. state (dict): The optimizer's state. NotImplementedErrorrrQrDs&&&rrIOptimizer.init_singleK "##r)c&<V^8dQhRS[RS[/#r!r#r0r1)r%r&s"rr'r(UsNNN4Nr)c NVP'gVPV4VPP4F$wr4V!VP4VP V&K& VP^,VP R&\ VPWVP 4#)aApply the gradients to the parameters and return the updated parameters. Can be used to update a model via ``model.update(opt.apply_gradients(grads, model))`` which is precisely how :meth:`Optimizer.update` is implemented. Args: gradients (dict): A Python tree of gradients. parameters (dict): A Python tree of parameters. It can be a superset of the gradients. In that case the returned python tree will be of the same structure as the gradients. r)rrMrrrrDr apply_single)rr#r0param schedulers&&& rr,Optimizer.apply_gradientsUs   IIi !% 0 0 6 6 8 E )$)) 4DJJu !9"YY] 6))9$**MMr)cT<V^8dQhRS[PRS[PRS[/#r!gradientrQrDrR)r%r&s"rr'r(os*$$RXX$"(($4$r)c \4h)zTo be extended by derived classes to implement the optimizer's update. Args: gradient (mx.array): The ``parameter`` gradient. parameter (mx.array): The ``parameter`` to update. state (dict): The optimizer's state. rT)rrbrQrDs&&&&rr\Optimizer.apply_singleorXr)c VP#)z!The optimizer's state dictionary.)rrs&rrDOptimizer.stateys{{r)c <V^8dQhRS[/#r!rDr1)r%r&s"rr'r(s4r)c RVnWnR#)FN)rr)rrDs&&rrDrg~s! r)c(VPR,#)rrDrfs&rrOptimizer.stepszz&!!r)c(VPR,#) learning_raterlrfs&rroOptimizer.learning_rateszz/**r)cJ<V^8dQhRS[S[S[P3,/#r!rorfloatrr)r%r&s"rr'r(s!>>5+A>r)cL\P!V4VPR&R#roN)rrrD)rros&&rrorps&(hh}&= ?#r)c|<V^8dQhRS[RS[S[S[S[P .S[P 3,3,/#)r!namer])strrrtrrr)r%r&s"rr'r(s; % % % %eXrxxj"((6J-K&K L %r)c \V\4'd"W PV&V!VP4pM\P !V4pW0P V&R#)zL To be used by derived classes to optionally put a parameter on a schedule. N)r<rrrrrrD)rrxr]rQs&&& r_maybe_scheduleOptimizer._maybe_schedulesF eX & &%*  T "dii(II$ 4r))rrrr5)__name__ __module__ __qualname____firstlineno____doc__rr+rMrIr,r\propertyrDsetterrror{__static_attributes____classdictcell__)r&s@rrr sI = =*!*!X$$NN4$$ \\""++>> % %r)rcaa]tRt^toRt.3V3RlV3RllltV3RlRltV3RlRltV3RlR lt] R 4t ] PV3R lR l4t ] R 4t ] PV3RlRl4t Rt VtV;t#)MultiOptimizeraDWraps a list of optimizers with corresponding weight predicates/filters to make it easy to use different optimizers for different weights. The predicates take the full "path" of the weight and the weight itself and return True if it should be considered for this optimizer. The last optimizer in the list is a fallback optimizer and no predicate should be given for it. Args: optimizers (list[Optimizer]): A list of optimizers to delegate to filters (list[Callable[[str, array], bool]): A list of predicates that should be one less than the provided optimizers. c <V^8dQhRS[/#)r!filters)r=)r%r&s"rr'MultiOptimizer.__annotate__s @ @D @r)c<\SV`4/Vn\V4\V4^, 8wd,\ R\V4 R\V4^, R24hWnVR.,VnR#)zGiven z filters but z needed.cR#)Tr6)argskwargss*,rr9)MultiOptimizer.__init__..s$r)N)superrrr@ ValueError optimizersr)rrr __class__s&&&rrMultiOptimizer.__init__sn  w<3z?Q. .WmC OA4E3FhO %">!?? r)c <V^8dQhRS[/#)r!r#r1)r%r&s"rr'rs 2 24 2r)c\VP4^8XdV.#\\VP44Uu.uFp.NK pp\V4pVFLwrV\ VP 4F.wrxV!WV4'gKW7,P WV34KJ KN VU u.uFp \V 4NK up #uupiuup i)r)r@rr?r enumeraterappendr ) rr#r8partsflat_gradientsrgrEfnrJs && r_split_dictionary MultiOptimizer._split_dictionarys t 1 $; "3t#789889%i0"DA"4<<0a88HOOQF+1# ,115aq!511:2s C,Cc <V^8dQhRS[/#r/r1)r%r&s"rr'rstr)c\VPVPV44Fwr#VPV4K R#r5)ziprrrM)rr0orJs&& rrMMultiOptimizer.inits0)?)? )KLDA FF1IMr)c&<V^8dQhRS[RS[/#rZr1)r%r&s"rr'rs4r)c/p\VPVPV44FwrE\W4P WR44pK! V#r5)rrrr r,)rr#r0treerrs&&& rr,MultiOptimizer.apply_gradientssD)?)? )JKDAd$5$5a$DEDL r)cZRVPUu.uFqPNK up/#uupi)states)rrD)rrs& rrDMultiOptimizer.states%DOO> \\00,,,r)rcjaa]tRt^toRtR V3RlV3RllltV3RlRltV3RlRltRtVt V;t #) SGDabThe stochastic gradient descent optimizer. Updates a parameter :math:`w` with a gradient :math:`g` as follows .. math:: v_{t+1} &= \mu v_t + (1 - \tau) g_t \\ w_{t+1} &= w_t - \lambda v_{t+1} Args: learning_rate (float or callable): The learning rate :math:`\lambda`. momentum (float, optional): The momentum strength :math:`\mu`. Default: ``0`` weight_decay (float, optional): The weight decay (L2 penalty). Default: ``0`` dampening (float, optional): Dampening for momentum :math:`\tau`. Default: ``0`` nesterov (bool, optional): Enables Nesterov momentum. Default: ``False`` c <V^8dQhRS[S[S[S[P.S[P3,3,RS[RS[RS[RS[/#)r!romomentum weight_decay dampeningnesterov)rrtrrrbool)r%r&s"rr'SGD.__annotate__s[!!UHbhhZ-A$BBC!! !  !  !r)c<V'dV^8:gV^8wd \R4h\SV` 4VPRV4W nW0nW@nWPnR#)rz9Nesterov momentum requires a momentum and zero dampening.roN)rrrr{rrrr)rrorrrrrs&&&&&&rr SGD.__init__sV Q)q.K   _m< (" r)c:<V^8dQhRS[PRS[/#rPrR)r%r&s"rr'r ..RXX.d.r)c 8\P!V4VR&R#Initialize optimizer staterNr zeros_likerVs&&&rrISGD.init_single ]]9-c r)cT<V^8dQhRS[PRS[PRS[/#rarR)r%r&s"rr'rs/NNRXXN"((N4Nr)c ^VP^8wdWPV,, pVP^8:d3W PPVP4V,, #VPVP R4,pVP ^8d#V^VP , V,, pMWA, pVP'dWPV,,pMTpWCR&W PPVP4V,, #)zNPerforms the SGD parameter update and stores :math:`v` in the optimizer state.r)rrroastypedtypegetrr)rrbrQrDrr+s&&&& rr\SGD.apply_singles    ! ))I5 5H ==A 1188H8SS S MMEIIcN * >>A  !dnn$0 0A MA ===  11FFc --44X^^DvMMMr))rrrr)rrF r}r~rrrrrIr\rrrrs@@rrrs0"!!(..NNNr)rcjaa]tRtRtoRtR V3RlV3RllltV3RlRltV3RlRltR tVt V;t #) RMSpropi)a;The RMSprop optimizer [1]. [1]: Tieleman, T. and Hinton, G. 2012. Lecture 6.5-rmsprop, coursera: Neural networks for machine learning .. math:: v_{t+1} &= \alpha v_t + (1 - \alpha) g_t^2 \\ w_{t+1} &= w_t - \lambda \frac{g_t}{\sqrt{v_{t+1}} + \epsilon} Args: learning_rate (float or callable): The learning rate :math:`\lambda`. alpha (float, optional): The smoothing constant :math:`\alpha`. Default: ``0.99`` eps (float, optional): The term :math:`\epsilon` added to the denominator to improve numerical stability. Default: ``1e-8`` c<V^8dQhRS[S[S[S[P.S[P3,3,RS[RS[/#)r!roalphaepsrrtrrr)r%r&s"rr'RMSprop.__annotate__;sGUHbhhZ-A$BBC r)c<\SV`4VPRV4W nW0nVPR8d\ RVP R24hVPR8d\ RVP R24hR#)rorzRMSprop alpha should be >=0,  was provided insteadzRMSprop epsilon should be >0, N)rrr{rrr)rrorrrs&&&&rrRMSprop.__init__;s  _m<  :: / |;PQ  88c>0 :OP  r)c:<V^8dQhRS[PRS[/#rPrR)r%r&s"rr'rPrr)c 8\P!V4VR&R#rrrVs&&&rrIRMSprop.init_singlePrr)cT<V^8dQhRS[PRS[PRS[/#rarR)r%r&s"rr'rTs* > >RXX >"(( >4 >r)c ZVPPVP4pVPpVPpVR,pWW,^V, \ P !V4,,pWsR&W$V,\ P!V4V,, , #)zRPerforms the RMSprop parameter update and stores :math:`v` in the optimizer state.r)rorrrrrsquaresqrt)rrbrQrDlrrrrs&&&& rr\RMSprop.apply_singleTs{    & &x~~ 6 hh #J IUbii&99 9c =BGGAJ,<===r))rr)Gz?:0yE>rrs@@rrr)s-"*.. > > >r)rcjaa]tRtRtoRtR V3RlV3RllltV3RlRltV3RlRltR tVt V;t #) Adagradiaa,The Adagrad optimizer [1]. Our Adagrad implementation follows the original paper. In detail, [1]: Duchi, J., Hazan, E. and Singer, Y., 2011. Adaptive subgradient methods for online learning and stochastic optimization. JMLR 2011. .. math:: v_{t+1} &= v_t + g_t^2 \\ w_{t+1} &= w_t - \lambda \frac{g_t}{\sqrt{v_{t+1}} + \epsilon} Args: learning_rate (float or callable): The learning rate :math:`\lambda`. eps (float, optional): The term :math:`\epsilon` added to the denominator to improve numerical stability. Default: ``1e-8`` c|<V^8dQhRS[S[S[S[P.S[P3,3,RS[/#)r!rorr)r%r&s"rr'Adagrad.__annotate__ts=  UHbhhZ-A$BBC  r)c<\SV`4VPRV4W nVPR8d\ RVP R24hR#)rorzAdagrad epsilon should be >0, rN)rrr{rr)rrorrs&&&rrAdagrad.__init__tsS  _m< 88c>0 :OP  r)c:<V^8dQhRS[PRS[/#rPrR)r%r&s"rr'rrr)c 8\P!V4VR&R#rrrVs&&&rrIAdagrad.init_singlerr)cT<V^8dQhRS[PRS[PRS[/#rarR)r%r&s"rr'rs* > >RXX >"(( >4 >r)c VPPVP4pVPpVR,\P !V4,pWcR&W$V,\P !V4V,, , #)zRPerforms the Adagrad parameter update and stores :math:`v` in the optimizer state.r)rorrrrrr)rrbrQrDrrrs&&&& rr\Adagrad.apply_singlesd   & &x~~ 6hh #J8, ,c =BGGAJ,<===r))r)rrrs@@rrras-$  .. > > >r)rcjaa]tRtRtoRtR V3RlV3RllltV3RlRltV3RlRltR tVt V;t #) AdaDeltaia0The AdaDelta optimizer with a learning rate [1]. Our AdaDelta implementation follows the original paper. In detail, [1]: Zeiler, M.D., 2012. ADADELTA: an adaptive learning rate method. arXiv preprint arXiv:1212.5701. .. math:: v_{t+1} &= \rho v_t + (1 - \rho) g_t^2 \\ \Delta w_{t+1} &= \frac{\sqrt{u_t + \epsilon}}{\sqrt{v_{t+1} + \epsilon}} g_t \\ u_{t+1} &= \rho u_t + (1 - \rho) \Delta w_{t+1}^2 \\ w_{t+1} &= w_t - \lambda \Delta w_{t+1} Args: learning_rate (float or callable): The learning rate :math:`\lambda`. rho (float, optional): The coefficient :math:`\rho` used for computing a running average of squared gradients. Default: ``0.9`` eps (float, optional): The term :math:`\epsilon` added to the denominator to improve numerical stability. Default: `1e-8` c<V^8dQhRS[S[S[S[P.S[P3,3,RS[RS[/#)r!rorhorr)r%r&s"rr'AdaDelta.__annotate__sGUHbhhZ-A$BBC r)c<\SV`4VPRV4W nW0nVPR8d\ RVP R24hVPR8d\ RVP R24hR#)rorzAdaDelta rho should be >=0, rzAdaDelta epsilon should be >0, N)rrr{rrr)rrorrrs&&&&rrAdaDelta.__init__s  _m< 88c>.txxj8MN  88c>1$((;PQ  r)c:<V^8dQhRS[PRS[/#rPrR)r%r&s"rr'r..RXX.d.r)c j\P!V4VR&\P!V4VR&R#)rruNrrVs&&&rrIAdaDelta.init_single&]]9-c ]]9-c r)cT<V^8dQhRS[PRS[PRS[/#rarR)r%r&s"rr'rs*""RXX""(("4"r)c VPPVP4pVPpVPpVR,pVR,pWW,^V, \ P !V4,,p\ P!W,4\ P!Wv,4, V,p WX,^V, \ P !V 4,,pWsR&WR&W$V ,, #)zaPerforms the AdaDelta parameter update and stores :math:`v` and :math:`u` in the optimizer state.rr)rorrrrrrr) rrbrQrDrrrrrds &&&& rr\AdaDelta.apply_singles   & &x~~ 6hhhh #J #J Gq3w"))H"55 5 GGAG rwwqw/ /( : Gq3w"))A,. .c c 6!!r))rr)?ư>rrs@@rrrs-*(.. """r)rctaa]tRtRtoRtRR.RR3V3RlV3RllltV3R lR ltV3R lR ltR tVt V;t #)Adamia?The Adam optimizer [1]. In detail, [1]: Kingma, D.P. and Ba, J., 2015. Adam: A method for stochastic optimization. ICLR 2015. .. math:: m_{t+1} &= \beta_1 m_t + (1 - \beta_1) g_t \\ v_{t+1} &= \beta_2 v_t + (1 - \beta_2) g_t^2 \\ w_{t+1} &= w_t - \lambda \frac{m_{t+1}}{\sqrt{v_{t+1}} + \epsilon} Args: learning_rate (float or callable): The learning rate :math:`\lambda`. betas (Tuple[float, float], optional): The coefficients :math:`(\beta_1, \beta_2)` used for computing running averages of the gradient and its square. Default: ``(0.9, 0.999)`` eps (float, optional): The term :math:`\epsilon` added to the denominator to improve numerical stability. Default: ``1e-8`` bias_correction (bool, optional): If set to ``True``, bias correction is applied. Default: ``False`` r+?rFc<V^8dQhRS[S[S[S[P.S[P3,3,RS[S[,RS[RS[/#)r!robetasrbias_correctionrrtrrrrr)r%r&s"rr'Adam.__annotate__sU / /UHbhhZ-A$BBC /E{ / /  /r)cl<\SV`4VPRV4W nW0nW@nR#rv)rrr{r rr )rror rr rs&&&&&rr Adam.__init__s0  _m< .r)c:<V^8dQhRS[PRS[/#rPrR)r%r&s"rr'r rr)c j\P!V4VR&\P!V4VR&R#rmrNrrVs&&&rrIAdam.init_singlerr)cT<V^8dQhRS[PRS[PRS[/#rarR)r%r&s"rr'r s*;;RXX;"((;4;r)c >VPPVP4pVPwrVVPpVP pVP p VR,p VR,p WZ,^V, V,,p Wk,^V, \P!V4,,p WR&WR&V'dV^WY,, , PVP4p \P!^Wi,, 4PVP4p W,p\P!V 4V ,V,pW.V, , #W$V ,\P!V 4V,, , #)z]Performs the Adam parameter update and stores :math:`v` and :math:`m` in the optimizer state.rr) rorrr rr rrrrsqrtr)rrbrQrDrb1b2rr rrrc1c2 numerator denominators&&&& rr\Adam.apply_singles(   & &x~~ 6hh..yy #J #J Fa"f( ( Fa"f ( 33 3c c BH %--hnn=B!bh,'..x~~>BI''!*r/C/K;66 6Avc)9:: :r))r r rrrs@@rrrs<2"5\ % / /.. ;;;r)rcjaa]tRtRtoRtRR.RRR3V3RlV3R llltV3R lV3R lltR tVtV;t #) AdamWiaThe AdamW optimizer [1]. We update the weights with a weight_decay (:math:`\lambda`) value: [1]: Loshchilov, I. and Hutter, F., 2019. Decoupled weight decay regularization. ICLR 2019. .. math:: m_{t+1} &= \beta_1 m_t + (1 - \beta_1) g_t \\ v_{t+1} &= \beta_2 v_t + (1 - \beta_2) g_t^2 \\ w_{t+1} &= w_t - \alpha (\frac{m_{t+1}}{\sqrt{v_{t+1}} + \epsilon} + \lambda w_t) Args: learning_rate (float or callable): The learning rate :math:`\alpha`. betas (Tuple[float, float], optional): The coefficients :math:`(\beta_1, \beta_2)` used for computing running averages of the gradient and its square. Default: ``(0.9, 0.999)`` eps (float, optional): The term :math:`\epsilon` added to the denominator to improve numerical stability. Default: ``1e-8`` weight_decay (float, optional): The weight decay :math:`\lambda`. Default: ``0.01``. bias_correction (bool, optional): If set to ``True``, bias correction is applied. Default: ``False`` rrr{Gz?Fc <V^8dQhRS[S[S[S[P.S[P3,3,RS[S[,RS[RS[RS[/#)r!ror rrr r )r%r&s"rr'AdamW.__annotate__4s_))UHbhhZ-A$BBC)E{) )  )  )r)c:<\SV`VVVVR7W@nR#))ror rr N)rrr)rror rrr rs&&&&&&rrAdamW.__init__4s+ '+  )r)cT<V^8dQhRS[PRS[PRS[/#rarR)r%r&s"rr'r"Ds*  RXX "(( 4 r)c <VPPVP4p\SV`W^W@P ,, ,V4#)zRPerforms the AdamW parameter update by modifying the parameters passed into Adam. )rorrrr\r)rrbrQrDrrs&&&& rr\AdamW.apply_singleDsI    & &x~~ 6w# 1r,=,='=#=>  r))r) r}r~rrrrr\rrrrs@@rrrs58"5\" % ))    r)rcraa]tRtRtoRtRR.R3V3RlV3RllltV3RlR ltV3R lR ltR tVt V;t #) AdamaxiOalThe Adamax optimizer, a variant of Adam based on the infinity norm [1]. Our Adam implementation follows the original paper and omits the bias correction in the first and second moment estimates. In detail, [1]: Kingma, D.P. and Ba, J., 2015. Adam: A method for stochastic optimization. ICLR 2015. .. math:: m_{t+1} &= \beta_1 m_t + (1 - \beta_1) g_t \\ v_{t+1} &= \max(\beta_2 v_t, |g_t|) \\ w_{t+1} &= w_t - \lambda \frac{m_{t+1}}{v_{t+1} + \epsilon} Args: learning_rate (float or callable): The learning rate :math:`\lambda`. betas (Tuple[float, float], optional): The coefficients :math:`(\beta_1, \beta_2)` used for computing running averages of the gradient and its square. Default: ``(0.9, 0.999)`` eps (float, optional): The term :math:`\epsilon` added to the denominator to improve numerical stability. Default: ``1e-8`` rrrc<V^8dQhRS[S[S[S[P.S[P3,3,RS[S[,RS[/#)r!ror rrrtrrrr)r%r&s"rr'Adamax.__annotate__gsK  UHbhhZ-A$BBC E{  r)ch<\SV`WV4RV8:g\RVP R24hR#)rzEpsilon value should be >=0, rN)rrrr)rror rrs&&&&rrAdamax.__init__gs> s3cz/z9NO r)c:<V^8dQhRS[PRS[/#rPrR)r%r&s"rr'r,srr)c j\P!V4VR&\P!V4VR&R#rrrVs&&&rrIAdamax.init_singlesrr)cT<V^8dQhRS[PRS[PRS[/#rarR)r%r&s"rr'r,xs*..RXX."((.4.r)c VPPVP4pVPwrVVPpVR,pVR,p WX,^V, V,,p\ P !Wi,\ P!V44p WR&WR&W$V,W,, , #)z_Performs the Adamax parameter update and stores :math:`v` and :math:`m` in the optimizer state.rr)rorrr rrmaximumabs) rrbrQrDrrrrrrs &&&& rr\Adamax.apply_singlexs   & &x~~ 6hh #J #J Fa"f( ( JJrvrvvh/ 0c c 6QW---r)r6rrs@@rr)r)Os94"5\  .. ...r)r)craa]tRtRtoRtRR.R3V3RlV3RllltV3RlR ltV3R lR ltR tVt V;t #) LioniaThe Lion optimizer [1]. Since updates are computed through the sign operation, they tend to have larger norm than for other optimizers such as SGD and Adam. We recommend a learning rate that is 3-10x smaller than AdamW and a weight decay 3-10x larger than AdamW to maintain the strength (lr * wd). Our Lion implementation follows the original paper. In detail, [1]: Chen, X. Symbolic Discovery of Optimization Algorithms. arXiv preprint arXiv:2302.06675. .. math:: c_{t + 1} &= \beta_1 m_t + (1 - \beta_1) g_t \\ m_{t + 1} &= \beta_2 m_t + (1 - \beta_2) g_t \\ w_{t + 1} &= w_t - \eta (\text{sign}(c_t) + \lambda w_t) Args: learning_rate (float or callable): The learning rate :math:`\eta`. betas (Tuple[float, float], optional): The coefficients :math:`(\beta_1, \beta_2)` used for computing the gradient momentum and update direction. Default: ``(0.9, 0.99)`` weight_decay (float, optional): The weight decay :math:`\lambda`. Default: ``0.0`` rrrc<V^8dQhRS[S[S[S[P.S[P3,3,RS[S[,RS[/#)r!ror rr+)r%r&s"rr'Lion.__annotate__sK ) )UHbhhZ-A$BBC )E{ ) )r)c`<\SV`4VPRV4W nW0nR#rv)rrr{r r)rror rrs&&&&rr Lion.__init__s+  _m< (r)c:<V^8dQhRS[PRS[/#rPrR)r%r&s"rr'r:rr)c 8\P!V4VR&R#)rrNrrVs&&&rrILion.init_singlerr)cT<V^8dQhRS[PRS[PRS[/#rarR)r%r&s"rr'r:s* + +RXX +"(( +4 +r)c VPPVP4pVPwrVVPpVR,pWX,^V, V,,p Wh,^V, V,,VR&V^8d^WG,, V,pW$\ P !V 4,, #)zOPerforms the Lion parameter update and stores :math:`m` in the optimizer state.r)rorrr rrsign) rrbrQrDrrrrrcs &&&& rr\Lion.apply_singles   & &x~~ 6(( #J Fa"f( (Vq2v11c ! R..);I ?**r))r rrrs@@rr8r8s9:"4[! ) ).. + + +r)r8c|aa]tRtRtoRtR V3RlV3RllltV3RlRltRtRtR t V3R lR lt R t Vt V;t #) AdafactoriaThe Adafactor optimizer. Our Adafactor implementation follows the original paper: `Adafactor: Adaptive Learning Rates with Sublinear Memory Cost `_ Args: learning_rate (float or callable, optional): The learning rate. Default: ``None``. eps (tuple(float, float), optional): The first term :math:`\epsilon_1` added to the square of the gradients to improve numerical stability and the second term :math:`\epsilon_2` is used for parameter scaling if ``parameter_scale`` is set to ``True``. Default: ``(1e-30, 1e-3)``. clip_threshold (float, optional): Clips the unscaled update at ``clip_threshold``. Default: ``1.0``. decay_rate (float, optional): Coefficient for the running average of the squared gradient. Default: ``-0.8``. beta_1 (float, optional): If set to a value bigger than zero then first moment will be used. Default: ``None``. weight_decay (float, optional): The weight decay :math:`\lambda`. Default: ``0.0``. scale_parameter (bool, optional): If set to ``True`` the learning rate will be scaled by :math:`\max(\epsilon_1, \text{RMS}(w_{t-1}))`. Default: ``True``. relative_step (bool, optional): If set to ``True`` the ``learning_rate`` will be ignored and relative step size will be computed. Default: ``True``. warmup_init (bool, optional): If set to ``True`` then the relative step size will be calculated by the current step. Default: ``False``. c<V^8dQhRS[S[S[S[P.S[P3,R3,RS[S[S[3,RS[RS[RS[S[,RS[RS[R S[R S[/ #) r!roNrclip_threshold decay_ratebeta_1rscale_parameter relative_step warmup_init)rrtrrrrrr)r%r&s"rr'Adafactor.__annotate__s''UHbhhZ-A$BDHI'5%< ' '  '  '''''r)c <\S V`4VeVPRV4W nW0nW@nWPnW`nWpnWn Wn R#)Nro) rrr{rrHrIrJrrKrLrM) rrorrHrIrJrrKrLrMrs &&&&&&&&&&rrAdafactor.__init__sS   $  - @,$ (.*&r)c:<V^8dQhRS[PRS[/#rPrR)r%r&s"rr'rNs 8 8RXX 8d 8r)c jVP^8d`VPpVPp\P!VRRVR7VR&\P!VRRVRR,VR7VR&M\P !V4VR&VP e\P !V4VR&R#R#) rN)rexp_avg_sq_rowexp_avg_sq_col exp_avg_sqexp_avg)ndimshaperrzerosrrJ)rrQrDrZrs&&& rrIAdafactor.init_singles >>Q OOEOOE&(hhuSbz&GE" #&(hhuSbzE"#J/Fe&TE" #"$-- ":E,  ;; "!}}Y7E)  #r)c~\P!\P!\P!V444#r5)rrmeanr)rinputss&&r _compute_rmsAdafactor._compute_rms s#wwrwwryy0122r)cVP'dJVP'd RV,MRp\P!V\P!V44pM VP pVP VP4pRpVP'd)\P!VP^,V4pWT,#)rr ?) rLrMrminimumrrorrrKr4r)rr parameter_rmsmin_steprelative_step_sizeparameter_scales&&& r_compute_learning_rate Adafactor._compute_learning_rates    &*&6&6&6td{DH!#Hbhhtn!E !%!3!3 /66}7J7JK     jj!mDO33r)c \P!V\P!VRRR7, 4p\P!V4p\P!\P!VRR7\P!V^R74#)rT)axiskeepdimsrlrW)rrr^matmul expand_dims)rrSrTr_factorc_factors&&& r_approximate_exp_moving_avg%Adafactor._approximate_exp_moving_avgsa88 RWW^"tL L 88N+yy NN8" -r~~hQ/O  r)cT<V^8dQhRS[PRS[PRS[/#rarR)r%r&s"rr'rN$s*,","RXX,""((,"4,"r)c VP^8pVPpVPRJpVPV4pVP WW4pRWPP ,P VP4, p \P!V4VP^,,p V'dVR,p VR,p W,^V , \P!V RR7,,p W,^V , \P!V R R7,,p WR&WR&VPW4p W,p MGVR,p W,^V , V ,,p WR&\P!V 4V,p V \P!RVPV 4VP, 4, p W,p V'dBVR,pVPV,^VP, V ,,pWR&Tp VP ^8wd"W"VP )V,,, pW*, #) z2Performs the Adafactor parameter and state update.NrcrSrTrnrUrVrWrX)rYrrJr`rirIrrrrrr^rsrr4rHr)rrbrQrDfactoredruse_first_momentrerobeta_2r+rSrTrUrVs&&&& rr\Adafactor.apply_single$s==A%yy;;d2)))4 33DH oo-55m6I6IJJ8$txx{2 "#34N"#34N$5VrwwvB77N%5VrwwvB77N'5" #&4" #55nUF&F|,J -1v:2GHJ",, XXj)H4F"** ""6*T-@-@@  ' I&G{{W,!dkk/V1KLG&) F    ! t'8'8&8=&HI II!!r))rJrHrIrrLrKrMr) N)gKH9gMbP?rcgNrTTF)r}r~rrrrrIr`rirsr\rrrrs@@rrFrFs=B''0 8 83 4 ,",","r)rFc|aa]tRtRtoRtR V3RlV3RllltV3RlRltV3RlRltV3R lR ltR t Vt V;t #) MuoniSaThe Muon optimizer. Our Muon (MomentUm Orthogonalized by Newton-schulz) optimizer follows the original implementation: `Muon: An optimizer for hidden layers in neural networks `_ Note: - Muon may be sub-optimal for the embedding layer, the final fully connected layer, or any 0D/1D parameters. Those should be optimized by a different method (e.g., :class:`AdamW`). - For 4D convolutional filters, it works by flattening their last dimensions. Args: learning_rate (float or callable): The learning rate. momentum (float, optional): The momentum strength. Default: ``0.95`` weight_decay (float, optional): The weight decay (L2 penalty). Default: ``0.01`` nesterov (bool, optional): Enables Nesterov momentum. Recommended for better performance. Default: ``True`` ns_steps (int, optional): Number of Newton-Schulz iteration steps for orthogonalization. Default: ``5`` c <V^8dQhRS[S[S[S[P.S[P3,3,RS[RS[RS[RS[/#)r!rorrrns_steps)rrtrrrrint)r%r&s"rr'Muon.__annotate__ls[!!UHbhhZ-A$BBC!! !  !  !r)cx<\SV`4VPRV4W nW0nW@nWPnR#rv)rrr{rrrr~)rrorrrr~rs&&&&&&rr Muon.__init__ls5  _m< (  r)c:<V^8dQhRS[PRS[/#rPrR)r%r&s"rr'r|rr)c 8\P!V4VR&R#rrrVs&&&rrIMuon.init_single|rr)c <V^8dQhRS[/#)r!steps)r)r%r&s"rr'rsSr)c  VP^8XgQRVP R24hRR RrTpVPR ,VPR ,8pV'd VPpV\PP VRR7R,, p\ V4FUpWP,p\P!WH,WRVR 7p \P!W1,WRRR 7pKW V'd VPpV#) r!z;Expected a 2D array for Newton-Schulz iteration, got shape z instead.guV @gn@@T)rmgHz>rc)betargrXrW)rYrZTrlinalgnormr?addmm) rXrabrCtranspose_neededr8ABs &&& r_zeropower_via_newtonschulz5!Muon._zeropower_via_newtonschulz5s FFaK \ H QZ [ \ 7Fa772;4 A D1D8 9uACCA3a8A3c:A Ar)cT<V^8dQhRS[PRS[PRS[/#rarR)r%r&s"rr'rs*''RXX'"(('4'r)c |VP^8wdWPV,,pVPVR,,pV^VP, V,,pWCR&VP'd3V^VP, ,W@P,,pMTpVPP VP 4pVP ^8dVPpVP ^8pV'd*\P!WUP^,R34pVPWPPR7pV'd\P!WW4pV\^VPR,VPR,, 4R,,pW&V,, #)z"Performs the Muon parameter updater)rg?rWrX) rrrrorrrYrZrreshaperr~max) rrbrQrDrr+roriginal_shapereshape_neededs &&&& rr\Muon.apply_singles?    !"3"3i"??H MME#J & T]]"h. .c ===T]]!23a--6GGFF    & &x~~ 6 ;;! #\\N#[[1_NF\\!_b,AB66v]]6SFF; #ab)FLL,<<=D DB;&&r))rrr~r)gffffff?r T) r}r~rrrrrIrr\rrrrs@@rr|r|Ss70!! ..*'''r)r|ca\RVR4p\P!V4p\P!WR,, R4o\ V3RlV4pWC3#)aClips the global norm of the gradients. This function ensures that the global norm of the gradients does not exceed ``max_norm``. It scales down the gradients proportionally if their norm is greater than ``max_norm``. Example: >>> grads = {"w1": mx.array([2, 3]), "w2": mx.array([1])} >>> clipped_grads, total_norm = clip_grad_norm(grads, max_norm=2.0) >>> print(clipped_grads) {"w1": mx.array([...]), "w2": mx.array([...])} Args: grads (dict): A dictionary containing the gradient arrays. max_norm (float): The maximum allowed global norm of the gradients. Returns: (dict, float): The possibly rescaled gradients and the original gradient norm. cJWP4P4,#r5)rsum)accrs&&rr9 clip_grad_norm..scHHJNN4D.Dr)rrrcc<VS,#r5r6)r normalizers&rr9rs q:~r))r rrrdr )gradsmax_norm norm_squared total_norm clipped_gradsrs&& @rclip_grad_normrsP*DeSQL&JHT(9:C@J5u=M  $$r))typingrrrrrmlx.corecorermlx.nnr mlx.utilsr r r r r rrrrrrrrr)r8rFr|rr6r)rrs:9UUP%P%fF,YF,R@N)@NF5>i5>p/>i/>d@"y@"FA;9A;H2 D2 j8.T8.v7+97+tL" L"^a'9a'H%r)