+ &jN*^RIt^RIHt^RIHtHtHt^RIHt ^RI H t ^RI H t ^RIHt^RIHt]R4tR tRR R lltR tR tRtR^RR/RRlltR^RR/RRllt!RR] 4t!RR] 4t!RR] 4t!RR] 4tR#)N) lru_cache)CallableOptionalUnion)Module)Linear)QuantizedLinear)tree_map_with_pathcaSP4^8XdR#\PR4pVPV3Rl4pV#)cV#Nxs&q/Users/jameslopez/projects/CWCArchive/cwc-podcast/.venv/lib/python3.14/site-packages/mlx/nn/layers/distributed.pysum_gradients..scV#rrrs&rfsum_gradients..fsrcF<\PPVSR7#)group)mx distributedall_sum)rdx_rs&&&rrrs~~%%b%66r)sizercustom_functionvjp)rrsf r sum_gradientsr$sN zz|qUU7 7 Hrc8\V\4'g\V^,\4'd\P!WVR7#VPV,pVUu.uFp\WC,4NK pp\P!WVR7#uupi)z:Equivalent to mx.split but allows for fractional segments.axis) isinstanceintrsplitshape)weightsegmentsr'Nsindicess&&& r_splitr1sn(C  Jx{C$@$@xxt44 TA#+,8as15z8G, 88F$ //-s#BczV^8dQhR\R\R\\PP ,/#) parameterssharding_predicater)dictrrrrGroup)formats"r __annotate__r9(s5)5)5)5 )5 BNN(( ))5rcaaaT;'g\PP4pVP4oVP 4oVVV3Rlp\ W04#)zReturns a new parameter tree with the weights sharded according to the sharding_predicate. The sharding predicate should return the sharding axis and optionally also the segments that comprise the weight. c <\V\P4'gV#S!W4pVfV#Rp^p\V\4'dTpM&\V\4'dVwr4M \ R4h\P !\P!\WV4Uu.uFp\VSV4S,NK upVR74#uupi)Nz;The sharding function should return int or tuple[int, list]r&) r(rarrayr)tuple ValueError contiguous concatenater1) pathr,r/r'r-partr.rr5s && r _shard_fn_shard.._shard_fn7s&"((++M t , 9M a  D 5 ! !ND(M }} NN6._shard_fnXs6 == x< 6;;?A&00rrr-rDsf r_all_to_shardedrSTs1 rcaV3RlpV#)ztSimple predicate to shard fully connected layers such that a sharded representation becomes a common representation.c><VPR4'dR#RS3#)rKNrL)rMrPs&&rrD"_sharded_to_all.._shard_fnds == 8|rrrRsf r_sharded_to_allrW`s rc4VR9d\RV: R24hR#)all-to-shardedzSharding type sharding=zB not supported, choose one of 'all-to-sharded' or 'sharded-to-all'N)rYsharded-to-all)r>)shardings&r_check_shardingr\ls0;;* ,EE  ?rc V^8dQhR\R\R\\\3,R\ \ PP,/#r^) rr`rr)rarrrr7)r8s"rr9r9sH     CI  BNN(( ) rc\V4R\PR\PR\ PR\ P/pWA\V\43,!WVR7#)aCreate a new linear layer that has its parameters sharded and also performs distributed communication either in the forward or backward pass. .. note:: Contrary to ``shard_inplace``, the original layer is not changed but a new layer is returned. Args: module (mlx.nn.Module): The linear layer to be sharded. sharding (str): One of "all-to-sharded" and "sharded-to-all" that defines the type of sharding to perform. segments (int or list): The segments to use. Default: ``1``. group (mlx.core.distributed.Group): The distributed group to shard across. If not set, the global group will be used. Default: ``None``. )r-r)rYT)rYF)rZT)rZF) r\AllToShardedLinear from_linearQuantizedAllToShardedLinearfrom_quantized_linearShardedToAllLinearQuantizedShardedToAllLinearr(r)r_r[r-rfnss&&$$ r shard_linearrnsf.H "4"@"@!#>#T#T "4"@"@!#>#T#T  C FF33 4 rcaa]tRt^toRtRV3RlV3RllltV3RlRltV3RlRlt]R ^R R/V3R lR ll4t R t Vt V;t #)rga]Each member of the group applies part of the affine transformation such that the result is sharded across the group. The gradients are automatically aggregated from each member of the group. Args: input_dims (int): The dimensionality of the input features output_dims (int): The dimensionality of the output features bias (bool, optional): If set to ``False`` the the layer will not use a bias. Default is ``True``. group (mx.distributed.Group, optional): The sharding will happen across this group. If not set then the global group is used. Default is ``None``. Nc j<V^8dQhRS[RS[RS[RS[S[PP ,/#r3 input_dims output_dimsrKrr)boolrrrr7)r8 __classdict__s"rr9AllToShardedLinear.__annotate__B  ,,- rc<\SV`4\P!RV, 4pT;'g\P P 4VnVPP4pW&,^8wd\RV RV R24h\PPV)VW&,V3R7Vn V'd2\PPV)VW&,3R7Vn R#R#)? Cannot shard the output of size  across devices.lowhighr+Nsuper__init__mathsqrtrrrFrr!r>randomuniformr,rKselfrrrsrKrscaler. __class__s&&&&& rrAllToShardedLinear.__init__s  # *+33bnn113 JJOO  O !2;-xs)T ii''#Z0(   ))F"')*DI rc <V^8dQhRS[/#r3returnr`)r8rvs"rr9rwVVSVrcVPPwrVPP4pW,pRV RV RRV9 2# input_dims=, output_dims=, bias=rK)r,r+rr!)rout_dimsin_dimsr.s& r _extra_reprAllToShardedLinear._extra_reprsI KK-- JJOO  WI^H:WVt^DTUUrcN<V^8dQhRS[PRS[P/#r3rrrr<)r8rvs"rr9rws#  "(( rxx rc\VP4!V4pRV9d2\P!VR,WR,P4pV#WR,P,pV#)rKr,)r$rraddmmTrrs&&r__call__AllToShardedLinear.__call__s\ $** %a ( T>fqx.*:*:;AN$$$Arr-rcz<V^8dQhRS[RS[S[S[3,RS[S[P P,/#r3 linear_layerr-rrrr)rarrrr7)r8rvs"rr9rwA   T "  ,,- rcT;'g\PP4pVPPwrEV!WT\ VR4V4pVP \VP4\V4V44V#rK) rrrFr,r+hasattrrcrHr4rSclsrr-rrsrrsls&&$$ rrhAllToShardedLinear.from_linearo..,,."."5"5";"; ',*G O &002OH4MuUV rrKrr,TN __name__ __module__ __qualname____firstlineno____doc__rrr classmethodrh__static_attributes____classdictcell__ __classcell__rrvs@@rrgrgs\ >VV   &'  15    rrgcaa]tRtRtoRtRV3RlV3RllltV3RlRltV3RlR lt]R ^R R/V3R lR ll4t Rt Vt V;t #)rkiaEach member of the group applies part of the affine transformation and then aggregates the results. All nodes will have the same exact result after this layer. :class:`ShardedToAllLinear` provides a classmethod :meth:`from_linear` to convert linear layers to sharded :obj:`ShardedToAllLinear` layers. Args: input_dims (int): The dimensionality of the input features output_dims (int): The dimensionality of the output features bias (bool, optional): If set to ``False`` the the layer will not use a bias. Default is ``True``. group (mx.distributed.Group, optional): The sharding will happen across this group. If not set then the global group is used. Default is ``None``. Nc j<V^8dQhRS[RS[RS[RS[S[PP ,/#rqrt)r8rvs"rr9ShardedToAllLinear.__annotate__%rxrc<\SV`4\P!RV, 4pT;'g\P P 4VnVPP4pW,^8wd\RV RV R24h\PPV)VW!V,3R7Vn V'd,\PPV)VV3R7Vn R#R#)rzThe input of size  cannot be sharded across r}r~Nrrs&&&&& rrShardedToAllLinear.__init__%s  # *+33bnn113 JJOO  Nq $ZL0J1#YW ii''a0(   ))F"n*DI rc <V^8dQhRS[/#rr)r8rvs"rr9rDrrcVPP4pVPPwr#W1,pRV RV RRV9 2#r)rr!r,r+)rr.rrs& rrShardedToAllLinear._extra_reprDsI JJOO  KK-- WI^H:WVt^DTUUrcN<V^8dQhRS[PRS[P/#rr)r8rvs"rr9rJs#"((rxxrcWR,P,p\PPWPR7pRV9dWR,,pV#)r,rrK)rrrrrrs&&rrShardedToAllLinear.__call__JsG X   NN " "1JJ " 7 T>L Arr-rcz<V^8dQhRS[RS[S[S[3,RS[S[P P,/#rr)r8rvs"rr9rUrrcT;'g\PP4pVPPwrEV!WT\ VR4V4pVP \VP4\V4V44V#r) rrrFr,r+rrcrHr4rWrs&&$$ rrhShardedToAllLinear.from_linearTrrrrrrs@@rrkrks\$>VV  &'  15    rrkcaa]tRtRtoRtRV3RlV3RllltV3RltV3RlRltV3R lR lt] R ^R R/V3R lRll4t Rt Vt V;t #)riieaqEach member of the group applies part of the affine transformation with a quantized matrix such that the result is sharded across the group. It is the quantized equivalent of :class:`mlx.nn.AllToShardedLinear`. Similar to :class:`mlx.nn.QuantizedLinear` its parameters are frozen and will not be included in any gradient computation. Args: input_dims (int): The dimensionality of the input features. output_dims (int): The dimensionality of the output features. bias (bool, optional): If set to ``False`` then the layer will not use a bias. Default: ``True``. group_size (int, optional): The group size to use for the quantized weight. See :func:`~mlx.core.quantize`. Default: ``64``. bits (int, optional): The bit width to use for the quantized weight. See :func:`~mlx.core.quantize`. Default: ``4``. mode (str, optional): The quantization method to use (see :func:`~mlx.core.quantize`). Default: ``"affine"``. group (mx.distributed.Group, optional): The sharding will happen across this group. If not set then the global group is used. Default is ``None``. Nc|<V^8dQhRS[RS[RS[RS[RS[RS[RS[S[P P ,/#r3rrrsrK group_sizebitsmoderr)rur`rrrr7)r8rvs"rr9(QuantizedAllToShardedLinear.__annotate__}`**** *  *  **,,-*rc<\S V`4W@nWPnW`n\ P !RV, 4pT;'g\PP4Vn VPP4p W),^8wd\RV RV R24h\PPV)VW),V3R7p \P!WWVR7vVnVnp V 'd V ^,MRVnV'd#\P&!W),34VnVP+4R#)rzr{r|r}r~rNrrrrrrrrrrFrr!r>rrquantizer,scalesbiaseszerosrKfreeze rrrrsrKrrrrrr.r,rrs &&&&&&&& rr$QuantizedAllToShardedLinear.__init__}s %   # *+33bnn113 JJOO  O !2;-xs)T ""#Z0#  -/KK - ) T[6$*fQit  +"2!45DI rc L<\SV`!V/VBVPRR7R#zdWrap unfreeze so that we unfreeze any layers we might contain but our parameters will remain frozen.F)recurseNrunfreezerrargskwargsrs&*,rr$QuantizedAllToShardedLinear.unfreeze% $)&) E "rc <V^8dQhRS[/#rr)r8rvs"rr9rs  S rc VPPwrV^ ,VP,pWPP 4,pRV RV RRV9 RVP RVP RVP 2 # rrrrKz , group_size=z, bits=z, mode=r,r+rrr!rrrrrs& rr'QuantizedAllToShardedLinear._extra_reprs KK--R88H%  T>L Arr-rcz<V^8dQhRS[RS[S[S[3,RS[S[P P,/#r3quantized_linear_layerr-rr)r8rvs"rr9rA &T "  ,,- rc T;'g\PP4pVPPwrEV^ ,VP ,pV!VV\ VR4VPVP \VRR4VR7pVP\VP4\V4V44V#rrKraffine)rrrr) rrrFr,r+rrrgetattrrcrHr4rSrrr-rrsrrrs&&$$ rrj1QuantizedAllToShardedLinear.from_quantized_linear..,,."8"?"?"E"E  2o*@*E*EE    *F 3-88',,/B  &113)   rrKrrrrrrr,T@rNrrrrrrrrrrrjrrrrs@@rriries`.**X#   $&'  15 rricaa]tRtRtoRtRV3RlV3RllltV3RltV3RlRltV3R lR lt] R ^R R/V3R lRll4t Rt Vt V;t #)rliaEach member of the group applies part of the affine transformation using the quantized matrix and then aggregates the results. All nodes will have the same exact result after this layer. It is the quantized equivalent of :class:`mlx.nn.ShardedToAllLinear`. Similar to :class:`mlx.nn.QuantizedLinear` its parameters are frozen and will not be included in any gradient computation. Args: input_dims (int): The dimensionality of the input features. output_dims (int): The dimensionality of the output features. bias (bool, optional): If set to ``False`` then the layer will not use a bias. Default: ``True``. group_size (int, optional): The group size to use for the quantized weight. See :func:`~mlx.core.quantize`. Default: ``64``. bits (int, optional): The bit width to use for the quantized weight. See :func:`~mlx.core.quantize`. Default: ``4``. mode (str, optional): The quantization method to use (see :func:`~mlx.core.quantize`). Default: ``"affine"``. group (mx.distributed.Group, optional): The sharding will happen across this group. If not set then the global group is used. Default is ``None``. Nc|<V^8dQhRS[RS[RS[RS[RS[RS[RS[S[P P ,/#rr)r8rvs"rr9(QuantizedShardedToAllLinear.__annotate__rrc<\S V`4W@nWPnW`n\ P !RV, 4pT;'g\PP4Vn VPP4p W,^8wd\RV RV R24h\PPV)VW!V ,3R7p \P!WWVR7vVnVnp V 'd V ^,MRVnV'd\P&!V34VnVP+4R#)rzrrr}r~rNrrs &&&&&&&& rr$QuantizedShardedToAllLinear.__init__s  %   # *+33bnn113 JJOO  Nq $ZL0J1#YW ""a0#  -/KK - ) T[6$*fQit  +0DI rc L<\SV`!V/VBVPRR7R#rrrs&*,rr$QuantizedShardedToAllLinear.unfreeze0rrc <V^8dQhRS[/#rr)r8rvs"rr9r 6s  S rc VPPwrV^ ,VP,VPP 4,pRV RV RRV9 RVP RVP RVP 2 #rrrs& rr'QuantizedShardedToAllLinear._extra_repr6s} KK--Rs#"((rxxrc 8\P!VVR,VR,VPR4RVPVPVP R7p\P PWPR7pRV9dWR,,pV#)r,rrTrrrK) rrrrrrrrrrs&&rr$QuantizedShardedToAllLinear.__call__>s|    N>88H%  NN " "1JJ " 7 T>L Arr-rcz<V^8dQhRS[RS[S[S[3,RS[S[P P,/#rr)r8rvs"rr9r Orrc T;'g\PP4pVPPwrEV^ ,VP ,pV!VV\ VR4VPVP \VRR4VR7pVP\VP4\V4V44V#r) rrrFr,r+rrrrrcrHr4rWrs&&$$ rrj1QuantizedShardedToAllLinear.from_quantized_linearNrrrrrrs@@rrlrls`2**X#    &'  15 rrlr)r functoolsrtypingrrrmlx.corecorermlx.nn.layers.basermlx.nn.layers.linearrmlx.nn.layers.quantizedr mlx.utilsr r$r1rHrSrWr\rdrnrgrkrirlrrrr!s ,,%'3(      0)5X   %@"# %@ -1 %@P "#  -1 FNNbPPfB&BJA&Ar