+ &jG^RIt^RIt^RIHt^RIHt^RIHtHtRR/Rlt Rt ]P3Rlt ^]P3Rlt ^]P3R lt]P]P3R lt]P]P3R lt]P3R lt]P3R lt]P3RltRR]P3Rlt]P]P3Rlt!RR]4t!RR]4t!RR]4t!RR]4t!RR]4t!RR]4t!RR]4t!RR ]4t!R!R"]4t!R#R$]4t R#)%N)Function)groupReduceOp suggestioncNRV R2pV'dVRV R2, p\V4h) torch.distributed.nn.functional.z& is not supported under torch.compile.z Use instead.) RuntimeError)namermsgs&$ w/Users/jameslopez/projects/CWCArchive/cwc-podcast/.venv/lib/python3.14/site-packages/torch/distributed/nn/functional.py_not_supported_under_compilers7 *4&0VW zl),, s cN\P!RV RV R2\^R7R#)rz is deprecated, use r )category stacklevelN)warningswarn FutureWarning)r rs&&r _deprecatedrs, MM *4&1l) % rc\PP4'd\RRR7\ RR4\ P WV4#)a Broadcasts the tensor to the whole group. ``tensor`` must have the same number of elements in all processes participating in the collective. Arguments: tensor (Tensor): Data to be sent if ``src`` is the rank of current process. src (int): Source rank. group (ProcessGroup, optional): The process group to work on. Returns: Tensor: Received tensor from the broadcast op. broadcastz3torch.distributed._functional_collectives.broadcastr)torchcompiler is_compilingrr _Broadcastapply)tensorsrcrs&&&r rr sE" ~~""$$$ L  RS   C //rc\PP4'd \R4\P WV4#)a4 Gathers a list of tensors in a single process. Arguments: tensor (Tensor): Input tensor. dst (int, optional): Destination rank (default is 0). group (ProcessGroup, optional): The process group to work on. Returns: tuple[Tensor]: List of appropriately-sized tensors with the gathered data. gather)rrrr_Gatherr)rdstrs&&&r r"r":s0 ~~""$$$X. ==V ,,rc\PP4'd \R4\P !W.VO5!#)a Scatters a list of tensors to all processes in a group. Each process will receive exactly one tensor and store its data in the ``tensor`` argument. Arguments: tensors (list[Tensor]): List of tensors to scatter on the source rank. Receivers must pass ``None`. src (int, optional): Source rank (default is 0). group (ProcessGroup, optional): The process group to work on. Returns: Tensor: Output tensor from the scatter operation. scatter)rrrr_Scatterr)tensorsr rs&&&r r&r&Ks3" ~~""$$$Y/ >># /w //rc\PP4'd \R4\P WW04#)a Reduces the tensor data across all machines. Only the process with rank ``dst`` is going to receive the final result. Arguments: tensor (Tensor): Input of the collective. dst (int): Destination rank. op (optional): One of the values from ``torch.distributed.ReduceOp`` enum. Specifies an operation used for element-wise reductions. group (ProcessGroup, optional): The process group to work on. Returns: Tensor: Output of the collective. reduce)rrrr_Reducer)rr$oprs&&&&r r*r*as0$ ~~""$$$X. ==% 00rc\PP4'd\RRR7\ RR4\ P !W#V.VO5!#)a Reduces, then scatters a list of tensors to all processes in a group. Arguments: output (Tensor): Output tensor. input_list (list[Tensor]): List of tensors to reduce and scatter. op (optional): One of the values from ``torch.distributed.ReduceOp`` enum. Specifies an operation used for element-wise reductions. group (ProcessGroup, optional): The process group to work on. Returns: Tensor: Output of the collective. reduce_scatterz?torch.distributed._functional_collectives.reduce_scatter_tensorr)rrrrr_Reduce_Scatterr)output input_listr,rs&&&&r r.r.xsO  ~~""$$$ X I  F @Z @@rc\PP4'd\RRR7\ RR4\ P W4#)z Gathers tensors from the whole group in a list. Arguments: tensor (Tensor): Tensor to be broadcast from current process. group (ProcessGroup, optional): The process group to work on. Returns: tuple([Tensor]): Output of the collective. all_gatherz;torch.distributed._functional_collectives.all_gather_tensorr)rrrrr _AllGatherr)rrs&&r r3r3sH ~~""$$$ T S   E **rc\PP4'd \R4\P WV4#)av Single tensor all gather. Gathers a single tensor from all ranks, and puts them in a single output tensor. Args: output_tensor (Tensor): Output tensor. It should contain correctly-sized tensors to be used for output of the collective. input_tensor (Tensor): Tensor to be broadcast from current process. group (ProcessGroup, optional): The process group to work on. If None, the default process group will be used. Examples: >>> # All tensors below are of torch.int64 dtype. >>> # We have 2 process groups, 2 ranks. >>> # xdoctest: +SKIP("incorrect want text") >>> output_tensor = torch.zeros(2, dtype=torch.int64) >>> output_tensor [tensor([0, 0])] # Rank 0 and 1 >>> tensor = torch.arange(1, dtype=torch.int64) + 1 + rank >>> tensor tensor([1]) # Rank 0 tensor([2]) # Rank 1 >>> dist.all_gather_base(output_tensor, tensor) >>> output_tensor tensor([1,2]) # Rank 0 tensor([1,2]) # Rank 1 .. warning:: `_all_gather_base` is experimental and subject to change. It is the caller's responsibility to ensure the output_tensor is correctly sized. _all_gather_base)rrrr_AllGatherBaser) output_tensor input_tensorrs&&&r r6r6s4B ~~""$$$%78    U CCrc\PP4'd \R4\P !W .VO5!#)a Each process scatters list of input tensors to all processes in a group and return gathered list of tensors in output list. Arguments: output_tensor_list (list[Tensor]): list of tensors to gather one per rank. input_tensor_list (list[Tensor]): List of tensors to scatter one per rank. group (ProcessGroup, optional): The process group to work on. Returns: tuple([Tensor]): Output of the collective. all_to_all)rrrr _AlltoAllr)output_tensor_listinput_tensor_listrs&&&r r;r;s4 ~~""$$$\2 ??5 I7H IIrc\PP4'd\RRR7\ RR4\ P W@W#V4#)a Each process splits input tensor and then scatters the split list to all processes in a group. Then concatenate the received tensors from all the processes in the group and return single output tensor. Arguments: output (Tensor): Gathered concatenated output tensor. input (Tensor): Input tensor to scatter. output_split_sizes: (list[Int], optional): Output split sizes for dim 0 if specified None or empty, dim 0 of ``output`` tensor must divide equally by ``world_size``. input_split_sizes: (list[Int], optional): Input split sizes for dim 0 if specified None or empty, dim 0 of ``input`` tensor must divide equally by ``world_size``. Returns: Tensor: Output of the collective. all_to_all_singlez;torch.distributed._functional_collectives.all_to_all_singler)rrrrr_AlltoAllSingler)r0inputoutput_split_sizesinput_split_sizesrs&&&&&r r@r@sR4 ~~""$$$ T E   )e rc\PP4'd\RRR7\ RR4\ P WV4#)a Reduces the tensor data across all machines in such a way that all get the final result. After the call the returned tensor is going to be bitwise identical in all processes. Arguments: tensor (Tensor): Input of the collective. op (optional): One of the values from ``torch.distributed.ReduceOp`` enum. Specifies an operation used for element-wise reductions. group (ProcessGroup, optional): The process group to work on. Returns: Tensor: Output of the collective all_reducez4torch.distributed._functional_collectives.all_reducer)rrrrr _AllReducer)rr,rs&&&r rFrF sE$ ~~""$$$ M  TU   Bv ..rc@a]tRtRto]R4t]R4tRtVtR#)ri&cWnW n\P!VR7VnVP 4p\P !W1VR7V#r)r rdistget_rankrankcloner)ctxr rrs&&&&r forward_Broadcast.forward's> ==u- v%0 rc\PVP\PVP V4pVPVP 8wdVP4RRV3#N)r+rr rSUMrrNzero_)rP grad_outputgxs&& r backward_Broadcast.backward3sJ]]377HLL#))[ I 77chh  HHJdBrN __name__ __module__ __qualname____firstlineno__ staticmethodrQrY__static_attributes____classdictcell__ __classdict__s@r rr&s+  rrc@a]tRtRto]R4t]R4tRtVtR#)r#i<c|WnW n\\P!VR74Uu.uFp\ P !V4NK ppVP4p\P!VR7V8Xd\P!W5WR7M\P!VRWR7\V4#uupi)rKN) r$rrangerLget_world_sizer zeros_like contiguousrMr"tuple)rPr$rri tensor_lists&&&& r rQ_Gather.forward=s /4D4G4Ge4T.U .UE  V $.U  ""$ ==u % , KKS > KKc 7[!! sB9clR\P!VPVP.VO5!3,#NNN)r'rr$r)rP grad_outputss&*r rY_Gather.backwardQs(x~~cggsyyP<PRRRrr[Nr\rds@r r#r#<s-""$SSrr#c@a]tRtRto]R4t]R4tRtVtR#)r'iVcaWnW n\;QJdV3RlS4F 'dK RM RM!V3RlS44'g\h\P !S^,4p\ P!VR7V8Xd%\ P!V\S4WR7V#\ P!VRWR7V#)c3r<"TF,qP4S^,P48HxK. R#5i)rN)size).0tr(s& r #_Scatter.forward..\s%B'Q668wqz00's47FTrKN) r rallAssertionErrorrrjrLrMr&list)rPr rr(r0s&&&j r rQ_Scatter.forwardWs sB'BsssB'BBB !!'!*- ==u % , LLg A  LLs 8 rcfR\PVPVPV4,#rq)r#rr rrPrWs&&r rY_Scatter.backwardes$gmmCGGSYY LLLrr[Nr\rds@r r'r'Vs-  MMrr'c@a]tRtRto]R4t]R4tRtVtR#)r+ikcnWnW0nVP4p\P!WAW#R7V#)r,r)r rrOrLr*)rPr r,rrs&&&&&r rQ_Reduce.forwardls,  FB4 rchR\PVPVPV43,#N)NNN)rrr rrs&&r rY_Reduce.backwardus)"Z%5%5cggsyy+%V$XXXrr[Nr\rds@r r+r+ks-YYrr+c@a]tRtRto]R4t]R4tRtVtR#)r/i{cW nVP4p\;QJd.RV4FNK 5M !RV44p\P!V\ V4WR7V#)c3@"TFqP4xK R#5irTrkryrzs& r r{*_Reduce_Scatter.forward..s!L:KQ,,..:Kr)rrkrlrLr.r)rPr,rrr>s&&&&*r rQ_Reduce_Scatter.forward|sU ""$!E!L:K!LEE!L:K!LL FD):$;P rcPR\PVPV4,#r)r4rrrs&&r rY_Reduce_Scatter.backwards "J$4$4SYY $LLLrr[Nr\rds@r r/r/{s-MMrr/c@a]tRtRto]R4t]R4tRtVtR#)r4icVP4pWn\\P!VR74Uu.uFp\ P !V4NK pp\P!WBVR7\V4#uupirJ) rkrrhrLrir empty_liker3rl)rPrr_out_tensor_lists&&& r rQ_AllGather.forwardso""$ .3D4G4Ge4T.U .UE  V $.U   u=_%%  sA=c\P!VPR7\PP\PP 39dt\P !VPR7p\P!W,4p\P!\PVPV.VO5!pRV3#VUu.uFp\P!V4NK pp\P!VPV.VO5!p\P!\P!V4^R7pRV3#uupi)rK)dimN)rL get_backendrBackendNCCLXCCLrMrrr/rrrUr<sumstack)rPrsrNrXrrngxss&* r rY_AllGather.backwards   #)) ,1B1BDLLDUDU0V V==syy1D!!,"45B &&x||SYYR\RBbzCOO,5++F3,KO//#))[H<HC5;;s+3BbzPsEr[Nr\rds@r r4r4s+ & &  rr4c@a]tRtRto]R4t]R4tRtVtR#)r7ic^W0n\P!WP4VR7V#rJ)rrLr6rk)rPr8r9rs&&&&r rQ_AllGatherBase.forwards(  m-D-D-FeTrc\P!VPR7\PP\PP 39d\P !VPR7p\VP44pV^,V,^8wd\RV RV 24hV^,\P !VPR7,V^&\P!W1PVPR7p\P!WA\P VP4M \R4hRVR3#)rKzTensor with dimensions: z8 does not have first dimension divisible by world_size: devicedtypezBackend not supported!N)rLrrrrrrirrxr remptyrr_reduce_scatter_baserrU)rPrW world_sizeout_sizerXs&& r rY_AllGatherBase.backwards   #)) ,1B1BDLLDUDU0V V,,399=JK,,./H{Z'1,".xj9IIS V#1+)<)<399)MMHQK!3!3;;L;LB  % %bx||SYY O78 8b$rr[Nr\rds@r r7r7s+   rr7c@a]tRtRto]R4t]R4tRtVtR#)r<icWn\\P!VR74Uu.uFqCV,P 4NK upVn\P !VR7p\;QJd.RV4FNK 5M !RV44p\P!VR7\PPJd[\\P!VR74F5pRpWE8Xd \V4p\P!W$,WdVR7K7 M"\P!V\V4VR7\V4#uupi)rKc3@"TFqP4xK R#5irTrrs& r r{$_AlltoAll.forward..s81 rN)rrhrLrirxinput_tensor_size_listrMrlrrGLOOrr&r;)rPrrr(rmmy_rankto_sends&&&* r rQ_AlltoAll.forwards ',T-@-@u-M'N& 'N!AJOO 'N& "--e,%88%%888   % (DLL,=,= =4..U;<<"7mG _/5I = OOW   _%%%& sE c VPUu.uF=p\P!W!^,PV^,PR7NK? ppR\ P !VPV.VO5!,#uupi)rrrr)rrrrrr<rr)rPrsrxrns&* r rY_AlltoAll.backwardsv 22  3 KK!_33<?;P;P 3  ioociiT|TTT  sAA>r[Nr\rds@r r<r<s-&&,UUrr<c@a]tRtRto]R4t]R4tRtVtR#)rAicWnVP4VnW@nW0n\ P !VVVVVR7V#))rCrDr)rrx input_sizerCrDrLr@)rPrr0rCrDrBs&&&&&&r rQ_AlltoAllSingle.forwardsF !2 2   1/   rc  \P!VPVPVPR7pR\ P VPVVPVPVP443,#)r)NNNN) rrrrrrArrrCrDrk)rPrWrs&& r rY_AlltoAllSingle.backwardss NN;#5#5[=N=N (  ! ! &&%%&&(  +   rr[Nr\rds@r rArAs+      rrAc@a]tRtRto]R4t]R4tRtVtR#)rGi cW nWnVP\PR7p\ P !W1VR7V#)) memory_formatr)rr,rOrcontiguous_formatrLrF)rPr,rrs&&&&r rQ_AllReduce.forwards7 E,C,CD U3 rchR\PVPVPV43,#rq)rGrr,rrs&&r rY_AllReduce.backwards)z// ;OQQQrr[Nr\rds@r rGrG s-RRrrG)!rrtorch.distributed distributedrLtorch.autogradrrrrrWORLDrr"r&rUr*r.r3r6r;r@rFrr#r'r+r/r4r7r<rArGr[rr rs[ # .T"'04 -"%++0,$< #DL=BKKJ* ++ %P#,,ekk/6  ,ShS4MxM* Yh Y MhM"< X :!U!UH h D R Rr