+ &jFZ^RIt^RIt^RIHtHt^RIHtHtHtH t ^RI H t H t ^RI Ht.R*OtRRltR R ltR R ltR Rlt!RR]4tRRlt!RR]4t!RR]4tRRltRRltRRltRRlt]!R4RR3RRllt]P<]P>0]n ]PB0]n"]PFR R!l4t$R"R#lt%R$R%lt&R+R&R'llt'!R(R)4t(R#),N)ABCabstractmethod)Callable GeneratorIterableSequence)Anycast CustomPolicyModuleWrapPolicycV^8dQhR\PR\\P.\PR,3,/#) root_modulefnNnnModuler)formats"s/Users/jameslopez/projects/CWCArchive/cwc-podcast/.venv/lib/python3.14/site-packages/torch/distributed/fsdp/wrap.py __annotate__rs9)3)3)3"))bii$../)3c<aaaV0oRVVV3RlloS!VRR4R#)a9 This applies ``fn`` to every module in the module tree of ``root_module`` following a post-order traversal. If ``fn`` returns an :class:`nn.Module`, then this replaces the original module with the newly returned one in the tree. Otherwise, ``fn`` should return ``None``, in which case the module is not changed. crV^8dQhR\PR\R\PR,/#)rmodule module_name parent_moduleN)rrstr)rs"rr'_post_order_apply..__annotate__-s8AA AAyy4'Arc<VP4F(wr4VS9gKSPV4S!WCV4K* S!V4pVe\V\P4'g\ RV RV 24hV'g\ RV 24h\V\P4'g\ RV 24h\ W!V4R#R#)Nz=Non-root modules should have their parent module set but got z for zTNon-root modules should have their module name set but got an empty module name for z.fn should return None or an nn.Module but got )named_childrenadd isinstancerrAssertionErrorsetattr) rrrchild_module_name child_moduleoptional_module_post_order_apply_innerrvisited_moduless &&& rr(2_post_order_apply.._post_order_apply_inner-s 06/D/D/F + ?2##L1' P0GV*  &mRYY77$S$oU6(4$006x9oryy99$D_DUV M @ 'rN)rrr(r)s&f@@r_post_order_applyr-s&(3mOAA6KT2rc V^8dQhR\PR\\P\\\3,3,R\ R\ \P.\PR,3,/#)rrtarget_module_to_kwargsfsdp_fnreturnN)rrdictrr r)rs"rrrKsc!"))T#s(^";<ryyk299t++, rc"aaaRVVV3RllpV#)z This constructs the "wrap" function to pass to :func:`_post_order_apply` based on ``target_module_to_kwargs``, which should be constructed from the wrapping policy. cfV^8dQhR\PR\PR,/#)rrr1N)rr)rs"rr(_construct_wrap_fn..__annotate__Vs'299T!1rcF<VS9dVSJdSV,pS!V3/VB#R#Nr,)rkwargsr0rr/s& rr_construct_wrap_fn..fnVs2 , ,{1J,V4F6,V, ,rr,)rr/r0rsfff r_construct_wrap_fnr:Ks IrcLV^8dQhR\PR\\\P,,R\\P,R\ \ \3,R\ \P\ \ \3,3,/#)rrmodule_classesignored_modules root_kwargsr/)rrrtypesetr2rr )rs"rrrasq###T"))_-#^#c3h # ""))T#s(^";< #rc\\V44pVP4F3pWb9dK \We4'gKWd9dW4V&RWF,R&K5 V#)Nmixed_precision)tupler@modulesr")rr<r=r>r/module_classes_tuplers&&&&& r$_run_mixed_precision_override_policyrFas`!^!45%%'  $   5 542=/AE # +,= >( #"rc$V^8dQhR\/#rr1)bool)rs"rrrus4rcR#)z A simple recursive wrap policy that always returns ``True``. This means that every submodule is wrapped by the wrapper class in :func:`_recursive_wrap`. Tr,)argsr8s*,ralways_wrap_policyrLus rc@a]tRt^~toRt]V3RlRl4tRtVtR#)_Policyz_ This defines an abstract base class that represents a policy for applying a module-level API. c <V^8dQhRS[PRS[S[P,RS[S[S[3,RS[S[PS[S[S[3,3,/#rrr=r>r1rrr@r2rr )r __classdict__s"rr_Policy.__annotate__s]   YY  RYY  #s(^  biic3h' (  rc R#)zp This should return a dict ``target_module_to_kwargs`` that maps from each target module to wrap to its kwargs. Nr,)selfrr=r>s&&&&r _run_policy_Policy._run_policys rr,N) __name__ __module__ __qualname____firstlineno____doc__rrV__static_attributes____classdictcell__rRs@rrNrN~s     rrNc V^8dQhR\PR\R\R\\ \P,,R\/#)rrrecursenonwrapped_numelr<r1rrrIintr@r?)rs"rrrsK55 II5 55RYY( 5  5rc@V'dR#\V\V44#)a This auto wrap policy wraps every module that is an instance of any type in ``module_classes`` as its own FSDP instance. The root module given by ``module`` is always wrapped as an FSDP instance regardless. Since the wrapping proceeds bottom up, each FSDP instance manages the parameters in its subtree excluding any already managed by a child FSDP instance. Args: module (nn.Module): Current module being considered. recurse (bool): If ``False``, then this function must decide whether ``module`` should be wrapped as an FSDP instance or not. If ``True``, then the function is still recursing down the module tree as a part of the DFS. nonwrapped_numel (int): Parameter numel not yet wrapped. module_classes (Set[Type[nn.Module]]): Set of module classes that are wrapped as FSDP instances. Returns: ``True`` if ``recurse=True``, and whether ``module`` should be wrapped if ``recurse=False``. Tr"rC)rrarbr<s&&&&r_module_wrap_policyrgs6 feN3 44rclaa]tRt^toRtV3RlRltV3RlRltRtV3RlV3RlltR t Vt V;t #) r zo This policy applies to every module of the specified module classes, passing in the kwargs given to the root. cT<V^8dQhRS[S[S[P,,/#)rr<)rr?rr)rrRs"rrModuleWrapPolicy.__annotate__s!;;xRYY'@;rcH\V4pW n\V4VnR#r7)r@_module_classesr_module_classes_str)rUr<module_classes_sets&& r__init__ModuleWrapPolicy.__init__s! 01#&'9#: rc <V^8dQhRS[PRS[S[P,RS[S[S[3,RS[S[PS[S[S[3,3,/#rPrQ)rrRs"rrrjs]''YY'RYY'#s(^ ' biic3h' ( 'rc\VP4p/pVP4F6pWb9dK \Wd4'gK\P!V4WV&K8 V#r7)rCrlrDr"copy)rUrr=r>r<r/rs&&&& rrVModuleWrapPolicy._run_policys[ t334CE!))+F(F3326))K2H'/ , '&rc2\WRVPR7#))rbr<)rgrl)rUrrarKr8s&&&*,r__call__ModuleWrapPolicy.__call__s" bAUAU  rc <V^8dQhRS[/#rH)r)rrRs"rrrjsDD#DrcJ<\SV`4RVP R2,#)())super__repr__rm)rU __class__s&rrModuleWrapPolicy.__repr__s&w!a(@(@'A$CCCr)rlrm) rXrYrZr[r\rorVrxrr]r^ __classcell__)rrRs@@rr r s5 ;; ''  DDDrcHa]tRt^toRtV3RlRltV3RlRltRtVtR#)r a This policy takes in a lambda function that maps a given ``nn.Module`` to either ``False``, ``True``, or a kwarg dictionary. - If the function returns ``False`` or an empty dictionary, then the module does not have the API applied. - If the function returns ``True``, then the module has the API applied with the root's kwargs. - If the function returns a non-empty dictionary, then the module has the API applied, and the dictionary overrides the root's kwargs. Example:: >>> # xdoctest: +SKIP("undefined variables") >>> model = init_transformer_model(...) >>> def lambda_fn(module: nn.Module): >>> if module is model.lm_head: >>> return {"sharding_strategy": ShardingStrategy.SHARD_GRAD_OP} >>> elif isinstance(module, TransformerBlock): >>> return True >>> return False >>> policy = CustomPolicy(lambda_fn) >>> fsdp_model = FSDP(model, auto_wrap_policy=policy) cr<V^8dQhRS[S[P.S[S[S[S[3,,3,/#)r lambda_fn)rrrrIr2rr )rrRs"rrCustomPolicy.__annotate__s1$$(BII;tCH~8M+M"N$rcWnR#r7 _lambda_fn)rUrs&&rroCustomPolicy.__init__s#rc <V^8dQhRS[PRS[S[P,RS[S[S[3,RS[S[PS[S[S[3,3,/#rPrQ)rrRs"rrrs]''YY'RYY'#s(^ ' biic3h' ( 'rcT/pVP4FpWR9dK VPV4p\V\\34'g\ RV 24hV'gKP\ P !V4p\V\4'dVPV4WtV&K V#)z_The lambda_fn passed to CustomPolicy should return False/True or a kwarg dict, but it returned )rDrr"r2rI ValueErrorrsupdate)rUrr=r>r/rresr8s&&&& rrVCustomPolicy._run_policys DF!))+F(//&)CcD$<00 CCF%IYY{+F#t$$ c".4F +!,"'&rrN) rXrYrZr[r\rorVr]r^r_s@rr r s0$$''rc hV^8dQhR\PR\R\R\R\/#)rrrarbrr1)rrrIrdr)rs"rrrs8 II $8;HP rc&V'dR#V!V4#)a A convenient auto wrap policy to wrap submodules based on an arbitrary user function. If `lambda_fn(submodule) == True``, the submodule will be wrapped as a `wrapper_cls` unit. Return if a module should be wrapped during auto wrapping. The first three parameters are required by :func:`_recursive_wrap`. Args: module (nn.Module): Current module being considered. recurse (bool): If ``False``, then this function must decide whether ``module`` should be wrapped as an FSDP instance or not. If ``True``, then the function is still recursing down the module tree as a part of the DFS. nonwrapped_numel (int): Parameter numel not yet wrapped. lambda_fn (Callable[[nn.Module], bool]): If this returns ``True``, then this module will be wrapped. Tr,)rrarbrs&&&&rlambda_auto_wrap_policyrs. V rc V^8dQhR\PR\R\R\\ \P,,R\/#)rrrarbtransformer_layer_clsr1rc)rs"rrr*sR Y Y II Y  Y YtBII/ Y  Yrc\WW#4#)a See :func:`_module_wrap_policy`, where ``transformer_layer_cls`` is the same as ``module_classes``. Note that shared parameters must be wrapped in the same FSDP instance, so this auto wrap policy can help wrap shared embeddings into the same FSDP instance for transformer models. )rg)rrarbrs&&&&rtransformer_auto_wrap_policyr*s v0@ XXrcfV^8dQhR\PR\\,R\/#)rrr<ra)rrrr?rI)rs"rrr9s, 9 9 II 9'/~ 9@D 9rc@V'dR#\V\V44#)Trf)rr<rarKr8s&&&*,r_wrap_module_cls_individuallyr9s&%"788rc\V^8dQhR\PR\R\R\/#)rrrarbr1)rrrIrd)rs"rrrEs4   II      rcaaa\;QJd!VVV3RlV4F 'gK R# R#!VVV3RlV44#)zj A policy that wraps ``module`` if any policy in the passed in iterable of ``policies`` returns ``True``. c3:<"TFpV!SSSR7xK R#5i)rrarbNr,).0policyrrbras& r _or_policy..Os%F fg@PQQsTF)any)rrarbpoliciessfff&r _or_policyrEs@ 33333 rgחAcV^8dQhR\PR\R\R\R\\ \P,,R,R\\ \P,,R,R\/#) rrrarbmin_num_paramsforce_leaf_modulesNexclude_wrap_modulesr1rc)rs"rrrUs3P3P II3P 3P3P  3P DO,t3 3Pd299o.53P 3PrcVf\PMTpVf\PMTpTpW&8pV'd%T;'d\V\ V44'*#T;'d\V\ V44'*#)a{ A size-based auto wrap policy. Args: module (nn.Module): Current module being considered. recurse (bool): If ``False``, then this function must decide whether ``module`` should be wrapped as an FSDP instance or not. If ``True``, then the function is still recursing down the module tree as a part of the DFS. nonwrapped_numel (int): Parameter numel not yet wrapped. min_num_params (int): Customizable policy input that controls the size threshold over which a module is ready to be wrapped. This is in units of numel. force_leaf_modules (Optional[set[type[nn.Module]]]): Set of module types to keep as leaves, i.e. their children will never be wrapped. exclude_wrap_modules (Optional[set[type[nn.Module]]]): Set of module types to be excluded in wrapping. Returns: Whether ``module`` should be wrapped. )size_based_auto_wrap_policyFORCE_LEAF_MODULESEXCLUDE_WRAP_MODULESr"rC)rrarbrrrmin_nonwrapped_numelis_larges&&&&&& rrrUsB  % $66  ' $88 !*7HMM 659K3L MMMOO 659M3N OOOrcJV^8dQhR\R\R\R,/#)r wrapper_clswrapper_kwargsr1)NNN)r r)rs"rrrs*   +.   rc+"RV/VCp\R/VB;_uu_4RxRRR4R# +'giR#;i5i)a Context manager to wrap modules using a wrapper. Useful for when you'd like to apply the same configuration arguments to all child modules that you wrap. A particularly important use case is wrapping large layers so that they get sharded (in-place) during initialization, to avoid running out of system memory. Large layers can indicate that they should be sharded via the ``wrap`` annotation and this context manager can provide the exact configuration for these nested instances. Usage:: with enable_wrap(wrapper_cls, **params): # Wraps layer in FSDP by default if within context self.l1 = wrap(torch.nn.Linear(5, 5)) Args: wrapper_cls: Class that `wrap` annotation will `wrap` modules with, such as `FullyShardedDataParallel`. **wrapper_kwargs: Configuration settings that will be passed to all ``wrap`` instances inside the context rNr,)_ConfigAutoWrap)rrr8s$, r enable_wraprs<: { F  "6 " "  # " " "s?+ ? < ?cdV^8dQhR\PR\R\P/#)rrwrap_overridesr1)rrr )rs"rrrs)###c#bii#rc \P'dM\Pf \R4h/\PCVCp\ V\P3/VB#V#)a Annotate that a module should be wrapped. Annotated modules will only be wrapped if inside of an :func:`enable_wrap` context manager. This allows a module to be initialized both with and without a wrapper without code change. The class that this function wraps the passed in ``nn.Module`` with is the passed in ``wrapper_cls`` argument into ``enable_wrap``. Both ``enable_wrap`` and ``wrap`` can take in kwargs specifying how to construct the ``wrapper_cls`` instance. In the case of duplicate kwargs in ``enable_wrap`` and ``wrap``, the argument passed into ``wrap`` will be respected. Usage:: with enable_wrap(wrapper_cls=FSDP, **fsdp_config): # Wraps layer in FSDP by default if within context self.l1 = wrap(torch.nn.Linear(5, 5)) Args: module (nn.Module): module to wrap (if in :func:`enable_wrap` context) **wrap_overrides: configuration overrides that will take priority over the values provided by the :func:`enable_wrap` context z.Expected _ConfigAutoWrap.wrapper_cls to be set)rin_autowrap_contextrr#r8_wrap)rrs&,rwraprsg2***  & & . !QR REO22EnE   ' '   MrcdV^8dQhR\PR\R\P/#)rrrr1r)rs"rrrs) ) )")) )( ) )rc Vf \R4h\VR4'd/VCVPCpV!V3/VB#V!V3/VB#)NzExpected wrapper_cls to be set_wrap_overrides)r#hasattrr)rrr8 overridess&&, rrrsY=>>v()) 9v8!7!78 6/Y// v ( ((rcV^8dQhR\PR\R\R\\P,R\\P,R\ R\ R\\P\3,/#) rrauto_wrap_policyrr=ignored_paramsonly_wrap_childrenr8r1) rrrr@ ParameterrIr rCrd)rs"rrrsMM IIMMM^ M  % M  MM 299c>Mrc aVf \R4hVf \R4hVP4F?wrxW9dK \V\\V44'd\RV RV 24hKA \ V3RlVP444p Vf \R4hV!VRV R7'd^p VP4F8wrW9dK \RR VR VR VR VR S/VBwr\W V 4W, p K: W, pV'g V!VRVR7'd\W3/VBV 3#W 3#V^3# \ dEKi;i)a Wraps submodules of ``module`` for which ``auto_wrap_policy`` returns ``True`` with ``wrapper_cls``. Args: module (nn.Module): Module to recursively wrap. auto_wrap_policy (Callable): A callable representing a policy that determines which modules to recursively wrap with ``wrapper_cls``. ignored_modules (set[torch.nn.Module]): Modules to ignore when wrapping. ignored_params (set[torch.nn.Parameter]): Parameters to ignore when wrapping; these should be the parameters contained in the modules in ``ignored_modules``. Returns: (nn.Module, int): ``module`` after wrapping and the numel recursively wrapped. zMust specify auto_wrap_policy.zMust specify wrapper_clsz Child module z is already wrapped by c3T<"TFqS9gK VP4xK R#5ir7)numel)rprs& rr"_recursive_wrap..s$.a>2I  .s((z#Expected auto_wrap_policy to be setTrrrrr=rFr,) r# named_modulesr"r r? TypeErrorsum parametersr _recursive_wrapr$r)rrrr=rrr8_childrbtotal_wrapped_numelname wrapped_childnum_wrapped_params remainders&&&&f&, rrrs4=>>788((*  #  %dK!899$#E7*A+O: +!,,.BCCvtFVWW!002KD'0?11!11(1!0 1 . 1  1 -M F- 0  5 3 %: !&659' ' 779II I. . 19M   s1D:: E  E ca]tRtRtoRtRtRt/tV3RlRlt] V3RlRl4t ] V3R lR l4t V3R lR lt V3R lRlt V3RltRtVtR#)ri8z Helper class to wrap modules based on default config args via a context manager. See :func:`enable_wrap` for more information. FNc6<V^8dQhRS[S[S[3,/#)rr8)r2rr )rrRs"rr_ConfigAutoWrap.__annotate__Bsc3hrc WnR#r7r8)rUr8s&,rro_ConfigAutoWrap.__init__Bs rc$<V^8dQhRS[RR/#)rr8r1Nr )rrRs"rrrFs((((rc\P'd \R4hR\nRV9d \R4h\ \ VR,4\nVRV\nR#)z]You are already within an autowrap context and we currently do not supported nested autowrap.Trz9Expected to pass in wrapper_cls arg into _ConfigAutoWrap.N)rrNotImplementedErrorr#r rrr8rs&renable_autowrap_context'_ConfigAutoWrap.enable_autowrap_contextEsf  . . .%o /3+  & K '+8VM5J&K# = !!'rc<V^8dQhRR/#rr1Nr,)rrRs"rrrWs$$d$rcHR\nR\n/\nR#)FN)rrrr8r,rrdisable_autowrap_context(_ConfigAutoWrap.disable_autowrap_contextVs.3+&*#!#rc<V^8dQhRR/#rr,)rrRs"rrr\s2242rc<VPVP4R#r7)rr8)rUs&r __enter___ConfigAutoWrap.__enter__\s $$T[[1rc0<V^8dQhRS[RS[RS[RR/#)rexc_typeexc_valexc_tbr1Nr)rrRs"rrr_s)(((s(C(D(rc&VP4R#r7)r)rUrrrs&&&&r__exit___ConfigAutoWrap.__exit___s %%'rcb<V^8dQh/S[;R&S[R,;R&S[S[S[3,;R&#)rrNrr8)rIrr2rr )rrRs"rrr8s8 % D' cNrr)rXrYrZr[r\rrr8ro staticmethodrrrr__annotate_func__r]r^r_s@rrr8sj !&#'KF(( $$ 22((Orr)rLrrrrrr r )F)) contextlibrsabcrrcollections.abcrrrrtypingr r torch.nnr__all__r-r:rFrLrNrgr r rrrrrdr ModuleList ModuleDictrMultiheadAttentionrcontextmanagerrrrrrr,rrrs #CC )3X,#( c (5@"Dw"DJ4'74'n8 Y 9 *c(6:8<3Pn57MM2==3Q0242G2G1H.   F#L )M`((((r