+ &j,aRt40t^RIt^RIt^RIHt^RIHt^RIHt^RI H t ^RI t ^RI H t Ht^RIHt^RIHt^R IHt^R IHt^R IHtR .t!R R]4tRtRRlt]!RR7!RR] P<P>44t Rt!Rt"] PFPHPJPL] PFPHPNPL] PFPHPPPL] PFPHPRPT] PFPHPVPL] PFPHPXPZ] PFPHP\PL] PFPHP^PL/t0]^kRRlt1RRlt2]!RR7R4t3R#)N) defaultdict)Callable)Enum)Any) FakeTensorFakeTensorModeNode) compatibility)StorageWeakRef)_pytree) tree_map_only reinplacec"]tRt^t^t^t^tRtR#) _ViewTypeN)__name__ __module__ __qualname____firstlineno__NonViewSingleOutputViewMultiOutputView__static_attributes__rq/Users/jameslopez/projects/CWCArchive/cwc-podcast/.venv/lib/python3.14/site-packages/torch/fx/passes/reinplace.pyrrsGOrrc>Ve\V\PP4'dmVPp\ VP 4^8dEVP ^,pVPRJ;'dVPP'*#R#R#R#N) isinstancetorch_ops OpOverload_schemalen arguments alias_infois_writetgtschema first_args& r _is_view_opr,s :c5::+@+@AA v 1 $((+I$$D0VV9M9M9V9V5V  %Brc$V^8dQhR\/#)return)r)formats"r __annotate__r1&s  9 rcVe\V\PP4'dVPp\ VP 4^8dyVP ^,pVPeXVPP'g<RVPP9d\P#\P#\P#)N*)rr r!r"r#r$r%r&r' after_setrrrrr(s& r_get_view_typer5&s :c5::+@+@AA v 1 $((+I##/ 8L8L8U8U8U)..888$444$555   rF)is_backward_compatiblecJaa]tRt^>toV3RlV3RlltV3RltRtVtV;t#)_FunctionalizationMetadataPropc <V^8dQhRS[/#)r.noder )r0 __classdict__s"rr1+_FunctionalizationMetadataProp.__annotate__@s;;T;rc<V;P^, un\S V` V4pW!PR&VPVPR&VPpVP \ PPPPJd VR,pVPR8XEd]\VP 4pV\P8Xdq\VP^,\ 4'g)\#R\%VP^,4 24hVP^,VPR&MV\P&8Xd"VP^,VP(V&MVP \*P,JdoVP^,pVP(P/VR4pVe<\V\ 4'g\#R\%V4 24hWaPR&RVP9d\VPR,\04'g \#R4h\VPR,PR,\04'g \#R 4h\3VPR,P544p\3VPR,PR,P544pWx8wd \#R 4hV#)  fake_resultnode_idxr>NN call_functionzExpected Node, got view_ofNz"Expected FakeTensor in fake_resultz*Expected FakeTensor in view_of fake_resultzview_storage != base_storage) node_countersuperrun_nodemetaargstargetr opsatencopy_defaultopr5rrrr AssertionErrortypermulti_output_view_nodes _operatorgetitemgetrr _typed_storage) selfr:result node_args view_typelist_argmaybe_base_of_view view_storage base_storage __class__s && rrF'_FunctionalizationMetadataProp.run_node@sL Q!$'#) - $ 1 1 * II ;;%))....66 6!" I 77o %&t{{3II666!$))A,55(+>tDIIaL?Q>R)STT'+yy| )$i77759YYq\,,T2 1 1199Q<%)%A%A%E%EhPT%U"%1&&8$??,1$7I2J1KL,>IIi(  !dii 6 CC$%IJJdii 277 F SS$%QRR)$))M*B*Q*Q*STL) )$))-8GGIL+$%CDD rc ,</VnRVn\4;_uu_4pVUu.uF6p\V\P 4'dVP V4MTNK8 pp\SV` !V!uuRRR4#uupi +'giR#;i)r>N) rQrDrrr Tensor from_tensorrErun)rVrHmodea fake_argsr^s&* r propagate(_FunctionalizationMetadataProp.propagate}s')$   SWSWaz!U\\'B'B  #ISW 7; *    sBs;;z++rr8cZVPPR4;'dVPRR VP8Hp\VP4\VP48H;'dq\;QJd:R\ VPVP44F 'dK RM/ RM+!R\ VPVP444pVP^,P e.VP^,P P'g \R4h\;QJd-RVPR,4F 'dK RM" RM!RVPR,44'g \R4hT;'dT#) _Nc3X"TF wrVPVP8HxK" R#5ir)rP).0a1a2s& r !_schemas_match..s&PFB 277Ps(*FTz,First argument of inplace op must be mutablec3<"TFqPRJxK R#5ir)r&)rorfs& rrrrssJ-I||t#-IsrAz5Remaining arguments of inplace op must not be mutablera) nameendswithr$r%allzipr&r'rO)functional_schemainplace_schema names_matcharg_types_matchs&& r_schemas_matchr}s[$$S) ? ?    $(9(>(> >+556#  ; #+55~7O7OP###+55~7O7OP    #..:  $ $Q ' 2 2 ; ; ;KLL 3J^-E-Eb-IJ333J^-E-Eb-IJ J JTUU  * *?*rc\V\PP4'gR#\ V4'dR#VP P R4R,pVPPp\\PV4pVfRM\W2 R2R4pVfR#VP4Uu.uFp\WE4NK ppVUu.uF-p\VPVP4'gK+VNK/ pp\V4^8XdR#\V4^8wd\R\V4 24hV^,p V #uupiuupi)N.rmz2Expected exactly 1 matching inplace overload, got ra)rr r!r"r,rsplitoverloadpacketrgetattrrJ overloadsr}r#r$rO) rN op_namespace op_base_namemaybe_namespace_modulemaybe_inplace_op overload_nameinplace_overloadsf'inplace_overloads_with_matching_schemas inplace_ops & r_maybe_get_inplace_oprsU b%**// 0 02==&&s+B/L$$--L$UYY = " ) +~Q-? F .7799M  09 %/$arzz199(M$,/ 23q8 23q8@:;< >  9;J '/s?E(EEcFV^8dQhR\\,R\/#)r.tensor_aliasesop_index)setr int)r0s"rr1r1ss4yCrcRp\4pVFpVPpVFpRVP9gVPR,V8:dK.W`9dU\VP\ P P4'gVP\PJdKVPV4K K V#)c\V\4'd+VP\VP 444R#R#r)rraddr rU)xset_s&&r_add_if_tensor2_get_all_later_node_usages.._add_if_tensors. a $ $ HH^A$4$4$67 8 %rr@) rusersrGrrIr r!r"rRrSr)rrrnodes_used_aftert usage_nodesns&& r_get_all_later_node_usagesrs9u gg A'166*+=+I"qxx)>)>??xx9#4#44   #" rc~V^8dQhR\\,R\\,R\\,/#)r.later_node_usages self_aliasesr/)rr )r0s"rr1r1s0::4y:03D :Y:rcRp\4p\VRR7EFpVP\9dKVP^,pVP^,p\ V\ 4'g\R\V4 24h\ VPR,\4'g \R4h\ V\ 4'g\R\V4 24h\ VPR,\4'g \R4h\ VP\4'd \R4h\VP,pVFpR VP9dKVPR ,p V!V PR,.VPR ,O5/VPBp VPR,p V!V PR,VPR,4'd$V!W4'dVPV4KKK EK V# \dKi;i) cVP4VP48H;'dKVP4VP48H;'d"VP4VP48H#r)sizestridestorage_offset)rfbs&&rmatching_view_metadata=_get_view_inverse_node_usages..matching_view_metadatas] FFH  9 9 ahhj( 9 9  "a&6&6&88 rc(VPR,#)r@)rG)rs&r/_get_view_inverse_node_usages..s QVVJ5Gr)keyzExpected Node for base, got r?z/Expected FakeTensor in base.meta['fake_result']z$Expected Node for mutated_view, got z7Expected FakeTensor in mutated_view.meta['fake_result']zn.target should not be a stringrCr.NN)rsortedrI_VIEW_INVERSE_MAPrHrr rOrPrGrstrkwargsr Exception) rrrview_inverse_nodesrbase mutated_view original_view self_aliasself_alias_baseview_replay_metadataexpected_metadatas && r_get_view_inverse_node_usagesrs  %+G H H 88, , vvayvvay $%% #?T |!LM M$))M2J?? !RS S,-- 6tL7I6JK ,++M:JGG I  ahh $ $ !BC C*!((3 &J  /(ooi8O (5#((7(:;&&*(HI($%/OOM$B!)#((7=9Q,-AUU&**1-V#'-I\   s A;H3H33 IITc aa%a&a'\V4P!V!VPPUu0uFspVPR8XgK\ VP R,\P4'gKI\VP R,P44kKu pp\\4o'VPPFEoRSP 9gKVV'3Rlp\P!VSP R,4KG \4pVPPEF<pVPR8XgK\ VP\P P"4'gKM\%VPP&P(4^8dK}\+VPP&P(^,P*4\P,JdKVP.^,p\P0!VP R,4p\P0!VP R,4pRp \%V4\%V48Xdv\3Wx4FfwrV P54V P548wdRp V P6V P68wdRp \P8!V 4^8XgKdRp Kh V 'dAVP\P:P<P>P@8wdEK\VP R,P44p W9dEK7\%VP.U u.uF qVJgK V NK up 4^8dEKk\VP R,P44p S'V ,p\CWP R,4p\EW4p\%VV, 4^8HpV'gEKVP\F9EdW%9Ed\FVP,pVPPIV4;_uu_4VP.^,pVP.R,pVPPKRVV3\MV4,VPN4pVPPKR\P:P<PPP@VV3/4R R R 4VPSV4M#\UVP4pVfEKVVn\VP R,P44pS'V ,PWS'V,4S'V,PWS'V ,4VPWV4\XPZ!V.V4EFo&S&P.^,o%S&P\Uu.uF0pVP R,VP R,8gK.VNK2 ppVEFpV%V&3R lp\_\`VVP.4Vn\_\`VVPN4Vn'\P0!S&P R,4p\P0!VP R,4pVU u0uF4p \ V \b4'gK\V P44kK6 pp VU u0uF4p \ V \b4'gK\V P44kK6 pp \%V4^8XgEK,\%V4^8XgEK?VV8XgEKI\P0!S%P R,4p V U u0uF4p \ V \b4'gK\V P44kK6 p!p \%V!4^8wd\eR \%V!4 24hV!wp"Vwp#S'V#,PWS'V",4S'V",PWS'V#,4EK EK EK? VFp$VPPgV$4K VPi4V#uupiuup i +'giEL;iuupiuup iuup iuup i) a Given an fx.GraphModule, modifies it to perform "reinplacing", mutating the nodes of the graph. We look for out-of-place op call sites like ``b = a.add(...)``, and convert them to be inplace (``b = a.add_(...)``), as long as the input to the current operator ("a") isn't reused anywhere later in the graph. This pass currently expects to operate on a **functional, ATen** graph. This can be obtained by running ``make_fx(functionalize(f))``. Sample inputs are needed to determine aliasing relationships of the inputs. In general, we can't reinplace node ``b = a.add(...)`` if "a" aliases any of the inputs to the program. Given a node ``b = foo(a, args...)`` the algorithm for re-inplacing is as follows: **(1)** Perform some initial checks on the metadata of "a" and "args..." that can disqualify them from being reinplaced. - **(1a)** Check that the self argument we're attempting to reinplace has acceptable dtype/size metadata to reinplace with. For example, if we have:: a = torch.ones(1) b = torch.ones(10) out = torch.add(a, b) We can't turn that into ``a.add_(b)`` because that would require resizing "a". Similarly, we can't convert ``torch.ge(a, b)`` into ``a.ge_(b)``, because that would require changing a's dtype (from e.g. float32 to bool). Note that in this specific example, we could technically do better.. If we see the pattern:: a_1 = a.ge(b) a_2 = aten._to_copy(a_1, a.dtype) Then this should be valid to completely re-inplace (this is exactly what functionalization will emit when it sees ``a.ge_(b)``). This optimization is only really important for user programs that directly use inplace comparison ops though. We also cannot re-inplace on tensors that have overlapping memory, e.g. ``torch.ones(1).expand(4, 4).add_(1)``. - **(1b)** Check if "a" is an alias of any of the program inputs. If it is, skip and move to the next node. Inplace'ing an op that would cause it to mutate a program is not sound, because that would be a side effect visible to the user. NOTE: there's a future optimization that we should make: if "a" is a (alias of a) program input, but later in the program there is a node that looks like ``a.copy_(...)``, then re-inplacing is ok to do - we are temporarily reusing a's buffer, which will later be overwritten by the ``copy_()`` call. This will be an important optimization to have for programs that mutate their inputs. It currently isn't implemented though. - **(1c)** Check if "a" and "args..." alias. For example, re-inplacing to create code like the below isn't guaranteed to be sound:: aten.mul_(a, a) **(2)** Check that "a" and all of its outstanding aliases are not used anywhere later in the graph. If this is the case, then it's safe to re-inplace to ``b = foo_(a)``. There are a few caveats to this, explained in more detail below: - (a) If "a" is used later as an argument to a view op, that is okay. It's only a problem if "a" (or that view) is later passed into a normal operator, or if it is returned as the program output. - (b) If "a" is a repeat argument in ``foo()``, then don't reinplace. Most ATen kernels don't make any guarantees that this is sound, e.g. if you do ``aten.mul_(a, a)``. So we'll just ban re-inplacing in this case. - (c) If "a" is used as an input into a view "inverse" / "scatter" operator, it is potentially fine to re-inplace (and remove that scatter operator from the graph). See below for a more detailed example. NOTE: there is an optimization in this step that is crucial to fully recovering performance from functionalization. Given this program:: def f(x): a = torch.ops.aten.add(x, x) b = torch.ops.aten.diagonal(a) torch.ops.aten.fill_(b, 0) return d Functionalization will emit the following:: def f(x): a = torch.ops.aten.add(x, x) b = torch.ops.aten.diagonal(a, 0, 1) b_updated = torch.ops.aten.fill(b, 0) a_updated = torch.ops.aten.diagonal_scatter(a, b_updated, 0, 1) return a_updated Ordinarily, we would not be able to reinplace the fill, because "b" aliases with "a" which is used by the diagonal_scatter call. "re-inplacing" is on the hook for figuring out that it is ok to completely remove the expensive diagonal_scatter call, if we re-inplace the add(). So, for every ``alias in alias_set(a)``, instead of checking that "alias" is not used anywhere later in the graph, we check that EITHER: - (a) alias is not used anywhere later in the graph, OR - (b) alias is used exactly once later on in the graph, in the following op:: out = foo_scatter(alias, x, args...) where the following must hold: - (i) ``foo_scatter`` is the "inverse" operator for foo. This only applies to "foo" ops that are view operators, which view into a subset of the original tensor's memory. In practice, there are ~4 operators where this applies:: diagonal -> diagonal_scatter slice -> slice_scatter select -> select_scatter as_strided -> as_strided_scatter - (ii) "args..." are the same between the ``foo()`` and ``foo_scatter()`` calls. **(3)** Perform the actual re-inplacing on foo! (3b) is the common case, but special care is needed for ``{view}_scatter`` (3a). - **(3a)** ``{view}_scatter`` ops. Consider this program:: a = torch.zeros(2, 2) b = torch.ones(2) a[0] = b Post functionalization, that will look like:: a = torch.zeros(2) b = torch.ones(1) a_updated = torch.select_scatter(a, b, 0, 0) In this case though, there is no "functional" op to re-inplace! Instead, we'd like to directly remove the select_scatter call. We already know from (3) that this is valid, because "a" has no later usages in the graph. We perform the re-inplacing on the ``{view}_scatter`` op like so. Before:: a_updated = torch.select_scatter(a, b, args...) After:: a_slice = a.select(a, args...) a_slice.copy_(b) - **(3b)** Otherwise, replace the functional op with its inplace variant. Before:: b = foo(a, args...) After:: a.foo_(args...) **(4)** Finally, after converting either:: # Before: # After: b = foo(a) foo_(a) or:: # Before: b = {slice}_scatter(a, mutated_slice, args...) # After: slice = {slice}(a, args...) slice.copy_(mutated_slice) We now need to find all later nodes that use "b" as an argument and update them to take in "a" instead. Note that for the majority of inplace ops, this isn't actually necessary (because most inplace ops return "self" as their output). This isn't generally true for all mutable ops though, which is why we need to actually replace all of the arguments. We also need to update our metadata of ``Dict[StorageWeakRef, Set[Node]]``, that maps a given tensor storage to the set of all nodes that take in that storage as an input. Specifically, re-inplacing ``b = foo(a)`` causes "a" and "b"'s sets to get fused together. **(5)** Any ``view_inverse/scatter`` nodes that were identified as "it's ok to ignore them" during step (3) get manually deleted from the graph. Their outputs are no longer used, so technically standard DCE would be able to do this, but we can no longer run FX's DCE pass now that we have mutable ops in the graph. placeholderr?c<\V\4'd2S\VP44,P S4R#R#r)rrr rUr)rrstorage_to_nodess&r _add_to_mapreinplace.._add_to_map6s6a,,$^A4D4D4F%GHLLQO-rrBFTr@rNc<VS8XdS#V#rr)rfnewolds&r replace_argreinplace..replace_args8#&J rzExpected 1 storage, got )5r8rhgraphnodesrNrrGr rbr rUrrpytree tree_map_rIr!r"r$r#r%rP TensorTyperH tree_leavesrxnumeldtype_debug_has_internal_overlaprJrKresizerMrrrinserting_before create_nodetuplerrLrrupdate itertoolschainrrr rrO erase_node recompile)(gm sample_argsr:input_storagesr&all_later_view_inverse_nodes_to_deleteself_argself_flattenednode_flattenedself_has_wrong_metadata self_meta node_metaself_arg_storagerrrlater_view_inverse_node_usages can_reinplaceview_opmutated_slice_noderemaining_slice_args slice_nodercurr_node_storagernodes_to_updatenode_to_updaterold_flattened_resnode_flattened_resold_res_storagenode_res_storagenew_flattened_resnew_res_storagenew_refnode_ref to_deleterrrs(&* ` @@@rrr6s5z#2&00+>.HHNN"D GG} $ B499]3U\\B Btyy/>>@A"9DC8H XX^^ AFF " P   [!&&*? @.1U* 77o %dkk5::+@+@AA4;;&&001A5DKK''11!499:%BRBRRyy|H#// m0LMN#// -0HIN&+ #>"c.&99,/,O(I (IOO,==26/ )//926/88CqH26/-P'4;;%))..:O:O:W:W+W . m,;;=   1tyy:y!MAAy:;a?- m,;;=  ,,<=L!;ii 3!  .K!. * 14R RSWXXM  00F+DKK8 XX..t44)-1&+/99R=(!#!5!5'! e,@&AA "J HH((' ,,44&.5&7::4@$9#E #+. !/ -(779!  - . 5 5 !23  . / 6 6 !12  3 9 9. !v/MNhhqk"yy#(!AFF:,>:AV,VAAy #'6N! +8k>+>+>+N'-:k>+@+@-N) )/(:(:388M;R(S%)/););&++M:*& "3'!2A%a4;q'7'7'9:!2$'"4(!3A%a4;q'7'7'9:!3%("O,1 01Q6+/??-3,>,>sxx ?V,W)&7+%6)!Z8?N1+;+;+=>%6(+ /14"0":3;O:P Q#&5 &6  )299:J7:ST(1889I(9STC'6 Oi~<  I&<LLN IsP;Z544n#0' (.+sZb6.b6.b6 b; b; ;B,c+c c1c c 1c c c# %c# c cV^8dQh/^\9d5\\R\3,\R\3,3,;R&#)r..r)__conditional_annotations__dictrr)r0s"rr1r1s<V4c*HS#X,>>?Wr)5rrRr collectionsrcollections.abcrenumrtypingrr torch._subclasses.fake_tensorrrtorch.fxr torch.fx._compatibilityr torch.multiprocessing.reductionsr torch.utilsr rtorch.utils._pytreer__all__rr,r5fx Interpreterr8r}rrJrKdiagonal_scatterrMdiagonalselect_scatterselectr slice_scatterslicerbas_strided_scatter as_stridedrrrrr1)rs@rrss#$ D1;)- -  0e,F+UXX%9%9F+-F+R+6'V IINN##++UYY^^-D-D-L-L IINN!!))599>>+@+@+D+D IINN  ((%))..*>*>*E*E IINN%%--uyy~~/H/H/P/P CB:zd+j,jr