iB UddlZddlZddlZddlZddlmZmZmZddlm Z m Z m Z ddl m Z ddlmZmZmZmZmZddlZddlmZddlmZddlmZddlmZmZmZm Z m!Z!m"Z"ddl#m$Z$dd l%m&Z&m'Z'm(Z)m*Z*m+Z+m,Z,m-Z-m.Z.dd l/m0Z0m1Z1dd l2m3Z3dd l4m5Z5dd l6m7Z8ddl9m:Z:gdZ;dZdZ?e@eAZBee3eejeDeEeAfZFeeFeGeFeHeFeIeAdffZJeIeAeJfZKeGeKZLeIeAeeKeLffZMe@ZNe@eeOd<ejdZQe GddZRe GddeRZS dIdejdeAdeAdeUdeUd eBf d!ZVGd"d#ZWdJd$ZXddd%dejd&eHejjd'fd(eUd)ee@ejd*eeRd eSf d+Z[d,eIeAeJfd-eMd.eSd dfd/Z\d0eejejjfd1eAd efd2Z]d3eIeAefd.eSd eIeAeffd4Z^ejdejd.eSd eIeAeJffd5Z`ejdejd3eIeAeJfd.eSd e5fd6Zad7ejjd dfd8Zbd3eMd eIeAeJffd9Zcd7ejjd3eIeAeJfd.eSd eMfd:Zdejdejd;eHejjd'fd.eSd eMfd<Zedejd7ejjd-eMd.eSd eMf d=Zfejdejd;eHejjd'fd3eMd.eSd df d>Zgddd%dejd)ee@ejd*eeRd eIeAeJffd?Zhddd%dejd;eejjeejjfd)ee@ejd*eeRd eMf d@Ziddd%dejd;eejjeejjfd)ee@ejd*eeRd eHeIeAeJfeMff dAZjdejd3eeIejeIeAeJffeIeAeJffd eIeAeJffdBZkddCdejd,eIeAeJfd*eeRd e5fdDZlddCdejd;eejjeejjfd-eMd*eeRd df dEZmddCdejd;eejjeejjfd,eIeAeJfd-eMd*eeRd e5f dFZneddCdejd*eeRd dfdGZoeddCdejd;eHejjd'fd*eeRd dfdHZpy)KN)Callable GeneratorIterable)asdict dataclassfield)chain)Anycast no_type_checkOptionalUnion) ShardedTensor)_broadcast_state_dict_distribute_state_dict_flatten_state_dict_gather_state_dict_offload_state_dict_to_cpu_unflatten_state_dict)_CHECKPOINT_PREFIX)FullOptimStateDictConfigFullStateDictConfigFullyShardedDataParallelOptimStateDictConfigShardedOptimStateDictConfigShardedStateDictConfigStateDictConfig StateDictType)._get_module_fsdp_state_if_fully_sharded_moduleFSDP_WRAPPED_MODULE)DTensor)_IncompatibleKeys)DistributedDataParallel) tree_map_only) FQNS_T PrimitiveType ValueType DictValueTypeListDictValueTypeOptimizerStateTypeStateDictOptionsget_model_state_dictget_optimizer_state_dictget_state_dictset_model_state_dictset_optimizer_state_dictset_state_dict _flat_param param_groupsparamsstater'_patched_state_dictc#Ktj}tj d|rtjyy#|rtjwwxYwwN)gc isenableddisableenable) is_enableds a/var/www/html/engine/venv/lib/python3.12/site-packages/torch/distributed/checkpoint/state_dict.py _gc_contextr?QsDJJJL  IIK : IIK s)A$AA$A!!A$ceZdZUdZdZeed<dZeed<dZeed<dZ eed<dZ eed<dZ eed <dZ eed <d Z eed <y )r+ap This dataclass specifies how get_state_dict/set_state_dict will work. - ``full_state_dict``: if this is set to True, all the tensors in the returned state_dict will be gathered. No ShardedTensor and DTensor will be in the returned state_dict. - ``cpu_offload``: offload all the tensors to cpu. To prevent CPU OOM, if ``full_state_dict`` is also true, then only the rank0 will get the state_dict and all other ranks will get empty state_dict. - ``ignore_frozen_params``: if the value is True, the returned state_dict won't contain any frozen parameters -- the ``requires_grad`` is False. The default value is False. - ``keep_submodule_prefixes`` (deprecated): when ``submodules`` is not None, this option indicates whether to keep the submodule prefixes from the state_dict keys. or example, if the submodule is ``module.pretrain`` and the full FQN of the parameter is ``pretrain.layer1.weight`` of the param. When this option is True, the parameter's key in the returned state_dict will be ``pretrain.layer1.weight``. If the options is False, the key will be ``layer1.weight``. Note that if ``keep_submodule_prefixes`` is False, there may be conflicted FQNs, hence there should be only one submodule in ``submodules``. - ``strict``: the ``strict`` option when ``set_state_dict`` calls model.load_state_dict(). - ``broadcast_from_rank0``: when the option is True, rank0 should receive a full state_dict and will broadcast the tensors in the state_dict/ optim_state_dict one by one to other ranks. Other ranks will receive the tensors and shard according to the local shards in the model and optimizer. ``full_state_dict`` must be set to True when using this option. This option currently only supports DTensor, not the legacy ShardedTensor. Ffull_state_dict cpu_offloadignore_frozen_paramsTkeep_submodule_prefixesstrictbroadcast_from_rank0flatten_optimizer_state_dict_fqn_modifiersdsd_fqn_modifiersN)__name__ __module__ __qualname____doc__rAbool__annotations__rBrCrDrErFrGrIstrr>r+r+\s_"H"OT!K!&$&$(T(FD!&$&). $.-s-rRr+cheZdZUeeZeeeejfee ejffe d<eeZ eeeejfee ejffe d<ee Ze ee d<dZee d<dZee d<ej&Zee d<eeZeej2e d <y ) _StateDictInfo)default_factoryfqn_param_mappingshared_params_mappingsubmodule_prefixesT handle_model handle_optim fsdp_context fsdp_modulesN)rJrKrLrdictrVrrPtorchTensorr%rOrWsetrXrYrNrZ contextlib nullcontextr[rlistr\nnModulerQrRr>rTrTs d#t c5<<  fell"# %$ d#4 c5<<  fell"# %$$)#=C=L$L$'33L(3$)$$?L$ryy/?rRrTmodelnamerIskip_ddp_prefixskip_compiler_prefixreturnc|jtd}d|vr|hS|jd}g}|}t|D]\}} t |t r5| dk7rt d| d|j}|r:|j| Lt |tr|t|dz krW||dztk(rHdj|} t|t} | r| d} | jD chc]} | |  c} cSt|t}| tk7s|j| t|| }t |t j"j$j&r7| dk7rt d| d|j(}|rN|j| at+||r:t||j-| x} rt+|| r t|| }|j| | t.j0jj2k(r|t|dz k7st5d t|| } dj|jtdhScc} w) a This API is used to convert the name of a parameter to the FQNs. For FSDP without `use_orig_params`, the name of FlatParameter can be mapped to multiple original parameters. As a result, the return type of this function is `set[str]`. Args: module (nn.Module): the root model. name (str): the name skip_ddp_prefix (bool): whether to skip DDP's `module` prefix Returns: The canonical FQNs based on the model traversal. .modulezExpected 'module', got '' _orig_modzExpected '_orig_mod', got 'z-Expect `_extra_state` to be the last obj name)replacersplit enumerate isinstanceDDPAssertionErrorrnappendFSDPlen _FLAT_PARAMjoingetattr_fqnsr r^_dynamo eval_frameOptimizedModulerqhasattrgetrdmodules_EXTRA_STATE_KEY_SUFFIX RuntimeError)rfrgrIrhri obj_names fqn_obj_namescurr_obji curr_obj_nameprefix flat_paramfqn removed_fqns r> _get_fqnsrsS. <<*B /D $v  3IMH%i0(<= h $($'? a%PQQH"$$]3 $ '3y>A%%)AE*:k*I-0$X{;  &xq\F4>4D4DES6(3%(EEx)<=H 33$$]3"8]; %--":":"J"J K +$'B=/QR%STT))H'$$]3x!23"F'(4E"F"H"L"L!#;x5#*8[#A   / 1 1 I III**&'VWW"8];Q( C DD;Fs0 I0c eZdZy) _EXTRA_STATEN)rJrKrLrQrRr>rrsrRrc#Ktdtjdtdtffd |dEd{y7w)Nrncurr_fqnrjc3Kj||r|dnd}|jD]T\}}|vr t|r'|t|j vr|dd}n||}||Ed{Vt |j d|jdD] \}}||jvr||}||f"t|jdtjjtjjk7r7|tjjj}|t!fyy7׭w)NrmrlF)recurseget_extra_state)addnamed_childrenrr}valuesr named_buffersnamed_parameters_non_persistent_buffers_set __class__rdrerrrnrr) rnrrg submodulenew_fqnobjrIrvisited_moduless r>rz+_iterate_valid_model_state..recursesrF#%-hZq>2%446 3OD)O+ 12>GF,=>@GGII#3B-%Jtf-y'2 2 2 3   /1H1HQV1H1W  ID#v999! 4&)G3,    F$$&79R9R Syy(( )" 2::#4#4#L#L"MNG<>) )  ) 3sA;E>E?CErl)r`rdrerPr)rfrIrrs `@@r>_iterate_valid_model_staters?&)eO * *S *Y *Dub!!!s;AAA) submodulesoptionsoptims. optim_onlyrrc |rtjdtd|r |s td|xs t }i}i}t |D]\}}t |trt||} |j|} | 2ttt||j| ||||<n| j||<| D]} t |tr||| <t|j!D])\} } | D]} tt"j$| || <!+t} |ret|}|j'D]G\}}||vr t||} t)| dk7r t+d| jd| DI|j,r|j.s t1dt3j4|}|r|j.rat7|j8|j8 }t;|j8|j8xs |j, }t<j>}n z"_verify_options..Qs%@CQi%@sz?full_state_dict must be True when broadcast_from_rank0 is True.)offload_to_cpu rank0_only)rc3Ktj5tjddtt j ||||5dddddddy#1swYxYw#1swYyxYww)NignorezFSDP.state_dict_type)messagecategoryrnstate_dict_typestate_dict_configoptim_state_dict_config)warningscatch_warningsfilterwarnings FutureWarningryrrs r>$fsdp_state_dict_type_without_warningz=_verify_options..fsdp_state_dict_type_without_warningosy((* ''&<}))!$3&7,C       s4A;6A/ A#A/ A;#A, (A//A84A;rr)rVrWrXr[r\rYrZrQ),rwarnrrr+rrurrrr r`rPupdatecopyrcitemsr^r_ named_modulesrzrwrFrA ValueErrorryr\rrBrrFULL_STATE_DICTrrSHARDED_STATE_DICTracontextmanager functoolspartialrbrTrrdre)rfrrrrrVrWrgparamfqnsrparam_fqns_rXrnr\rrrrr[s r>_verify_optionsrs-  I   & I  +)+G  2%8/ e e\ * %##E* ? S,U3 4 ; ;D A+>O  " "  # $!(( 0+/$;  "--   / +3-!$ryy/<8#^&kAo  rRmodel_state_dictoptim_state_dictinfoc |jD]}t|}|td|jrk|si|js]|j sQ|j r |js9|jr-|js!tdtjd|jr4|s2|j r |js|jstd||D]}t|vs t|dtdy)Nz)Expected a fsdp_state with a fsdp module.z}The option indicates that model state_dict is required to save or load, but model state_dict is empty.rank = dist.get_rank()=rmzgThe option indicates that model state_dict is required to save, or load but optim state_dict is empty. z contains z6. This can happen if the model is not the root module.)r\rrwrYrXrCrBrArErFrdistget_rankrZr{)rrrrn fsdp_statekeys r>_verify_state_dictrs ##NCFK   !LM MN  ''))!!d&:&: KK)) 'mmo'q *    %%$*>*>..::J9KM    # %z+/** rRrapict||}|tvr+tjt|j||}|S)N)self)r}r6rrr)rrcalls r>_state_dict_fnrs9 3 D ""  !<3G KrR state_dictc|jrF|jrtjj sdnd}t ||j|S|jr t |S|S)NrQ)r)rB ranks_only)rArBr^ distributedis_initializedrr)rrrs r>_maybe_full_or_cpu_state_dictrsn $$E,=,=,L,L,N   " D$4$4     )*55rRc f|jsiS|j5t|d}dddtj D]}t ||}t |dk7rtd|dt |d|tt|}||k7sTdtfd}|||std|d ||j|||<|jrYi}|D]P}|jD]?}|j|s|jr ||||<*|t |d} |||| <AR|}|j rI|j#D]6\}} | j$rt ||}|D]}|j|8t'||S#1swYwxYw) NrrpExpected 1 FQN for key '', got z: rjc t|t|k\ry|jd}|jd}d}t|D]:\}}|||k(r'|dz }|t|k(s"|t|dz k(cS|dvr:yy)NFrmrrp)rnrqT)rzrsrt)rr fqn_split key_splitfqn_idxkey_idxkey_names r>verifyz%_get_model_state_dict..verifyss8s3x' IIcN IIcN )29)=%%GX9W#551 "c)n4#*c)nq.@#@@!%<< $%rRzAn unexpected key, z, exists. FQN is )rYr[rrckeysrrzrwnextiterrNrpoprX startswithrDrCr requires_gradr) rfrrrrrrnew_state_dictrrrs r>_get_model_state_dictrs      ;8^E<8: ;JOO%&2$ t9> *3%ws4ykD6J 4: #: D "#s#"%8=Nse#TUU(nnS1JsO=2@ /1 >C11 >~~f-//*4S/N3'!#f+-0G.8oN7+ > >$    002 $JC""UC(D $s# $  $ )T ::s;;s F&&F0c |jr|s|js tiiSi}t||jD]\}}t |||j}t |||jdd}t ||D]f\}} |jrtjdk(r9|| k7r4|j|d} | |jrtd|d| || <||| <hd} |js |jrnt} |jD]G}tj |s|j#dkDs-| j%|j&Itj&d| vr&| j)tj&dd} t+| dk(r.| j%tj,j/nt+| dkDr t1d |jr3t3||| j|j|j4 n(|jrt7||| j |j9||j;5t=tt?|d ||j| cdddS#1swYyxYw)NF)rhrirz Missing key: rmmetaTrpzMultiple devices found)devicerErBrload_state_dict)rrEassign) rYrFr"rrIrziprrrrErrAr`rr^ is_tensordimrrremoverzdistributed_c10d_get_pg_default_devicerrrBrrr[r r) rfrrlocal_state_dictrvaluerfqns_with_prefixrfqn_with_prefix load_valuerdevicess r>_load_model_state_dictr"sq   Z8Q8Q R((08N8NO6 UT%;%;<$    " "!!&  %(.>$? 6 C--A1E('^^C6 %{{*]3%q+ABB2_init_optim_statergsB {{ ))  ) Ezz%  ))5  ) 5E"""--e4  55 C)) ;  JJ{4( )k$/> S!   JJtJ))+ ;  # K + OOO%rRc < dtttfdtdtttff fd d i}t t |t jD]C\}}t d|}|j t tttf||Et t|tD]\}|jt}t tt|D]+}|jD]\}}||td|d|<-^|S)a3 This API flattens the optimizer state_dict to support optimizer resharding for MPMD, e.g., pipeline parallelism. Without the API, the original optimizer state_dict looks like: { "state": { "layer1.weight": { "step": 10, "exp_avg": SomeTensor, "exp_avg_sq": SomeTensor }, "layer2.weight": { "step": 10, "exp_avg": SomeTensor, "exp_avg_sq": SomeTensor }, }, "param_groups": [ { "lr": 0.0, "betas": (0.9, 0.95), ..., "params": ["layer1.weight", "layer2.weight"] } ] } With this API, the optimizer state_dict looks like: { "state.layer1.weight.step": 10, "state.layer2.weight.step": 10, "state.layer1.weight.exp_avg": SomeTensor, "state.layer2.weight.exp_avg": SomeTensor, "state.layer1.weight.exp_avg_sq": SomeTensor, "state.layer2.weight.exp_avg_sq": SomeTensor, "param_groups.layer1.weight.lr": 0.1, "param_groups.layer2.weight.lr": 0.1, "param_groups.layer1.weight.betas": (0.9, 0.95), "param_groups.layer2.weight.betas": (0.9, 0.95), } The "state" section supports arbitrary levels of nesting for optimizers like Shampoo. nested_dictrrjci}|jD]Q\}}t|}|r|d|n|}t|tr|j ||E||||<S|S)a Recursively flatten a nested dictionary with dot-separated keys. Args: nested_dict: The dictionary to flatten prefix: The prefix to prepend to all keys Returns: Flattened dictionary with dot-separated keys rm)rrPrur]r) rr flattenedrrstr_keyfull_key_flatten_state_nested_dict_raise_if_type_not_supporteds r>rz=_flatten_optim_state_dict.._flatten_state_nested_dicts+- %++- ,JC#hG06&7),GH%&  !;E8!LM-U3&+ (# ,rRct|tjttt fst dt|dy)Nz[Flattening optimizer state_dict only supports tensor, int, float, dict states now. Type is rm)rur^r_intfloatr]NotImplementedErrortype)vs r>rz?_flatten_optim_state_dict.._raise_if_type_not_supporteds@!ellC=>%7)1& ?rRrm)r]rPr r'r r(_STATErrr)_PGrr rc) rretrr5 state_prefixrrkr#rrs @@r>_flatten_optim_state_dictr)s R#s(^-0 c9n <!#C=*V*<=CCE U 3%(  &tDcNE'BL Q  -z#?, w'S 4( ,C#))+ ,1*+se1SE1#&' , ,, JrRc dtdtttfdtttffd}i}g}t|t|i}|j D]}|j tgi|tD]}|j|D]} | |jvr)d} |D]!} | tk(r td| d| } | |vrd} nnd} | s?|dt} t| tstd t| | j | |jsi|| <|j|D]M}td| d|}||vr |||}|t!t"|| |<5||t!t"|| |<O t!tt|dtd }|D]V} | tk(r |td|d| }| |dvr ||d| <.|d| |k7s:t%d |d| d |d |d| d |S)z This API unflattens the state_dict generated by _flatten_optim_state_dict(). Supports arbitrary levels of nesting in the state section through recursive reconstruction. See the docstring of _flatten_optim_state_dict() for more detail. flattened_keyflattened_dictrjc|d}i}|jD]m\}}|j|s|t|d}|jd}|}|ddD]%} | |vri|| <t || t sJ|| }'|||d<o|S)z Reconstructs a potentially nested value from flattened keys. For non-nested values, returns the value directly. For nested values, reconstructs the nested structure with string keys. rmNr)rrrzrsrur]) r+r,rrrr remaining_keypartscurrentparts r>_reconstruct_nested_dictz=_unflatten_optim_state_dict.._reconstruct_nested_dicts"?!$&( )..0 'JC>>&) F .M"'',E!Gcr  (w&$&GDM!'$-666!$-  ("'GE"I 9 '>rRFrmTrExpected list, got rzaAll the parameters in the same parameter group should have the same saved param_group value. But z is z while other(s) is )rPr]r'r$r%r3rxr rVrWrurcrwr"rr5r r(r)rrrr2r5pg_state return_osdrrr in_paramsr( flatten_keyr4 state_nameflattened_state_keyreconstructed_valuefirst_param_fqnrs r>_unflatten_optim_state_dictr<s//,0i,@/ c9n /bE"$H&,eS(%CJ))> " & ). E--e4- $444 %I(<$),Qse1QC&8 &*4(,I !%I !"g.!&$/(+>tF|n)MNN c"** c "'++e"4J-3HAcU!J<*H'**<.F//+0]E#J7 C GQ/G]E#J7 C=- . `tCy(2,w*?@C AG|#a'8!=>E $"' Q"aE)"==L@ rR optimizersc |jsiStitgi}|D]}t|t |d}|j r|j 5tj|||}ddd|s_t|tjD]9}d|vs|tj||t|jdd<;|tD]1}|tDcgc]}|jdd}}||t<3nGttjd|j D}t#t%|t't)|} i} |j+D]a\} } t-|| } t)| dk7rt/d| dt)| t1t3| }| | vrS| | }|| |<|| |<ct|tjD])} | | }|tj| |t|<+|tD]#}|tDcgc]}| | c}|t<%|sEt5t6|tj9|tt5t:|tj=|t|j>rt5t@tC|}tE||S#1swYxYwcc}wcc}w) Nrrqz _orig_mod.rlc3.K|] }|tywr8)r )rgs r>rz(_get_optim_state_dict..s-UQaj-Usrprr)#rZr$r%rrr\r[ryrrcrrrrr r from_iterabler3r]rrangerzrrrwrrr r(rr)extendrGr*r)r)rfr=rrrosdr(r@r4param_pid_mappingfqn_pid_mappingrrrrpidgroups r>_get_optim_state_dictrIws    ,2BR+@2H% 1nUL13   ""$ ?++E5#> ?#f+**,- R!#?B6{q?QCK , ;< RX $?@zJ!!))L"5JJ#'  $%---U%BTBT-UUVF $Ss6{1C%D E  O#446 + U ,t9>(23%ws4ykJ4:& 11'.'*$'*$ +CK,,./ 8%c*$'v;??3#7F C  8 S RBG.!Q3/#"6!Qg R  ],V45<} t'ttt| td k(s.|j | @t'|dk7r t+d t'|tt'|j k7r t+d t'|tdz |t) <wtt|tD]M}|j-t)|d}|dk(r$|j/D]\}}|tk(r||||<O|S) a Extract the corresponding optim state_dict from ``optim_state_dict`` for ``optim`` and return the result optim state_dict. Args: model (nn.Module): the root model. optim (torch.optim.Optimizer): the optimizer. optim_state_dict (Dict[str, ValueType]): the superset optim state_dict that contains the optim state_dict of ``optim``. info (_StateDictInfo): state dict information. Returns: The optim state_dict of ``optim``. c3<K|]}t|tywr8)rur)rr(s r>rz*_split_optim_state_dict..s U!:a  UsFTrr3z'Missing optimizer state for parameter 'z' in checkpoint. The parameter requires gradients but has no saved optimizer state. To load anyway, use StateDictOptions(strict=False).rprzThere are param groups that have zero parameters. In such a case, DSD only support exactly one param group with zero parameters.But the loaded state_dict has zero or more than one param groups that have zero parameters.z`When there is a parameter group that has zero parameters, multiple optimizers are not supported.)r$r%allr r(r3rxr rVrWr)rcrPrurwr"rrErrzidrrr)rfrrrr5r4r5 pg_mappingrrrr6loaded_param_groupr4r&pg_idxrrs r>_split_optim_state_dictrQsI*E"$H&,eS(%CJ!#J UtM;KF;S'T UU))7J " & ) VE--e4 V$444 %I.2)+;C+@/"*$tCy2DW2M"NN(,I! "!%I !"g.!&$/(+>tF|n)MNN c"&&d=2B62JKK%)-9I&9Q%RSV%Wc *EcUKRR +/%'7'<+V&d49.@.IJJ=@C=QTU=U 2&8#9: V7 V VD {7# $ )C&*+<>Ns>S&T 3"tDI'9''BCDIJJ12 33x1} 1#C()S1C1C-DD =25Z_1E1IJr,- .o7Jr-/?/DE * ;4 R< %++- *JCg~$)HV S !  * * rRc 0|jsy|D]}t||r@t|vrt||||}n+t |t t ttf||}ni}|jrl|jD]&\}}t||}t||d} || k(r't|dk7rtd|dt||j} | j} |tD]N} t t tt f| } | t"Dcgc]}|j%| | }}|| t"<Pt t&|t}t)|j+D]+}| |vs|j|||j%| | <-)|j-5t/j0|||}dddn|j2rd|_t5||f|}d|_dfd}t7t8j:||} tdt=|\}}t=|\}}|j>rtA|| ntC|| |D]*}||vs||vrtd |d ||||<||||<,tE||}|tD]/}t"|vs gt t ttf|t"<1tG|d | ycc}w#1swY%xYw)NF)rirpzExpected 1 FQN for 'rTc|jdkDr*|j|S|jk7r td|S)NrzDevice mismatch)rrr)trs r>_devicez'_load_optim_state_dict.._device^sD557Q;~!" 188+():;;rRzExpected device to be setrzExpected key 'z' in osd_mappingr)r)$rZrr$rQr<r r]rPr'r\rrrzrwrr%r r rrr(rcrr[ryoptim_state_dict_to_loadrArIr$r^r_rrFrrrr)rfr=rrrr original_fqn_rfqns_with_compilerrfqn_with_compilerr@valrr4 osd_stater(rrU flatten_osd osd_mappingflatten_local_osdlocal_osd_mapping optim_keypgrs @r>_load_optim_state_dictrc!st   [N% ##:5*d$ $?4S)^ 4jA4$  "    $)#9#9#; X a  5%.(.|nGCI;Ohhj$6$:$:$<!)#.*AtCH~q1CGJ7|@C C):;F$*CL * !0@0HI inn./XAaxGP}}UVGW !))C1B"CDX+ X2""$ #'#@#@5"2$    ! !#(D 4UUHdK #'D F ellG5EFA~$%@AA':;K'L $K3FGW3X 0 0((%k3DVT&{4EfU ) J $55 3,,YK7GH4?y3I%i03>y3I%i0 J 5!#4  's+ A"$>@Dc9n-r27; A 1u/0r,r,sV0     !  1=+R6    s +AA ct5t|tjjr|fn t |}t ||d||}t|||}ti|||cdddS#1swYyxYw)a Return the combined state_dict for optimizers. See ``get_state_dict`` for the detail usage. Args: model (nn.Module): the nn.Module to the model. optimizers (Union[None, Optimizer, Iterable[Optimizer]]): The optimizers that are used to optimize ``model``. submodules (deprecated): Optional[set[nn.Module]]: only return the model parameters that belong to the submodules. options (StateDictOptions): the options to control how model state_dict and optimizer state_dict should be returned. See `StateDictOptions` for the details. Returns: The state_dict for ``optimizers``. :rtype: OptimizerStateType TreN) r?rur^r OptimizertuplerrIr)rfr=rrrrs r>r-r-s6  *ekk&;&;<Mz"    !  1 DI2/6   s AA33A<ct5t|tjjr|fn t |}t ||d||}t||}t|||}t|||||fcdddS#1swYyxYw)a Return the model state_dict and optimizers state_dict. ``get_state_dict`` can process any module that is parallelized by PyTorch FSDP/fully_shard, DDP/replicate, tensor_parallel/parallelize_module, and any combination of these parallelisms. The main functions of ``get_state_dict`` are: 1.) returning a model and optimizer state_dict that can be resharded with a different number of trainers and/or different parallelisms. 2.) hiding the parallelism-specific state_dict APIs. Users don't have to call these APIs. 3.) sanity checking the result state_dict. The keys of the result state dictionary are the canonical FQNs (Fully Qualified Names). A canonical FQN refers to the FQN based on a parameter's position in an nn.Module hierarchy. More specifically, a canonical FQN to a parameter is the FQN returned by ``module.named_parameters()`` or ``module.named_buffers()`` when the module is not distributed by any parallelisms. Since the optimizer internally uses parameter IDs to represent a parameter, there will be a conversion from the parameter IDs to the canonical FQNs when calling this API. ``get_state_dict`` can also process a module that is not parallelized. In such a case, ``get_state_dict`` only performs one function -- converting the optimizer parameter IDs to the canonical FQNs. Example: >>> # xdoctest: +SKIP >>> import torch >>> from torch.distributed.fsdp import FullyShardedDataParallel as FSDP >>> from torch.nn.parallel import DistributedDataParallel as DDP >>> from torch.distributed.checkpoint.state_dict import get_state_dict >>> fsdp_model = FSDP(copy.deepcopy(model)) >>> fsdp_optim = torch.optim.Adam(model.parameters(), lr=1e-3) >>> ddp_model = DDP(copy.deepcopy(model)) >>> ddp_optim = torch.optim.Adam(model.parameters(), lr=1e-3) >>> ddp_state_dict, ddp_optim_state_dict = get_state_dict(ddp_model, ddp_optim) >>> fsdp_state_dict, fsdp_optim_state_dict = get_state_dict( ... fsdp_model, fsdp_optim ... ) >>> # if we simply call ddp_model.state_dict() and fsdp_model.state_dict(), >>> # the asserts will fail. >>> assert ddp_state_dict == fsdp_state_dict >>> assert ddp_optim_state == fsdp_optim_state_dict Args: model (nn.Module): the nn.Module to the model. optimizers (Union[None, Optimizer, Iterable[Optimizer]]): The optimizers that are used to optimize ``model``. submodules (deprecated): Optional[set[nn.Module]]: only return the model parameters that belong to the submodules. options (StateDictOptions): the options to control how model state_dict and optimizer state_dict should be returned. See `StateDictOptions` for the details. Returns: ``Tuple`` that contain model state_dict and optimizer state_dict. :rtype: typing.Tuple[typing.Dict[str, ValueType], OptimizerStateType] FreN) r?rur^rrgrhrrrIr)rfr=rrrrrs r>r.r.sP 2*ekk&;&;<Mz"    !  1=0 DI+-=tD!11!222s A,BB c |siSttt|jtj rt jdtdtttj tttff|}i}|jD]\}}|jD]}\}}||k7r t||}t!|dk7r t#dtt|d} |j%|jD cic] \} } | | z|  c} } |Sttttf|Scc} } w)NzPassing model_state_dict as a ``Dict[nn.Module, Dict[str, Any]]``is deprecated and will be removed in 2.5. If you need this feature, please preprocessing the model_state_dict to achieve the same functionality.rrrpz/FQNs for a submodule should only have 1 elementrm)rurrrrdrerrrr r]rPr'rrrrzrwr) rfrcast_state_dictrrsub_state_dictrgmrrsubfqnrs r>_unflatten_model_state_dictro6sF  $tJOO-./;  "   tBIItCN/C$CDjQ/1)8)>)>)@  %I~ ..0 a > -t9>(I!d,-Q/%%AOAUAUAWX Vf_e+X  Di(*55 YsE)rct||}t5t|dd|}t|i|t |||cdddS#1swYyxYw)a=Load the model state_dict. The counterpart of ``get_model_state_dict`` to set the state_dict to the model. See ``set_state_dict`` for the detail usage. Args: model (nn.Module): the nn.Module to the model. model_state_dict: (Dict[str, ValueType]): the model state_dict to load. If the key of the ``model_state_dict`` is nn.Module, the key is a submodule of ``model`` and the value should be the state_dict of the submodule. When loading the state_dict, the prefix of the submodule will be append to the state_dict. options (StateDictOptions): the options to control how model state_dict and optimizer state_dict should be loaded. See `StateDictOptions` for the details. Returns: ``NamedTuple`` with ``missing_keys`` and ``unexpected_keys`` fields: * **missing_keys** is a list of str containing the missing keys * **unexpected_keys** is a list of str containing the unexpected keys :type model_state_dict: typing.Dict[str, ValueType] rQFrrN)ror?rrr)rfrrrs r>r/r/[s`:.I . EubUGL+R6%e-=tD EEEs )A  Act5t|tjjr|fn t |}t ||d|}ti||t||||dddy#1swYyxYw)aLoad the optimizers state_dict. The counterpart of ``get_optimizer_state_dict`` to set the state_dict to the optimizers. See ``set_state_dict`` for the detail usage. WARN: ``set_optimizer_state_dict`` can only be called before ``backward()`` or after ``step()`` is called on the optimizers. Otherwise, the optimizer states won't be initialized correctly. Args: model (nn.Module): the nn.Module to the model. optimizers (Union[Optimizer, Iterable[Optimizer]]): The optimizers that are used to optimize ``model``. optim_state_dict: OptimizerStateType: the optimizer state_dict to load. options (StateDictOptions): the options to control how model state_dict and optimizer state_dict should be loaded. See `StateDictOptions` for the details. Returns: None :type optim_state_dict: typing.OptimizerStateType TrqN) r?rur^rrgrhrrrc)rfr=rrrs r>r0r0sz>  J*ekk&;&;<Mz"  ujT7S2/6uj2BDI J J Js AA11A:c.t||}t5t|tjj r|fn t |}t||| |}t|||t||||t|||cdddS#1swYyxYw)aLoad the model state_dict and optimizers state_dict. The counterpart of ``get_state_dict`` to set the state_dict to the model and optimizers. The given ``model_state_dict`` and ``optim_state_dict`` do not have to be returned by ``get_state_dict`` but must meet the following requirements: 1) all FQNs are canonical FQNs as defined in ``get_state_dict``, 2) if a tensor is sharded, it must be either a ShardedTensor or DTensor, 3) optimizer state_dict cannot contain the parameter IDs; the keys should be the canonical FQNs. WARN: ``set_state_dict`` can only be called before ``backward()`` or after ``step()`` is called on the optimizers. Otherwise, the optimizer states won't be initialized correctly. Args: model (nn.Module): the nn.Module to the model. optimizers (Union[Optimizer, Iterable[Optimizer]]): The optimizers that are used to optimize ``model``. model_state_dict: (Union[Dict[nn.Module, Dict[str, ValueType]], Dict[str, ValueType]]): the model state_dict to load. If the key of the ``model_state_dict`` is nn.Module, the key is a submodule of ``model`` and the value should be the state_dict of the submodule. When loading the state_dict, the prefix of the submodule will be append to the state_dict. optim_state_dict: OptimizerStateType: the optimizer state_dict to load. options (StateDictOptions): the options to control how model state_dict and optimizer state_dict should be loaded. See `StateDictOptions` for the details. Returns: ``NamedTuple`` with ``missing_keys`` and ``unexpected_keys`` fields: * **missing_keys** is a list of str containing the missing keys of the model state_dict. * **unexpected_keys** is a list of str containing the unexpected keys of the model state_dict. :type model_state_dict: typing.Dict[str, ValueType] :type optim_state_dict: typing.OptimizerStateType rqN) ror?rur^rrgrhrrrcr)rfr=rrrrs r>r1r1s\.I .  E*ekk&;&;<Mz"   :.>*>  +-=tDuj2BDI%e-=tD E E Es A*B  Bc$tjt||fd}||_tjt||dt t tfffd }||_tj|tj|y)aPatch the ``state_dict`` and ``load_state_dict`` attributes of ``model``. Patch the ``state_dict`` and ``load_state_dict`` attributes of ``model`` to be a partial function to call ``get_state_dict`` and ``set_state_dict``. Example: from torch.distributed.fsdp import FullyShardedDataParallel as FSDP from torch.distributed.checkpoint.state_dict import patch_model_state_dict model = fsdp(model) patch_model_state_dict(model) Args: model (nn.Module): the nn.Module to the model. options (StateDictOptions): the options to control how model state_dict and optimizer state_dict should be loaded. See `StateDictOptions` for the details. Returns: None )rfrcSr8rQ_state_dict_callsr>state_dict_callz0_patch_model_state_dict..state_dict_call !!rRrc|y)N)rrQr_load_state_dict_calls r>load_state_dict_callz5_patch_model_state_dict..load_state_dict_call z:rRN) rrr,rr/r]rPr rr6r)rfrrxr}r|rws @@r>_patch_model_state_dictrs6!(( "'E%-- ;c3h;1EO,01rRctjt|||fd}tjt|||dtt t fffd }tj|tj|t|tjjr|fn t|}|D]}||_||_y)aPatch the ``state_dict`` and ``load_state_dict`` attributes of ``optimizers``. Patch the ``state_dict`` and ``load_state_dict`` attributes of ``optimizers`` to be a partial function to call ``get_state_dict`` and ``set_state_dict``. Note that if there are multiple optimizers, all of the optimizers will be patched. So users only need to call one of the state_dict() to get the full result. Example: from torch.distributed.fsdp import FullyShardedDataParallel as FSDP from torch.distributed.checkpoint.state_dict import patch_model_state_dict model = fsdp(model) patch_model_state_dict(model) Args: model (nn.Module): the nn.Module to the model. options (StateDictOptions): the options to control how model state_dict and optimizer state_dict should be loaded. See `StateDictOptions` for the details. Returns: None )rfr=rcSr8rQrvsr>rxz4_patch_optimizer_state_dict..state_dict_callLryrRrc|y)N)rrQr{s r>r}z9_patch_optimizer_state_dict..load_state_dict_callVr~rRN)rrr-r0r]rPr r6rrur^rrgrhrr)rfr=rrxr}rr|rws @@r>_patch_optimizer_state_dictr&s>!((  "&--  ;c3h;O,01 j%++"7"7 8  :  5* 45rR)rHTT)rH)qrarr9rcollections.abcrrr dataclassesrrr itertoolsr typingr r r r rr^torch.distributedrrtorch.nnrd'torch.distributed._shard.sharded_tensorr#torch.distributed._state_dict_utilsrrrrrr;torch.distributed.algorithms._checkpoint.checkpoint_wrapperrtorch.distributed.fsdprrrryrrrrr$torch.distributed.fsdp._common_utilsrr torch.distributed.tensorr!torch.nn.modules.moduler"torch.nn.parallelr#rvtorch.utils._pytreer$__all__r{r%r r$r`rPr%r_rr r&rcrhr]r'r(r)r*r6rOrr?r+rTrerNrrrrrgrrrrno_gradrrrr)r<rIrQrcr,r-r.ror/r0r1rrrQrRr>rs 9900<< A   -5<- "    Sg}ellCKL 4 &m(TT  S)^$ '#u]4E%EFFG&)US]*  ,.,. ,.^  @% @  @&. !% HE 99HE HEHE HE  HE  HEV  %"Z,0*. A 99A %++'', -AA RYY( A & ' AAH+3 >*+(+ + +\bii)>)>>?chS#X&4 #s(^$?; 99?;*?; #y.?;?;DA 99A S)^$A  A  A A H'&U[[22'&t'&T_*<_c9nAU_DA ;; AS)^$A A AHBA 99BAekk++S01BA BA BABAJa 99a ;; a)a  a  aHdN 99dNekk++S01dN#dN  dN  dNdNT,0*. " 99" RYY(" & ' "  #y. " R,0*. * 99* ekk++Xekk6K6K-LLM* RYY( * & ' *  * b,0*. X2 99X2ekk++Xekk6K6K-LLMX2RYY( X2 & ' X2  4Y !3 34 X2v"6 99"6d299d3 >&::;T#y.=QQR"6 #y."6R+/ $E 99$E3 >*$E& ' $E  $EX+/ (J 99(Jekk++Xekk6K6K-LLM(J)(J & ' (J  (Jb+/ =E 99=Eekk++Xekk6K6K-LLM=E3 >* =E ) =E & ' =E=ED+/12 9912& '12 1212l +/ ;5 99;5ekk++S01;5& ' ;5  ;5;5rR