i hddlZddlZddlmZddlmZddlZddlmZm Z ddl m Z m Z m Z gdZde dee fd Zde d eedee fd Zd e defd Ze j,fd e dedede fdZd e defdZdd e dedzde fdZeded e dede fdZd e defdZd e dej:de fdZy)N)Any) deprecated)_ShardingIterDataPipeSHARDING_PRIORITIES)DataPipe DataPipeGraph traverse_dps)apply_random_seedapply_shardingapply_shuffle_seedapply_shuffle_settingsget_all_graph_pipesgraphreturnc*t|tSN)_get_all_graph_pipes_helperset)rs Y/var/www/html/engine/venv/lib/python3.12/site-packages/torch/utils/data/graph_settings.pyrrs &uce 44id_cachecg}|jD]J\}\}}||vr|j||j||jt ||L|Sr)itemsaddappendextendr)rrresultsdp_iddatapipe sub_graphs rrrsh!G(- I$$) H   Ux 29hGH I Nrrct|txs-t|dxrtj|j S)Nr ) isinstancerhasattrinspectismethodr rs r_is_sharding_datapiper')s; h 5 6 *+ 6   X44 5rnum_of_instances instance_idcHt|}ddfd ||S)z Apply dynamic sharding over the ``sharding_filter`` DataPipe that has a method ``apply_sharding``. RuntimeError will be raised when multiple ``sharding_filter`` are presented in the same branch. cP|jD]\}}d}t|rs|td|d|tj|j }t |jdkr|j n|j  |}||}||y)NzoSharding twice on a single pipeline is likely unintended and will cause data loss. Sharding already applied to z while trying to apply to )sharding_group)valuesr' RuntimeErrorr$ signaturer len parameters) r prev_applieddpr appliedsig_helperr)r(r-s rr7zapply_sharding.._helper=s"\\^ (MB G$R(+&77CnD^_a^bd ''(9(9:s~~&*%%&6 D%%(+n&& Iw '' (rr)rN)r )rr(r)r-rr7s ``` @rr r 0s'  "E((, EN Orct|dxrNt|dxr@tj|jxrtj|jS)N set_shuffleset_seed)r#r$r%r9r:r&s r_is_shuffle_datapiper;XsT-( 0 Hj ) 0   X11 2 0   X.. / rshufflec||St|}t|}|Dcgc]}t|s|}}|s,|r*tjdd|j }|g}|D]}|j ||Scc}w)aE Traverse the graph of ``DataPipes`` to find and set shuffle attribute. Apply the method to each `DataPipe` that has APIs of ``set_shuffle`` and ``set_seed``. Args: datapipe: DataPipe that needs to set shuffle attribute shuffle: Shuffle option (default: ``None`` and no-op to the graph) z`shuffle=True` was set, but the datapipe does not contain a `Shuffler`. Adding one at the end. Be aware that the default buffer size might not be sufficient for your task.) stacklevel)r rr;warningswarnr<r9)rr<r all_pipespipe shufflersshufflers rr r as  "E#E*I"+J$/CD/IJIJ   [ ##%  &W%& OKs A>A>z`apply_shuffle_seed` is deprecated since 1.12 and will be removed in the future releases. Please use `apply_random_seed` instead.)categoryrngct||Sr)r )rrGs rr r s Xs ++rc\t|dxrtj|jS)Nr:)r#r$r%r:r&s r_is_random_datapiperJs% 8Z ( PW-=-=h>O>O-PPrct|}t|}t}g}|D]G}t||vrt |s|j ||j t|I|D]_}ttjdtjj|j}|j|a|S)a6 Traverse the graph of ``DataPipes`` to find random ``DataPipe`` with an API of ``set_seed``. Then set the random seed based on the provided RNG to those ``DataPipe``. Args: datapipe: DataPipe that needs to set randomness rng: Random number generator to generate random seeds )dtype) generator)r rridrJrrinttorchemptyint64random_itemr:)rrGrrBcacherandom_datapipesrC random_seeds rr r s  "E#E*I EE  d8u   t $  # #D ) IIbh   !# KK%++ . 6 6 6 E J J L  k" # Orr)r$r@typingrtyping_extensionsrrQ(torch.utils.data.datapipes.iter.shardingrrtorch.utils.data.graphrrr __all__listrrrPrboolr'DEFAULTr r;r FutureWarningr rJ Generatorr rLrrrcsI( IH 5}5h5   $'H  (^ H'.. %%%%  %P8Xt xD .  ,,,,  ,Q(QtQu8r