i0dZddlZddlmZddlmZddlmZddlZddl m Z m Z m Z m Z mZmZmZmZmZmZeddGd d eZy) zj Definition of CuTe inspired Layouts for DeviceMesh internal bookkeeping and functions to manipulate them N)Iterator) dataclass)product) as_tuplecoalesce complement compositionflattenIntTupleis_intis_tupleLayoutmatch_structureT)frozeninitczeZdZUdZeed<eed<ddZedefdZedefdZ ede e e e ffdZ ede e d ffd Zde fd Zd e ddffd ZddZddZddZde ddfdZde de ddddfdZdee fdZde deee fdZdefdZdej8dej8fdZxZS) _MeshLayoutax Utility class for representing an integer layout by borrowing ideas from CuTe Layout Algebra. See https://docs.nvidia.com/cutlass/media/docs/cpp/cute/02_layout_algebra.html for more details. Each layout is represented as a list of sizes and strides. We use it as a way for mechanical bookkeeping of the integers such as ranks in a SPMD mesh, and the transformation on top of it. Lots of methods of layout like coalesce, composition, complement, etc. are borrowed from pycute. https://github.com/NVIDIA/cutlass/blob/6dd13d42784ee5bfa232d2441e6b9a021c5c6290/python/pycute/layout.py#L137,L257 Note this is a CuTe-inspired layout, because CuTe uses co-lexicographic way in linearization while PyTorch is using lexicographic. So even though the CuTe documentation can still be referenced, the implementation will be different from that of PyCute's. shapestridereturnct|js6t|js!tdt |jt|j s6t|j s!tdt |j t |j|j s&td|jd|j dy)Nz"shape must be a tuple or int, got z#stride must be a tuple or int, got zsizes z and strides z don't match)r rr TypeErrortyperr ValueErrorselfs X/var/www/html/engine/venv/lib/python3.12/site-packages/torch/distributed/_mesh_layout.py __post_init__z_MeshLayout.__post_init__/s #F4::,>@djjAQ@RST T $VDKK-@A$t{{BSATUV Vtzz4;;7 M$++lK 8c|jSN)rrs rsizesz_MeshLayout.sizes9s zzrc|jSr!)rrs rstridesz_MeshLayout.strides=s {{rcftt|jt|jSr!)zipr rrrs rsizes_and_stridesz_MeshLayout.sizes_and_stridesAs!74::& (<==r.cPtfdttDS)Nc3DK|]}|jywr!)numel).0irs r z._MeshLayout.top_level_sizes..Gs?T!W]]_?s )tuplerangelenrs`rtop_level_sizesz_MeshLayout.top_level_sizesEs?eCI.>???rcRtjt|jSr!)mathprodr rrs rr*z_MeshLayout.numelIsyy,--rr,c  |t| ks|t|k\r7td|dt|dt| dt|dz d t| |}t |j |j S)NzDim z! is out of range for layout with z* dimensions. Expected dim to be in range [z, z].)r0 IndexErrorsuper __getitem__rrr)rr,layout __class__s rr9z_MeshLayout.__getitem__Ms D z>Q#d)^qc:3t9+F014T |2c$i!m_BP $Q'6<<77rcFt|jf|jfSr!)rrrrs rnestz_MeshLayout.nestVsDJJ=4;;.99rcXt|}t|j|jS)u A layout is represented by (sizes):(strides), e.g. (3,2):(4,2). Two consecutive dimensions can be "merged" into one if their strides are contiguous/multiplicative (i.e., the inner stride * inner size equals the next stride), we perform this kind of merge inside coalesce. Example 1 (simple): (3,2):(2,1) - inner dimension: has stride=1, size=2 - outer dimension: stride = inner_stride * inner_size = 2 → coalesced = (6:1) # acts like a flat 1D array of length 6 Example 2 (non-coalescible): (3,2):(4,1) - inner dimension: stride=1, size=2 → 2*1 = 2 - outer dimension: stride=4, mismatch (≠ 2) → cannot merge; result stays (3,2):(4,1) )rrrr)rr:s rrz_MeshLayout.coalesceYs""$6<<77rr:cZt||}t|j|jS)u By-dimension composition allows one layout to "select from" or "filter through" another layout. Think of it as function composition: (self ∘ layout)(input) = self(layout(input)) between two layouts. This function is a wrapper of pycute's composition. Mental model about how to understand the composition logic: - The LEFT layout (self) defines the "output space" - what indices are possible - The RIGHT layout (layout parameter) acts as a "selector" - which specific indices to pick - The composition only generates indices that the left layout could originally produce, but the right layout determines which indices to be picked. - The stride of the composition layout will not be smaller than the stride of the right layout, because when picking the indices the composition will at least follow the the right layout's stride to move forward. Example: self = (6,2):(2,1) # sizes=(6,2), strides=(2,1) layout = (3:2) # sizes=(3,), stride=(2,) self o layout = (3:2) Returns: Layout being composed. )r rrr)rr:results rr z_MeshLayout.compositionms%.T6*6<<77r world_sizecZt||}t|j|jS)uG Compute the "complement layout" relative to a given world_size. A complement layout fills in the "missing" factor so that: self repeat a layout of complement(self, world_size) will get a complete world_size. We use ⊗ to denote the repeat operation. Example: self = (4:1) # size=4, stride=1 world_size = 8 Then: complete needed factor = 8 / 4 = 2 complement(self, 8) = (2:1) Together they form: (4:1) ⊗ (2:1) = (4,2):(2,1) which has world_size = 4 * 2 = 8, as required. In distributed terms, complement() is often used to derive the "other" rank grouping when splitting processes into 2D meshes. For a visualized explanation, see https://x.com/ezyang/status/1962364978393981433/ )rrrr)rrAr:s rrz_MeshLayout.complements%,D*-6<<77rstartendc:tt|j}tt|j}tt|j|||tt|j|||t t |t |Sr!)listrr"r$rr.)rrCrDr:r"r$s rsplicez_MeshLayout.splicesqXdjj)*x -. 67eC!(6>>":;c5<w88rc tdt|jDDcgc]1}tdt |t|j D3c}Scc}w)a This function computes the all ranks specified by the layout staring from zero. How it works: 1. we enumerates every possible coordinate (like a nested for-loop). If sizes = (2, 3), we get the following coordinates: (0,0), (0,1), (0,2), (1,0), (1,1), (1,2) 2. For each coordinate, we compute a linear rank index as: all_ranks_from_zero = sum(coord[i] * strides[i] for i in range(ndim)) Example A: sizes = (2, 3) # 2 rows, 3 cols strides = (3, 1) # row-major layout coords = (0,0) -> 0*3 + 0*1 = 0 (0,1) -> 0*3 + 1*1 = 1 (0,2) -> 0*3 + 2*1 = 2 (1,0) -> 1*3 + 0*1 = 3 (1,1) -> 1*3 + 1*1 = 4 (1,2) -> 1*3 + 2*1 = 5 result = [0, 1, 2, 3, 4, 5] Example B: sizes = (2, 3) strides = (1, 2) # non-standard / strided layout coords = (0,0) -> 0*1 + 0*2 = 0 (0,1) -> 0*1 + 1*2 = 2 (0,2) -> 0*1 + 2*2 = 4 (1,0) -> 1*1 + 0*2 = 1 (1,1) -> 1*1 + 1*2 = 3 (1,2) -> 1*1 + 2*2 = 5 result = [0, 2, 4, 1, 3, 5] c32K|]}t|ywr!)r/)r+ss rr-z2_MeshLayout.all_ranks_from_zero..s"I58"Isc3,K|] \}}||zywr!)r+crJs rr-z2_MeshLayout.all_ranks_from_zero..sD$!QADs)rr r"sumr&r$)rcoords rall_ranks_from_zeroz_MeshLayout.all_ranks_from_zerosTH!"IWTZZ5H"IJ  D#eWT\\-B"CD D   s6Ac |j|jDcgc]#}|jDcgc]}||z c}%c}}Scc}wcc}}w)us Build global ranks specified by the layout via two-level ranks composition. The nested list forms the Cartesian product of all ranks for one layout and offset regarding filling up the world_size with the layout. The final global ranks are the addition of these two. The result is a list of lists: one sublist per layout. This rank list will be used to build the communicator underlying the layout and the given `world_size`. Example: world_size = 16 self.size = 4 self.stride = 1 ranks = [0, 1, 2, 3] offsets = [0, 4, 8, 12] result = [ [0+0, 0+1, 0+2, 0+3], # → [0, 1, 2, 3] [4+0, 4+1, 4+2, 4+3], # → [4, 5, 6, 7] [8+0, 8+1, 8+2, 8+3], # → [8, 9, 10,11] [12+0, 12+1, 12+2, 12+3], # → [12,13,14,15] ] )rrP)rrAoffsetranks r global_ranksz_MeshLayout.global_rankssQ2//*5IIK (,'?'?'A BtVd] B  B sA A AAcb|j}t|tt|k(S)u Check if the layout has any overlap between the ranks it generates. If there is overlap, we return False, otherwise True. The layout is supposed to be injective i.e, aside from indice 0, indices from each dim of the layout must be non-overlapping. Example 1 - Valid (no overlap): Layout: sizes=(2,3), strides=(6,1) - Dim 1: stride=1, span=3*1=3, covers indices [0,1,2] - Dim 0: stride=6, span=2*6=12, covers indices [0,6] → No overlap since 6 > 3 Example 2 - Invalid (overlap): Layout: sizes=(2,3), strides=(2,1) - Dim 1: stride=1, span=3*1=3, covers indices [0,1,2] - Dim 0: stride=2, span=2*2=4, covers indices [0,2] → Overlap! stride=2 < span=3, so indices [0,2] are duplicated Example 3 - Invalid (overlap): Layout: sizes=(4,2), strides=(1,1) - Dim 1: stride=1, span=4, covers indices [0,1,2,3] - Dim 0: stride=1, span=2, covers indices [0,1] → Overlap! stride is same for two dims, so indices [0,2] are duplicated Returns: bool: True if no overlap, False if overlap detected )rPr0set)rrankss rcheck_non_overlapz_MeshLayout.check_non_overlaps):((*5zSU_,,rrank_mapc|jdk(sJ|jsJ|j|jk\sJ|j |j}|j t |jt |jzt |jt |jzjdg|jS)a Leverage layout as an index for mesh tensor that re-maps the indexes after layout transformation to actual device ranks. With this method, the cute layout serves as the backend of indices bookkeeping for the mesh tensor when it comes to flatten, unflatten and slicing operations. The actual mesh tensor still represents the actual device assignment and ranks. We need this function to specify device allocation and create backend for a mesh. Although any transform of mesh tensors can be treated as a view or subset of mesh tensor, we do need to use the actual view or sub-tensor for DeviceMesh and its backend creation. The shape of the `rank_map` must be 1D and contiguous. Examples: Case 1 - Consecutive ranks, full world: original_mesh_tensor = [[0,1],[2,3]] # 2x2 mesh, ranks 0-3 world_size = 4 layout = Layout(2:2) Return: [[0,2],[1,3]] Case 2 - Non-consecutive ranks: original_mesh_tensor = [[10,20],[30,40]] # custom rank assignment world_size = 4 layout = Layout(2:2) Return: [[[10,30],[20,40]]] Args: rank_map: The concrete mesh tensor with actual device ranks Returns: torch.Tensor: A tensor representing the actual device allocation from rank_map r6) ndim is_contiguousr*cosizer as_stridedr r"r$reshaper1)rrYcomplement_layouts rremap_to_tensorz_MeshLayout.remap_to_tensor sD}}!!!%%'''~~4;;=000 OOHNN,<=x"" %++ ,wtzz/B B %-- .1F F  '"-++- -r)rN)rr)r:rrr)__name__ __module__ __qualname____doc__r __annotations__rpropertyr"r$rr.intr'r1r*r9r=rr rrGrFrPrTboolrXtorchTensorrb __classcell__)r;s@rrrsT  O x>8E#s(O#<>>@sCx@@.s.8S8]8:8(848S8]829C9c9=9]9% T#Y% N s tDI 8-4-@+- +-+-rr)rfr3collections.abcr dataclassesr itertoolsrrktorch.distributed._pycuterrrr r r r r rrrrLrrrrsO $!     $T"[-&[-#[-r