i.DUdZddlmZddlZddlmZddlmZddlm Z m Z ddl m Z m Z dd lmZe rdd lmZddlZdd lmZd gZdad ed<eGddZed%dZ d& d'dZed(dZd)dZ d* d+dZ d,dZ d-dZe dZ d.dZ! d/ d0dZ" d1dZ#ddddddd d2dZ$dddd d3dZ% d4dd  d5d!Z&dd  d6d"Z'ejPd#e$y)7zUBER PROTOTYPE!!!) annotationsN) dataclass)cache)Any TYPE_CHECKING) TypeVarTupleUnpack) _registry) ModuleType)Libraryregister_flash_attention_fa4 str | None_FA4_MODULE_PATHc eZdZUded<ddZy) _FA4HandlezLibrary | Nonelibrarycd|_yN)r)selfs Q/var/www/html/engine/venv/lib/python3.12/site-packages/torch/nn/attention/_fa4.pyremovez_FA4Handle.remove"s  N)returnNone)__name__ __module__ __qualname____annotations__rrrrrs rrcJtjj|\}}|Sr)torchcudaget_device_capability)devicemajor_s r_get_device_majorr(&szz//7HE1 LrcBt|}|attS)z Register FA4 flash attention kernels with the PyTorch dispatcher. Args: module_path: Python module path to the FA4 implementation. )_fa4_import_modulerr_fa4_register_kernels) module_pathr's rrr,s# ;'A" +- ..rc~tj|}t|dr t|dstd|d|S)N_flash_attn_fwd_flash_attn_bwdzModule 'z' does not expose FA4 kernels) importlib import_modulehasattr RuntimeError)r,modules rr*r*;sA  $ $[ 1F 6, -WVEV5WXk]2OPQQ Mrctddd}|jdtd|jdtd|jdtd|jdt d|S)NatenIMPLCUDA_flash_attention_forward_flash_attention_backward#_scaled_dot_product_flash_attention,_scaled_dot_product_flash_attention_backward)r impl!_fa4_flash_attention_forward_impl"_fa4_flash_attention_backward_impl4_fa4_scaled_dot_product_flash_attention_forward_impl5_fa4_scaled_dot_product_flash_attention_backward_impl)libs rr+r+Csg &&& )CHH ')JFSHH (*LfUHH-< HH6= Jrctd|Dsyt|Dchc]}|jc}dk7ry|jtj tj fvry|D])\}}|jtjk7s$|dcS||jdk7ry||jd k7ry tjjsy t|jd vry ycc}w)Nc34K|]}|jywr)is_cuda.0ts r z,_fa4_common_support_error..Zs*Qqyy*szinputs must be CUDA tensorsr inputs must share devicez'query dtype must be float16 or bfloat16z dtype must be float32zdense query must be 4Dzragged query must be 3DzCUDA not available) z+FA4 requires compute capability 9.0 or 10.0) alllenr%dtyper"float16bfloat16float32dimr# is_availabler()querytensors cum_seq_q require_fp32rHnametensors r_fa4_common_support_errorr]Ts *'* *, g &AHH &'1,) {{5==%..998$3 f <<5== (V12 23UYY[A-'!1( :: " " $#&g5<  'sC:c|dk7ry|ry|y|+|jtjk7ry|jsyt ||||f|}||dk(ry|Sy) Ndropout_p must be 0zreturn_debug_mask must be Falsezalibi_slopes not supportedzseqused_k must be int32zseqused_k must be CUDArJz(query, key, value must be on same device)rQr"int32rEr]) rWkeyvalue dropout_preturn_debug_mask alibi_slopes seqused_krYerrors r_fa4_forward_support_errorrinsyC$0+ ??ekk ),  + % U E   . .= rc R|dk7ry|| yt|||||||f|d|ff} | | Sy)Nr_r`z windowed attention not supported logsumexp)rZ)r]) grad_outrWrbrcoutrkrdrYwindow_size_leftwindow_size_rightrhs r_fa4_backward_support_errorrpsYC$#'8'D1 % 5#uc95"I.0  E   rTsc&td|DS)Nc3@K|]}|jddyw)r N) transposerFs rrIz#_transpose_dense..s4qQ"4s)tuple)rXs r_transpose_denserws 4G4 44rc t tdtt} ||||d||| | jndd} | | | d<| j|||fi| \} } | | jfS)NFA4 not registeredT) softmax_scalecausalrnro return_lse cu_seqlens_q cu_seqlens_krgrm)rr3r* contiguousr.)rWrbrccu_seq_qcu_seq_kscale is_causalrnrorgrmr4kwargslses r_fa4_run_forwardrs/00  0 1F,.  /8/DY))+$ F u %v%%eS%B6BHC   rc t tdtt} | j||||||j || || \} } } | | | fS)Nry)rzr{r}r~)rr3r*r/r)rlrWrbrcrmrkrrrrr4dqdkdvs r_fa4_run_backwardrsq/00  0 1F''    ( JBB r2:r)rrnrorgrfrmc  t||||| || |}|td|t|||||| || | | | \}}tjdtj |j }tjdtj |j }tjd|j|j }|||||fS)Nz)FA4 flash_attention forward unsupported: )rt)rQr%r r) rir3rr"zerosuint64r%emptyrQ)rWrbrcrY cum_seq_kmax_qmax_krdrrerrnrorgrfrmrhr rng_state philox_offset debug_masks rr>r>s& '    E FugNOO      HC D U\\JIKK%,,u||LMQekk%,,GJ Y z 99r)rrnroc t||||||| ||| }|td|t|||||||||| \}}}|||fS)Nz*FA4 flash_attention backward unsupported: )rpr3r)rlrWrbrcrmrkrYrrrrdrrunusedrrnrorhrrrs rr?r?"s( (     E GwOPP"      JBB r2:rrct|||||ddd}|td|t|||\}} } tj|} | j dd} |j d} | j d}t|| | dd| ||||||  \}}}}}|j d}|j d}| |dd|||||f S)NzFA4 SDPA forward unsupported: r rt)rrm)rir3rwr" empty_likerusizer>)rWrbrcrdrrerrhqkvout_bhsdout_bshd max_q_flash max_k_flashr'rrrrrrs rr@r@Ss  '       E ;E7CDDuc51GAq! &H!!!Q'H&&)K&&)K3T       40AsI}j JJqME HHQKE       rc t||||||| ddd }|td|t|||||\}}}}}|jd}|jd} t ||||||dd|| | | | | |\}}}t|||\}}}|||fS)NzFA4 SDPA backward unsupported: rtr)rpr3rwrr?)rlrWrbrcrmrkrYrrrrdr philox_seedrrrhrrrogorrrs rrArAs$ (        E r?r@rAregister_flash_attention_implr rrrs%#!%2 !#  $*#   3 / / /*:<   %#7   4       & ##B        #!"6$5 $! ! ! !" ! " !  !!!!"!#! !'!B        "" 5T#'$(%)(,##/: /: /: /:# /: # /:  /: /:/:/:/: /:!/:"/:#/: &!/:" #/:D#'$(%.. . .  .  .  .#.#. . .... . !."!#.$"%.j# :: : : : :  :  : :Z!55 5 5  5  5  5#5#5 5 5555 5 !5p( '';WXr