glossary/tooling/triton

Triton

also OpenAI Triton

A Python-embedded language and compiler for writing GPU kernels at the level of blocks of data rather than individual threads. It handles memory coalescing, shared memory and scheduling within a block, so a fused kernel that would take a CUDA expert a week is an afternoon. Most custom kernels in serving engines are Triton, with CUDA or CUTLASS reserved for the few that need the last percent.

Related

• glossary/
• Kernel fusion

  Combining several operations into one kernel so intermediate results stay in registers or on-chip memory instead of making a round trip to HBM.

• CUTLASS

  NVIDIA's C++ template library for building GEMM and related kernels that hit tensor core peak, and CuTe, its layer for describing the layouts and tilings that make that possible.

• GEMM

  General matrix multiply, the operation the projections and feed-forward layers of a transformer reduce to.

• Nsight

  NVIDIA's profilers.

• sources/
• Triton tutorials
• Triton: An Intermediate Language and Compiler for Tiled Neural Network Computations (Tillet et al., 2019)
• topics/
• #kernels#gpu