Basics
The whole idea first, then the terms the rest of this page uses.
1.1Whole idea
@triton.jit doesn't run your function in Python. It parses the function into an intermediate form, lowers it through GPU-aware stages, and produces binary code at first launch. Later launches with the same shape reuse the cache.
Follows one Python function down to GPU binary, which explains caching, specialisation and autotuning.
1.2Key concepts
One idea per card, in plain words.
Compiles at first use, when argument types and constants are known.
Captures your kernel at the level of blocks and tensor operations.
Adds layout decisions: how data maps onto threads and warps.
Compiles a separate version per data type and constexpr value.
Question
What exactly you must answer, and what to settle before you start.
2.1Question
A Triton kernel takes seconds on its first call and milliseconds afterwards, then slows again when the block size changes. Explain why.
Detailed answer
How it works, and how to build or explain it.
3.1Step by step
The mechanism, in the order you would explain it.
- Parse the Python
Triton reads the function's source and builds its intermediate representation.
- Lower to TritonGPU IR
The compiler assigns layouts and fuses compatible operations.
- Lower to LLVM IR
Standard compiler optimisations run here.
- Emit PTX
LLVM's NVIDIA backend emits GPU assembly.
- Assemble to binary
ptxas produces the binary, which is cached on disk.
- Reuse the cache
Same source, types and constexprs load instantly. A new block size means a new compile.
3.2Code
A working implementation, then what each part does and why.
A fused softmax in Triton
You write one block of work, and Triton compiles it for the GPU.
1import torch2import triton3import triton.language as tl4 5 6@triton.jit7def softmax_kernel(output_ptr, input_ptr, n_elements, stride, BLOCK_SIZE: tl.constexpr):8 row = tl.program_id(0) # one program per row9 col = tl.arange(0, BLOCK_SIZE)10 mask = col < n_elements11 12 x = tl.load(input_ptr + row * stride + col, mask=mask, other=-float("inf"))13 x = x - tl.max(x, axis=0) # all in one pass: no extra trips to memory14 numerator = tl.exp(x)15 denominator = tl.sum(numerator, axis=0)16 17 tl.store(output_ptr + row * stride + col, numerator / denominator, mask=mask)18 19 20def softmax(x: torch.Tensor) -> torch.Tensor:21 rows, cols = x.shape22 block = triton.next_power_of_2(cols)23 out = torch.empty_like(x)24 softmax_kernel[(rows,)](out, x, cols, x.stride(0), BLOCK_SIZE=block,25 num_warps=4 if block <= 256 else 8)26 return outtl.program_id(0)- Each parallel program handles one row.
x - tl.max(x, axis=0)- Max, exp and sum run in one pass, with no extra trips to memory. That's the point of fusion.
mask- Handles rows that don't fill a power-of-two block.
Syntax-checked. The GPU libraries it needs don't run in our tests.
Q&A
What interviewers ask next. Answer out loud, then open.
4.1Interview questions
Why is the first call slow?
Why do many warp configs slow start-up?
How do you debug a slow kernel?
Extra depth
Edge cases, counterpoints and what to read next.
5.1Edge cases and counterpoints
Includes the strongest argument against this approach.
Show detailsHide details
- Cache in containers
- Ship ~/.triton/cache in your image, or every new replica pays the compile cost.
- Counterpoint
- Most engineers never need these stages. Learn them when kernels misbehave.