Triton 中的多头注意力
今天写了一个小型 Triton 多头注意力 kernel:
import torch
import triton
import triton.language as tl
import math
@triton.jit
def MHA(Q, K, V, output, N, h, d_head, d_model, scale, HEAD_DIM: tl.constexpr, N_BLOCK: tl.constexpr):
row_pid = tl.program_id(0)
h_pid = tl.program_id(1) # head index since we partition by head
row_indexes = row_pid * N_BLOCK + tl.arange(0, N_BLOCK)
# NOTE: HEAD_DIM needs to be a constexpr to compile
h_indexes = h_pid * d_head + tl.arange(0, HEAD_DIM)
h_masks = tl.arange(0, HEAD_DIM) < d_head
# consider Q, K, V as N * d_head tensors
# when doing QK^T, we do not do tiling again. Instead, we just use full d_head as one tile
running_sum = tl.zeros([N_BLOCK], dtype=tl.float32)
running_max = tl.full([N_BLOCK], float('-inf'), dtype=tl.float32)
# load complete d_head as a tile
q_offsets = row_indexes[:, None] * d_model + h_indexes[None, :]
row_mask = row_indexes < N
q_mask = row_mask[:, None] & h_masks[None, :]
q_tile = tl.load(Q + q_offsets, mask=q_mask, other=0.0)
out_acc = tl.zeros((N_BLOCK, HEAD_DIM), dtype=tl.float32)
for n in tl.range(0, N, N_BLOCK):
acc = tl.zeros((N_BLOCK, N_BLOCK), dtype=tl.float32)
# online softmax for each N_BLOCK columns in K
# use row_stride==h for simplicity
# nth tile of K
# h indexes are aligned with Q
k_indexes = n + tl.arange(0, N_BLOCK)
k_offsets = k_indexes[:, None] * d_model + h_indexes[None, :]
k_row_mask = k_indexes < N
k_mask = k_row_mask[:, None] & h_masks[None, :]
k_tile = tl.load(K + k_offsets, mask=k_mask, other=0.0)
acc = tl.dot(q_tile, tl.trans(k_tile)) * scale
acc = tl.where(row_mask[:, None] & k_row_mask[None, :], acc, float('-inf'))
# compute running max for this tile
current_max = tl.max(acc, axis=-1)
new_running_max = tl.maximum(running_max, current_max)
# compute error correction
alpha = tl.exp(running_max - new_running_max)
# compute increment to running sum
weights = tl.exp(acc - new_running_max[:, None])
incre = tl.sum(weights, axis=-1)
# update
running_sum = tl.fma(running_sum, alpha, incre)
running_max = new_running_max
# load v, same offsets as k because we are loading a complete head as a tile
v_offsets = k_offsets
v_mask = k_mask
v_tile = tl.load( V+ v_offsets, mask=v_mask, other=0.0)
weighted_v = tl.dot(weights, v_tile)
out_acc = tl.fma(out_acc, alpha[:, None], weighted_v)
out_acc /= running_sum[:, None]
out_offsets = q_offsets
out_mask = q_mask
tl.store(output + out_offsets, out_acc, mask=out_mask)
# Q, K, V, output are tensors on the GPU
def solve(
Q: torch.Tensor,
K: torch.Tensor,
V: torch.Tensor,
output: torch.Tensor,
N: int,
d_model: int,
h: int,
):
d_head = d_model // h
N_BLOCK = 32
HEAD_DIM = max(16, triton.next_power_of_2(d_head))
grid = (triton.cdiv(N, N_BLOCK), h)
# do this kind of compute in driver
scale= 1 / math.sqrt(d_head)
MHA[grid](
Q=Q,
K=K,
V=V,
output=output,
N=N,
h=h,
d_head=d_head,
d_model=d_model,
scale=scale,
HEAD_DIM=HEAD_DIM,
N_BLOCK=N_BLOCK
)
一个 program 负责一个 query 块和一个头
启动 grid 是二维的:
grid = (triton.cdiv(N, N_BLOCK), h)
第一个 program ID 选择一组 query 行,第二个选择注意力头:
row_pid = tl.program_id(0)
h_pid = tl.program_id(1)
因此,一个 Triton program 计算:
one head
one N_BLOCK x d_head slice of output
N_BLOCK = 32 时,每个 program 最多处理单个头的 32 个 query token:
output, one head
row_pid=0 row_pid=1 row_pid=2
+-----------+-----------+-----------+
head 0 | 32 x Dh | 32 x Dh | 32 x Dh |
+-----------+-----------+-----------+
head 1 | 32 x Dh | 32 x Dh | 32 x Dh |
+-----------+-----------+-----------+
head 2 | 32 x Dh | 32 x Dh | 32 x Dh |
+-----------+-----------+-----------+
行偏移为:
row_indexes = row_pid * N_BLOCK + tl.arange(0, N_BLOCK)
头偏移为:
h_indexes = h_pid * d_head + tl.arange(0, HEAD_DIM)
kernel 假设 Q、K、V、output 都按 [N, d_model] 存储,头维度包含在 d_model 中。同一个 token 的所有头在一行内相邻排列:
token row
+---------+---------+---------+---------+
| head 0 | head 1 | head 2 | head 3 |
+---------+---------+---------+---------+
对于固定的 h_pid,h_indexes 选择对应切片。
为什么 HEAD_DIM 与 d_head 分开?
Triton 要求块形状是编译期常量。真正的头大小是运行时数据:
d_head = d_model // h
而 tile 形状为:
HEAD_DIM = max(16, triton.next_power_of_2(d_head))
因此分配的块可能略大于真实头大小。例如 d_head = 48 时,HEAD_DIM = 64,额外 lane 用 mask 排除:
h_masks = tl.arange(0, HEAD_DIM) < d_head
q_mask = row_mask[:, None] & h_masks[None, :]
这是常见 Triton 模式:
- 使用静态的 2 的幂块形状,方便编译。
- 用 mask 排除额外元素,使其不参与实际计算。
加载 Q
遍历 key/value 之前,只加载一次 query tile:
q_offsets = row_indexes[:, None] * d_model + h_indexes[None, :]
q_tile = tl.load(Q + q_offsets, mask=q_mask, other=0.0)
其形状为:
q_tile: [N_BLOCK, HEAD_DIM]
它包含当前输出行、当前头的 query 向量。在 program 流式遍历所有 key/value 块时,它始终保持活跃。
流式遍历 K 和 V
循环按块遍历序列维度:
for n in tl.range(0, N, N_BLOCK):
k_indexes = n + tl.arange(0, N_BLOCK)
对于每一块,program 加载:
k_tile: [N_BLOCK, HEAD_DIM]
v_tile: [N_BLOCK, HEAD_DIM]
分数 tile 为:
acc = tl.dot(q_tile, tl.trans(k_tile)) * scale
其形状为:
acc: [N_BLOCK, N_BLOCK]
每行对应当前 query 块中的一个 token,每列对应当前 key 块中的一个 token。
概念上,program 执行的是:
for this head:
for 32 query tokens:
compare against key tokens 0..31
compare against key tokens 32..63
compare against key tokens 64..95
...
mask 将无效行或 key 位置设为 -inf:
acc = tl.where(row_mask[:, None] & k_row_mask[None, :], acc, float('-inf'))
当 N 不能被 N_BLOCK 整除时,尾部块需要这样处理。
与简单 Softmax Attention 的区别
有趣的部分并不是在线 Softmax,它与前面的 attention/softmax kernel 相同:扫描 key 块时维护每行最大值与分母。
区别在于 tile 形状。简单 attention 通常从如下注意力矩阵出发:
scores: [N, N]
只沿序列维度分块。多头注意力的逻辑输入则是:
Q, K, V: [N, h, d_head]
这个 kernel 将一个 program 分配给一个头和一个 query 块,并将整个头维度作为特征 tile:
q_tile: [N_BLOCK, HEAD_DIM]
k_tile: [N_BLOCK, HEAD_DIM]
v_tile: [N_BLOCK, HEAD_DIM]
因此 d_head 不再拆分,QK^T 的归约维度覆盖整个头:
[N_BLOCK, HEAD_DIM] x [HEAD_DIM, N_BLOCK] -> [N_BLOCK, N_BLOCK]
第二次乘法消费 value tile:
[N_BLOCK, N_BLOCK] x [N_BLOCK, HEAD_DIM] -> [N_BLOCK, HEAD_DIM]
这让实现与数学表达十分接近:
for one head:
Q block compares with every K block
softmax weights immediately multiply the matching V block
final output is one [N_BLOCK, d_head] tile
头维度为 32、64、128 等较小数值时,这种模式很直接。如果 d_head 很大,q_tile、v_tile 和 out_acc 都携带完整头维度,tile 就会变得沉重。
直接进行指针运算的版本
第一版手动选择头切片:
h_indexes = h_pid * d_head + tl.arange(0, HEAD_DIM)
h_masks = tl.arange(0, HEAD_DIM) < d_head
每次加载都通过广播行索引和头索引构建偏移:
q_offsets = row_indexes[:, None] * d_model + h_indexes[None, :]
q_mask = row_mask[:, None] & h_masks[None, :]
q_tile = tl.load(Q + q_offsets, mask=q_mask, other=0.0)
K 和 V 的加载也一样:
k_offsets = k_indexes[:, None] * d_model + h_indexes[None, :]
k_mask = k_row_mask[:, None] & h_masks[None, :]
k_tile = tl.load(K + k_offsets, mask=k_mask, other=0.0)
这种方式显式且容易调试,但每个指针表达式都混入了边界处理。每个 tile 都要记住两种 mask:
- 序列 mask:
row_indexes < N或k_indexes < N。 - 头维度 mask:
tl.arange(0, HEAD_DIM) < d_head。
小 kernel 尚可,但布局一变,代码就容易繁杂。
更清晰的块指针版本
第二版在启动前修改数据布局:
Q = Q.reshape(N, h, d_key).permute(1, 0, 2).contiguous()
K = K.reshape(N, h, d_key).permute(1, 2, 0).contiguous()
V = V.reshape(N, h, d_key).permute(1, 0, 2).contiguous()
output = output.reshape(N, h, d_key).permute(1, 0, 2)
现在头位于最外层:
Q: [h, N, d_key]
K: [h, d_key, N]
V: [h, N, d_key]
output: [h, N, d_key]
这种布局更适合两次点积。Q 和 V 自然加载为 [BLOCK_SIZE_N, HEAD_DIM],K 则加载为 [HEAD_DIM, BLOCK_SIZE_N],无需先加载 [N_BLOCK, HEAD_DIM] 再在点积中转置:
k = tl.load(K_block_ptr, boundary_check=(0, 1), padding_option="zero")
qkt = tl.dot(q, k) * scale
更大的变化是用 tl.make_block_ptr 一次性描述 tile 形状和内存布局:
Q_block_ptr = tl.make_block_ptr(
base=Q_ptr + start_h * stride_qh,
shape=(N, d_key),
strides=(stride_qn, stride_qk),
offsets=(start_n * BLOCK_SIZE_N, 0),
block_shape=(BLOCK_SIZE_N, HEAD_DIM),
order=(1, 0),
)
它表达的是:
inside this head:
logical matrix shape is [N, d_key]
current tile starts at [start_n * BLOCK_SIZE_N, 0]
tile shape is [BLOCK_SIZE_N, HEAD_DIM]
于是加载变为:
q = tl.load(Q_block_ptr, boundary_check=(0, 1), padding_option="zero")
边界行为附着在 tile 本身。超出 N 或 d_key 的元素由 Triton 补零,从而移除手写加载 mask:
q_mask = row_mask[:, None] & h_masks[None, :]
改由块指针声明的形状表示:
shape=(N, d_key)
boundary_check=(0, 1)
padding_option="zero"
这样只需用逻辑张量形状表达一次有效区域,加载位置不必反复重建相同 mask。
推进 tile,而不是重建偏移
第一版每轮都重新构造 K、V 偏移:
k_indexes = n + tl.arange(0, N_BLOCK)
k_offsets = k_indexes[:, None] * d_model + h_indexes[None, :]
k_tile = tl.load(K + k_offsets, mask=k_mask, other=0.0)
块指针版本只创建一次 K、V tile 指针:
K_block_ptr = tl.make_block_ptr(
base=K_ptr + start_h * stride_kh,
shape=(d_key, N),
strides=(stride_kk, stride_kn),
offsets=(0, 0),
block_shape=(HEAD_DIM, BLOCK_SIZE_N),
order=(1, 0),
)
V_block_ptr = tl.make_block_ptr(
base=V_ptr + start_h * stride_vh,
shape=(N, d_key),
strides=(stride_vn, stride_vk),
offsets=(0, 0),
block_shape=(BLOCK_SIZE_N, HEAD_DIM),
order=(1, 0),
)
然后在内层循环推进逻辑 tile 位置:
K_block_ptr = tl.advance(K_block_ptr, (0, BLOCK_SIZE_N))
V_block_ptr = tl.advance(V_block_ptr, (BLOCK_SIZE_N, 0))
这与数学过程更直接地对应:
K moves across columns: [d_key, N]
V moves down rows: [N, d_key]
循环只需表达“将 tile 移到下一个 K/V 块”,块指针始终携带形状、步长、块形状和边界规则。
块指针改善了加载边界处理:
tl.load(K_block_ptr, boundary_check=(0, 1), padding_option="zero")
tl.load(V_block_ptr, boundary_check=(0, 1), padding_option="zero")
但 attention 的分数 tile 仍需要逻辑 mask:
qkt = tl.where((offset_m[:, None] < N) & (offset_n[None, :] < N), qkt, 0.0)
加载边界检查保护访存,分数 mask 保护数学语义,二者职责不同:
boundary_check:不读取逻辑 Q/K/V 范围之外的地址。tl.where:不让补齐的 query/key 位置参与 Softmax 分子。
一个细节是,本版本在应用分数 mask 之前就计算了 m_i:
m_i = tl.maximum(m, tl.max(qkt, axis=1))
qkt = tl.exp(qkt - m_i[:, None])
qkt = tl.where((offset_m[:, None] < N) & (offset_n[None, :] < N), qkt, 0.0)
越界 K 加载补零后,无效 key 列可能向行最大值贡献零分数。由于分子和分母一起缩放,最终归一化结果仍可能正确,但先屏蔽无效分数再求最大值会更清楚:
qkt = tl.where((offset_m[:, None] < N) & (offset_n[None, :] < N), qkt, -float("inf"))
m_i = tl.maximum(m, tl.max(qkt, axis=1))
qkt = tl.exp(qkt - m_i[:, None])
qkt = tl.where((offset_m[:, None] < N) & (offset_n[None, :] < N), qkt, 0.0)
完整代码:
import torch
import triton
import triton.language as tl
import math
@triton.jit
def mha_inner_fwd(
q, K_block_ptr, V_block_ptr, m, l,
offset_m, acc, N, scale, BLOCK_SIZE_N
):
for n_idx in range(0, tl.cdiv(N, BLOCK_SIZE_N)):
offset_n = n_idx * BLOCK_SIZE_N + tl.arange(0, BLOCK_SIZE_N)
k = tl.load(K_block_ptr, boundary_check = (0, 1), padding_option = 'zero')
qkt = tl.dot(q, k) * scale
m_i = tl.maximum(m, tl.max(qkt, axis = 1))
qkt = tl.exp(qkt - m_i[:, None])
qkt = tl.where((offset_m[:, None] < N) & (offset_n[None, :] < N), qkt, 0.0)
l_i = tl.sum(qkt, axis = 1)
alpha = tl.exp(m - m_i)
l = alpha * l + l_i
v = tl.load(V_block_ptr, boundary_check = (0, 1), padding_option = 'zero')
acc = alpha[:, None] * acc
acc = tl.dot(qkt, v, acc)
m = m_i
K_block_ptr = tl.advance(K_block_ptr, (0, BLOCK_SIZE_N))
V_block_ptr = tl.advance(V_block_ptr, (BLOCK_SIZE_N, 0))
return l, acc
@triton.jit
def mha_fwd(
Q_ptr, K_ptr, V_ptr, output_ptr,
N, d_model, h, d_key, scale,
stride_qh, stride_qn, stride_qk,
stride_kh, stride_kk, stride_kn,
stride_vh, stride_vn, stride_vk,
stride_oh, stride_on, stride_ok,
BLOCK_SIZE_N: tl.constexpr,
HEAD_DIM: tl.constexpr,
):
start_n = tl.program_id(axis = 0)
start_h = tl.program_id(axis = 1)
offset_m = start_n * BLOCK_SIZE_N + tl.arange(0, BLOCK_SIZE_N)
Q_block_ptr = tl.make_block_ptr(
base = Q_ptr + start_h * stride_qh,
shape = (N, d_key),
strides = (stride_qn, stride_qk),
offsets = (start_n * BLOCK_SIZE_N, 0),
block_shape = (BLOCK_SIZE_N, HEAD_DIM),
order = (1, 0),
)
K_block_ptr = tl.make_block_ptr(
base = K_ptr + start_h * stride_kh,
shape = (d_key, N),
strides = (stride_kk, stride_kn),
offsets = (0, 0),
block_shape = (HEAD_DIM, BLOCK_SIZE_N),
order = (1, 0),
)
V_block_ptr = tl.make_block_ptr(
base = V_ptr + start_h * stride_vh,
shape = (N, d_key),
strides = (stride_vn, stride_vk),
offsets = (0, 0),
block_shape = (BLOCK_SIZE_N, HEAD_DIM),
order = (1, 0),
)
m = tl.full((BLOCK_SIZE_N,), -float('inf'), dtype = tl.float32)
l = tl.zeros((BLOCK_SIZE_N,), dtype = tl.float32)
acc = tl.zeros((BLOCK_SIZE_N, HEAD_DIM), dtype = tl.float32)
q = tl.load(Q_block_ptr, boundary_check = (0, 1), padding_option = 'zero')
l, acc = mha_inner_fwd(
q, K_block_ptr, V_block_ptr, m, l,
offset_m, acc, N, scale, BLOCK_SIZE_N
)
acc = acc / l[:, None]
offset_d = tl.arange(0, HEAD_DIM)
output_ptrs = output_ptr + start_h * stride_oh + offset_m[:, None] * stride_on + offset_d[None, :] * stride_ok
tl.store(output_ptrs, acc, mask = (offset_m[:, None] < N) & (offset_d[None, :] < d_key))
# Q, K, V, output are tensors on the GPU
def solve(
Q: torch.Tensor,
K: torch.Tensor,
V: torch.Tensor,
output: torch.Tensor,
N: int,
d_model: int,
h: int,
):
assert d_model % h == 0, "d_model must be divisible by num_head"
d_key = d_model // h
BLOCK_SIZE_N = 32
HEAD_DIM = max(16, triton.next_power_of_2(d_key))
Q = Q.reshape(N, h, d_key).permute(1, 0, 2).contiguous()
K = K.reshape(N, h, d_key).permute(1, 2, 0).contiguous()
V = V.reshape(N, h, d_key).permute(1, 0, 2).contiguous()
output = output.reshape(N, h, d_key).permute(1, 0, 2)
grid = (triton.cdiv(N, BLOCK_SIZE_N), h)
scale = 1 / math.sqrt(d_key)
mha_fwd[grid](
Q, K, V, output, N, d_model, h, d_key, scale,
Q.stride(0), Q.stride(1), Q.stride(2),
K.stride(0), K.stride(1), K.stride(2),
V.stride(0), V.stride(1), V.stride(2),
output.stride(0), output.stride(1), output.stride(2),
BLOCK_SIZE_N, HEAD_DIM, num_warps = 2, num_stages = 1
)
output = output.transpose(0, 1).reshape(N, -1)
数据复用
query tile 在全部 key/value 块之间复用:
q_tile [32, d_head]
dot K block 0
dot K block 1
dot K block 2
...
每个 key tile 用于生成 32 x 32 分数 tile。对应权重算出后,立刻使用 value tile:
weighted_v = tl.dot(weights, v_tile)
因此,kernel 的访存行为为:
Q:每个 program 加载一次。K:流式遍历所有序列块。V:流式遍历所有序列块。output:最后写入一次。
关键是中间注意力矩阵不会被存储。直接实现单个头需要 N x N 分数矩阵,而这里每个 program 只保留 N_BLOCK x N_BLOCK 分数 tile,以及 N_BLOCK x d_head 输出累加器。
块指针版本保留相同复用模式,但表达更清楚:
Q tile: loaded once
K tile: block pointer advances across sequence
V tile: block pointer advances across sequence
O tile: one manual masked store at the end
最终存储仍手动完成:
output_ptrs = (
output_ptr
+ start_h * stride_oh
+ offset_m[:, None] * stride_on
+ offset_d[None, :] * stride_ok
)
tl.store(output_ptrs, acc, mask=(offset_m[:, None] < N) & (offset_d[None, :] < d_key))
这很合理。存储通常需要显式 mask,这里清楚限定了真实序列行和真实头维度 lane 才是有效输出。
形状的权衡
本版本沿序列分块,不沿头维度分块,会加载完整的 d_head:
q_tile: [N_BLOCK, HEAD_DIM]
k_tile: [N_BLOCK, HEAD_DIM]
v_tile: [N_BLOCK, HEAD_DIM]
out_acc: [N_BLOCK, HEAD_DIM]
这保持了代码简洁。两次点积为:
[N_BLOCK, HEAD_DIM] x [HEAD_DIM, N_BLOCK] -> [N_BLOCK, N_BLOCK]
[N_BLOCK, N_BLOCK] x [N_BLOCK, HEAD_DIM] -> [N_BLOCK, HEAD_DIM]
对于 32、64、128 等常见小头维度,这种模型很合理。d_head 很大时,累加器和 tile 成本升高,可能需要进一步拆分头维度。
N_BLOCK = 32 同样是调优选择。更大的块改善复用,却也增大分数 tile 和寄存器压力,因为 program 需要维护:
acc:N_BLOCK x N_BLOCK。weights:N_BLOCK x N_BLOCK。out_acc:N_BLOCK x HEAD_DIM。