跳转到正文
Retep's
返回

[TIL] 多头注意力

编辑文章

本文由英文原文翻译。代码、公式与技术名称保留原文。

Triton 中的多头注意力

今天写了一个小型 Triton 多头注意力 kernel:

import torch
import triton
import triton.language as tl
import math

@triton.jit
def MHA(Q, K, V, output, N, h, d_head, d_model, scale, HEAD_DIM: tl.constexpr, N_BLOCK: tl.constexpr):
    row_pid = tl.program_id(0)
    h_pid = tl.program_id(1) # head index since we partition by head

    row_indexes = row_pid * N_BLOCK + tl.arange(0, N_BLOCK)
    # NOTE: HEAD_DIM needs to be a constexpr to compile
    h_indexes = h_pid * d_head + tl.arange(0, HEAD_DIM)

    h_masks = tl.arange(0, HEAD_DIM) < d_head


    # consider Q, K, V as N * d_head tensors
    # when doing QK^T, we do not do tiling again. Instead, we just use full d_head as one tile


    running_sum = tl.zeros([N_BLOCK], dtype=tl.float32)
    running_max = tl.full([N_BLOCK], float('-inf'), dtype=tl.float32)

    # load complete d_head as a tile
    q_offsets = row_indexes[:, None] * d_model + h_indexes[None, :]
    row_mask = row_indexes < N
    q_mask = row_mask[:, None] & h_masks[None, :]
    q_tile = tl.load(Q + q_offsets, mask=q_mask, other=0.0)


    out_acc = tl.zeros((N_BLOCK, HEAD_DIM), dtype=tl.float32)
    for n in tl.range(0, N, N_BLOCK):
        acc = tl.zeros((N_BLOCK, N_BLOCK), dtype=tl.float32)
        # online softmax for each N_BLOCK columns in K
        # use row_stride==h for simplicity
        # nth tile of K
        # h indexes are aligned with Q
        k_indexes = n + tl.arange(0, N_BLOCK)
        k_offsets = k_indexes[:, None] * d_model + h_indexes[None, :]
        k_row_mask = k_indexes < N
        k_mask = k_row_mask[:, None] & h_masks[None, :]
        k_tile = tl.load(K + k_offsets, mask=k_mask, other=0.0)

        acc = tl.dot(q_tile, tl.trans(k_tile)) * scale
        acc = tl.where(row_mask[:, None] & k_row_mask[None, :], acc, float('-inf'))
        # compute running max for this tile
        current_max = tl.max(acc, axis=-1)
        new_running_max = tl.maximum(running_max, current_max)
        # compute error correction
        alpha = tl.exp(running_max - new_running_max)

        # compute increment to running sum
        weights = tl.exp(acc - new_running_max[:, None])
        incre = tl.sum(weights, axis=-1)

        # update
        running_sum = tl.fma(running_sum, alpha, incre)
        running_max = new_running_max

        # load v, same offsets as k because we are loading a complete head as a tile
        v_offsets = k_offsets
        v_mask = k_mask
        v_tile = tl.load( V+ v_offsets, mask=v_mask, other=0.0)

        weighted_v = tl.dot(weights, v_tile)
        out_acc = tl.fma(out_acc, alpha[:, None], weighted_v)

    out_acc /= running_sum[:, None]
    out_offsets = q_offsets
    out_mask = q_mask
    tl.store(output + out_offsets, out_acc, mask=out_mask)


# Q, K, V, output are tensors on the GPU
def solve(
    Q: torch.Tensor,
    K: torch.Tensor,
    V: torch.Tensor,
    output: torch.Tensor,
    N: int,
    d_model: int,
    h: int,
):
    d_head = d_model // h
    N_BLOCK = 32
    HEAD_DIM = max(16, triton.next_power_of_2(d_head))
    grid = (triton.cdiv(N, N_BLOCK), h)
    # do this kind of compute in driver
    scale= 1 / math.sqrt(d_head)
    MHA[grid](
        Q=Q,
        K=K,
        V=V,
        output=output,
        N=N,
        h=h,
        d_head=d_head,
        d_model=d_model,
        scale=scale,
        HEAD_DIM=HEAD_DIM,
        N_BLOCK=N_BLOCK
    )

一个 program 负责一个 query 块和一个头

启动 grid 是二维的:

grid = (triton.cdiv(N, N_BLOCK), h)

第一个 program ID 选择一组 query 行,第二个选择注意力头:

row_pid = tl.program_id(0)
h_pid = tl.program_id(1)

因此,一个 Triton program 计算:

one head
one N_BLOCK x d_head slice of output

N_BLOCK = 32 时,每个 program 最多处理单个头的 32 个 query token:

output, one head

          row_pid=0     row_pid=1     row_pid=2
        +-----------+-----------+-----------+
head 0  | 32 x Dh   | 32 x Dh   | 32 x Dh   |
        +-----------+-----------+-----------+
head 1  | 32 x Dh   | 32 x Dh   | 32 x Dh   |
        +-----------+-----------+-----------+
head 2  | 32 x Dh   | 32 x Dh   | 32 x Dh   |
        +-----------+-----------+-----------+

行偏移为:

row_indexes = row_pid * N_BLOCK + tl.arange(0, N_BLOCK)

头偏移为:

h_indexes = h_pid * d_head + tl.arange(0, HEAD_DIM)

kernel 假设 Q、K、V、output 都按 [N, d_model] 存储,头维度包含在 d_model 中。同一个 token 的所有头在一行内相邻排列:

token row
+---------+---------+---------+---------+
| head 0  | head 1  | head 2  | head 3  |
+---------+---------+---------+---------+

对于固定的 h_pid,h_indexes 选择对应切片。

为什么 HEAD_DIM 与 d_head 分开?

Triton 要求块形状是编译期常量。真正的头大小是运行时数据:

d_head = d_model // h

而 tile 形状为:

HEAD_DIM = max(16, triton.next_power_of_2(d_head))

因此分配的块可能略大于真实头大小。例如 d_head = 48 时,HEAD_DIM = 64,额外 lane 用 mask 排除:

h_masks = tl.arange(0, HEAD_DIM) < d_head
q_mask = row_mask[:, None] & h_masks[None, :]

这是常见 Triton 模式:

  1. 使用静态的 2 的幂块形状,方便编译。
  2. 用 mask 排除额外元素,使其不参与实际计算。

加载 Q

遍历 key/value 之前,只加载一次 query tile:

q_offsets = row_indexes[:, None] * d_model + h_indexes[None, :]
q_tile = tl.load(Q + q_offsets, mask=q_mask, other=0.0)

其形状为:

q_tile: [N_BLOCK, HEAD_DIM]

它包含当前输出行、当前头的 query 向量。在 program 流式遍历所有 key/value 块时,它始终保持活跃。

流式遍历 K 和 V

循环按块遍历序列维度:

for n in tl.range(0, N, N_BLOCK):
    k_indexes = n + tl.arange(0, N_BLOCK)

对于每一块,program 加载:

k_tile: [N_BLOCK, HEAD_DIM]
v_tile: [N_BLOCK, HEAD_DIM]

分数 tile 为:

acc = tl.dot(q_tile, tl.trans(k_tile)) * scale

其形状为:

acc: [N_BLOCK, N_BLOCK]

每行对应当前 query 块中的一个 token,每列对应当前 key 块中的一个 token。

概念上,program 执行的是:

for this head:
    for 32 query tokens:
        compare against key tokens 0..31
        compare against key tokens 32..63
        compare against key tokens 64..95
        ...

mask 将无效行或 key 位置设为 -inf:

acc = tl.where(row_mask[:, None] & k_row_mask[None, :], acc, float('-inf'))

当 N 不能被 N_BLOCK 整除时,尾部块需要这样处理。

与简单 Softmax Attention 的区别

有趣的部分并不是在线 Softmax,它与前面的 attention/softmax kernel 相同:扫描 key 块时维护每行最大值与分母。

区别在于 tile 形状。简单 attention 通常从如下注意力矩阵出发:

scores: [N, N]

只沿序列维度分块。多头注意力的逻辑输入则是:

Q, K, V: [N, h, d_head]

这个 kernel 将一个 program 分配给一个头和一个 query 块,并将整个头维度作为特征 tile:

q_tile:  [N_BLOCK, HEAD_DIM]
k_tile:  [N_BLOCK, HEAD_DIM]
v_tile:  [N_BLOCK, HEAD_DIM]

因此 d_head 不再拆分,QK^T 的归约维度覆盖整个头:

[N_BLOCK, HEAD_DIM] x [HEAD_DIM, N_BLOCK] -> [N_BLOCK, N_BLOCK]

第二次乘法消费 value tile:

[N_BLOCK, N_BLOCK] x [N_BLOCK, HEAD_DIM] -> [N_BLOCK, HEAD_DIM]

这让实现与数学表达十分接近:

for one head:
    Q block compares with every K block
    softmax weights immediately multiply the matching V block
    final output is one [N_BLOCK, d_head] tile

头维度为 32、64、128 等较小数值时,这种模式很直接。如果 d_head 很大,q_tile、v_tile 和 out_acc 都携带完整头维度,tile 就会变得沉重。

直接进行指针运算的版本

第一版手动选择头切片:

h_indexes = h_pid * d_head + tl.arange(0, HEAD_DIM)
h_masks = tl.arange(0, HEAD_DIM) < d_head

每次加载都通过广播行索引和头索引构建偏移:

q_offsets = row_indexes[:, None] * d_model + h_indexes[None, :]
q_mask = row_mask[:, None] & h_masks[None, :]
q_tile = tl.load(Q + q_offsets, mask=q_mask, other=0.0)

K 和 V 的加载也一样:

k_offsets = k_indexes[:, None] * d_model + h_indexes[None, :]
k_mask = k_row_mask[:, None] & h_masks[None, :]
k_tile = tl.load(K + k_offsets, mask=k_mask, other=0.0)

这种方式显式且容易调试,但每个指针表达式都混入了边界处理。每个 tile 都要记住两种 mask:

小 kernel 尚可,但布局一变,代码就容易繁杂。

更清晰的块指针版本

第二版在启动前修改数据布局:

Q = Q.reshape(N, h, d_key).permute(1, 0, 2).contiguous()
K = K.reshape(N, h, d_key).permute(1, 2, 0).contiguous()
V = V.reshape(N, h, d_key).permute(1, 0, 2).contiguous()
output = output.reshape(N, h, d_key).permute(1, 0, 2)

现在头位于最外层:

Q:      [h, N, d_key]
K:      [h, d_key, N]
V:      [h, N, d_key]
output: [h, N, d_key]

这种布局更适合两次点积。Q 和 V 自然加载为 [BLOCK_SIZE_N, HEAD_DIM],K 则加载为 [HEAD_DIM, BLOCK_SIZE_N],无需先加载 [N_BLOCK, HEAD_DIM] 再在点积中转置:

k = tl.load(K_block_ptr, boundary_check=(0, 1), padding_option="zero")
qkt = tl.dot(q, k) * scale

更大的变化是用 tl.make_block_ptr 一次性描述 tile 形状和内存布局:

Q_block_ptr = tl.make_block_ptr(
    base=Q_ptr + start_h * stride_qh,
    shape=(N, d_key),
    strides=(stride_qn, stride_qk),
    offsets=(start_n * BLOCK_SIZE_N, 0),
    block_shape=(BLOCK_SIZE_N, HEAD_DIM),
    order=(1, 0),
)

它表达的是:

inside this head:
    logical matrix shape is [N, d_key]
    current tile starts at [start_n * BLOCK_SIZE_N, 0]
    tile shape is [BLOCK_SIZE_N, HEAD_DIM]

于是加载变为:

q = tl.load(Q_block_ptr, boundary_check=(0, 1), padding_option="zero")

边界行为附着在 tile 本身。超出 N 或 d_key 的元素由 Triton 补零,从而移除手写加载 mask:

q_mask = row_mask[:, None] & h_masks[None, :]

改由块指针声明的形状表示:

shape=(N, d_key)
boundary_check=(0, 1)
padding_option="zero"

这样只需用逻辑张量形状表达一次有效区域,加载位置不必反复重建相同 mask。

推进 tile,而不是重建偏移

第一版每轮都重新构造 K、V 偏移:

k_indexes = n + tl.arange(0, N_BLOCK)
k_offsets = k_indexes[:, None] * d_model + h_indexes[None, :]
k_tile = tl.load(K + k_offsets, mask=k_mask, other=0.0)

块指针版本只创建一次 K、V tile 指针:

K_block_ptr = tl.make_block_ptr(
    base=K_ptr + start_h * stride_kh,
    shape=(d_key, N),
    strides=(stride_kk, stride_kn),
    offsets=(0, 0),
    block_shape=(HEAD_DIM, BLOCK_SIZE_N),
    order=(1, 0),
)

V_block_ptr = tl.make_block_ptr(
    base=V_ptr + start_h * stride_vh,
    shape=(N, d_key),
    strides=(stride_vn, stride_vk),
    offsets=(0, 0),
    block_shape=(BLOCK_SIZE_N, HEAD_DIM),
    order=(1, 0),
)

然后在内层循环推进逻辑 tile 位置:

K_block_ptr = tl.advance(K_block_ptr, (0, BLOCK_SIZE_N))
V_block_ptr = tl.advance(V_block_ptr, (BLOCK_SIZE_N, 0))

这与数学过程更直接地对应:

K moves across columns: [d_key, N]
V moves down rows:      [N, d_key]

循环只需表达“将 tile 移到下一个 K/V 块”,块指针始终携带形状、步长、块形状和边界规则。

块指针改善了加载边界处理:

tl.load(K_block_ptr, boundary_check=(0, 1), padding_option="zero")
tl.load(V_block_ptr, boundary_check=(0, 1), padding_option="zero")

但 attention 的分数 tile 仍需要逻辑 mask:

qkt = tl.where((offset_m[:, None] < N) & (offset_n[None, :] < N), qkt, 0.0)

加载边界检查保护访存,分数 mask 保护数学语义,二者职责不同:

一个细节是,本版本在应用分数 mask 之前就计算了 m_i:

m_i = tl.maximum(m, tl.max(qkt, axis=1))
qkt = tl.exp(qkt - m_i[:, None])
qkt = tl.where((offset_m[:, None] < N) & (offset_n[None, :] < N), qkt, 0.0)

越界 K 加载补零后,无效 key 列可能向行最大值贡献零分数。由于分子和分母一起缩放,最终归一化结果仍可能正确,但先屏蔽无效分数再求最大值会更清楚:

qkt = tl.where((offset_m[:, None] < N) & (offset_n[None, :] < N), qkt, -float("inf"))
m_i = tl.maximum(m, tl.max(qkt, axis=1))
qkt = tl.exp(qkt - m_i[:, None])
qkt = tl.where((offset_m[:, None] < N) & (offset_n[None, :] < N), qkt, 0.0)

完整代码:

import torch
import triton
import triton.language as tl
import math

@triton.jit
def mha_inner_fwd(
    q, K_block_ptr, V_block_ptr, m, l,
    offset_m, acc, N, scale, BLOCK_SIZE_N
):
    for n_idx in range(0, tl.cdiv(N, BLOCK_SIZE_N)):
        offset_n = n_idx * BLOCK_SIZE_N + tl.arange(0, BLOCK_SIZE_N)
        k = tl.load(K_block_ptr, boundary_check = (0, 1), padding_option = 'zero')
        qkt = tl.dot(q, k) * scale
        m_i = tl.maximum(m, tl.max(qkt, axis = 1))
        qkt = tl.exp(qkt - m_i[:, None])
        qkt = tl.where((offset_m[:, None] < N) & (offset_n[None, :] < N), qkt, 0.0)
        l_i = tl.sum(qkt, axis = 1)
        alpha = tl.exp(m - m_i)
        l = alpha * l + l_i
        v = tl.load(V_block_ptr, boundary_check = (0, 1), padding_option = 'zero')
        acc = alpha[:, None] * acc
        acc = tl.dot(qkt, v, acc)
        m = m_i 
        K_block_ptr = tl.advance(K_block_ptr, (0, BLOCK_SIZE_N))
        V_block_ptr = tl.advance(V_block_ptr, (BLOCK_SIZE_N, 0))
    
    return l, acc

@triton.jit
def mha_fwd(
    Q_ptr, K_ptr, V_ptr, output_ptr,
    N, d_model, h, d_key, scale,
    stride_qh, stride_qn, stride_qk,
    stride_kh, stride_kk, stride_kn,
    stride_vh, stride_vn, stride_vk,
    stride_oh, stride_on, stride_ok,
    BLOCK_SIZE_N: tl.constexpr,
    HEAD_DIM: tl.constexpr,
):
    start_n = tl.program_id(axis = 0)
    start_h = tl.program_id(axis = 1)
    offset_m = start_n * BLOCK_SIZE_N + tl.arange(0, BLOCK_SIZE_N)

    Q_block_ptr = tl.make_block_ptr(
        base = Q_ptr + start_h * stride_qh,
        shape = (N, d_key),
        strides = (stride_qn, stride_qk),
        offsets = (start_n * BLOCK_SIZE_N, 0),
        block_shape = (BLOCK_SIZE_N, HEAD_DIM),
        order = (1, 0),
    )
    K_block_ptr = tl.make_block_ptr(
        base = K_ptr + start_h * stride_kh,
        shape = (d_key, N),
        strides = (stride_kk, stride_kn),
        offsets = (0, 0),
        block_shape = (HEAD_DIM, BLOCK_SIZE_N),
        order = (1, 0),
    )
    V_block_ptr = tl.make_block_ptr(
        base = V_ptr + start_h * stride_vh,
        shape = (N, d_key),
        strides = (stride_vn, stride_vk),
        offsets = (0, 0),
        block_shape = (BLOCK_SIZE_N, HEAD_DIM),
        order = (1, 0),
    )

    m = tl.full((BLOCK_SIZE_N,), -float('inf'), dtype = tl.float32)
    l = tl.zeros((BLOCK_SIZE_N,), dtype = tl.float32)
    acc = tl.zeros((BLOCK_SIZE_N, HEAD_DIM), dtype = tl.float32)

    q = tl.load(Q_block_ptr, boundary_check = (0, 1), padding_option = 'zero')

    l, acc = mha_inner_fwd(
        q, K_block_ptr, V_block_ptr, m, l,
        offset_m, acc, N, scale, BLOCK_SIZE_N
    )

    acc = acc / l[:, None]
    offset_d = tl.arange(0, HEAD_DIM)
    output_ptrs = output_ptr + start_h * stride_oh + offset_m[:, None] * stride_on + offset_d[None, :] * stride_ok
    tl.store(output_ptrs, acc, mask = (offset_m[:, None] < N) & (offset_d[None, :] < d_key))

# Q, K, V, output are tensors on the GPU
def solve(
    Q: torch.Tensor,
    K: torch.Tensor,
    V: torch.Tensor,
    output: torch.Tensor,
    N: int,
    d_model: int,
    h: int,
):
    assert d_model % h == 0, "d_model must be divisible by num_head"
    d_key = d_model // h
    BLOCK_SIZE_N = 32
    HEAD_DIM = max(16, triton.next_power_of_2(d_key))

    Q = Q.reshape(N, h, d_key).permute(1, 0, 2).contiguous()
    K = K.reshape(N, h, d_key).permute(1, 2, 0).contiguous()
    V = V.reshape(N, h, d_key).permute(1, 0, 2).contiguous()
    output = output.reshape(N, h, d_key).permute(1, 0, 2)
    grid = (triton.cdiv(N, BLOCK_SIZE_N), h)
    scale = 1 / math.sqrt(d_key)
    mha_fwd[grid](
        Q, K, V, output, N, d_model, h, d_key, scale,
        Q.stride(0), Q.stride(1), Q.stride(2),
        K.stride(0), K.stride(1), K.stride(2),
        V.stride(0), V.stride(1), V.stride(2),
        output.stride(0), output.stride(1), output.stride(2),
        BLOCK_SIZE_N, HEAD_DIM, num_warps = 2, num_stages = 1
    )
    output = output.transpose(0, 1).reshape(N, -1)

数据复用

query tile 在全部 key/value 块之间复用:

q_tile [32, d_head]
    dot K block 0
    dot K block 1
    dot K block 2
    ...

每个 key tile 用于生成 32 x 32 分数 tile。对应权重算出后,立刻使用 value tile:

weighted_v = tl.dot(weights, v_tile)

因此,kernel 的访存行为为:

关键是中间注意力矩阵不会被存储。直接实现单个头需要 N x N 分数矩阵,而这里每个 program 只保留 N_BLOCK x N_BLOCK 分数 tile,以及 N_BLOCK x d_head 输出累加器。

块指针版本保留相同复用模式,但表达更清楚:

Q tile: loaded once
K tile: block pointer advances across sequence
V tile: block pointer advances across sequence
O tile: one manual masked store at the end

最终存储仍手动完成:

output_ptrs = (
    output_ptr
    + start_h * stride_oh
    + offset_m[:, None] * stride_on
    + offset_d[None, :] * stride_ok
)

tl.store(output_ptrs, acc, mask=(offset_m[:, None] < N) & (offset_d[None, :] < d_key))

这很合理。存储通常需要显式 mask,这里清楚限定了真实序列行和真实头维度 lane 才是有效输出。

形状的权衡

本版本沿序列分块,不沿头维度分块,会加载完整的 d_head:

q_tile:   [N_BLOCK, HEAD_DIM]
k_tile:   [N_BLOCK, HEAD_DIM]
v_tile:   [N_BLOCK, HEAD_DIM]
out_acc:  [N_BLOCK, HEAD_DIM]

这保持了代码简洁。两次点积为:

[N_BLOCK, HEAD_DIM] x [HEAD_DIM, N_BLOCK] -> [N_BLOCK, N_BLOCK]
[N_BLOCK, N_BLOCK] x [N_BLOCK, HEAD_DIM] -> [N_BLOCK, HEAD_DIM]

对于 32、64、128 等常见小头维度,这种模型很合理。d_head 很大时,累加器和 tile 成本升高,可能需要进一步拆分头维度。

N_BLOCK = 32 同样是调优选择。更大的块改善复用,却也增大分数 tile 和寄存器压力,因为 program 需要维护:


编辑文章
分享文章:

上一篇
[Book Club] FlashAttention:具备 IO 感知的快速、省内存精确注意力
下一篇
[Book Club] TileLang:连接现代神经网络 Kernel 的可编程性与性能