跳转到正文
Retep's
返回

[TIL] 学习笔记:TileLang 中的 RoPE

编辑文章

本文由英文原文翻译。代码、公式与技术名称保留原文。

TileLang 中的 RoPE

今天为使用 TileLang 实现 Qwen 推理的项目编写了一个旋转位置编码 kernel。

输入张量布局为:

X: [N, S, H, D]

RoPE 概览

RoPE 通过旋转成对的隐藏维度加入位置信息。对于位置 pp 的头向量,将其分成两半:

x=[x0,x1,…,xd/2−1,xd/2,…,xd−1]x = [x_0, x_1, \ldots, x_{d/2-1}, x_{d/2}, \ldots, x_{d-1}]

将每对元素视为:

(ai,bi)=(xi,xi+d/2)(a_i, b_i) = (x_i, x_{i + d/2})

每对元素的旋转角度由 token 位置 pp 和旋转维度 ii 决定:

θi=10000−i/(d/2)\theta_i = 10000^{-i / (d/2)} ϕp,i=p⋅θi\phi_{p,i} = p \cdot \theta_i

旋转为:

[ai′bi′]=[cos⁡ϕp,i−sin⁡ϕp,isin⁡ϕp,icos⁡ϕp,i][aibi]\begin{bmatrix} a'_i \\ b'_i \end{bmatrix} = \begin{bmatrix} \cos \phi_{p,i} & -\sin \phi_{p,i} \\ \sin \phi_{p,i} & \cos \phi_{p,i} \end{bmatrix} \begin{bmatrix} a_i \\ b_i \end{bmatrix}

逐元素公式为:

ai′=aicos⁡ϕp,i−bisin⁡ϕp,ia'_i = a_i \cos \phi_{p,i} - b_i \sin \phi_{p,i} bi′=bicos⁡ϕp,i+aisin⁡ϕp,ib'_i = b_i \cos \phi_{p,i} + a_i \sin \phi_{p,i}

对整个向量,这等价于乘以块对角旋转矩阵:

Rp=[Rp,00⋯00Rp,1⋯0⋮⋮⋱⋮00⋯Rp,d/2−1]R_p = \begin{bmatrix} R_{p,0} & 0 & \cdots & 0 \\ 0 & R_{p,1} & \cdots & 0 \\ \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & \cdots & R_{p,d/2-1} \end{bmatrix}

其中:

Rp,i=[cos⁡ϕp,i−sin⁡ϕp,isin⁡ϕp,icos⁡ϕp,i]R_{p,i} = \begin{bmatrix} \cos \phi_{p,i} & -\sin \phi_{p,i} \\ \sin \phi_{p,i} & \cos \phi_{p,i} \end{bmatrix}

本 kernel 采用前后两半配对的布局,不会实际构造矩阵,而是计算两个基张量:

Cp,i=cos⁡(p⋅10000−i/(d/2))C_{p,i} = \cos(p \cdot 10000^{-i/(d/2)}) Sp,i=sin⁡(p⋅10000−i/(d/2))S_{p,i} = \sin(p \cdot 10000^{-i/(d/2)})

再通过逐元素操作完成相同旋转:

real′=real⊙C−imag⊙S\text{real}' = \text{real} \odot C - \text{imag} \odot S imag′=imag⊙C+real⊙S\text{imag}' = \text{imag} \odot C + \text{real} \odot S

这里 real/imag 与 C/S 逐元素相乘。C、S 的形状为 [S, D],其他维度通过广播参与计算。映射到 kernel 后,每个 program 负责 [N, S, H, D // 2] 的一个 tile,加载实部和虚部,计算对应的正弦余弦,再写出旋转后两半。在 TileLang 中可直接通过 T.Parallel 表达所有元素,因此逐元素操作很容易编写。

RoPE 将每个头向量视为前后两半:

x_real = X[..., : D // 2]
x_imag = X[..., D // 2 :]

再按位置相关角度旋转每一对:

real = x_real * cos - x_imag * sin
imag = x_imag * cos + x_real * sin

合并 Grid 维度

最初尝试为每个张量轴设置一个 grid 维度:

with T.Kernel(
    T.ceildiv(N, BLOCK_N),
    T.ceildiv(S, BLOCK_S),
    T.ceildiv(H, BLOCK_H),
    T.ceildiv(half_D, BLOCK_D),
    threads=256,
) as (pid_n, pid_s, pid_h, pid_d):

这很自然,但 TileLang 遵循 CUDA 启动模型:grid 最多只有三维。

修正方式是将头块和维度块打包到同一个物理 grid 轴:

num_h_blocks = T.ceildiv(H, BLOCK_H)
num_d_blocks = T.ceildiv(half_D, BLOCK_D)

with T.Kernel(
    T.ceildiv(N, BLOCK_N),
    T.ceildiv(S, BLOCK_S),
    num_h_blocks * num_d_blocks,
    threads=256,
) as (pid_n, pid_s, pid_hd):
    pid_h = pid_hd // num_d_blocks
    pid_d = pid_hd % num_d_blocks

这样 N、S 保留独立 grid 轴,pid_hd 则解码为逻辑 (H, D // 2) tile。

位置索引与维度索引

另一个容易犯的错误是混淆序列索引和维度索引。

RoPE 角度为:

angle = position * base ^ (-dim / half_dim)

序列位置决定 token 在哪里,维度索引决定频率:

seq_idx = offset + pid_s * BLOCK_S + s
dim_idx = pid_d * BLOCK_D + d
freq = T.pow(10000.0, -dim_idx.astype("float32") / half_D)

cos_basis[s, d] = T.cos(seq_idx * freq)
sin_basis[s, d] = T.sin(seq_idx * freq)

基函数必须同时由 s 和 d 索引。每个序列位置只计算一个值是不够的,因为各旋转维度具有不同频率。

整数除法与浮点除法

TVM 不允许含糊的整数除法:

freq = T.pow(10000.0, -dim_idx / half_D)

这会失败,因为 dim_idx 和 half_D 都是整数表达式。RoPE 需要实数指数,所以除法前必须转换 dim_idx:

freq = T.pow(10000.0, -dim_idx.astype("float32") / half_D)

从 PyTorch 迁移时很容易忽略这个编译器细节,因为 PyTorch 的 / 自动表示浮点除法。

写入两半结果

kernel 对头维度前后两半的元素配对旋转,因此输出也分两部分写入:

T.copy(O_local_real, O[..., pid_d * BLOCK_D : (pid_d + 1) * BLOCK_D])
T.copy(
    O_local_imag,
    O[..., pid_d * BLOCK_D + half_D : (pid_d + 1) * BLOCK_D + half_D],
)

这比尝试将两半结果放进末维只有 BLOCK_D 的局部缓冲区更清楚。

附录

完整 kernel:


@tilelang.jit
def rope(X, offset, BLOCK_N, BLOCK_S, BLOCK_H, BLOCK_D):
    N, S, H, D = T.const("N, S, H, D")
    dtype = T.float32
    X: T.Tensor((N, S, H, D), dtype)
    O = T.empty((N, S, H, D), dtype)
    # can I compute constants here?
    half_D = D // 2

    num_h_blocks = T.ceildiv(H, BLOCK_H)
    num_d_blocks = T.ceildiv(half_D, BLOCK_D)
    with T.Kernel(
        T.ceildiv(N, BLOCK_N),
        T.ceildiv(S, BLOCK_S),
        num_h_blocks * num_d_blocks,
        # tilelang support 3 dimension at most, since hardware usually have at most 3 level of parallelism. So we need to combine H and D dimensions together.
        threads=256,
    ) as (
        pid_n,
        pid_s,
        pid_hd,
    ):
        pid_h = pid_hd // num_d_blocks
        pid_d = pid_hd % num_d_blocks
        X_local_real = T.alloc_fragment((BLOCK_N, BLOCK_S, BLOCK_H, BLOCK_D), dtype)
        X_local_imag = T.alloc_fragment((BLOCK_N, BLOCK_S, BLOCK_H, BLOCK_D), dtype)
        O_local_real = T.alloc_fragment((BLOCK_N, BLOCK_S, BLOCK_H, BLOCK_D), dtype)
        O_local_imag = T.alloc_fragment((BLOCK_N, BLOCK_S, BLOCK_H, BLOCK_D), dtype)
        cos_basis = T.alloc_fragment((BLOCK_S, BLOCK_D), dtype)
        sin_basis = T.alloc_fragment((BLOCK_S, BLOCK_D), dtype)
        for s, d in T.Parallel(BLOCK_S, BLOCK_D):
            seq_idx = offset + pid_s * BLOCK_S + s
            dim_idx = pid_d * BLOCK_D + d
            freq = T.pow(10000.0, -dim_idx.astype("float32") / half_D)
            cos_basis[s, d] = T.cos(seq_idx * freq)
            sin_basis[s, d] = T.sin(seq_idx * freq)
        # for each block, we process
        # x[..., BLOCK_D * pid_d : BLOCK_D * (pid_d + 1)]
        # and x[..., BLOCK_D * pid_d + half_D : BLOCK_D * (pid_d + 1) + half_D]
        n_blk_id = pid_n
        s_blk_id = pid_s
        T.copy(
            X[
                n_blk_id * BLOCK_N : (n_blk_id + 1) * BLOCK_N,
                s_blk_id * BLOCK_S : (s_blk_id + 1) * BLOCK_S,
                pid_h * BLOCK_H : (pid_h + 1) * BLOCK_H,
                pid_d * BLOCK_D : (pid_d + 1) * BLOCK_D,
            ],
            X_local_real,
        )
        T.copy(
            X[
                n_blk_id * BLOCK_N : (n_blk_id + 1) * BLOCK_N,
                s_blk_id * BLOCK_S : (s_blk_id + 1) * BLOCK_S,
                pid_h * BLOCK_H : (pid_h + 1) * BLOCK_H,
                pid_d * BLOCK_D + half_D : (pid_d + 1) * BLOCK_D + half_D,
            ],
            X_local_imag,
        )

        for n, s, h, d in T.Parallel(BLOCK_N, BLOCK_S, BLOCK_H, BLOCK_D):
            real = (
                X_local_real[n, s, h, d] * cos_basis[s, d]
                - X_local_imag[n, s, h, d] * sin_basis[s, d]
            )
            imag = (
                X_local_imag[n, s, h, d] * cos_basis[s, d]
                + X_local_real[n, s, h, d] * sin_basis[s, d]
            )
            O_local_real[n, s, h, d] = real
            O_local_imag[n, s, h, d] = imag

        T.copy(
            O_local_real,
            O[
                n_blk_id * BLOCK_N : (n_blk_id + 1) * BLOCK_N,
                s_blk_id * BLOCK_S : (s_blk_id + 1) * BLOCK_S,
                pid_h * BLOCK_H : (pid_h + 1) * BLOCK_H,
                pid_d * BLOCK_D : (pid_d + 1) * BLOCK_D,
            ],
        )
        T.copy(
            O_local_imag,
            O[
                n_blk_id * BLOCK_N : (n_blk_id + 1) * BLOCK_N,
                s_blk_id * BLOCK_S : (s_blk_id + 1) * BLOCK_S,
                pid_h * BLOCK_H : (pid_h + 1) * BLOCK_H,
                pid_d * BLOCK_D + half_D : (pid_d + 1) * BLOCK_D + half_D,
            ],
        )
    return O

用于对照测试的 Torch 实现:

half_d = D // 2
positions = torch.arange(offset, offset + S, dtype=torch.float32, device=X.device)
dims = torch.arange(half_d, dtype=torch.float32, device=X.device)
freqs = torch.pow(10000.0, -dims / half_d)
angles = torch.outer(positions, freqs)

cos_basis = torch.cos(angles).reshape(1, S, 1, half_d)
sin_basis = torch.sin(angles).reshape(1, S, 1, half_d)

x_real = X[..., :half_d]
x_imag = X[..., half_d:]
real = x_real * cos_basis - x_imag * sin_basis
imag = x_imag * cos_basis + x_real * sin_basis
out = torch.cat([real, imag], dim=-1)

编辑文章
分享文章:

上一篇
如何正确地写 Layout:深入理解 TileLang 布局推导
下一篇
[TIL] 理解 IM2COL