[TIL] IREE 编译流水线
本文用一个小型向量程序,介绍 IREE 各流水线阶段的工作,以及生成的 IR。
核心设计是分离主机编排与设备计算。二者按不同速度降级:GPU kernel 已经变成 LLVM/NVVM IR 时,外围程序可能仍使用 Stream 操作;之后 kernel 成为不透明载荷,而主机程序变成调用 HAL 运行时的 VM 字节码。
程序与分析依据
输入是一个简单的四元素向量程序:
func.func @simple_mul(%arg0: tensor<4xf32>,
%arg1: tensor<4xf32>) -> tensor<4xf32> {
%0 = "arith.mulf"(%arg0, %arg1)
: (tensor<4xf32>, tensor<4xf32>) -> tensor<4xf32>
return %0 : tensor<4xf32>
}
实际计算为 C[i] = A[i] * B[i]。输入 [1, 2, 3, 4] 与 [10, 20, 30, 40] 时,结果为 [10, 40, 90, 160]。
各阶段概览
使用 --compile-to 查看各阶段结果。一个阶段可能执行多个 pass,同一份产物通常同时包含多种方言,这就是渐进式降级。
| 阶段边界 | 主要职责 | 代表操作或属性 |
|---|---|---|
input | 将受支持输入规范化为公共表示 | util.func、张量运算 |
abi | 建立面向应用的调用约定 | hal.tensor.import、hal.tensor.export |
preprocessing | 全局优化前的预处理 | ABI 包装中的张量计算 |
global-optimization | 优化结构化张量计算 | linalg.generic、计算屏障 |
dispatch-creation | 划分可调度工作区域 | dispatch 区域与计算边界 |
flow | 提取和组织可执行计算 | flow.executable、flow.dispatch |
stream | 规划资源与执行依赖 | stream.resource.alloca、stream.cmd.execute |
executable-sources | 具体化目标变体和接口 | hal.executable.variant、绑定布局 |
executable-configurations | 选择代码生成策略和参数 | translation_info、lowering_config |
executable-targets | 将 kernel 降级为面向目标的 IR | llvm.func、nvvm.read.ptx.sreg.tid.x |
hal | 降级编排并序列化设备程序 | 队列、fence、命令缓冲区、二进制 |
vm | 用 VM 指令和导入表示主机控制 | vm.func、vm.call、vm.rodata |
| 最终输出 | 编码字节码并封装运行时模块 | .vmfb |
1. Input、ABI 与预处理:让程序可调用
Input
程序从 func.func 和张量运算开始,公共函数导入后使用 IREE utility 方言表示:
util.func public @simple_mul(%arg0: tensor<4xf32>,
%arg1: tensor<4xf32>) -> tensor<4xf32> {
%0 = arith.mulf %arg0, %arg1 : tensor<4xf32>
util.return %0 : tensor<4xf32>
}
输入转换将受支持的源表示转换为后续编译器可理解的形式。PyTorch 程序需要先经过受支持的导入或导出路径,再进入类似流水线。这里手写 MLIR 已经是简单张量计算,因此这一步很直接。
ABI 与预处理
下一步建立 ABI 边界,区分外部对象与内部表示,定义程序如何与外部交互。公共签名接收并返回 !hal.buffer_view:
%a = hal.tensor.import %arg0 "input0"
: !hal.buffer_view -> tensor<4xf32>
%b = hal.tensor.import %arg1 "input1"
: !hal.buffer_view -> tensor<4xf32>
%c = arith.mulf %a, %b : tensor<4xf32>
%view = hal.tensor.export %c "output0"
: tensor<4xf32> -> !hal.buffer_view
buffer view 是面向运行时的对象,描述缓冲区及张量元数据。函数内部仍基于张量优化,reflection 保留原签名:
sync func @simple_mul(%input0: tensor<4xf32>,
%input1: tensor<4xf32>) -> (%output0: tensor<4xf32>)
这样分离了外部对象模型与内部数学表示。import、export 标记边界,出现它们不意味着必然发生物理数据复制。
预处理通常涉及规范化。这里程序太小,几乎没有可简化的内容,因此与 ABI 阶段结果相同。完整结果如下:
module {
util.func public @simple_mul(%arg0: !hal.buffer_view, %arg1: !hal.buffer_view) -> !hal.buffer_view attributes {iree.abi.stub, iree.reflection = {iree.abi.declaration = "sync func @simple_mul(%input0: tensor<4xf32>, %input1: tensor<4xf32>) -> (%output0: tensor<4xf32>)"}} {
%0 = hal.tensor.import %arg0 "input0" : !hal.buffer_view -> tensor<4xf32>
%1 = hal.tensor.import %arg1 "input1" : !hal.buffer_view -> tensor<4xf32>
%2 = arith.mulf %0, %1 : tensor<4xf32>
%3 = hal.tensor.export %2 "output0" : tensor<4xf32> -> !hal.buffer_view
util.return %3 : !hal.buffer_view
}
}
2. 全局优化
全局优化通常包括图级改写,例如公共子表达式消除和常量折叠,与目标硬件无关。
示例中的逐元素乘法转换为结构化的 linalg.generic:
indexing_maps = [
affine_map<(d0) -> (d0)>,
affine_map<(d0) -> (d0)>,
affine_map<(d0) -> (d0)>
]
iterator_types = ["parallel"]
三个映射描述循环索引如何访问输入 A、输入 B 和输出 C,均使用相同索引。循环体执行标量乘法并返回结果:
^bb0(%a: f32, %b: f32, %out: f32):
%product = arith.mulf %a, %b : f32
linalg.yield %product : f32
结构化迭代使计算能够参与融合、分块和分发变换。这里使用第一个输入张量作为 destination-style outs。标量体忽略 %out,因此不依赖旧输出内容。但这一阶段的张量语义并不代表原地修改调用者输入;后续别名与缓冲区化决策仍须保持程序语义。
外围 iree_tensor_ext.compute_barrier.start/end 限制编译器变换,例如阻止 reshape 跨边界传播。它们是编译器层面的边界,不一定需要生成实际运行时操作。
全局优化发生在张量结构仍可见时。对于大模型,可以在 dispatch 边界限制组合范围之前,跨较大计算图进行优化。
全局优化结果:
module {
util.func public @simple_mul(%arg0: !hal.buffer_view, %arg1: !hal.buffer_view) -> !hal.buffer_view attributes {iree.abi.stub, iree.reflection = {iree.abi.declaration = "sync func @simple_mul(%input0: tensor<4xf32>, %input1: tensor<4xf32>) -> (%output0: tensor<4xf32>)"}} {
%0 = hal.tensor.import %arg0 "input0" : !hal.buffer_view -> tensor<4xf32>
%1 = iree_tensor_ext.compute_barrier.start %0 : tensor<4xf32> -> tensor<4xf32>
%2 = hal.tensor.import %arg1 "input1" : !hal.buffer_view -> tensor<4xf32>
%3 = iree_tensor_ext.compute_barrier.start %2 : tensor<4xf32> -> tensor<4xf32>
%4 = linalg.generic {indexing_maps = [affine_map<(d0) -> (d0)>, affine_map<(d0) -> (d0)>, affine_map<(d0) -> (d0)>], iterator_types = ["parallel"]} ins(%1, %3 : tensor<4xf32>, tensor<4xf32>) outs(%1 : tensor<4xf32>) {
^bb0(%in: f32, %in_0: f32, %out: f32):
%7 = arith.mulf %in, %in_0 : f32
linalg.yield %7 : f32
} -> tensor<4xf32>
%5 = iree_tensor_ext.compute_barrier.end %4 : tensor<4xf32> -> tensor<4xf32>
%6 = hal.tensor.export %5 "output0" : tensor<4xf32> -> !hal.buffer_view
util.return %6 : !hal.buffer_view
}
}
3. Flow
Flow 围绕 dispatch 组织可执行工作,决定哪些操作属于一个执行单元。此前只有计算图,尚未明确哪些计算应组合为设备上的 SPMD 工作。flow.executable 容纳相应可执行计算,dispatch 成为调度单位。例如逐元素操作自然适合 SPMD,而归约会改变执行边界、grid 和 block 形状,需要另外处理。
Flow 后,外层函数调用:
%c = flow.dispatch
@simple_mul_dispatch_0::@simple_mul_dispatch_0_elementwise_4_f32(%a, %b)
: (tensor<4xf32>, tensor<4xf32>) -> tensor<4xf32>
计算位于 flow.executable 中,包含导出入口和嵌套函数。后者接收两个只读 dispatch tensor 和一个只写输出 tensor,加载输入切片、执行 Linalg,再存储结果。
这建立了清楚的分工:外围程序指定调用哪段计算,executable 描述如何计算结果。dispatch 边界同时是调度与编译边界。
导出中仍包含:
%x, %y, %z = iree_tensor_ext.dispatch.workgroup_count_from_slice()
这是 workgroup 数量计算的占位。workgroup 是分布式执行单位,在 CUDA 中对应线程块。组数和每组线程数是不同概念,此时都不能直接由四元素张量形状推出。
Flow 阶段完整结果:
module {
flow.executable private @simple_mul_dispatch_0 {
flow.executable.export public @simple_mul_dispatch_0_elementwise_4_f32 workgroups() -> (index, index, index) {
%x, %y, %z = iree_tensor_ext.dispatch.workgroup_count_from_slice()
flow.return %x, %y, %z : index, index, index
}
builtin.module {
func.func @simple_mul_dispatch_0_elementwise_4_f32(%arg0: !iree_tensor_ext.dispatch.tensor<readonly:tensor<4xf32>>, %arg1: !iree_tensor_ext.dispatch.tensor<readonly:tensor<4xf32>>, %arg2: !iree_tensor_ext.dispatch.tensor<writeonly:tensor<4xf32>>) {
%0 = iree_tensor_ext.dispatch.tensor.load %arg0, offsets = [0], sizes = [4], strides = [1] : !iree_tensor_ext.dispatch.tensor<readonly:tensor<4xf32>> -> tensor<4xf32>
%1 = iree_tensor_ext.dispatch.tensor.load %arg1, offsets = [0], sizes = [4], strides = [1] : !iree_tensor_ext.dispatch.tensor<readonly:tensor<4xf32>> -> tensor<4xf32>
%2 = tensor.empty() : tensor<4xf32>
%3 = linalg.generic {indexing_maps = [affine_map<(d0) -> (d0)>, affine_map<(d0) -> (d0)>, affine_map<(d0) -> (d0)>], iterator_types = ["parallel"]} ins(%0, %1 : tensor<4xf32>, tensor<4xf32>) outs(%2 : tensor<4xf32>) {
^bb0(%in: f32, %in_0: f32, %out: f32):
%4 = arith.mulf %in, %in_0 : f32
linalg.yield %4 : f32
} -> tensor<4xf32>
iree_tensor_ext.dispatch.tensor.store %3, %arg2, offsets = [0], sizes = [4], strides = [1] : tensor<4xf32> -> !iree_tensor_ext.dispatch.tensor<writeonly:tensor<4xf32>>
return
}
}
}
util.func public @simple_mul(%arg0: !hal.buffer_view, %arg1: !hal.buffer_view) -> !hal.buffer_view attributes {iree.abi.stub, iree.reflection = {iree.abi.declaration = "sync func @simple_mul(%input0: tensor<4xf32>, %input1: tensor<4xf32>) -> (%output0: tensor<4xf32>)"}} {
%0 = hal.tensor.import %arg0 "input0" : !hal.buffer_view -> tensor<4xf32>
%1 = hal.tensor.import %arg1 "input1" : !hal.buffer_view -> tensor<4xf32>
%2 = flow.dispatch @simple_mul_dispatch_0::@simple_mul_dispatch_0_elementwise_4_f32(%0, %1) : (tensor<4xf32>, tensor<4xf32>) -> tensor<4xf32>
%3 = hal.tensor.export %2 "output0" : tensor<4xf32> -> !hal.buffer_view
util.return %3 : !hal.buffer_view
}
}
4. Stream
Stream 让资源大小和调度关系显式可见。调度工作时需要知道:
- 输入在哪里,是否准备好?
- 可以在哪里执行?
- 主机何时能读取结果? 因此必须通过分配和依赖跟踪资源生命周期。
Stream 将外部资源访问改写为带显式缓冲区大小的 stream.tensor.import:
%a = stream.tensor.import %arg0
: !hal.buffer_view -> tensor<4xf32>
in !stream.resource<external>{%c16}
external 资源类别表示生命周期跨越程序边界,例如由主机分配后复制到设备,或来自其他 kernel 的资源。
输出被显式分配:
%result, %ready = stream.resource.alloca uninitialized
: !stream.resource<external>{%c16} => !stream.timepoint
分配同时返回资源及其可用时间点。这里的 alloca 是 Stream 资源操作,并不承诺使用 CPU 栈内存。
执行等待资源可用,并声明访问:
stream.cmd.dispatch @simple_mul_dispatch_0::@... {
ro %a[0 for 16] : ...,
ro %b[0 for 16] : ...,
wo %result[0 for 16] : ...
}
它标注两次 16 字节读取和一次 16 字节写入。外围 stream.cmd.execute 返回完成时间点,stream.timepoint.await 将后续结果使用与执行完成关联起来。
Stream 将数据依赖与具体队列、同步 API 分离。较早的异步形式可能包含 stream.async.transfer、stream.async.dispatch,通过资源值表达依赖。调度与分配再逐步降级为命令区域和时间点。stream.async.transfer 表示资源状态或 affinity 转换,可能需要跨内存空间复制,但优化也可以改写或移除它。
executable 内的 !stream.binding 参数和 stream.binding.subspan 描述如何从绑定资源获得张量视图。Linalg 计算仍然处于高层,使 IREE 能在确定 kernel 最终实现前先分析资源调度。
Stream 完整结果:
module {
stream.executable private @simple_mul_dispatch_0 {
stream.executable.export public @simple_mul_dispatch_0_elementwise_4_f32 workgroups() -> (index, index, index) {
%x, %y, %z = iree_tensor_ext.dispatch.workgroup_count_from_slice()
stream.return %x, %y, %z : index, index, index
}
builtin.module {
func.func @simple_mul_dispatch_0_elementwise_4_f32(%arg0: !stream.binding {stream.alignment = 64 : index}, %arg1: !stream.binding {stream.alignment = 64 : index}, %arg2: !stream.binding {stream.alignment = 64 : index}) {
%c0 = arith.constant 0 : index
%0 = stream.binding.subspan %arg0[%c0] : !stream.binding -> !iree_tensor_ext.dispatch.tensor<readonly:tensor<4xf32>>
%1 = stream.binding.subspan %arg1[%c0] : !stream.binding -> !iree_tensor_ext.dispatch.tensor<readonly:tensor<4xf32>>
%2 = stream.binding.subspan %arg2[%c0] : !stream.binding -> !iree_tensor_ext.dispatch.tensor<writeonly:tensor<4xf32>>
%3 = iree_tensor_ext.dispatch.tensor.load %0, offsets = [0], sizes = [4], strides = [1] : !iree_tensor_ext.dispatch.tensor<readonly:tensor<4xf32>> -> tensor<4xf32>
%4 = iree_tensor_ext.dispatch.tensor.load %1, offsets = [0], sizes = [4], strides = [1] : !iree_tensor_ext.dispatch.tensor<readonly:tensor<4xf32>> -> tensor<4xf32>
%5 = tensor.empty() : tensor<4xf32>
%6 = linalg.generic {indexing_maps = [affine_map<(d0) -> (d0)>, affine_map<(d0) -> (d0)>, affine_map<(d0) -> (d0)>], iterator_types = ["parallel"]} ins(%3, %4 : tensor<4xf32>, tensor<4xf32>) outs(%5 : tensor<4xf32>) {
^bb0(%in: f32, %in_0: f32, %out: f32):
%7 = arith.mulf %in, %in_0 : f32
linalg.yield %7 : f32
} -> tensor<4xf32>
iree_tensor_ext.dispatch.tensor.store %6, %2, offsets = [0], sizes = [4], strides = [1] : tensor<4xf32> -> !iree_tensor_ext.dispatch.tensor<writeonly:tensor<4xf32>>
return
}
}
}
util.func public @simple_mul(%arg0: !hal.buffer_view, %arg1: !hal.buffer_view) -> !hal.buffer_view attributes {iree.abi.stub, iree.reflection = {iree.abi.declaration = "sync func @simple_mul(%input0: tensor<4xf32>, %input1: tensor<4xf32>) -> (%output0: tensor<4xf32>)"}} {
%c0 = arith.constant 0 : index
%c16 = arith.constant 16 : index
%c4 = arith.constant 4 : index
%element_type_f32 = hal.element_type<f32> : i32
%dense_row_major = hal.encoding_type<dense_row_major> : i32
hal.buffer_view.assert<%arg0 : !hal.buffer_view> message("input0") shape([%c4]) type(%element_type_f32) encoding(%dense_row_major)
%0 = stream.tensor.import %arg0 : !hal.buffer_view -> tensor<4xf32> in !stream.resource<external>{%c16}
hal.buffer_view.assert<%arg1 : !hal.buffer_view> message("input1") shape([%c4]) type(%element_type_f32) encoding(%dense_row_major)
%1 = stream.tensor.import %arg1 : !hal.buffer_view -> tensor<4xf32> in !stream.resource<external>{%c16}
%result, %result_timepoint = stream.resource.alloca uninitialized : !stream.resource<external>{%c16} => !stream.timepoint
%2 = stream.cmd.execute await(%result_timepoint) => with(%0 as %arg2: !stream.resource<external>{%c16}, %1 as %arg3: !stream.resource<external>{%c16}, %result as %arg4: !stream.resource<external>{%c16}) {
stream.cmd.dispatch @simple_mul_dispatch_0::@simple_mul_dispatch_0_elementwise_4_f32 {
ro %arg2[%c0 for %c16] : !stream.resource<external>{%c16},
ro %arg3[%c0 for %c16] : !stream.resource<external>{%c16},
wo %arg4[%c0 for %c16] : !stream.resource<external>{%c16}
}
} => !stream.timepoint
%3 = stream.timepoint.await %2 => %result : !stream.resource<external>{%c16}
%4 = stream.tensor.export %3 : tensor<4xf32> in !stream.resource<external>{%c16} -> !hal.buffer_view
util.return %4 : !hal.buffer_view
}
}
5. Executable Sources
这个阶段后,程序拥有默认设备 affinity,以及面向 CUDA 的设备声明。分配和执行等操作都关联到该设备。
executable 变为:
hal.executable @simple_mul_dispatch_0
└─ hal.executable.variant @cuda_nvptx_fb
├─ hal.executable.export
└─ builtin.module containing the kernel
variant 是面向某个执行目标的实现。即使本例只有一种,也允许存在多个变体。导出建立包含三个 storage-buffer binding 的 pipeline layout:
| 绑定 | 含义 | 访问 |
|---|---|---|
| 0 | A | 只读 |
| 1 | B | 只读 |
| 2 | C | kernel 写入 |
kernel 通过 hal.interface.binding.subspan binding(0) 获取数据,绑定 1、2 同理。设备元数据记录 subgroup 大小、最大 block 维度和可用 workgroup 内存,作为代码生成约束。
这个阶段不修改 kernel 内部计算。
module attributes {stream.affinity.default = #hal.device.affinity<@__device_0>} {
util.global private @__device_0 = #hal.device.target<"cuda", [#hal.executable.target<"cuda", "cuda-nvptx-fb", {iree_codegen.target_info = #iree_gpu.target<arch = "sm_60", features = "+ptx76", wgp = <compute = fp64|fp32|fp16|int64|int32|int16|int8, storage = b64|b32|b16|b8, subgroup = shuffle|arithmetic, subgroup_size_choices = [32], max_workgroup_sizes = [1024, 1024, 1024], max_thread_count_per_workgroup = 1024, max_workgroup_memory_bytes = 49152, max_workgroup_counts = [2147483647, 65535, 65535]>>}>]> : !hal.device
hal.executable private @simple_mul_dispatch_0 {
hal.executable.variant public @cuda_nvptx_fb target(<"cuda", "cuda-nvptx-fb", {iree_codegen.target_info = #iree_gpu.target<arch = "sm_60", features = "+ptx76", wgp = <compute = fp64|fp32|fp16|int64|int32|int16|int8, storage = b64|b32|b16|b8, subgroup = shuffle|arithmetic, subgroup_size_choices = [32], max_workgroup_sizes = [1024, 1024, 1024], max_thread_count_per_workgroup = 1024, max_workgroup_memory_bytes = 49152, max_workgroup_counts = [2147483647, 65535, 65535]>>}>) {
hal.executable.export public @simple_mul_dispatch_0_elementwise_4_f32 ordinal(0) layout(#hal.pipeline.layout<bindings = [#hal.pipeline.binding<storage_buffer, "ReadOnly|Indirect">, #hal.pipeline.binding<storage_buffer, "ReadOnly|Indirect">, #hal.pipeline.binding<storage_buffer, Indirect>], flags = Indirect>) count(%arg0: !hal.device) -> (index, index, index) {
%x, %y, %z = iree_tensor_ext.dispatch.workgroup_count_from_slice()
hal.return %x, %y, %z : index, index, index
}
builtin.module {
func.func @simple_mul_dispatch_0_elementwise_4_f32() {
%c0 = arith.constant 0 : index
%0 = hal.interface.binding.subspan layout(<bindings = [#hal.pipeline.binding<storage_buffer, "ReadOnly|Indirect">, #hal.pipeline.binding<storage_buffer, "ReadOnly|Indirect">, #hal.pipeline.binding<storage_buffer, Indirect>], flags = Indirect>) binding(0) alignment(64) offset(%c0) flags("ReadOnly|Indirect") : !iree_tensor_ext.dispatch.tensor<readonly:tensor<4xf32>>
%1 = hal.interface.binding.subspan layout(<bindings = [#hal.pipeline.binding<storage_buffer, "ReadOnly|Indirect">, #hal.pipeline.binding<storage_buffer, "ReadOnly|Indirect">, #hal.pipeline.binding<storage_buffer, Indirect>], flags = Indirect>) binding(1) alignment(64) offset(%c0) flags("ReadOnly|Indirect") : !iree_tensor_ext.dispatch.tensor<readonly:tensor<4xf32>>
%2 = hal.interface.binding.subspan layout(<bindings = [#hal.pipeline.binding<storage_buffer, "ReadOnly|Indirect">, #hal.pipeline.binding<storage_buffer, "ReadOnly|Indirect">, #hal.pipeline.binding<storage_buffer, Indirect>], flags = Indirect>) binding(2) alignment(64) offset(%c0) flags(Indirect) : !iree_tensor_ext.dispatch.tensor<writeonly:tensor<4xf32>>
%3 = iree_tensor_ext.dispatch.tensor.load %0, offsets = [0], sizes = [4], strides = [1] : !iree_tensor_ext.dispatch.tensor<readonly:tensor<4xf32>> -> tensor<4xf32>
%4 = iree_tensor_ext.dispatch.tensor.load %1, offsets = [0], sizes = [4], strides = [1] : !iree_tensor_ext.dispatch.tensor<readonly:tensor<4xf32>> -> tensor<4xf32>
%5 = tensor.empty() : tensor<4xf32>
%6 = linalg.generic {indexing_maps = [affine_map<(d0) -> (d0)>, affine_map<(d0) -> (d0)>, affine_map<(d0) -> (d0)>], iterator_types = ["parallel"]} ins(%3, %4 : tensor<4xf32>, tensor<4xf32>) outs(%5 : tensor<4xf32>) {
^bb0(%in: f32, %in_0: f32, %out: f32):
%7 = arith.mulf %in, %in_0 : f32
linalg.yield %7 : f32
} -> tensor<4xf32>
iree_tensor_ext.dispatch.tensor.store %6, %2, offsets = [0], sizes = [4], strides = [1] : tensor<4xf32> -> !iree_tensor_ext.dispatch.tensor<writeonly:tensor<4xf32>>
return
}
}
}
}
util.func public @simple_mul(%arg0: !hal.buffer_view, %arg1: !hal.buffer_view) -> !hal.buffer_view attributes {iree.abi.stub, iree.reflection = {iree.abi.declaration = "sync func @simple_mul(%input0: tensor<4xf32>, %input1: tensor<4xf32>) -> (%output0: tensor<4xf32>)"}} {
%c0 = arith.constant 0 : index
%c16 = arith.constant 16 : index
%c4 = arith.constant 4 : index
%element_type_f32 = hal.element_type<f32> : i32
%dense_row_major = hal.encoding_type<dense_row_major> : i32
hal.buffer_view.assert<%arg0 : !hal.buffer_view> message("input0") shape([%c4]) type(%element_type_f32) encoding(%dense_row_major)
%0 = stream.tensor.import on(#hal.device.affinity<@__device_0>) %arg0 : !hal.buffer_view -> tensor<4xf32> in !stream.resource<external>{%c16}
hal.buffer_view.assert<%arg1 : !hal.buffer_view> message("input1") shape([%c4]) type(%element_type_f32) encoding(%dense_row_major)
%1 = stream.tensor.import on(#hal.device.affinity<@__device_0>) %arg1 : !hal.buffer_view -> tensor<4xf32> in !stream.resource<external>{%c16}
%result, %result_timepoint = stream.resource.alloca uninitialized on(#hal.device.affinity<@__device_0>) : !stream.resource<external>{%c16} => !stream.timepoint
%2 = stream.cmd.execute on(#hal.device.affinity<@__device_0>) await(%result_timepoint) => with(%0 as %arg2: !stream.resource<external>{%c16}, %1 as %arg3: !stream.resource<external>{%c16}, %result as %arg4: !stream.resource<external>{%c16}) {
stream.cmd.dispatch @simple_mul_dispatch_0::@cuda_nvptx_fb::@simple_mul_dispatch_0_elementwise_4_f32 {
ro %arg2[%c0 for %c16] : !stream.resource<external>{%c16},
ro %arg3[%c0 for %c16] : !stream.resource<external>{%c16},
wo %arg4[%c0 for %c16] : !stream.resource<external>{%c16}
}
} => !stream.timepoint
%3 = stream.timepoint.await %2 => %result : !stream.resource<external>{%c16}
%4 = stream.tensor.export on(#hal.device.affinity<@__device_0>) %3 : tensor<4xf32> in !stream.resource<external>{%c16} -> !hal.buffer_view
util.return %4 : !hal.buffer_view
}
}
6. Executable Configurations:选择降级策略
该阶段加入了对预期 GPU 映射最精炼的说明:
translation_info = #iree_codegen.translation_info<
pipeline = #iree_gpu.pipeline<TileAndFuse>
workgroup_size = [32, 1, 1]
subgroup_size = 32
>
Linalg 操作获得:
lowering_config = #iree_gpu.lowering_config<{
thread = [1], workgroup = [32]
}>
这些属性分别回答不同问题:
workgroup_size:三维线程组织,此处每块 32 个线程。workgroup = [32]:将一维迭代空间按 32 元素的 workgroup tile 划分。thread = [1]:每线程一个元素。subgroup_size = 32:选择 CUDA warp 大小的 subgroup。
四元素问题可以放进一个 block,仅四个线程需要进行有效乘法。这里由配置表达映射,下一份产物会展示生成的控制流。
TileAndFuse 选择代码生成流水线。名称并不意味着本例中发生了多操作融合,因为这里只有一个计算操作。策略选择与具体变换分离,让配置可检查,也为用户选择和调优提供入口。
缓冲区接口也被规范化:binding subspan 现在生成 memref<4xf32, ...>,iree_codegen.load_from_buffer、store_to_buffer 连接它们与剩余张量计算。边界出现 memref 并不意味着完整缓冲区化已完成。
module attributes {stream.affinity.default = #hal.device.affinity<@__device_0>} {
util.global private @__device_0 = #hal.device.target<"cuda", [#hal.executable.target<"cuda", "cuda-nvptx-fb", {iree_codegen.target_info = #iree_gpu.target<arch = "sm_60", features = "+ptx76", wgp = <compute = fp64|fp32|fp16|int64|int32|int16|int8, storage = b64|b32|b16|b8, subgroup = shuffle|arithmetic, subgroup_size_choices = [32], max_workgroup_sizes = [1024, 1024, 1024], max_thread_count_per_workgroup = 1024, max_workgroup_memory_bytes = 49152, max_workgroup_counts = [2147483647, 65535, 65535]>>}>]> : !hal.device
hal.executable private @simple_mul_dispatch_0 {
hal.executable.variant public @cuda_nvptx_fb target(<"cuda", "cuda-nvptx-fb", {iree_codegen.target_info = #iree_gpu.target<arch = "sm_60", features = "+ptx76", wgp = <compute = fp64|fp32|fp16|int64|int32|int16|int8, storage = b64|b32|b16|b8, subgroup = shuffle|arithmetic, subgroup_size_choices = [32], max_workgroup_sizes = [1024, 1024, 1024], max_thread_count_per_workgroup = 1024, max_workgroup_memory_bytes = 49152, max_workgroup_counts = [2147483647, 65535, 65535]>>}>) {
hal.executable.export public @simple_mul_dispatch_0_elementwise_4_f32 ordinal(0) layout(#hal.pipeline.layout<bindings = [#hal.pipeline.binding<storage_buffer, "ReadOnly|Indirect">, #hal.pipeline.binding<storage_buffer, "ReadOnly|Indirect">, #hal.pipeline.binding<storage_buffer, Indirect>], flags = Indirect>) count(%arg0: !hal.device) -> (index, index, index) {
%x, %y, %z = iree_tensor_ext.dispatch.workgroup_count_from_slice()
hal.return %x, %y, %z : index, index, index
}
builtin.module {
func.func @simple_mul_dispatch_0_elementwise_4_f32() attributes {translation_info = #iree_codegen.translation_info<pipeline = #iree_gpu.pipeline<TileAndFuse> workgroup_size = [32, 1, 1] subgroup_size = 32>} {
%c0 = arith.constant 0 : index
%0 = hal.interface.binding.subspan layout(<bindings = [#hal.pipeline.binding<storage_buffer, "ReadOnly|Indirect">, #hal.pipeline.binding<storage_buffer, "ReadOnly|Indirect">, #hal.pipeline.binding<storage_buffer, Indirect>], flags = Indirect>) binding(0) alignment(64) offset(%c0) flags("ReadOnly|Indirect") : memref<4xf32, #hal.descriptor_type<storage_buffer>>
%1 = hal.interface.binding.subspan layout(<bindings = [#hal.pipeline.binding<storage_buffer, "ReadOnly|Indirect">, #hal.pipeline.binding<storage_buffer, "ReadOnly|Indirect">, #hal.pipeline.binding<storage_buffer, Indirect>], flags = Indirect>) binding(1) alignment(64) offset(%c0) flags("ReadOnly|Indirect") : memref<4xf32, #hal.descriptor_type<storage_buffer>>
%2 = hal.interface.binding.subspan layout(<bindings = [#hal.pipeline.binding<storage_buffer, "ReadOnly|Indirect">, #hal.pipeline.binding<storage_buffer, "ReadOnly|Indirect">, #hal.pipeline.binding<storage_buffer, Indirect>], flags = Indirect>) binding(2) alignment(64) offset(%c0) flags(Indirect) : memref<4xf32, #hal.descriptor_type<storage_buffer>>
%3 = iree_codegen.load_from_buffer %0 : memref<4xf32, #hal.descriptor_type<storage_buffer>> -> tensor<4xf32>
%4 = iree_codegen.load_from_buffer %1 : memref<4xf32, #hal.descriptor_type<storage_buffer>> -> tensor<4xf32>
%5 = tensor.empty() : tensor<4xf32>
%6 = linalg.generic {indexing_maps = [affine_map<(d0) -> (d0)>, affine_map<(d0) -> (d0)>, affine_map<(d0) -> (d0)>], iterator_types = ["parallel"]} ins(%3, %4 : tensor<4xf32>, tensor<4xf32>) outs(%5 : tensor<4xf32>) attrs = {lowering_config = #iree_gpu.lowering_config<{thread = [1], workgroup = [32]}>} {
^bb0(%in: f32, %in_0: f32, %out: f32):
%7 = arith.mulf %in, %in_0 : f32
linalg.yield %7 : f32
} -> tensor<4xf32>
iree_codegen.store_to_buffer %6, %2 : tensor<4xf32> into memref<4xf32, #hal.descriptor_type<storage_buffer>>
return
}
iree_codegen.dispatch_config @simple_mul_dispatch_0_elementwise_4_f32 {
^bb0(%arg0: !hal.device):
%x, %y, %z = iree_tensor_ext.dispatch.workgroup_count_from_slice()
iree_codegen.yield %x, %y, %z : index, index, index
}
}
}
}
util.func public @simple_mul(%arg0: !hal.buffer_view, %arg1: !hal.buffer_view) -> !hal.buffer_view attributes {iree.abi.stub, iree.reflection = {iree.abi.declaration = "sync func @simple_mul(%input0: tensor<4xf32>, %input1: tensor<4xf32>) -> (%output0: tensor<4xf32>)"}} {
%c0 = arith.constant 0 : index
%c16 = arith.constant 16 : index
%c4 = arith.constant 4 : index
%element_type_f32 = hal.element_type<f32> : i32
%dense_row_major = hal.encoding_type<dense_row_major> : i32
hal.buffer_view.assert<%arg0 : !hal.buffer_view> message("input0") shape([%c4]) type(%element_type_f32) encoding(%dense_row_major)
%0 = stream.tensor.import on(#hal.device.affinity<@__device_0>) %arg0 : !hal.buffer_view -> tensor<4xf32> in !stream.resource<external>{%c16}
hal.buffer_view.assert<%arg1 : !hal.buffer_view> message("input1") shape([%c4]) type(%element_type_f32) encoding(%dense_row_major)
%1 = stream.tensor.import on(#hal.device.affinity<@__device_0>) %arg1 : !hal.buffer_view -> tensor<4xf32> in !stream.resource<external>{%c16}
%result, %result_timepoint = stream.resource.alloca uninitialized on(#hal.device.affinity<@__device_0>) : !stream.resource<external>{%c16} => !stream.timepoint
%2 = stream.cmd.execute on(#hal.device.affinity<@__device_0>) await(%result_timepoint) => with(%0 as %arg2: !stream.resource<external>{%c16}, %1 as %arg3: !stream.resource<external>{%c16}, %result as %arg4: !stream.resource<external>{%c16}) {
stream.cmd.dispatch @simple_mul_dispatch_0::@cuda_nvptx_fb::@simple_mul_dispatch_0_elementwise_4_f32 {
ro %arg2[%c0 for %c16] : !stream.resource<external>{%c16},
ro %arg3[%c0 for %c16] : !stream.resource<external>{%c16},
wo %arg4[%c0 for %c16] : !stream.resource<external>{%c16}
}
} => !stream.timepoint
%3 = stream.timepoint.await %2 => %result : !stream.resource<external>{%c16}
%4 = stream.tensor.export on(#hal.device.affinity<@__device_0>) %3 : tensor<4xf32> in !stream.resource<external>{%c16} -> !hal.buffer_view
util.return %4 : !hal.buffer_view
}
}
7. Executable Targets:具体化 GPU 实现
张量操作、Linalg 和配置阶段函数体被 LLVM、NVVM 操作替换。入口接收三个 !llvm.ptr<1> 参数,对 NVVM 目标而言地址空间 1 是全局内存。导出的 workgroup 数量为 (1, 1, 1),每组线程数为 (32, 1, 1)。
kernel 读取线程 ID:
%tid = nvvm.read.ptx.sreg.tid.x range <i32, 0, 32> : i32
控制流等价于:
for (int i = threadIdx.x; i < 4; i += 32) {
C[i] = A[i] * B[i];
}
线程 0–3 各处理一个元素,线程 4–31 在初始边界检查后退出,不会发生越界访问。
llvm.getelementptr 计算地址,llvm.load、llvm.fmul、llvm.store 实现运算。中间 vector<1xf32> 只是单元素向量表示,并非每线程计算四个元素。
readonly、noalias 和对齐信息等指针属性帮助后续优化。乘法彼此独立,因此不需要共享内存协作或线程屏障。
这里仍是面向目标的 MLIR,还不是 PTX 文本。与此同时,外层函数仍包含 Stream 操作,清楚展示了 kernel 降级和编排降级彼此独立。
module attributes {stream.affinity.default = #hal.device.affinity<@__device_0>} {
util.global private @__device_0 = #hal.device.target<"cuda", [#hal.executable.target<"cuda", "cuda-nvptx-fb", {iree_codegen.target_info = #iree_gpu.target<arch = "sm_60", features = "+ptx76", wgp = <compute = fp64|fp32|fp16|int64|int32|int16|int8, storage = b64|b32|b16|b8, subgroup = shuffle|arithmetic, subgroup_size_choices = [32], max_workgroup_sizes = [1024, 1024, 1024], max_thread_count_per_workgroup = 1024, max_workgroup_memory_bytes = 49152, max_workgroup_counts = [2147483647, 65535, 65535]>>}>]> : !hal.device
hal.executable private @simple_mul_dispatch_0 {
hal.executable.variant public @cuda_nvptx_fb target(<"cuda", "cuda-nvptx-fb", {iree_codegen.target_info = #iree_gpu.target<arch = "sm_60", features = "+ptx76", wgp = <compute = fp64|fp32|fp16|int64|int32|int16|int8, storage = b64|b32|b16|b8, subgroup = shuffle|arithmetic, subgroup_size_choices = [32], max_workgroup_sizes = [1024, 1024, 1024], max_thread_count_per_workgroup = 1024, max_workgroup_memory_bytes = 49152, max_workgroup_counts = [2147483647, 65535, 65535]>>}>) {
hal.executable.export public @simple_mul_dispatch_0_elementwise_4_f32 ordinal(0) layout(#hal.pipeline.layout<bindings = [#hal.pipeline.binding<storage_buffer, "ReadOnly|Indirect">, #hal.pipeline.binding<storage_buffer, "ReadOnly|Indirect">, #hal.pipeline.binding<storage_buffer, Indirect>], flags = Indirect>) count(%arg0: !hal.device) -> (index, index, index) {
%c1 = arith.constant 1 : index
hal.return %c1, %c1, %c1 : index, index, index
} attributes {subgroup_size = 32 : index, workgroup_size = [32 : index, 1 : index, 1 : index]}
builtin.module {
llvm.func @simple_mul_dispatch_0_elementwise_4_f32(%arg0: !llvm.ptr<1> {llvm.align = 16 : i32, llvm.noalias, llvm.nonnull, llvm.noundef, llvm.readonly}, %arg1: !llvm.ptr<1> {llvm.align = 16 : i32, llvm.noalias, llvm.nonnull, llvm.noundef, llvm.readonly}, %arg2: !llvm.ptr<1> {llvm.align = 16 : i32, llvm.noalias, llvm.nonnull, llvm.noundef}) attributes {gpu.known_block_size = array<i32: 32, 1, 1>} {
%0 = llvm.mlir.constant(64 : i64) : i64
%1 = llvm.mlir.constant(true) : i1
%2 = llvm.mlir.constant(32 : i32) : i32
%3 = llvm.mlir.constant(4 : i32) : i32
%4 = nvvm.read.ptx.sreg.tid.x range <i32, 0, 32> : i32
%5 = llvm.sext %4 : i32 to i64
llvm.intr.assume %1 ["align"(%arg0, %0 : !llvm.ptr<1>, i64)] : i1
llvm.intr.assume %1 ["align"(%arg1, %0 : !llvm.ptr<1>, i64)] : i1
llvm.intr.assume %1 ["align"(%arg2, %0 : !llvm.ptr<1>, i64)] : i1
%6 = llvm.trunc %5 : i64 to i32
llvm.br ^bb1(%6 : i32)
^bb1(%7: i32): // 2 preds: ^bb0, ^bb2
%8 = llvm.icmp "slt" %7, %3 : i32
llvm.cond_br %8, ^bb2, ^bb3
^bb2: // pred: ^bb1
%9 = llvm.zext %7 : i32 to i64
%10 = llvm.getelementptr %arg0[%9] : (!llvm.ptr<1>, i64) -> !llvm.ptr<1>, f32
%11 = llvm.load %10 <alignment = 4> : !llvm.ptr<1> -> vector<1xf32>
%12 = llvm.getelementptr %arg1[%9] : (!llvm.ptr<1>, i64) -> !llvm.ptr<1>, f32
%13 = llvm.load %12 <alignment = 4> : !llvm.ptr<1> -> vector<1xf32>
%14 = llvm.fmul %11, %13 : vector<1xf32>
%15 = llvm.getelementptr %arg2[%9] : (!llvm.ptr<1>, i64) -> !llvm.ptr<1>, f32
llvm.store %14, %15 <alignment = 4> : vector<1xf32>, !llvm.ptr<1>
%16 = llvm.add %7, %2 : i32
llvm.br ^bb1(%16 : i32)
^bb3: // pred: ^bb1
llvm.return
}
}
}
}
util.func public @simple_mul(%arg0: !hal.buffer_view, %arg1: !hal.buffer_view) -> !hal.buffer_view attributes {iree.abi.stub, iree.reflection = {iree.abi.declaration = "sync func @simple_mul(%input0: tensor<4xf32>, %input1: tensor<4xf32>) -> (%output0: tensor<4xf32>)"}} {
%c0 = arith.constant 0 : index
%c16 = arith.constant 16 : index
%c4 = arith.constant 4 : index
%element_type_f32 = hal.element_type<f32> : i32
%dense_row_major = hal.encoding_type<dense_row_major> : i32
hal.buffer_view.assert<%arg0 : !hal.buffer_view> message("input0") shape([%c4]) type(%element_type_f32) encoding(%dense_row_major)
%0 = stream.tensor.import on(#hal.device.affinity<@__device_0>) %arg0 : !hal.buffer_view -> tensor<4xf32> in !stream.resource<external>{%c16}
hal.buffer_view.assert<%arg1 : !hal.buffer_view> message("input1") shape([%c4]) type(%element_type_f32) encoding(%dense_row_major)
%1 = stream.tensor.import on(#hal.device.affinity<@__device_0>) %arg1 : !hal.buffer_view -> tensor<4xf32> in !stream.resource<external>{%c16}
%result, %result_timepoint = stream.resource.alloca uninitialized on(#hal.device.affinity<@__device_0>) : !stream.resource<external>{%c16} => !stream.timepoint
%2 = stream.cmd.execute on(#hal.device.affinity<@__device_0>) await(%result_timepoint) => with(%0 as %arg2: !stream.resource<external>{%c16}, %1 as %arg3: !stream.resource<external>{%c16}, %result as %arg4: !stream.resource<external>{%c16}) {
stream.cmd.dispatch @simple_mul_dispatch_0::@cuda_nvptx_fb::@simple_mul_dispatch_0_elementwise_4_f32 {
ro %arg2[%c0 for %c16] : !stream.resource<external>{%c16},
ro %arg3[%c0 for %c16] : !stream.resource<external>{%c16},
wo %arg4[%c0 for %c16] : !stream.resource<external>{%c16}
}
} => !stream.timepoint
%3 = stream.timepoint.await %2 => %result : !stream.resource<external>{%c16}
%4 = stream.tensor.export on(#hal.device.affinity<@__device_0>) %3 : tensor<4xf32> in !stream.resource<external>{%c16} -> !hal.buffer_view
util.return %4 : !hal.buffer_view
}
}
8. HAL:封装 Kernel 并构造运行时执行
HAL 完成下一次重要转换:序列化设备实现,将外围调度转换为显式 HAL 操作。
Kernel 成为可执行数据
variant 中的 LLVM/NVVM 模块被替换为:
hal.executable.binary public @cuda_nvptx_fb attributes {
data = dense<"0x..."> : vector<1484xi8>,
format = "cuda-nvptx-fb",
mime_type = "application/x-flatbuffers"
}
这份 1,484 字节载荷包含 CUDA executable FlatBuffer,包括 PTX 和入口元数据。解码后可见:
.version 7.6
.target sm_60
.address_size 64
核心指令读取 %tid.x,跳过 ID 大于 3 的线程,加载 A、B,执行 mul.rn.f32,再存储 C。由于每个参与线程最多执行一次迭代,早先的循环简化为一次边界检查。
包内是 PTX,之后由 CUDA 加载路径和驱动继续处理为可执行设备代码。
初始化创建可复用状态
util.initializer 枚举运行时设备,检查 CUDA 和可执行格式支持,再创建 !hal.executable。另一个初始化器通过 @__simple_mul_memoize_apply 录制并缓存命令缓冲区。
录制使用绑定槽 0、1、2 和固定 workgroup 数量 (1, 1, 1),并记录执行屏障、结束命令缓冲区。该屏障约束命令执行顺序,不是 CUDA 线程间屏障。
初始化准备 executable 与命令对象,并不执行模型乘法。
每次调用提供资源并提交工作
公共函数检查输入视图和底层缓冲区,用 hal.device.queue.alloca 获取 16 字节输出,再通过 hal.device.queue.execute.indirect 提交缓存命令缓冲区。
间接提交为已录制槽位提供实际 A、B、C 缓冲区,将稳定命令结构与逐次调用数据分离。依赖可以概括为:
Allocate C ──signal allocation fence──► Execute cached command buffer
│
signal completion fence
│
Await completion, return C
hal.buffer_view.create 将结果包装为形状 [4]、类型 f32 的视图。返回视图本身不会复制到 CPU。函数期待兼容输入缓冲区,调用工具执行的上传在本函数 IR 之外。
HAL 完整结果:
module {
util.global private @__device_0 : !hal.device
util.initializer {
%c18_i32 = arith.constant 18 : i32
%false = arith.constant false
%c0 = arith.constant 0 : index
%c1 = arith.constant 1 : index
%0 = util.null : !hal.device
%device_count = hal.devices.count : index
cf.br ^bb1(%c0, %c0, %0 : index, index, !hal.device)
^bb1(%1: index, %2: index, %3: !hal.device): // 2 preds: ^bb0, ^bb4
%4 = util.cmp.eq %3, %0 : !hal.device
%5 = arith.cmpi slt, %1, %device_count : index
%6 = arith.andi %4, %5 : i1
cf.cond_br %6, ^bb2, ^bb5
^bb2: // pred: ^bb1
%device_n = hal.devices.get %1 : !hal.device
%ok, %value = hal.device.query<%device_n : !hal.device> key("hal.device.id" :: "cuda") : i1, i1 = false
cf.cond_br %value, ^bb3, ^bb4(%false : i1)
^bb3: // pred: ^bb2
%ok_0, %value_1 = hal.device.query<%device_n : !hal.device> key("hal.executable.format" :: "cuda-nvptx-fb") : i1, i1 = false
cf.br ^bb4(%value_1 : i1)
^bb4(%7: i1): // 2 preds: ^bb2, ^bb3
%8 = arith.cmpi eq, %2, %c0 : index
%9 = arith.select %7, %c1, %c0 : index
%10 = arith.addi %2, %9 : index
%11 = arith.andi %7, %8 : i1
%12 = arith.select %11, %device_n, %0 : !hal.device
%13 = arith.addi %1, %c1 : index
cf.br ^bb1(%13, %10, %12 : index, index, !hal.device)
^bb5: // pred: ^bb1
cf.cond_br %4, ^bb6, ^bb7
^bb6: // pred: ^bb5
util.status.check_ok %c18_i32, "HAL device `__device_0` not found or unavailable: #hal.device.target<\22cuda\22, [#hal.executable.target<\22cuda\22, \22cuda-nvptx-fb\22, {iree_codegen.target_info = #iree_gpu.target<arch = \22sm_60\22, features = \22+ptx76\22, wgp = <compute = fp64|fp32|fp16|int64|int32|int16|int8, storage = b64|b32|b16|b8, subgroup = shuffle|arithmetic, subgroup_size_choices = [32], max_workgroup_sizes = [1024, 1024, 1024], max_thread_count_per_workgroup = 1024, max_workgroup_memory_bytes = 49152, max_workgroup_counts = [2147483647, 65535, 65535]>>}>]>"
cf.br ^bb7
^bb7: // 2 preds: ^bb5, ^bb6
util.global.store %3, @__device_0 : !hal.device
util.return
}
util.global private @__device_0_query_0_hal_executable_format_cuda_nvptx_fb : i1
util.initializer {
%__device_0 = util.global.load @__device_0 : !hal.device
%ok, %value = hal.device.query<%__device_0 : !hal.device> key("hal.executable.format" :: "cuda-nvptx-fb") : i1, i1 = false
util.global.store %value, @__device_0_query_0_hal_executable_format_cuda_nvptx_fb : i1
util.return
}
util.global private @__device_0_executable_0_simple_mul_dispatch_0 : !hal.executable
util.initializer {
%c-1_i64 = arith.constant -1 : i64
%c-1 = arith.constant -1 : index
%c0 = arith.constant 0 : index
%c14_i32 = arith.constant 14 : i32
%0 = util.null : !hal.executable
%__device_0_query_0_hal_executable_format_cuda_nvptx_fb = util.global.load @__device_0_query_0_hal_executable_format_cuda_nvptx_fb : i1
%__device_0 = util.global.load @__device_0 : !hal.device
%1 = arith.select %__device_0_query_0_hal_executable_format_cuda_nvptx_fb, %c0, %c-1 : index
%2 = arith.cmpi eq, %1, %c0 : index
cf.cond_br %2, ^bb1, ^bb2
^bb1: // pred: ^bb0
%executable = hal.executable.create device(%__device_0 : !hal.device) affinity(%c-1_i64) target(@simple_mul_dispatch_0::@cuda_nvptx_fb) : !hal.executable
cf.br ^bb3(%executable : !hal.executable)
^bb2: // pred: ^bb0
util.status.check_ok %c14_i32, "HAL device `__device_0` does not support any variant of executable `simple_mul_dispatch_0`; available formats: [cuda-nvptx-fb]"
cf.br ^bb3(%0 : !hal.executable)
^bb3(%3: !hal.executable): // 2 preds: ^bb1, ^bb2
util.global.store %3, @__device_0_executable_0_simple_mul_dispatch_0 : !hal.executable
util.return
}
hal.executable private @simple_mul_dispatch_0 {
hal.executable.binary public @cuda_nvptx_fb attributes {data = dense<"..."> : vector<1484xi8>, format = "cuda-nvptx-fb", mime_type = "application/x-flatbuffers"}
}
util.func private @__simple_mul_memoize_apply() -> !hal.command_buffer attributes {inlining_policy = #util.inline.never} {
%c-1_i64 = arith.constant -1 : i64
%c3 = arith.constant 3 : index
%c0 = arith.constant 0 : index
%c1 = arith.constant 1 : index
%c2 = arith.constant 2 : index
%c16 = arith.constant 16 : index
%__device_0 = util.global.load immutable @__device_0 : !hal.device
%__device_0_executable_0_simple_mul_dispatch_0 = util.global.load immutable @__device_0_executable_0_simple_mul_dispatch_0 : !hal.executable
%cmd = hal.command_buffer.create device(%__device_0 : !hal.device) mode("None") categories("Transfer|Dispatch") affinity(%c-1_i64) bindings(%c3) : !hal.command_buffer
%function_id = hal.executable.lookup.function target(%__device_0_executable_0_simple_mul_dispatch_0 : !hal.executable) function(@simple_mul_dispatch_0::@cuda_nvptx_fb::@simple_mul_dispatch_0_elementwise_4_f32) : i64
hal.command_buffer.dispatch<%cmd : !hal.command_buffer> target(%__device_0_executable_0_simple_mul_dispatch_0 : !hal.executable)[%function_id] workgroups([%c1, %c1, %c1]) bindings([
(%c0 : index)[%c0, %c16],
(%c1 : index)[%c0, %c16],
(%c2 : index)[%c0, %c16]
]) flags("None")
hal.command_buffer.execution_barrier<%cmd : !hal.command_buffer> source("Dispatch|Transfer|CommandRetire") target("CommandIssue|Dispatch|Transfer") flags("None")
hal.command_buffer.finalize<%cmd : !hal.command_buffer>
util.return %cmd : !hal.command_buffer
}
util.global private @__simple_mul_memoize_result_0_device_0 : !hal.command_buffer
util.initializer {
%0 = util.call @__simple_mul_memoize_apply() : () -> !hal.command_buffer
util.global.store %0, @__simple_mul_memoize_result_0_device_0 : !hal.command_buffer
util.return
}
util.func public @simple_mul(%arg0: !hal.buffer_view, %arg1: !hal.buffer_view) -> !hal.buffer_view attributes {iree.abi.stub, iree.reflection = {iree.abi.declaration = "sync func @simple_mul(%input0: tensor<4xf32>, %input1: tensor<4xf32>) -> (%output0: tensor<4xf32>)"}} {
%c-1_i32 = arith.constant -1 : i32
%c0_i64 = arith.constant 0 : i64
%0 = util.null : !hal.fence
%c-1_i64 = arith.constant -1 : i64
%c0 = arith.constant 0 : index
%c16 = arith.constant 16 : index
%c4 = arith.constant 4 : index
%memory_type = hal.memory_type<"DeviceVisible|DeviceLocal"> : i32
%buffer_usage = hal.buffer_usage<"TransferSource|TransferTarget|Transfer|DispatchStorageRead|DispatchStorageWrite|DispatchStorage"> : i32
%__device_0 = util.global.load immutable @__device_0 : !hal.device
%__simple_mul_memoize_result_0_device_0 = util.global.load immutable @__simple_mul_memoize_result_0_device_0 : !hal.command_buffer
%element_type_f32 = hal.element_type<f32> : i32
%dense_row_major = hal.encoding_type<dense_row_major> : i32
hal.buffer_view.assert<%arg0 : !hal.buffer_view> message("input0") shape([%c4]) type(%element_type_f32) encoding(%dense_row_major)
%buffer = hal.buffer_view.buffer<%arg0 : !hal.buffer_view> : !hal.buffer
%allocator = hal.device.allocator<%__device_0 : !hal.device> : !hal.allocator
hal.buffer.assert<%buffer : !hal.buffer> message("tensor") allocator(%allocator : !hal.allocator) minimum_length(%c16) type(DeviceVisible) usage("TransferSource|TransferTarget|Transfer|DispatchStorageRead|DispatchStorageWrite|DispatchStorage")
hal.buffer_view.assert<%arg1 : !hal.buffer_view> message("input1") shape([%c4]) type(%element_type_f32) encoding(%dense_row_major)
%buffer_0 = hal.buffer_view.buffer<%arg1 : !hal.buffer_view> : !hal.buffer
hal.buffer.assert<%buffer_0 : !hal.buffer> message("tensor") allocator(%allocator : !hal.allocator) minimum_length(%c16) type(DeviceVisible) usage("TransferSource|TransferTarget|Transfer|DispatchStorageRead|DispatchStorageWrite|DispatchStorage")
%fence = hal.fence.create device(%__device_0 : !hal.device) flags("None") : !hal.fence
%transient_buffer = hal.device.queue.alloca<%__device_0 : !hal.device> affinity(%c-1_i64) wait(%0) signal(%fence) pool(%c0_i64) type(%memory_type) usage(%buffer_usage) flags("None") : !hal.buffer{%c16}
%fence_1 = hal.fence.create device(%__device_0 : !hal.device) flags("None") : !hal.fence
hal.device.queue.execute.indirect<%__device_0 : !hal.device> affinity(%c-1_i64) wait(%fence) signal(%fence_1) commands(%__simple_mul_memoize_result_0_device_0) bindings([
(%buffer : !hal.buffer)[%c0, %c16],
(%buffer_0 : !hal.buffer)[%c0, %c16],
(%transient_buffer : !hal.buffer)[%c0, %c16]
]) flags("None")
%status = hal.fence.await until([%fence_1]) timeout_millis(%c-1_i32) flags("None") : i32
util.status.check_ok %status, "failed to wait on timepoint"
%view = hal.buffer_view.create buffer(%transient_buffer : !hal.buffer)[%c0, %c16] shape([%c4]) type(%element_type_f32) encoding(%dense_row_major) : !hal.buffer_view
util.return %view : !hal.buffer_view
}
}
9. VM:将编排编码为可移植控制代码
最后 VM 流水线生成 vm.module,其中包含函数、导入、导出、全局变量和只读数据。
HAL 对象变为 VM 引用:
!hal.buffer_view → !vm.ref<!hal.buffer_view>
!hal.device → !vm.ref<!hal.device>
HAL 操作变为运行时导入调用:
vm.import private @hal.buffer_view.buffer(
%view: !vm.ref<!hal.buffer_view>) -> !vm.ref<!hal.buffer>
%buffer = vm.call @hal.buffer_view.buffer(%view)
: (!vm.ref<!hal.buffer_view>) -> !vm.ref<!hal.buffer>
这是运行时边界:VM 执行控制指令并调用管理设备资源的 HAL 模块。导入的 minimum_version 约束 HAL 接口版本,不是 LLVM 版本。字符串和 executable 字节变为 vm.rodata,CUDA 载荷仍为 1,484 字节。初始化调用 @hal.executable.create 时,vm.const.ref.rodata 提供其引用。
等待可以挂起 VM 执行
fence 等待转换为:
vm.call.variadic.yieldable @hal.fence.await(...)
-> ^resume(i32)
导入立即完成时,控制流携带结果进入后继;若让出执行,则等待后在该处恢复。后继检查状态,成功时创建输出视图,失败时清理并执行 vm.fail。
同步公共 ABI 要求返回结果前完成工作。内部 VM 可以支持挂起和恢复,HAL 同步模式也可能阻塞原生线程。同步的应用契约并不要求每一层采用相同的等待实现。
module attributes {vm.toplevel} {
vm.module public @module {
vm.global.ref private mutable @__device_0 : !vm.ref<!hal.device>
vm.global.ref private mutable @__device_0_executable_0_simple_mul_dispatch_0 : !vm.ref<!hal.executable>
vm.global.ref private mutable @__simple_mul_memoize_result_0_device_0 : !vm.ref<!hal.command_buffer>
vm.rodata private @_utf8_hal_device_id_C6650FF277232B5A {alignment = 1 : i64} "hal.device.id"
vm.rodata private @_utf8_cuda_342A062BE26F4639 {alignment = 1 : i64} "cuda"
vm.rodata private @_utf8_hal_executable_format_E03EECB63A2AAF52 {alignment = 1 : i64} "hal.executable.format"
vm.rodata private @_utf8_cuda_nvptx_fb_91E5BBD19AE47511 {alignment = 1 : i64} "cuda-nvptx-fb"
vm.rodata private @simple_mul_dispatch_0_cuda_nvptx_fb {alignment = 16 : i64, mime_type = "application/x-flatbuffers"} dense<"0x43444131000000008C0500000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000008000000434441319AFAFFFF08000000100100000100000004000000A6FAFFFF640000002000000001000000010000000C0000007C00000020000000030000000300000000000000030000000000000002000000000000002700000073696D706C655F6D756C5F64697370617463685F305F656C656D656E74776973655F345F663332002700000073696D706C655F6D756C5F64697370617463685F305F656C656D656E74776973655F345F6633320042FBFFFF40000000040000004EFBFFFF08000000010000002600000073616D706C65732F7374617469635F6C6962726172792F73696D706C655F6D756C2E6D6C697200002700000073696D706C655F6D756C5F64697370617463685F305F656C656D656E74776973655F345F663332000100000004000000C0FBFFFF04000000330400002F2F0A2F2F2047656E657261746564206279204C4C564D204E56505458204261636B2D456E640A2F2F0A0A2E76657273696F6E20372E360A2E74617267657420736D5F36300A2E616464726573735F73697A652036340A0A092F2F202E676C6F626C0973696D706C655F6D756C5F64697370617463685F305F656C656D656E74776973655F345F6633320A0A2E76697369626C65202E656E7472792073696D706C655F6D756C5F64697370617463685F305F656C656D656E74776973655F345F663332280A092E706172616D202E753634202E707472202E676C6F62616C202E616C69676E2031362073696D706C655F6D756C5F64697370617463685F305F656C656D656E74776973655F345F6633325F706172616D5F302C0A092E706172616D202E753634202E707472202E676C6F62616C202E616C69676E2031362073696D706C655F6D756C5F64697370617463685F305F656C656D656E74776973655F345F6633325F706172616D5F312C0A092E706172616D202E753634202E707472202E676C6F62616C202E616C69676E2031362073696D706C655F6D756C5F64697370617463685F305F656C656D656E74776973655F345F6633325F706172616D5F320A290A2E6D61786E7469642033322C20312C20310A7B0A092E726567202E70726564200925703C323E3B0A092E726567202E623332200925723C353E3B0A092E726567202E62363420092572643C383E3B0A0A096D6F762E75333220092572312C20257469642E783B0A09736574702E67742E75333220092570312C202572312C20333B0A0940257031206272612009244C5F5F4242305F323B0A096C642E706172616D2E6236342009257264342C205B73696D706C655F6D756C5F64697370617463685F305F656C656D656E74776973655F345F6633325F706172616D5F305D3B0A096C642E706172616D2E6236342009257264352C205B73696D706C655F6D756C5F64697370617463685F305F656C656D656E74776973655F345F6633325F706172616D5F315D3B0A096C642E706172616D2E6236342009257264362C205B73696D706C655F6D756C5F64697370617463685F305F656C656D656E74776973655F345F6633325F706172616D5F325D3B0A096D756C2E776964652E7533322009257264372C202572312C20343B0A096164642E7336342009257264312C20257264342C20257264373B0A096164642E7336342009257264322C20257264352C20257264373B0A096164642E7336342009257264332C20257264362C20257264373B0A096C642E676C6F62616C2E6E632E62333220092572322C205B257264315D3B0A096C642E676C6F62616C2E6E632E62333220092572332C205B257264325D3B0A096D756C2E726E2E66333220092572342C202572322C202572333B0A0973742E676C6F62616C2E62333220095B257264335D2C202572343B0A244C5F5F4242305F323A0A097265743B0A0A7D0A0006000800040008000C00040008001400200000000400080000000000140018001C000000"> : vector<1484xi8>
vm.rodata private @_utf8_simple_mul_dispatch_0_elem_BAECB3686C800B2E {alignment = 1 : i64} "simple_mul_dispatch_0_elementwise_4_f32"
vm.func private @__simple_mul_memoize_apply() -> !vm.ref<!hal.command_buffer> attributes {vm.unwind, inlining_policy = #util.inline.never} {
%c13 = vm.const.i32 13
%c28 = vm.const.i32 28
%c2 = vm.const.i32 2
%null = vm.const.ref.zero : !vm.ref<!hal.buffer>
%c1 = vm.const.i32 1
%c3 = vm.const.i32 3
%zero = vm.const.i32.zero
%c16 = vm.const.i64 16
%zero_0 = vm.const.i64.zero
%c-1 = vm.const.i64 -1
%__device_0 = vm.global.load.ref @__device_0 : !vm.ref<!hal.device>
%__device_0_executable_0_simple_mul_dispatch_0 = vm.global.load.ref @__device_0_executable_0_simple_mul_dispatch_0 : !vm.ref<!hal.executable>
%ref = vm.call @hal.command_buffer.create(%__device_0, %zero, %c3, %c-1, %c3) : (!vm.ref<!hal.device>, i32, i32, i64, i32) -> !vm.ref<!hal.command_buffer>
%_utf8_simple_mul_dispatch_0_elem_BAECB3686C800B2E = vm.const.ref.rodata @_utf8_simple_mul_dispatch_0_elem_BAECB3686C800B2E : !vm.buffer
%0 = vm.call @hal.executable.lookup.function(%__device_0_executable_0_simple_mul_dispatch_0, %_utf8_simple_mul_dispatch_0_elem_BAECB3686C800B2E) {nosideeffects} : (!vm.ref<!hal.executable>, !vm.buffer) -> i64
vm.call.variadic @hal.command_buffer.dispatch(%ref, %__device_0_executable_0_simple_mul_dispatch_0, %0, %c1, %c1, %c1, %zero_0, [], [(%zero, %zero, %null, %zero_0, %c16), (%zero, %c1, %null, %zero_0, %c16), (%zero, %c2, %null, %zero_0, %c16)]) : (!vm.ref<!hal.command_buffer>, !vm.ref<!hal.executable>, i64, i32, i32, i32, i64, i32 ..., tuple<i32, i32, !vm.ref<!hal.buffer>, i64, i64> ...)
vm.call @hal.command_buffer.execution_barrier(%ref, %c28, %c13, %zero_0) : (!vm.ref<!hal.command_buffer>, i32, i32, i64) -> ()
vm.call @hal.command_buffer.finalize(%ref) : (!vm.ref<!hal.command_buffer>) -> ()
vm.return %ref : !vm.ref<!hal.command_buffer>
}
vm.import private @hal.buffer.assert(%buffer : !vm.ref<!hal.buffer>, %message : !vm.buffer, %allocator : !vm.ref<!hal.allocator>, %minimum_length : i64, %memory_types : i32, %buffer_usage : i32)
vm.import private @hal.buffer_view.create(%buffer : !vm.ref<!hal.buffer>, %source_offset : i64, %source_length : i64, %element_type : i32, %encoding_type : i32, %shape : i64 ...) -> !vm.ref<!hal.buffer_view> attributes {nosideeffects}
vm.import private @hal.buffer_view.assert(%buffer_view : !vm.ref<!hal.buffer_view>, %message : !vm.buffer, %element_type : i32, %encoding_type : i32, %shape : i64 ...)
vm.import private @hal.buffer_view.buffer(%buffer_view : !vm.ref<!hal.buffer_view>) -> !vm.ref<!hal.buffer> attributes {nosideeffects}
vm.import private @hal.command_buffer.create(%device : !vm.ref<!hal.device>, %modes : i32, %command_categories : i32, %queue_affinity : i64, %binding_capacity : i32) -> !vm.ref<!hal.command_buffer> attributes {minimum_version = 6 : i32}
vm.import private @hal.command_buffer.finalize(%command_buffer : !vm.ref<!hal.command_buffer>)
vm.import private @hal.command_buffer.execution_barrier(%command_buffer : !vm.ref<!hal.command_buffer>, %source_stage_mask : i32, %target_stage_mask : i32, %flags : i64)
vm.import private @hal.command_buffer.dispatch(%command_buffer : !vm.ref<!hal.command_buffer>, %executable : !vm.ref<!hal.executable>, %function_id : i64, %workgroup_x : i32, %workgroup_y : i32, %workgroup_z : i32, %flags : i64, %constants : i32 ..., %bindings : tuple<i32, i32, !vm.ref<!hal.buffer>, i64, i64> ...) attributes {minimum_version = 7 : i32}
vm.import private @hal.device.allocator(%device : !vm.ref<!hal.device>) -> !vm.ref<!hal.allocator> attributes {nosideeffects}
vm.import private @hal.device.query.i64(%device : !vm.ref<!hal.device>, %category : !vm.buffer, %key : !vm.buffer) -> (i32, i64) attributes {nosideeffects}
vm.import private @hal.device.queue.alloca(%device : !vm.ref<!hal.device>, %queue_affinity : i64, %wait_fence : !vm.ref<!hal.fence>, %signal_fence : !vm.ref<!hal.fence>, %pool : i64, %memory_types : i32, %buffer_usage : i32, %allocation_size : i64, %flags : i64) -> !vm.ref<!hal.buffer>
vm.import private @hal.device.queue.execute.indirect(%device : !vm.ref<!hal.device>, %queue_affinity : i64, %wait_fence : !vm.ref<!hal.fence>, %signal_fence : !vm.ref<!hal.fence>, %command_buffer : !vm.ref<!hal.command_buffer>, %flags : i64, %binding_table : tuple<!vm.ref<!hal.buffer>, i64, i64> ...)
vm.import private @hal.devices.count() -> i32 attributes {nosideeffects}
vm.import private @hal.devices.get(%index : i32) -> !vm.ref<!hal.device> attributes {nosideeffects}
vm.import private @hal.executable.create(%device : !vm.ref<!hal.device>, %queue_affinity : i64, %executable_format : !vm.buffer, %executable_data : !vm.buffer, %constants : !vm.buffer) -> !vm.ref<!hal.executable> attributes {nosideeffects}
vm.import private @hal.executable.lookup.function(%executable : !vm.ref<!hal.executable>, %function_name : !vm.buffer) -> i64 attributes {nosideeffects, minimum_version = 7 : i32}
vm.import private @hal.fence.create(%device : !vm.ref<!hal.device>, %flags : i64) -> !vm.ref<!hal.fence>
vm.import private @hal.fence.await(%timeout_millis : i32, %flags : i64, %fences : !vm.ref<!hal.fence> ...) -> i32 attributes {vm.yield}
vm.rodata private @_utf8_input0_DCE99660CEB3F6B {alignment = 1 : i64} "input0"
vm.rodata private @_utf8_tensor_FC1814BC4A58F22A {alignment = 1 : i64} "tensor"
vm.rodata private @_utf8_input1_B898B726583C85DA {alignment = 1 : i64} "input1"
vm.func private @simple_mul(%arg0: !vm.ref<!hal.buffer_view>, %arg1: !vm.ref<!hal.buffer_view>) -> !vm.ref<!hal.buffer_view> attributes {iree.reflection = {iree.abi.declaration = "sync func @simple_mul(%input0: tensor<4xf32>, %input1: tensor<4xf32>) -> (%output0: tensor<4xf32>)"}, vm.unwind, vm.yield} {
%c16 = vm.const.i32 16
%c1 = vm.const.i32 1
%c553648160 = vm.const.i32 553648160
%c3075 = vm.const.i32 3075
%c48 = vm.const.i32 48
%c4 = vm.const.i64 4
%c16_0 = vm.const.i64 16
%zero = vm.const.i64.zero
%c-1 = vm.const.i64 -1
%null = vm.const.ref.zero : !vm.ref<!hal.fence>
%c-1_1 = vm.const.i32 -1
%__device_0 = vm.global.load.ref @__device_0 : !vm.ref<!hal.device>
%__simple_mul_memoize_result_0_device_0 = vm.global.load.ref @__simple_mul_memoize_result_0_device_0 : !vm.ref<!hal.command_buffer>
%_utf8_input0_DCE99660CEB3F6B = vm.const.ref.rodata @_utf8_input0_DCE99660CEB3F6B : !vm.buffer
vm.call.variadic @hal.buffer_view.assert(%arg0, %_utf8_input0_DCE99660CEB3F6B, %c553648160, %c1, [%c4]) : (!vm.ref<!hal.buffer_view>, !vm.buffer, i32, i32, i64 ...)
%ref = vm.call @hal.buffer_view.buffer(%arg0) {nosideeffects} : (!vm.ref<!hal.buffer_view>) -> !vm.ref<!hal.buffer>
%ref_2 = vm.call @hal.device.allocator(%__device_0) {nosideeffects} : (!vm.ref<!hal.device>) -> !vm.ref<!hal.allocator>
%_utf8_tensor_FC1814BC4A58F22A = vm.const.ref.rodata @_utf8_tensor_FC1814BC4A58F22A : !vm.buffer
vm.call @hal.buffer.assert(%ref, %_utf8_tensor_FC1814BC4A58F22A, %ref_2, %c16_0, %c16, %c3075) : (!vm.ref<!hal.buffer>, !vm.buffer, !vm.ref<!hal.allocator>, i64, i32, i32) -> ()
%_utf8_input1_B898B726583C85DA = vm.const.ref.rodata @_utf8_input1_B898B726583C85DA : !vm.buffer
vm.call.variadic @hal.buffer_view.assert(%arg1, %_utf8_input1_B898B726583C85DA, %c553648160, %c1, [%c4]) : (!vm.ref<!hal.buffer_view>, !vm.buffer, i32, i32, i64 ...)
%ref_3 = vm.call @hal.buffer_view.buffer(%arg1) {nosideeffects} : (!vm.ref<!hal.buffer_view>) -> !vm.ref<!hal.buffer>
vm.call @hal.buffer.assert(%ref_3, %_utf8_tensor_FC1814BC4A58F22A, %ref_2, %c16_0, %c16, %c3075) : (!vm.ref<!hal.buffer>, !vm.buffer, !vm.ref<!hal.allocator>, i64, i32, i32) -> ()
%ref_4 = vm.call @hal.fence.create(%__device_0, %zero) : (!vm.ref<!hal.device>, i64) -> !vm.ref<!hal.fence>
%ref_5 = vm.call @hal.device.queue.alloca(%__device_0, %c-1, %null, %ref_4, %zero, %c48, %c3075, %c16_0, %zero) : (!vm.ref<!hal.device>, i64, !vm.ref<!hal.fence>, !vm.ref<!hal.fence>, i64, i32, i32, i64, i64) -> !vm.ref<!hal.buffer>
%ref_6 = vm.call @hal.fence.create(%__device_0, %zero) : (!vm.ref<!hal.device>, i64) -> !vm.ref<!hal.fence>
vm.call.variadic @hal.device.queue.execute.indirect(%__device_0, %c-1, %ref_4, %ref_6, %__simple_mul_memoize_result_0_device_0, %zero, [(%ref, %zero, %c16_0), (%ref_3, %zero, %c16_0), (%ref_5, %zero, %c16_0)]) : (!vm.ref<!hal.device>, i64, !vm.ref<!hal.fence>, !vm.ref<!hal.fence>, !vm.ref<!hal.command_buffer>, i64, tuple<!vm.ref<!hal.buffer>, i64, i64> ...)
vm.call.variadic.yieldable @hal.fence.await(%c-1_1, %zero, %ref_6) {segment_sizes = dense<[-1, -1, 1]> : vector<3xi16>, segment_types = [i32, i64, !vm.ref<!hal.fence>]} : (i32, i64, !vm.ref<!hal.fence>) -> ^bb1 (i32)
^bb1(%0: i32): // pred: ^bb0
vm.cond_br %0, ^bb3, ^bb2
^bb2: // pred: ^bb1
%ref_7 = vm.call.variadic @hal.buffer_view.create(%ref_5, %zero, %c16_0, %c553648160, %c1, [%c4]) {nosideeffects} : (!vm.ref<!hal.buffer>, i64, i64, i32, i32, i64 ...) -> !vm.ref<!hal.buffer_view>
vm.return %ref_7 : !vm.ref<!hal.buffer_view>
^bb3: // pred: ^bb1
vm.discard.refs %ref_5 : !vm.ref<!hal.buffer>
vm.fail %0, "failed to wait on timepoint"
}
vm.export @simple_mul attributes {iree.abi.stub, iree.reflection = {iree.abi.declaration = "sync func @simple_mul(%input0: tensor<4xf32>, %input1: tensor<4xf32>) -> (%output0: tensor<4xf32>)"}}
vm.export @__init
vm.func private @__init() attributes {vm.unwind} {
%c1 = vm.const.i32 1
%null = vm.const.ref.zero : !vm.buffer
%c14 = vm.const.i32 14
%c-1 = vm.const.i64 -1
%c18 = vm.const.i32 18
%zero = vm.const.i32.zero
%zero_0 = vm.const.i64.zero
%c1_1 = vm.const.i64 1
%null_2 = vm.const.ref.zero : !vm.ref<!hal.device>
%0 = vm.call @hal.devices.count() {nosideeffects} : () -> i32
%1 = vm.ext.i32.i64.s %0 : i32 -> i64
vm.br ^bb1(%zero_0, %zero_0, %null_2 : i64, i64, !vm.ref<!hal.device>)
^bb1(%2: i64, %3: i64, %4: !vm.ref<!hal.device>): // 2 preds: ^bb0, ^bb4
%rnz = vm.cmp.nz.ref %4 : !vm.ref<!hal.device>
%5 = vm.xor.i32 %rnz, %c1 : i32
%slt = vm.cmp.lt.i64.s %2, %1 : i64
%6 = vm.and.i32 %5, %slt : i32
vm.cond_br %6, ^bb2, ^bb5
^bb2: // pred: ^bb1
vm.discard.refs %4 : !vm.ref<!hal.device>
%7 = vm.trunc.i64.i32 %2 : i64 -> i32
%ref = vm.call @hal.devices.get(%7) {nosideeffects} : (i32) -> !vm.ref<!hal.device>
%_utf8_hal_device_id_C6650FF277232B5A = vm.const.ref.rodata @_utf8_hal_device_id_C6650FF277232B5A : !vm.buffer
%_utf8_cuda_342A062BE26F4639 = vm.const.ref.rodata @_utf8_cuda_342A062BE26F4639 : !vm.buffer
%8:2 = vm.call @hal.device.query.i64(%ref, %_utf8_hal_device_id_C6650FF277232B5A, %_utf8_cuda_342A062BE26F4639) {nosideeffects} : (!vm.ref<!hal.device>, !vm.buffer, !vm.buffer) -> (i32, i64)
%nz = vm.cmp.nz.i64 %8#1 : i64
%9 = vm.select.i32 %8#0, %nz, %zero : i32
vm.cond_br %9, ^bb3, ^bb4(%zero : i32)
^bb3: // pred: ^bb2
%_utf8_hal_executable_format_E03EECB63A2AAF52 = vm.const.ref.rodata @_utf8_hal_executable_format_E03EECB63A2AAF52 : !vm.buffer
%_utf8_cuda_nvptx_fb_91E5BBD19AE47511 = vm.const.ref.rodata @_utf8_cuda_nvptx_fb_91E5BBD19AE47511 : !vm.buffer
%10:2 = vm.call @hal.device.query.i64(%ref, %_utf8_hal_executable_format_E03EECB63A2AAF52, %_utf8_cuda_nvptx_fb_91E5BBD19AE47511) {nosideeffects} : (!vm.ref<!hal.device>, !vm.buffer, !vm.buffer) -> (i32, i64)
%nz_3 = vm.cmp.nz.i64 %10#1 : i64
%11 = vm.select.i32 %10#0, %nz_3, %zero : i32
vm.br ^bb4(%11 : i32)
^bb4(%12: i32): // 2 preds: ^bb2, ^bb3
%eq = vm.cmp.eq.i64 %3, %zero_0 : i64
%13 = vm.select.i64 %12, %c1_1, %zero_0 : i64
%14 = vm.add.i64 %3, %13 : i64
%15 = vm.and.i32 %12, %eq : i32
%ref_4 = vm.select.ref %15, %ref, %null_2 : !vm.ref<!hal.device>
%16 = vm.add.i64 %2, %c1_1 : i64
vm.br ^bb1(%16, %14, %ref_4 : i64, i64, !vm.ref<!hal.device>)
^bb5: // pred: ^bb1
vm.discard.refs %null_2 : !vm.ref<!hal.device>
vm.cond_br %5, ^bb6, ^bb7
^bb6: // pred: ^bb5
vm.discard.refs %null, %4 : !vm.buffer, !vm.ref<!hal.device>
vm.fail %c18, "HAL device `__device_0` not found or unavailable: #hal.device.target<\22cuda\22, [#hal.executable.target<\22cuda\22, \22cuda-nvptx-fb\22, {iree_codegen.target_info = #iree_gpu.target<arch = \22sm_60\22, features = \22+ptx76\22, wgp = <compute = fp64|fp32|fp16|int64|int32|int16|int8, storage = b64|b32|b16|b8, subgroup = shuffle|arithmetic, subgroup_size_choices = [32], max_workgroup_sizes = [1024, 1024, 1024], max_thread_count_per_workgroup = 1024, max_workgroup_memory_bytes = 49152, max_workgroup_counts = [2147483647, 65535, 65535]>>}>]>"
^bb7: // pred: ^bb5
%_utf8_hal_executable_format_E03EECB63A2AAF52_5 = vm.const.ref.rodata @_utf8_hal_executable_format_E03EECB63A2AAF52 : !vm.buffer
%_utf8_cuda_nvptx_fb_91E5BBD19AE47511_6 = vm.const.ref.rodata @_utf8_cuda_nvptx_fb_91E5BBD19AE47511 : !vm.buffer
%17:2 = vm.call @hal.device.query.i64(%4, %_utf8_hal_executable_format_E03EECB63A2AAF52_5, %_utf8_cuda_nvptx_fb_91E5BBD19AE47511_6) {nosideeffects} : (!vm.ref<!hal.device>, !vm.buffer, !vm.buffer) -> (i32, i64)
%nz_7 = vm.cmp.nz.i64 %17#1 : i64
%18 = vm.select.i32 %17#0, %nz_7, %zero : i32
%19 = vm.select.i64 %18, %zero_0, %c-1 : i64
%eq_8 = vm.cmp.eq.i64 %19, %zero_0 : i64
vm.global.store.ref %4, @__device_0 : !vm.ref<!hal.device>
vm.cond_br %eq_8, ^bb8, ^bb9
^bb8: // pred: ^bb7
%simple_mul_dispatch_0_cuda_nvptx_fb = vm.const.ref.rodata @simple_mul_dispatch_0_cuda_nvptx_fb : !vm.buffer
%ref_9 = vm.call @hal.executable.create(%4, %c-1, %_utf8_cuda_nvptx_fb_91E5BBD19AE47511_6, %simple_mul_dispatch_0_cuda_nvptx_fb, %null) {nosideeffects} : (!vm.ref<!hal.device>, i64, !vm.buffer, !vm.buffer, !vm.buffer) -> !vm.ref<!hal.executable>
vm.global.store.ref %ref_9, @__device_0_executable_0_simple_mul_dispatch_0 : !vm.ref<!hal.executable>
%ref_10 = vm.call @__simple_mul_memoize_apply() : () -> !vm.ref<!hal.command_buffer>
vm.global.store.ref %ref_10, @__simple_mul_memoize_result_0_device_0 : !vm.ref<!hal.command_buffer>
vm.return
^bb9: // pred: ^bb7
vm.discard.refs %null, %4, %_utf8_cuda_nvptx_fb_91E5BBD19AE47511_6 : !vm.buffer, !vm.ref<!hal.device>, !vm.buffer
vm.fail %c14, "HAL device `__device_0` does not support any variant of executable `simple_mul_dispatch_0`; available formats: [cuda-nvptx-fb]"
}
}
}