clpeak result format v3

clpeak -o run.clpeak.json writes one JSON document describing one run. It is the only format clpeak writes and the only one it reads back (--compare, and the GUI’s run history).

The document is a tree — run → devices → tests → metrics — which is the shape the CLI prints and the GUI renders, so nothing has to regroup a flat table and guess at what belonged together. Beside the tree it carries the run’s diagnostic stream (log), so a file from a machine nobody can reach explains its own numbers: with --verbose it holds everything the terminal would have shown, scoped and timed, plus the device inventory.

Everything here is produced by src/common/run_document.cpp and modelled by include/common/run_document.h; the Dart side mirrors it in app/lib/src/model/result_model.dart and run_document.dart.

Example

{
  "schema": "clpeak/run",
  "format_version": 3,
  "clpeak_version": "3.0.0-12-gabc1234",
  "generated_at": "2026-08-29T14:03:11Z",
  "duration_s": 148.24,
  "cancelled": false,

  "build": { "backends": ["CUDA", "Vulkan", "OpenCL", "CPU", "ONNX"] },

  "host": {
    "os": "Macintosh", "os_version": "26.6.2", "arch": "arm64",
    "cpu": "Apple M1 Pro", "logical_cores": 10, "memory_bytes": 34359738368
  },

  "invocation": {
    "argv": ["clpeak", "--verbose", "-o", "run.clpeak.json"],
    "target_time_us": 500000, "target_time_us_cpu": 2000000, "warmup": 2,
    "categories": ["compute", "bandwidth", "latency"],
    "verbose": true
  },

  "devices": [{
    "backend": "CUDA", "platform": "CUDA",
    "name": "NVIDIA GeForce RTX 5060", "driver": "580.65.06",
    "type": "gpu", "device_index": 0,
    "properties": [ { "key": "Arch", "value": "sm_120" } ],

    "tests": [{
      "id": "cublas_gemm_fp",
      "title": "cuBLASLt GEMM peak",
      "category": "compute",
      "shape": "heterogeneous",
      "axis": "data type",
      "direction": "higher_is_better",
      "quantity": "flops",
      "unit": "FLOPS",
      "description": "Matrix-multiply speed through NVIDIA's own tuned library…",
      "duration_s": 3.42,
      "metrics": [
        { "id": "fp32", "value": 14.87 },
        { "id": "nvf4_e2m1", "value": 300.43,
          "description": "Four-bit floats with NVIDIA's own scaling…" },
        { "id": "mxf4_e2m1", "status": "unsupported",
          "reason": "FP4 tensor cores require Blackwell — unsupported on sm_120" }
      ]
    }]
  }],

  "inventory": [
    { "name": "CUDA", "flag": "cuda", "available": true,
      "platforms": [{ "index": 0, "name": "CUDA",
        "devices": [{ "index": 0, "name": "NVIDIA GeForce RTX 5060", "type": "GPU",
                      "arch": "sm_120", "compute_units": 36, "global_mem_bytes": 8589934592 }] }] },
    { "name": "ONNX", "flag": "onnx", "available": false,
      "reason": "onnxruntime library not found", "platforms": [] }
  ],

  "log": [
    {"elapsed_s": 0.31, "level": "warning", "backend": "ONNX", "message": "ONNX: onnxruntime library not found"},
    {"elapsed_s": 4.2, "level": "debug", "backend": "CUDA", "device": "NVIDIA GeForce RTX 5060", "device_index": 0, "test": "global_memory_bandwidth", "message": "global_memory_bandwidth: working set 1024 MB, L2 32 MB"},
    {"elapsed_s": 9.77, "level": "warning", "source": "vulkan", "backend": "Vulkan", "device": "NVIDIA GeForce RTX 5060", "device_index": 0, "test": "kernel_latency", "message": "query timeout (VK_ERROR_DEVICE_LOST)"},
    {"elapsed_s": 12.4, "level": "error", "backend": "OpenCL", "device": "…", "device_index": 1, "message": "OpenCL: program build failed on … (clBuildProgram -11):\n<kernel>:14:3: error: …"}
  ]
}

Run

invocation.tests is written only when the run was narrowed to specific tests; invocation.iters only when pinned with -i (absent means each test was calibrated to a time budget, which is the normal and comparable mode); invocation.verbose only when --verbose was on — so an empty debug log reads as “was not asked”, not “nothing happened”.

generated_at is the moment the run started: log[].elapsed_s counts from it.

Device

backend / platform / name / device_index identify it. device_index is the backend’s own numbering – the one --list-devices prints and --devices takes – and runs consecutively across an OpenCL backend’s platforms. All three strings are whitespace-trimmed: drivers pad them (Intel’s OpenCL runtime returns its CPU name with five trailing spaces), and padding in an identity is two names for one device the day a driver changes how much of it there is. The index is part of that identity because a name is not unique — MoltenVK exposes one GPU twice, and a multi-GPU box has N identical cards; without it their readings fold into one block and every test ends up with two of everything. driver is deliberately not identity, so a baseline stays comparable across a driver update. type is gpu | cpu | accelerator | unknown. properties are free-form facts the backend chose to report (compute units, VRAM, clocks).

Test

shape — the one thing that cannot be inferred

• homogeneous — the readings are interchangeable variants of one measurement: float / float2 / float4, int8_dp chain depths, a CPU kernel at one thread and at all of them, or a test with a single reading. The best of them is the test’s answer, so a presenter may collapse the test to that number.

• heterogeneous — each reading is its own measurement: cuBLASLt’s nine datatypes, memory_latency’s L1/L2/L3/DRAM, transfer’s h2d vs d2h, smt_scaling’s two thread counts (where the comparison is the result). There is no single answer, and picking the largest reading invents one.

Nothing else in the document determines it. mps_attention has one reading and is homogeneous; mps_gemm has three and is not; both are TFLOPS. The same tag even differs by backend — a GPU’s global_memory_bandwidth is a vector-width sweep, the CPU’s is read/copy/triad. So it is authored in the backend, at the beginTest() call site, next to the description.

heterogeneous is the default, which means an unclassified test is verbose rather than wrong.

axis — what varies

A short noun phrase, shown by the GUI as the header over a heterogeneous test’s readings and by --describe as “Readings vary by …”. The vocabulary in use:

It is optional, and left empty where no single noun covers the readings — kernel-launch latency measures a one-way cost and a full round trip, and an invented word for that pair would read worse than none.

Metric

The unit override is what lets one heterogeneous test hold both TFLOPS and TOPS readings, instead of being split into the -fp / -int twins that older clpeak emitted purely so the unit string could differ. It is set on skipped readings too: an unsupported int8 row still has to say it would have been ops and not flops, or it reads as its test’s unit and claims something false.

Units

A reading is stored in SI — FLOP/s, byte/s, s, Texel/s, ppm — with the unit that explains it:

"quantity": "flops",   // what is measured
"unit": "FLOPS"        // display symbol, ready to print

Values are already SI so the presenters pick G/T/P or µ/n via the SI ladder.

quantity is one of flops, ops, bytes_per_second, seconds, items_per_second, ratio, count, unknown. The last three have no SI ladder to slide along, so a presenter prints them exactly as measured.

The table mapping clpeak’s internal unit tokens (flops, bps, s, ppm, …) to these fields is src/common/units.cpp; a token missing from it passes through as its own symbol with quantity: "unknown", so a new unit appears correctly in the output before anyone adds it there.

Log

Everything clpeak had to say outside a reading, in emission order — the terminal transcript, as data. One JSON object per line in the pretty-printed file, so it reads and greps like a transcript while the rest of the document keeps one key per line; jq -r '.log[] | "\(.elapsed_s) [\(.level)] \(.message)"' run.clpeak.json reproduces what the CLI printed.

What each level means, and when it is in the file:

Every run also records where it is: an info line as each backend, device and test starts (Backend: …, Device 0: …, the test’s title). Under --verbose the readings join them, one line each as the terminal showed them, so the log is the run’s transcript.

The CLI prints its own warnings and errors inline with the results and everything else to stderr under --verbose; the GUI shows warnings and errors at the foot of a run’s results and folds the rest beneath them. The file has it all regardless.

The log is capped at 4 MiB of message text; past that, debug and info entries are dropped and a final warning says how many. Console captures record the first 200 lines of a call and count the rest.

The run-log sidecar

With -o, every entry is also appended — and flushed — to a sidecar as it happens: the output path with .json swapped for .log (run.clpeak.json → run.clpeak.log), one JSON object per line under a header line naming the run:

{"schema": "clpeak/run-log", "format_version": 3, "clpeak_version": "…", "generated_at": "…", "build": {…}, "host": {…}, "invocation": {…}}
{"elapsed_s": 0.31, "level": "warning", "backend": "ONNX", "message": "…"}
…

The document is written when the run ends. A native crash inside a driver — the case --verbose exists for — means it never is, and the sidecar is then the only record of the run; its last line is usually where. It is removed once the document has been saved, so one that outlives its run is the record of a run the process died in. The GUI adopts one it finds on its next launch and offers it for export.

Inventory

--verbose only. The backends array of the device catalog, exactly as the GUI’s run screen and --list-devices see it, taken before the run: every backend compiled in, whether it was available and if not its reason, any notes from enumeration (a provider the runtime names but nothing here can run), and every device each backend saw — including the ones a --devices list left out. It is the evidence for “my NPU is not listed”, which is a debugging question; in the CLI it costs an enumeration pass, which is why it is not on by default.

Per device: index, name, type, and whichever of arch, driver, api, origin (ONNX plugin EPs only, e.g. EP plugin (QNNExecutionProvider)), compute_units, clock_mhz, global_mem_bytes, max_alloc_bytes, fp16, fp64 the backend could answer. The keys are those of include/common/inventory.h, which is the one serializer for both uses.

Conventions

• Absent means default. Optional fields are omitted rather than written empty, so a file carries only facts. status absent ⇒ ok; label absent ⇒ id; metric unit fields absent ⇒ the test’s.
• Numbers use the classic locale, always . — the GUI hosts the writer inside toolkits that call setlocale(LC_ALL, ""), and a comma decimal separator would produce a file that is not JSON at all.
• Seven significant digits on every value: enough to round-trip a float’s precision, enough to keep a six-digit GFLOPS reading whole, and — unlike the fixed four decimals v2 used — it does not flatten the parts-per-million numeric-error readings to 0.0000.

What changed from v2

v2 wrote three formats (XML, CSV, JSON) of one flat table of rows, and carried no notion of whether a test’s readings were comparable to each other. v3 is one nested JSON document that says so.

There is no migration path and no compatibility shim: a v2 file is rejected with a message naming its version. Regenerate it.