What every section and number below means: reading-reports.md.
Cost model scaled to this device from one probe point: bidiag x1.00, block x0.24, cpu x0.08, gk x0.44, jacobi x0.10, qr x0.17, qrblock x0.22 (1.00 is an M1).
Machine state: load 4.7/18 at the start, load 8.4/18 at the end; power mains.
Probe point after the sweep relative to before it: block x1.00, cpu x0.96, gk x1.00, jacobi x1.04, qr x1.03, qrblock x1.01 (stable).
Generated by tuning/tune_svd.py from 295 (shape, batch) points, 45 shapes with M >= N, batch in [1, 4, 16, 64, 256, 1024, 4096], seven backends (and the CPU and bidiag again for singular values alone), two or more passes, min-of-repeats.
Row for kTuned[] in src/svd.mm:
// device, GPU cores, qr_min_rows, qr_min_k, block_min_k, block_min_k_batched, block_min_batch, gpu_max_k, gpu_min_batch_times_k, gpu_min_batch, gpu_max_l, bidiag_min_k, values_bidiag_min_k, bidiag_max_batch, values_bidiag_max_batch, gk_min_k, gk_max_k
{"Apple M5 Pro", 20, 512, 32, 192, 64, 64, 48, 4096, 1, 64, 1024, 2048, 1, 1, 4, 56},
To try it without rebuilding:
SVD_QR_MIN_ROWS=512 SVD_QR_MIN_K=32 SVD_BLOCK_MIN_K=192 SVD_BLOCK_MIN_K_BATCHED=64 SVD_BLOCK_MIN_BATCH=64 SVD_GPU_MAX_K=48 SVD_GPU_MIN_BATCH_TIMES_K=4096 SVD_GPU_MIN_BATCH=1 SVD_GPU_MAX_L=64 SVD_BIDIAG_MIN_K=1024 SVD_VALUES_BIDIAG_MIN_K=2048 SVD_BIDIAG_MAX_BATCH=1 SVD_VALUES_BIDIAG_MAX_BATCH=1 SVD_GK_MIN_K=4 SVD_GK_MAX_K=56
The policy in effect on this device came from tuned:Apple M5 Pro. Against the best measured backend at every point the fitted rule scores 1.0047 geometric-mean regret, worst 1.26x, 5 of 295 points losing more than 10%, and 1.099x the oracle’s total time.
Scored against the best GPU backend at each of the 295 points, as if there were no CPU: this is the rule a forced-GPU call (SVD_DEVICE=gpu) follows, and what a GPU with more cores will lean on. Two independent choices. Precondition with QR iff the long side is at least qr_min_rows, the short side at least qr_min_k, and the long side at least twice the short one. Block kernel iff the short side is at least block_min_k, or at least block_min_k_batched in a batch of block_min_batch or more.
| rule | geomean regret | worst | >10% | total time / oracle | est. picks |
|---|---|---|---|---|---|
| policy in effect (‘512’, ‘32’, ‘192’, ‘64’, ‘64’) | 1.0195 | 1.78x | 23 | 1.004 | 0 |
| fitted (‘512’, ‘32’, ‘192’, ‘64’, ‘64’) | 1.0195 | 1.78x | 23 | 1.004 | 0 |
Without a batch term, 7 of 735 combinations are within 0.5% of the best geomean: qr_min_rows 256 .. 512, qr_min_k 8 .. 32, block_min_k 96 .. 128.
With the batch term adopted (below), 3 combinations are within 0.5% of the best: block_min_k 192 .. 256, block_min_k_batched 56 .. 64, block_min_batch 64 .. 64. The curves vary one constant around the chosen combination.
xychart-beta
title "Regret by block_min_k"
x-axis "block_min_k" [32, 40, 48, 56, 64, 80, 96, 128, 192, 256, 384, 512, 768, 1024, none]
y-axis "geometric-mean regret" 1.0 --> 1.27
line [1.2581, 1.1439, 1.1173, 1.0988, 1.0851, 1.0417, 1.0332, 1.0274, 1.0195, 1.0238, 1.0589, 1.0747, 1.0956, 1.1223, 1.1527]
| block_min_k | 32 | 40 | 48 | 56 | 64 | 80 | 96 | 128 | 192 | 256 | 384 | 512 | 768 | 1024 | none |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| geomean | 1.2581 | 1.1439 | 1.1173 | 1.0988 | 1.0851 | 1.0417 | 1.0332 | 1.0274 | 1.0195 | 1.0238 | 1.0589 | 1.0747 | 1.0956 | 1.1223 | 1.1527 |
| worst | 5.85x | 4.25x | 3.29x | 2.91x | 2.71x | 2.35x | 1.80x | 1.78x | 1.78x | 1.78x | 2.65x | 5.07x | 8.17x | 15.82x | 24.52x |
xychart-beta
title "Regret by block_min_k_batched"
x-axis "block_min_k_batched" [32, 40, 48, 56, 64, 80, 96, 128]
y-axis "geometric-mean regret" 1.0 --> 1.07
line [1.0487, 1.0360, 1.0265, 1.0216, 1.0195, 1.0423, 1.0444, 1.0528]
| block_min_k_batched | 32 | 40 | 48 | 56 | 64 | 80 | 96 | 128 |
|---|---|---|---|---|---|---|---|---|
| geomean | 1.0487 | 1.0360 | 1.0265 | 1.0216 | 1.0195 | 1.0423 | 1.0444 | 1.0528 |
| worst | 2.66x | 2.57x | 1.93x | 1.78x | 1.78x | 2.31x | 2.31x | 2.31x |
xychart-beta
title "Regret by block_min_batch"
x-axis "block_min_batch" [4, 16, 64, 256, 1024, 4096]
y-axis "geometric-mean regret" 1.0 --> 1.09
line [1.0616, 1.0405, 1.0195, 1.0279, 1.0487, 1.0733]
| block_min_batch | 4 | 16 | 64 | 256 | 1024 | 4096 |
|---|---|---|---|---|---|---|
| geomean | 1.0616 | 1.0405 | 1.0195 | 1.0279 | 1.0487 | 1.0733 |
| worst | 2.62x | 2.61x | 1.78x | 2.29x | 2.45x | 2.49x |
xychart-beta
title "Regret by qr_min_rows"
x-axis "qr_min_rows" [16, 32, 64, 128, 256, 512, 1024, 2048, none]
y-axis "geometric-mean regret" 1.0 --> 1.19
line [1.0351, 1.0351, 1.0351, 1.0297, 1.0211, 1.0195, 1.0464, 1.1095, 1.1796]
| qr_min_rows | 16 | 32 | 64 | 128 | 256 | 512 | 1024 | 2048 | none |
|---|---|---|---|---|---|---|---|---|---|
| geomean | 1.0351 | 1.0351 | 1.0351 | 1.0297 | 1.0211 | 1.0195 | 1.0464 | 1.1095 | 1.1796 |
| worst | 1.66x | 1.66x | 1.66x | 1.66x | 1.66x | 1.78x | 2.34x | 4.13x | 6.56x |
xychart-beta
title "Regret by qr_min_k"
x-axis "qr_min_k" [8, 16, 32, 64, 128, 256]
y-axis "geometric-mean regret" 1.0 --> 1.16
line [1.0215, 1.0215, 1.0195, 1.0446, 1.0772, 1.1441]
| qr_min_k | 8 | 16 | 32 | 64 | 128 | 256 |
|---|---|---|---|---|---|---|
| geomean | 1.0215 | 1.0215 | 1.0195 | 1.0446 | 1.0772 | 1.1441 |
| worst | 1.78x | 1.78x | 1.78x | 3.94x | 4.53x | 6.56x |
Held-out check of a batch-dependent block crossover (block from a smaller k once the batch is large enough), fitted on 167 points and scored on the other 128; the verdict is a bootstrap over the test points.
| rule | fitted on train | train geomean | test geomean | test worst | verdict |
|---|---|---|---|---|---|
| one block crossover | [“256”, “32”, “96”, “none”, “none”] | 1.0477 | 1.0667 | 2.21x | baseline |
| batch-dependent block crossover | {“block_min”: “192”, “block_lo”: “64”, “batch_hi”: “64”} | 1.0183 | 1.0247 | 1.59x | justified (better in 100% of resamples, median gain 4.1%) |
Best GPU backend per point (J whole-matrix kernel, B block kernel, j and b the same after QR, g gk), then what the rule picks, the gk window included:
M x N \ batch 1 4 16 64 256 1024 4096
4 x 4 J g J g g g g
8 x 8 J J J J g g g
16 x 8 g g g J g g g
32 x 8 J g J g g g g
64 x 8 J J g J g g g
128 x 8 g J g g J J J
256 x 8 J J g g J J J
16 x 16 g J J J g g g
32 x 16 g g g g g g g
64 x 16 J g g J g g g
128 x 16 J g J J g g g
256 x 16 j J g J J J J
512 x 16 J J J J j g g
24 x 24 g J J g g g g
32 x 32 J J J g g g g
64 x 32 j J J J g g g
128 x 32 J J J J g g g
256 x 32 J J J J g g g
512 x 32 J j J j g g g
1024 x 32 j j J j g j g
40 x 40 J J J g g g g
48 x 48 J J J g g g g
56 x 56 J J J g g g g
64 x 64 J J J g g g g
128 x 64 J J J g B g g
256 x 64 j J J g B g B
512 x 64 j j j g g g g
1024 x 64 j j j g g g b
2048 x 64 j j j j g g b
80 x 80 J J J g g g g
96 x 96 J J J B B B B
128 x 128 J J J B B B B
256 x 128 j j j B b b b
512 x 128 j j j b b b b
1024 x 128 j j j b b b b
2048 x 128 j j j b b b .
192 x 192 B B B B B B .
256 x 256 B B B B B B .
512 x 256 B B b b b b .
1024 x 256 b b b b b b .
2048 x 256 b b b b b . .
384 x 384 B B B B B . .
512 x 512 B B B B . . .
768 x 768 B B B . . . .
1024 x 1024 B B B . . . .
M x N \ batch 1 4 16 64 256 1024 4096
4 x 4 g g g g g g g
8 x 8 g g g g g g g
16 x 8 g g g g g g g
32 x 8 g g g g g g g
64 x 8 g g g g g g g
128 x 8 g g g g g g g
256 x 8 g g g g g g g
16 x 16 g g g g g g g
32 x 16 g g g g g g g
64 x 16 g g g g g g g
128 x 16 g g g g g g g
256 x 16 g g g g g g g
512 x 16 g g g g g g g
24 x 24 g g g g g g g
32 x 32 g g g g g g g
64 x 32 g g g g g g g
128 x 32 g g g g g g g
256 x 32 g g g g g g g
512 x 32 g g g g g g g
1024 x 32 g g g g g g g
40 x 40 g g g g g g g
48 x 48 g g g g g g g
56 x 56 g g g g g g g
64 x 64 J J J B B B B
128 x 64 J J J B B B B
256 x 64 J J J B B B B
512 x 64 j j j b b b b
1024 x 64 j j j b b b b
2048 x 64 j j j b b b b
80 x 80 J J J B B B B
96 x 96 J J J B B B B
128 x 128 J J J B B B B
256 x 128 J J J B B B B
512 x 128 j j j b b b b
1024 x 128 j j j b b b b
2048 x 128 j j j b b b .
192 x 192 B B B B B B .
256 x 256 B B B B B B .
512 x 256 b b b b b b .
1024 x 256 b b b b b b .
2048 x 256 b b b b b . .
384 x 384 B B B B B . .
512 x 512 B B B B . . .
768 x 768 B B B . . . .
1024 x 1024 B B B . . . .
Inside a window of k = min(M, N), gk (Householder bidiagonalization and implicit QR, one threadgroup per matrix, k <= 83 on this device; on the matrix itself where it fits, else after a QR) instead of the Jacobi backend the split picks, fitted over the 295 points against the best GPU backend, gk included. Chosen: k = 4 .. 56.
| rule | geomean regret | worst | >10% | total time / oracle | est. picks |
|---|---|---|---|---|---|
| without gk (the split alone) | 1.1554 | 2.94x | 88 | 1.029 | 0 |
| with gk for k in (4, 56) | 1.1088 | 2.45x | 76 | 1.013 | 0 |
2 windows are within 0.5% of the best geomean: gk_min_k 4 .. 8, gk_max_k 56 .. 56.
Held out: fitted on 167 points (window (4, 56)), scored on the other 128: geomean 1.1175x, worst 2.45x, against 1.1305x, worst 2.94x without gk.
gk over the best Jacobi backend, M x N x batch: 4x4x1 0.87x, 4x4x4 1.07x, 4x4x16 0.99x, 4x4x64 1.23x, 4x4x256 1.01x, 4x4x1024 1.05x, 4x4x4096 1.10x, 8x8x1 0.93x, 8x8x4 0.96x, 8x8x16 0.89x, 8x8x64 0.48x, 8x8x256 1.72x, 8x8x1024 1.43x, 8x8x4096 1.51x, 16x8x1 1.17x, 16x8x4 1.01x, 16x8x16 1.56x, 16x8x64 0.88x, 16x8x256 2.34x, 16x8x1024 1.34x, 16x8x4096 1.39x, 32x8x1 0.87x, 32x8x4 1.22x, 32x8x16 0.91x, 32x8x64 1.52x, 32x8x256 2.38x, 32x8x1024 1.32x, 32x8x4096 1.21x, 64x8x1 0.78x, 64x8x4 0.84x, 64x8x16 1.53x, 64x8x64 0.87x, 64x8x256 1.10x, 64x8x1024 1.02x, 64x8x4096 1.04x, 128x8x1 1.38x, 128x8x4 0.88x, 128x8x16 1.13x, 128x8x64 1.54x, 128x8x256 0.89x, 128x8x1024 0.94x, 128x8x4096 0.90x, 256x8x1 0.69x, 256x8x4 0.79x, 256x8x16 1.03x, 256x8x64 1.24x, 256x8x256 0.87x, 256x8x1024 0.79x, 256x8x4096 0.74x, 16x16x1 1.38x, 16x16x4 0.72x, 16x16x16 0.76x, 16x16x64 0.93x, 16x16x256 1.66x, 16x16x1024 2.03x, 16x16x4096 2.39x, 32x16x1 1.23x, 32x16x4 1.33x, 32x16x16 1.37x, 32x16x64 1.12x, 32x16x256 1.46x, 32x16x1024 1.59x, 32x16x4096 1.75x, 64x16x1 0.69x, 64x16x4 1.20x, 64x16x16 1.15x, 64x16x64 0.87x, 64x16x256 1.60x, 64x16x1024 1.39x, 64x16x4096 1.47x, 128x16x1 0.55x, 128x16x4 1.03x, 128x16x16 0.65x, 128x16x64 0.86x, 128x16x256 1.05x, 128x16x1024 1.21x, 128x16x4096 1.17x, 256x16x1 0.86x, 256x16x4 0.70x, 256x16x16 1.08x, 256x16x64 0.83x, 256x16x256 0.84x, 256x16x1024 0.91x, 256x16x4096 0.89x, 512x16x1 0.65x, 512x16x4 0.59x, 512x16x16 0.55x, 512x16x64 0.59x, 512x16x256 0.86x, 512x16x1024 1.09x, 512x16x4096 1.08x, 24x24x1 1.53x, 24x24x4 0.65x, 24x24x16 0.99x, 24x24x64 1.02x, 24x24x256 2.18x, 24x24x1024 2.29x, 24x24x4096 2.50x, 32x32x1 0.48x, 32x32x4 0.47x, 32x32x16 0.65x, 32x32x64 1.14x, 32x32x256 2.30x, 32x32x1024 2.69x, 32x32x4096 2.43x, 64x32x1 0.66x, 64x32x4 0.48x, 64x32x16 0.53x, 64x32x64 0.95x, 64x32x256 1.48x, 64x32x1024 1.60x, 64x32x4096 1.55x, 128x32x1 0.44x, 128x32x4 0.44x, 128x32x16 0.57x, 128x32x64 0.99x, 128x32x256 1.04x, 128x32x1024 1.23x, 128x32x4096 1.07x, 256x32x1 0.47x, 256x32x4 0.44x, 256x32x16 0.41x, 256x32x64 0.77x, 256x32x256 1.20x, 256x32x1024 1.29x, 256x32x4096 1.25x, 512x32x1 0.57x, 512x32x4 0.61x, 512x32x16 0.53x, 512x32x64 0.96x, 512x32x256 1.16x, 512x32x1024 1.14x, 512x32x4096 1.08x, 1024x32x1 0.61x, 1024x32x4 0.65x, 1024x32x16 0.65x, 1024x32x64 0.98x, 1024x32x256 1.04x, 1024x32x1024 0.99x, 1024x32x4096 1.06x, 40x40x1 0.68x, 40x40x4 0.74x, 40x40x16 0.67x, 40x40x64 1.40x, 40x40x256 2.31x, 40x40x1024 2.88x, 40x40x4096 2.75x, 48x48x1 0.72x, 48x48x4 0.73x, 48x48x16 0.77x, 48x48x64 1.53x, 48x48x256 2.33x, 48x48x1024 2.94x, 48x48x4096 2.88x, 56x56x1 0.70x, 56x56x4 0.71x, 56x56x16 0.67x, 56x56x64 1.55x, 56x56x256 2.29x, 56x56x1024 2.35x, 56x56x4096 2.32x, 64x64x1 0.64x, 64x64x4 0.60x, 64x64x16 0.62x, 64x64x64 1.64x, 64x64x256 1.67x, 64x64x1024 1.55x, 64x64x4096 1.56x, 128x64x1 0.55x, 128x64x4 0.56x, 128x64x16 0.52x, 128x64x64 1.07x, 128x64x256 0.93x, 128x64x1024 1.02x, 128x64x4096 1.04x, 256x64x1 0.59x, 256x64x4 0.61x, 256x64x16 0.59x, 256x64x64 1.16x, 256x64x256 0.98x, 256x64x1024 1.03x, 256x64x4096 0.98x, 512x64x1 0.55x, 512x64x4 0.61x, 512x64x16 0.66x, 512x64x64 1.10x, 512x64x256 1.06x, 512x64x1024 1.07x, 512x64x4096 1.05x, 1024x64x1 0.61x, 1024x64x4 0.67x, 1024x64x16 0.75x, 1024x64x64 1.04x, 1024x64x256 1.07x, 1024x64x1024 1.02x, 1024x64x4096 1.00x, 2048x64x1 0.64x, 2048x64x4 0.77x, 2048x64x16 0.81x, 2048x64x64 0.96x, 2048x64x256 1.03x, 2048x64x1024 1.01x, 80x80x1 0.67x, 80x80x4 0.72x, 80x80x16 0.70x, 80x80x64 1.13x, 80x80x256 1.57x, 80x80x1024 1.78x, 80x80x4096 1.77x
gk over the CPU, M x N x batch: 4x4x1 0.02x, 4x4x4 0.10x, 4x4x16 0.24x, 4x4x64 0.64x, 4x4x256 0.67x, 4x4x1024 1.08x, 4x4x4096 1.83x, 8x8x1 0.05x, 8x8x4 0.10x, 8x8x16 0.36x, 8x8x64 0.32x, 8x8x256 0.81x, 8x8x1024 1.24x, 8x8x4096 1.52x, 16x8x1 0.04x, 16x8x4 0.11x, 16x8x16 0.42x, 16x8x64 0.59x, 16x8x256 0.93x, 16x8x1024 1.58x, 16x8x4096 1.83x, 32x8x1 0.04x, 32x8x4 0.12x, 32x8x16 0.34x, 32x8x64 0.61x, 32x8x256 0.98x, 32x8x1024 1.34x, 32x8x4096 1.58x, 64x8x1 0.05x, 64x8x4 0.13x, 64x8x16 0.35x, 64x8x64 0.65x, 64x8x256 0.97x, 64x8x1024 1.06x, 64x8x4096 1.48x, 128x8x1 0.06x, 128x8x4 0.16x, 128x8x16 0.35x, 128x8x64 0.65x, 128x8x256 0.82x, 128x8x1024 0.85x, 128x8x4096 1.09x, 256x8x1 0.06x, 256x8x4 0.18x, 256x8x16 0.41x, 256x8x64 0.63x, 256x8x256 0.74x, 256x8x1024 0.94x, 256x8x4096 0.73x, 16x16x1 0.06x, 16x16x4 0.13x, 16x16x16 0.27x, 16x16x64 0.52x, 16x16x256 1.05x, 16x16x1024 1.34x, 16x16x4096 1.84x, 32x16x1 0.08x, 32x16x4 0.15x, 32x16x16 0.38x, 32x16x64 0.62x, 32x16x256 1.30x, 32x16x1024 1.63x, 32x16x4096 2.09x, 64x16x1 0.09x, 64x16x4 0.16x, 64x16x16 0.37x, 64x16x64 0.65x, 64x16x256 1.22x, 64x16x1024 1.35x, 64x16x4096 1.64x, 128x16x1 0.09x, 128x16x4 0.15x, 128x16x16 0.37x, 128x16x64 0.61x, 128x16x256 0.75x, 128x16x1024 1.26x, 128x16x4096 1.13x, 256x16x1 0.10x, 256x16x4 0.15x, 256x16x16 0.37x, 256x16x64 0.56x, 256x16x256 0.60x, 256x16x1024 0.80x, 256x16x4096 0.72x, 512x16x1 0.13x, 512x16x4 0.17x, 512x16x16 0.31x, 512x16x64 0.45x, 512x16x256 0.66x, 512x16x1024 0.66x, 512x16x4096 0.74x, 24x24x1 0.07x, 24x24x4 0.13x, 24x24x16 0.31x, 24x24x64 0.54x, 24x24x256 1.24x, 24x24x1024 1.45x, 24x24x4096 1.66x, 32x32x1 0.08x, 32x32x4 0.12x, 32x32x16 0.24x, 32x32x64 0.51x, 32x32x256 1.15x, 32x32x1024 1.55x, 32x32x4096 1.65x, 64x32x1 0.09x, 64x32x4 0.14x, 64x32x16 0.26x, 64x32x64 0.52x, 64x32x256 0.93x, 64x32x1024 1.30x, 64x32x4096 1.30x, 128x32x1 0.10x, 128x32x4 0.13x, 128x32x16 0.27x, 128x32x64 0.49x, 128x32x256 0.69x, 128x32x1024 0.85x, 128x32x4096 0.85x, 256x32x1 0.11x, 256x32x4 0.14x, 256x32x16 0.25x, 256x32x64 0.41x, 256x32x256 0.69x, 256x32x1024 0.77x, 256x32x4096 0.84x, 512x32x1 0.15x, 512x32x4 0.21x, 512x32x16 0.26x, 512x32x64 0.49x, 512x32x256 0.65x, 512x32x1024 0.65x, 512x32x4096 0.60x, 1024x32x1 0.20x, 1024x32x4 0.29x, 1024x32x16 0.33x, 1024x32x64 0.59x, 1024x32x256 0.61x, 1024x32x1024 0.59x, 1024x32x4096 0.61x, 40x40x1 0.10x, 40x40x4 0.14x, 40x40x16 0.28x, 40x40x64 0.56x, 40x40x256 0.94x, 40x40x1024 1.35x, 40x40x4096 1.32x, 48x48x1 0.11x, 48x48x4 0.14x, 48x48x16 0.26x, 48x48x64 0.56x, 48x48x256 0.97x, 48x48x1024 1.25x, 48x48x4096 1.24x, 56x56x1 0.10x, 56x56x4 0.13x, 56x56x16 0.23x, 56x56x64 0.52x, 56x56x256 0.86x, 56x56x1024 0.91x, 56x56x4096 0.91x, 64x64x1 0.12x, 64x64x4 0.13x, 64x64x16 0.22x, 64x64x64 0.53x, 64x64x256 0.69x, 64x64x1024 0.73x, 64x64x4096 0.73x, 128x64x1 0.13x, 128x64x4 0.15x, 128x64x16 0.20x, 128x64x64 0.49x, 128x64x256 0.56x, 128x64x1024 0.62x, 128x64x4096 0.62x, 256x64x1 0.14x, 256x64x4 0.19x, 256x64x16 0.24x, 256x64x64 0.53x, 256x64x256 0.59x, 256x64x1024 0.61x, 256x64x4096 0.55x, 512x64x1 0.17x, 512x64x4 0.25x, 512x64x16 0.34x, 512x64x64 0.53x, 512x64x256 0.62x, 512x64x1024 0.61x, 512x64x4096 0.59x, 1024x64x1 0.23x, 1024x64x4 0.35x, 1024x64x16 0.46x, 1024x64x64 0.61x, 1024x64x256 0.65x, 1024x64x1024 0.60x, 1024x64x4096 0.61x, 2048x64x1 0.30x, 2048x64x4 0.43x, 2048x64x16 0.51x, 2048x64x64 0.64x, 2048x64x256 0.66x, 2048x64x1024 0.66x, 80x80x1 0.15x, 80x80x4 0.17x, 80x80x16 0.22x, 80x80x64 0.37x, 80x80x256 0.57x, 80x80x1024 0.59x, 80x80x4096 0.56x
GPU iff k <= gpu_max_k, l <= gpu_max_l (l = max(M, N)), batch * k >= gpu_min_batch_times_k and batch >= gpu_min_batch, with k = min(M, N), scored against the best of the CPU and the GPU backends (gk included). worst is over the points where the chosen backend was timed; a pick the cost model had to guess is listed in the warnings instead.
| rule | geomean regret | worst | >10% | total time / oracle | est. picks |
|---|---|---|---|---|---|
| oracle (best per point) | 1.0000 | 1.00x | 0 | 1.000 | 0 |
| policy in effect (‘8’, ‘4096’, ‘1’, ‘none’) | 1.0304 | 2.09x | 27 | 1.103 | 1 |
| fitted (‘48’, ‘4096’, ‘1’, ‘64’) | 1.0047 | 1.26x | 5 | 1.099 | 1 |
87 of 6171 combinations are within 0.5% of the best geomean: gpu_max_k 32 .. 56, gpu_min_batch_times_k 4096 .. 8192, gpu_min_batch 1 .. 16, gpu_max_l 64 .. 192.
xychart-beta
title "Regret by gpu_min_batch_times_k"
x-axis "gpu_min_batch_times_k" [0, 64, 128, 256, 512, 1024, 2048, 4096, 8192, 16384, none]
y-axis "geometric-mean regret" 1.0 --> 1.36
line [1.3478, 1.1487, 1.1133, 1.0687, 1.0512, 1.0232, 1.0144, 1.0047, 1.0075, 1.0109, 1.0414]
| gpu_min_batch_times_k | 0 | 64 | 128 | 256 | 512 | 1024 | 2048 | 4096 | 8192 | 16384 | none |
|---|---|---|---|---|---|---|---|---|---|---|---|
| geomean | 1.3478 | 1.1487 | 1.1133 | 1.0687 | 1.0512 | 1.0232 | 1.0144 | 1.0047 | 1.0075 | 1.0109 | 1.0414 |
| worst | 54.01x | 8.37x | 8.37x | 4.09x | 4.09x | 1.97x | 1.97x | 1.26x | 1.30x | 1.58x | 2.09x |
xychart-beta
title "Regret by gpu_max_l"
x-axis "gpu_max_l" [16, 24, 32, 40, 48, 56, 64, 80, 96, 128, 192, 256, 384, 512, 768, 1024, 2048, none]
y-axis "geometric-mean regret" 1.0 --> 1.04
line [1.0297, 1.0259, 1.0144, 1.0126, 1.0112, 1.0112, 1.0047, 1.0047, 1.0047, 1.0071, 1.0071, 1.0148, 1.0148, 1.0235, 1.0235, 1.0287, 1.0287, 1.0287]
| gpu_max_l | 16 | 24 | 32 | 40 | 48 | 56 | 64 | 80 | 96 | 128 | 192 | 256 | 384 | 512 | 768 | 1024 | 2048 | none |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| geomean | 1.0297 | 1.0259 | 1.0144 | 1.0126 | 1.0112 | 1.0112 | 1.0047 | 1.0047 | 1.0047 | 1.0071 | 1.0071 | 1.0148 | 1.0148 | 1.0235 | 1.0235 | 1.0287 | 1.0287 | 1.0287 |
| worst | 2.09x | 2.09x | 1.64x | 1.64x | 1.64x | 1.64x | 1.26x | 1.26x | 1.26x | 1.44x | 1.44x | 1.68x | 1.68x | 1.68x | 1.68x | 1.70x | 1.70x | 1.70x |
xychart-beta
title "Regret by gpu_min_batch"
x-axis "gpu_min_batch" [1, 4, 16]
y-axis "geometric-mean regret" 1.0 --> 1.02
line [1.0047, 1.0047, 1.0047]
| gpu_min_batch | 1 | 4 | 16 |
|---|---|---|---|
| geomean | 1.0047 | 1.0047 | 1.0047 |
| worst | 1.26x | 1.26x | 1.26x |
xychart-beta
title "Regret by gpu_max_k"
x-axis "gpu_max_k" [8, 16, 24, 32, 40, 48, 56, 64, 80, 96, 128, 192, 256, 384, 512, 768, 1024, none]
y-axis "geometric-mean regret" 1.0 --> 1.04
line [1.0288, 1.0169, 1.0132, 1.0079, 1.0062, 1.0047, 1.0059, 1.0186, 1.0186, 1.0186, 1.0186, 1.0186, 1.0186, 1.0186, 1.0186, 1.0186, 1.0186, 1.0186]
| gpu_max_k | 8 | 16 | 24 | 32 | 40 | 48 | 56 | 64 | 80 | 96 | 128 | 192 | 256 | 384 | 512 | 768 | 1024 | none |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| geomean | 1.0288 | 1.0169 | 1.0132 | 1.0079 | 1.0062 | 1.0047 | 1.0059 | 1.0186 | 1.0186 | 1.0186 | 1.0186 | 1.0186 | 1.0186 | 1.0186 | 1.0186 | 1.0186 | 1.0186 | 1.0186 |
| worst | 2.09x | 1.66x | 1.65x | 1.35x | 1.26x | 1.26x | 1.26x | 3.66x | 3.66x | 3.66x | 3.66x | 3.66x | 3.66x | 3.66x | 3.66x | 3.66x | 3.66x | 3.66x |
Held-out check of a per-k boundary against the product rule, fitted on 167 points and scored on the other 128; the verdict is a bootstrap.
| rule | fitted on train | train geomean | test geomean | test worst | verdict |
|---|---|---|---|---|---|
| product rule | [40, 4096, 1, 64] | 1.0026 | 1.0108 | 1.26x | baseline |
| per-k table | {“min_batch_by_k”: {“4”: null, “8”: 1024, “16”: 256, “24”: 256, “32”: 1024, “40”: 1024, “48”: null, “56”: null, “64”: null, “80”: null, “96”: null, “128”: null, “192”: null, “256”: null, “384”: null, “512”: null, “768”: null, “1024”: null}} | 1.0018 | 1.0173 | 1.83x | rejected (better in 0% of resamples, median gain -0.6%) |
Best backend per point (c CPU, J whole-matrix kernel, B block kernel, j and b the same after QR, g gk, . not measured), what the rule picks, and the speedup of the best GPU backend over the CPU:
M x N \ batch 1 4 16 64 256 1024 4096
4 x 4 c c c c c g g
8 x 8 c c c c c g g
16 x 8 c c c c c g g
32 x 8 c c c c c g g
64 x 8 c c c c c g g
128 x 8 c c c c c c J
256 x 8 c c c c c J c
16 x 16 c c c c g g g
32 x 16 c c c c g g g
64 x 16 c c c c g g g
128 x 16 c c c c c g g
256 x 16 c c c c c c c
512 x 16 c c c c c c c
24 x 24 c c c c g g g
32 x 32 c c c c g g g
64 x 32 c c c c c g g
128 x 32 c c c c c c c
256 x 32 c c c c c c c
512 x 32 c c c c c c c
1024 x 32 c c c c c c c
40 x 40 c c c c c g g
48 x 48 c c c c c g g
56 x 56 c c c c c c c
64 x 64 c c c c c c c
128 x 64 c c c c c c c
256 x 64 c c c c c c c
512 x 64 c c c c c c c
1024 x 64 c c c c c c c
2048 x 64 c c c c c c c
80 x 80 c c c c c c c
96 x 96 c c c c c c c
128 x 128 c c c c c c c
256 x 128 c c c c c c c
512 x 128 c c c c c c c
1024 x 128 c c c c c c c
2048 x 128 c c c c c c .
192 x 192 c c c c c c .
256 x 256 c c c c c c .
512 x 256 c c c c c c .
1024 x 256 c c c c c b .
2048 x 256 c c c c c . .
384 x 384 c c c c c . .
512 x 512 c c c c . . .
768 x 768 c c c . . . .
1024 x 1024 c c c . . . .
M x N \ batch 1 4 16 64 256 1024 4096
4 x 4 c c c c c g g
8 x 8 c c c c c g g
16 x 8 c c c c c g g
32 x 8 c c c c c g g
64 x 8 c c c c c g g
128 x 8 c c c c c c c
256 x 8 c c c c c c c
16 x 16 c c c c g g g
32 x 16 c c c c g g g
64 x 16 c c c c g g g
128 x 16 c c c c c c c
256 x 16 c c c c c c c
512 x 16 c c c c c c c
24 x 24 c c c c g g g
32 x 32 c c c c g g g
64 x 32 c c c c g g g
128 x 32 c c c c c c c
256 x 32 c c c c c c c
512 x 32 c c c c c c c
1024 x 32 c c c c c c c
40 x 40 c c c c g g g
48 x 48 c c c c g g g
56 x 56 c c c c c c c
64 x 64 c c c c c c c
128 x 64 c c c c c c c
256 x 64 c c c c c c c
512 x 64 c c c c c c c
1024 x 64 c c c c c c c
2048 x 64 c c c c c c c
80 x 80 c c c c c c c
96 x 96 c c c c c c c
128 x 128 c c c c c c c
256 x 128 c c c c c c c
512 x 128 c c c c c c c
1024 x 128 c c c c c c c
2048 x 128 c c c c c c .
192 x 192 c c c c c c .
256 x 256 c c c c c c .
512 x 256 c c c c c c .
1024 x 256 c c c c c c .
2048 x 256 c c c c c . .
384 x 384 c c c c c . .
512 x 512 c c c c . . .
768 x 768 c c c . . . .
1024 x 1024 c c c . . . .
M x N \ batch 1 4 16 64 256 1024 4096
4 x 4 0.02 0.10 0.25 0.64 0.67 1.08 1.83
8 x 8 0.05 0.10 0.41 0.67 0.81 1.24 1.52
16 x 8 0.04 0.11 0.42 0.67 0.93 1.58 1.83
32 x 8 0.05 0.12 0.37 0.61 0.98 1.34 1.58
64 x 8 0.06 0.16 0.35 0.75 0.97 1.06 1.48
128 x 8 0.06 0.18 0.35 0.65 0.91 0.91 1.21
256 x 8 0.09 0.23 0.41 0.63 0.85 1.19 0.99
16 x 16 0.06 0.18 0.35 0.56 1.05 1.34 1.84
32 x 16 0.08 0.15 0.38 0.62 1.30 1.63 2.09
64 x 16 0.14 0.16 0.37 0.75 1.22 1.35 1.64
128 x 16 0.16 0.15 0.57 0.71 0.75 1.26 1.13
256 x 16 0.12 0.22 0.37 0.67 0.71 0.87 0.81
512 x 16 0.20 0.30 0.56 0.76 0.77 0.66 0.74
24 x 24 0.07 0.19 0.32 0.54 1.24 1.45 1.66
32 x 32 0.17 0.26 0.37 0.51 1.15 1.55 1.65
64 x 32 0.13 0.28 0.49 0.55 0.93 1.30 1.30
128 x 32 0.23 0.29 0.48 0.50 0.69 0.85 0.85
256 x 32 0.22 0.33 0.62 0.53 0.69 0.77 0.84
512 x 32 0.26 0.34 0.49 0.52 0.65 0.65 0.60
1024 x 32 0.32 0.45 0.51 0.61 0.61 0.59 0.61
40 x 40 0.15 0.19 0.42 0.56 0.94 1.35 1.32
48 x 48 0.15 0.20 0.34 0.56 0.97 1.25 1.24
56 x 56 0.14 0.19 0.34 0.52 0.86 0.91 0.91
64 x 64 0.19 0.21 0.35 0.53 0.69 0.73 0.73
128 x 64 0.23 0.28 0.38 0.49 0.60 0.62 0.62
256 x 64 0.23 0.31 0.41 0.53 0.61 0.61 0.57
512 x 64 0.30 0.41 0.52 0.53 0.62 0.61 0.59
1024 x 64 0.37 0.52 0.62 0.61 0.65 0.60 0.61
2048 x 64 0.48 0.56 0.63 0.66 0.66 0.66 0.69
80 x 80 0.22 0.23 0.31 0.37 0.57 0.59 0.56
96 x 96 0.23 0.29 0.34 0.36 0.50 0.46 0.43
128 x 128 0.21 0.27 0.34 0.42 0.45 0.41 0.40
256 x 128 0.28 0.37 0.58 0.59 0.57 0.56 0.54
512 x 128 0.33 0.34 0.49 0.57 0.62 0.54 0.52
1024 x 128 0.39 0.41 0.54 0.62 0.66 0.58 0.53
2048 x 128 0.46 0.51 0.64 0.69 0.69 0.65 .
192 x 192 0.20 0.20 0.25 0.28 0.25 0.23 .
256 x 256 0.30 0.29 0.25 0.24 0.23 0.21 .
512 x 256 0.44 0.42 0.35 0.36 0.34 0.32 .
1024 x 256 0.46 0.46 0.44 0.44 0.41 gpu .
2048 x 256 0.55 0.55 0.53 0.51 0.52 . .
384 x 384 0.33 0.30 0.19 0.16 0.15 . .
512 x 512 0.52 0.38 0.18 0.16 . . .
768 x 768 0.52 0.35 0.15 . . . .
1024 x 1024 0.67 0.32 0.25 . . . .
Where the rule above chooses the CPU, the bidiag backend (GPU bidiagonalization, then LAPACK’s bidiagonal solve) from a threshold k = min(M, N) on (0: never), for batches up to a cap (0: any; it solves a batch one matrix after another, the CPU path spreads one over every core), fitted on the points where bidiag was timed (k >= 128, within the cost cap): the region the threshold decides. With vectors it is scored against the best of all backends, bidiag included; for singular values alone (svdvals), bidiag against the CPU at the points where the rule chooses the CPU.
| threshold | batch cap | geomean regret | worst | without bidiag: geomean | worst | held out (fitted on half) | |
|---|---|---|---|---|---|---|---|
| with vectors | 1024 | 1 | 1.0000 | 1.00x | 1.0263 | 2.03x | from 1024, batch <= 1: 1.0000 vs 1.0009 |
| singular values alone | 2048 | 1 | 1.0000 | 1.00x | 1.0195 | 1.95x | from 2048, batch <= 1: 1.0000 vs 1.0000 |
bidiag over the CPU (with vectors), M x N x batch: 128x128x1 0.24x, 128x128x4 0.08x, 128x128x16 0.02x, 128x128x64 0.02x, 128x128x256 0.02x, 256x128x1 0.29x, 256x128x4 0.10x, 256x128x16 0.05x, 256x128x64 0.03x, 256x128x256 0.03x, 512x128x1 0.33x, 512x128x4 0.10x, 512x128x16 0.04x, 512x128x64 0.04x, 512x128x256 0.03x, 1024x128x1 0.39x, 1024x128x4 0.12x, 1024x128x16 0.06x, 1024x128x64 0.05x, 1024x128x256 0.04x, 2048x128x1 0.45x, 2048x128x4 0.15x, 2048x128x16 0.08x, 2048x128x64 0.07x, 2048x128x256 0.06x, 192x192x1 0.27x, 192x192x4 0.07x, 192x192x16 0.03x, 192x192x64 0.02x, 192x192x256 0.02x, 256x256x1 0.38x, 256x256x4 0.10x, 256x256x16 0.04x, 256x256x64 0.03x, 512x256x1 0.44x, 512x256x4 0.12x, 512x256x16 0.04x, 512x256x64 0.04x, 1024x256x1 0.48x, 1024x256x4 0.14x, 1024x256x16 0.06x, 1024x256x64 0.05x, 2048x256x1 0.55x, 2048x256x4 0.16x, 2048x256x16 0.08x, 2048x256x64 0.07x, 384x384x1 0.44x, 384x384x4 0.13x, 384x384x16 0.05x, 384x384x64 0.04x, 512x512x1 0.65x, 512x512x4 0.18x, 512x512x16 0.07x, 512x512x64 0.06x, 768x768x1 0.73x, 768x768x4 0.22x, 768x768x16 0.11x, 1024x1024x1 1.03x, 1024x1024x4 0.31x, 1024x1024x16 0.25x, 1536x1536x1 1.08x, 1536x1536x2 0.64x, 1536x1536x4 0.34x, 2048x2048x1 1.40x, 2048x2048x2 0.95x, 2048x2048x4 0.63x, 3072x3072x1 1.86x, 4096x4096x1 2.03x
bidiag over the CPU (singular values alone), M x N x batch: 128x128x1 0.18x, 128x128x4 0.07x, 128x128x16 0.02x, 128x128x64 0.02x, 128x128x256 0.02x, 256x128x1 0.21x, 256x128x4 0.07x, 256x128x16 0.03x, 256x128x64 0.02x, 256x128x256 0.02x, 512x128x1 0.22x, 512x128x4 0.07x, 512x128x16 0.03x, 512x128x64 0.02x, 512x128x256 0.02x, 1024x128x1 0.25x, 1024x128x4 0.07x, 1024x128x16 0.04x, 1024x128x64 0.03x, 1024x128x256 0.02x, 2048x128x1 0.27x, 2048x128x4 0.10x, 2048x128x16 0.04x, 2048x128x64 0.03x, 2048x128x256 0.03x, 192x192x1 0.15x, 192x192x4 0.05x, 192x192x16 0.02x, 192x192x64 0.01x, 192x192x256 0.01x, 256x256x1 0.21x, 256x256x4 0.07x, 256x256x16 0.03x, 256x256x64 0.02x, 512x256x1 0.24x, 512x256x4 0.08x, 512x256x16 0.04x, 512x256x64 0.02x, 1024x256x1 0.27x, 1024x256x4 0.09x, 1024x256x16 0.03x, 1024x256x64 0.03x, 2048x256x1 0.30x, 2048x256x4 0.09x, 2048x256x16 0.04x, 2048x256x64 0.03x, 384x384x1 0.27x, 384x384x4 0.08x, 384x384x16 0.04x, 384x384x64 0.03x, 512x512x1 0.41x, 512x512x4 0.12x, 512x512x16 0.05x, 512x512x64 0.04x, 768x768x1 0.51x, 768x768x4 0.14x, 768x768x16 0.10x, 1024x1024x1 0.75x, 1024x1024x4 0.22x, 1024x1024x16 0.29x, 1536x1536x1 0.88x, 1536x1536x2 0.52x, 1536x1536x4 0.32x, 2048x2048x1 1.14x, 2048x2048x2 0.85x, 2048x2048x4 0.66x, 3072x3072x1 1.67x, 4096x4096x1 1.95x
Pass-to-pass ratio, 1489 measurements: median 1.010, p90 1.065, max 2.98.
| runtime | n | median | p90 | max |
|---|---|---|---|---|
| <1 ms | 385 | 1.026 | 1.394 | 2.98 |
| 1-3 ms | 242 | 1.008 | 1.044 | 1.29 |
| 3-10 ms | 255 | 1.009 | 1.042 | 1.37 |
| 10-30 ms | 174 | 1.008 | 1.033 | 1.24 |
| 30-100 ms | 164 | 1.008 | 1.034 | 1.16 |
| >100 ms | 269 | 1.006 | 1.035 | 1.12 |