What every section and number below means: reading-reports.md.
Cost model scaled to this device from one probe point: bidiag x1.00, block x0.24, cpu x0.07, gk x0.39, jacobi x0.09, qr x0.16, qrblock x0.22 (1.00 is an M1).
Machine state: load 2.2/18 at the start, load 7.5/18 at the end; power mains.
Probe point after the sweep relative to before it: block x1.05, cpu x0.99, gk x1.00, jacobi x1.04, qr x0.98, qrblock x1.00 (stable).
Generated by tuning/tune_svd.py from 295 (shape, batch) points, 45 shapes with M >= N, batch in [1, 4, 16, 64, 256, 1024, 4096], seven backends (and the CPU and bidiag again for singular values alone), two or more passes, min-of-repeats.
Row for kTuned[] in src/svd.mm:
// device, GPU cores, qr_min_rows, qr_min_k, block_min_k, block_min_k_batched, block_min_batch, gpu_max_k, gpu_min_batch_times_k, gpu_min_batch, gpu_max_l, values_gpu_max_k, values_gpu_min_batch_times_k, values_gpu_min_batch, values_gpu_max_l, bidiag_min_k, values_bidiag_min_k, bidiag_max_batch, values_bidiag_max_batch, gk_min_k, gk_max_k, share_min_batch, gpu_big_batch_max_k, gpu_big_batch_min, values_band_min_k, values_band_width, band_min_k
{"Apple M5 Pro", 20, 512, 32, 192, 64, 64, 56, 16384, 1, 256, 56, 16384, 1, 256, 1024, 1024, 4, 2, 8, 80, 1024, 80, 1024, 768, 16, 1024},
To try it without rebuilding:
SVD_QR_MIN_ROWS=512 SVD_QR_MIN_K=32 SVD_BLOCK_MIN_K=192 SVD_BLOCK_MIN_K_BATCHED=64 SVD_BLOCK_MIN_BATCH=64 SVD_GPU_MAX_K=56 SVD_GPU_MIN_BATCH_TIMES_K=16384 SVD_GPU_MIN_BATCH=1 SVD_GPU_MAX_L=256 SVD_BIDIAG_MIN_K=1024 SVD_VALUES_BIDIAG_MIN_K=1024 SVD_BIDIAG_MAX_BATCH=4 SVD_VALUES_BIDIAG_MAX_BATCH=2 SVD_GK_MIN_K=8 SVD_GK_MAX_K=80 SVD_SHARE_MIN_BATCH=1024 SVD_GPU_BIG_BATCH_MAX_K=80 SVD_GPU_BIG_BATCH_MIN=1024 SVD_VALUES_BAND_MIN_K=768 SVD_VALUES_BAND_WIDTH=16 SVD_BAND_MIN_K=1024 SVD_VALUES_GPU_MAX_K=56 SVD_VALUES_GPU_MIN_BATCH_TIMES_K=16384 SVD_VALUES_GPU_MIN_BATCH=1 SVD_VALUES_GPU_MAX_L=256
The policy in effect on this device came from tuned-stale:Apple M5 Pro. Against the best measured backend at every point the fitted rule scores 1.0196 geometric-mean regret, worst 1.55x, 22 of 295 points losing more than 10%, and 1.007x the oracle’s total time.
Scored against the best GPU backend at each of the 295 points, as if there were no CPU: this is the rule a forced-GPU call (SVD_DEVICE=gpu) follows, and what a GPU with more cores will lean on. Two independent choices. Precondition with QR iff the long side is at least qr_min_rows, the short side at least qr_min_k, and the long side at least twice the short one. Block kernel iff the short side is at least block_min_k, or at least block_min_k_batched in a batch of block_min_batch or more.
| rule | geomean regret | worst | >10% | total time / oracle | est. picks |
|---|---|---|---|---|---|
| policy in effect (‘512’, ‘32’, ‘192’, ‘64’, ‘64’) | 1.0271 | 1.91x | 25 | 1.008 | 0 |
| fitted (‘512’, ‘32’, ‘192’, ‘64’, ‘64’) | 1.0271 | 1.91x | 25 | 1.008 | 0 |
Without a batch term, 6 of 735 combinations are within 0.5% of the best geomean: qr_min_rows 128 .. 256, qr_min_k 16 .. 32, block_min_k 96 .. 128.
With the batch term adopted (below), 3 combinations are within 0.5% of the best: block_min_k 192 .. 256, block_min_k_batched 56 .. 64, block_min_batch 64 .. 64. The curves vary one constant around the chosen combination.
xychart-beta
title "Regret by block_min_k"
x-axis "block_min_k" [32, 40, 48, 56, 64, 80, 96, 128, 192, 256, 384, 512, 768, 1024, none]
y-axis "geometric-mean regret" 1.0 --> 1.31
line [1.2901, 1.1665, 1.1375, 1.1170, 1.1021, 1.0546, 1.0450, 1.0381, 1.0271, 1.0310, 1.0661, 1.0817, 1.1027, 1.1296, 1.1600]
| block_min_k | 32 | 40 | 48 | 56 | 64 | 80 | 96 | 128 | 192 | 256 | 384 | 512 | 768 | 1024 | none |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| geomean | 1.2901 | 1.1665 | 1.1375 | 1.1170 | 1.1021 | 1.0546 | 1.0450 | 1.0381 | 1.0271 | 1.0310 | 1.0661 | 1.0817 | 1.1027 | 1.1296 | 1.1600 |
| worst | 6.00x | 4.75x | 3.62x | 3.11x | 2.91x | 2.53x | 2.04x | 1.91x | 1.91x | 1.91x | 2.67x | 4.81x | 7.76x | 15.63x | 24.99x |
xychart-beta
title "Regret by block_min_k_batched"
x-axis "block_min_k_batched" [32, 40, 48, 56, 64, 80, 96, 128]
y-axis "geometric-mean regret" 1.0 --> 1.08
line [1.0637, 1.0461, 1.0350, 1.0293, 1.0271, 1.0482, 1.0500, 1.0581]
| block_min_k_batched | 32 | 40 | 48 | 56 | 64 | 80 | 96 | 128 |
|---|---|---|---|---|---|---|---|---|
| geomean | 1.0637 | 1.0461 | 1.0350 | 1.0293 | 1.0271 | 1.0482 | 1.0500 | 1.0581 |
| worst | 3.25x | 2.84x | 2.14x | 1.91x | 1.91x | 2.58x | 2.58x | 2.58x |
xychart-beta
title "Regret by block_min_batch"
x-axis "block_min_batch" [4, 16, 64, 256, 1024, 4096]
y-axis "geometric-mean regret" 1.0 --> 1.09
line [1.0745, 1.0507, 1.0271, 1.0333, 1.0523, 1.0785]
| block_min_batch | 4 | 16 | 64 | 256 | 1024 | 4096 |
|---|---|---|---|---|---|---|
| geomean | 1.0745 | 1.0507 | 1.0271 | 1.0333 | 1.0523 | 1.0785 |
| worst | 2.78x | 2.70x | 1.91x | 2.27x | 2.47x | 2.57x |
xychart-beta
title "Regret by qr_min_rows"
x-axis "qr_min_rows" [16, 32, 64, 128, 256, 512, 1024, 2048, none]
y-axis "geometric-mean regret" 1.0 --> 1.22
line [1.0370, 1.0370, 1.0370, 1.0315, 1.0260, 1.0271, 1.0583, 1.1275, 1.2020]
| qr_min_rows | 16 | 32 | 64 | 128 | 256 | 512 | 1024 | 2048 | none |
|---|---|---|---|---|---|---|---|---|---|
| geomean | 1.0370 | 1.0370 | 1.0370 | 1.0315 | 1.0260 | 1.0271 | 1.0583 | 1.1275 | 1.2020 |
| worst | 1.61x | 1.61x | 1.61x | 1.61x | 1.61x | 1.91x | 2.81x | 4.70x | 6.52x |
xychart-beta
title "Regret by qr_min_k"
x-axis "qr_min_k" [8, 16, 32, 64, 128, 256]
y-axis "geometric-mean regret" 1.0 --> 1.18
line [1.0281, 1.0281, 1.0271, 1.0539, 1.0920, 1.1661]
| qr_min_k | 8 | 16 | 32 | 64 | 128 | 256 |
|---|---|---|---|---|---|---|
| geomean | 1.0281 | 1.0281 | 1.0271 | 1.0539 | 1.0920 | 1.1661 |
| worst | 1.91x | 1.91x | 1.91x | 4.70x | 4.70x | 6.52x |
Held-out check of a batch-dependent block crossover (block from a smaller k once the batch is large enough), fitted on 167 points and scored on the other 128; the verdict is a bootstrap over the test points.
| rule | fitted on train | train geomean | test geomean | test worst | verdict |
|---|---|---|---|---|---|
| one block crossover | [“256”, “32”, “96”, “none”, “none”] | 1.0535 | 1.0730 | 2.27x | baseline |
| batch-dependent block crossover | {“block_min”: “192”, “block_lo”: “64”, “batch_hi”: “64”} | 1.0242 | 1.0285 | 1.61x | justified (better in 100% of resamples, median gain 4.3%) |
Best GPU backend per point (J whole-matrix kernel, B block kernel, j and b the same after QR, g gk, G gk shared with the CPU), then what the rule picks, the gk window included:
M x N \ batch 1 4 16 64 256 1024 4096
4 x 4 J J J J J g g
8 x 8 J g g g g g g
16 x 8 J g g g g g g
32 x 8 g J g g g g g
64 x 8 g g g g g g g
128 x 8 g g g g g J J
256 x 8 g g g g g J J
16 x 16 g J g g g g g
32 x 16 g g g g g g g
64 x 16 g g g J g g g
128 x 16 g g g g g g g
256 x 16 g g g g g g g
512 x 16 J j J J g g g
24 x 24 J g J g g g g
32 x 32 J J J J g g g
64 x 32 J J J J g g g
128 x 32 J J j g g g g
256 x 32 J J J J g g g
512 x 32 j J J j g g g
1024 x 32 j j J j g j g
40 x 40 J J J g g g g
48 x 48 J J J g g g g
56 x 56 J J J g g g g
64 x 64 J J J g g g g
128 x 64 J J J J B g g
256 x 64 j j J g g g g
512 x 64 j j j g g g g
1024 x 64 j j j g g g g
2048 x 64 j j j j g g b
80 x 80 J J J g g g g
96 x 96 J J J J B B B
128 x 128 J J J B B B B
256 x 128 j j j B b b b
512 x 128 j j j b b b b
1024 x 128 j j j b b b b
2048 x 128 j j j b b b .
192 x 192 B B B B B B .
256 x 256 B B B B B B .
512 x 256 b B b b b b .
1024 x 256 b b b b b b .
2048 x 256 b b b b b . .
384 x 384 B B B B B . .
512 x 512 B B B B . . .
768 x 768 B B B . . . .
1024 x 1024 B B B . . . .
M x N \ batch 1 4 16 64 256 1024 4096
4 x 4 J J J J J J J
8 x 8 g g g g g G G
16 x 8 g g g g g G G
32 x 8 g g g g g G G
64 x 8 g g g g g G G
128 x 8 g g g g g G G
256 x 8 g g g g g G G
16 x 16 g g g g g G G
32 x 16 g g g g g G G
64 x 16 g g g g g G G
128 x 16 g g g g g G G
256 x 16 g g g g g G G
512 x 16 g g g g g G G
24 x 24 g g g g g G G
32 x 32 g g g g g G G
64 x 32 g g g g g G G
128 x 32 g g g g g G G
256 x 32 g g g g g G G
512 x 32 g g g g g G G
1024 x 32 g g g g g G G
40 x 40 g g g g g G G
48 x 48 g g g g g G G
56 x 56 g g g g g G G
64 x 64 g g g g g G G
128 x 64 g g g g g G G
256 x 64 g g g g g G G
512 x 64 g g g g g G G
1024 x 64 g g g g g G G
2048 x 64 g g g g g G G
80 x 80 g g g g g G G
96 x 96 J J J B B B B
128 x 128 J J J B B B B
256 x 128 J J J B B B B
512 x 128 j j j b b b b
1024 x 128 j j j b b b b
2048 x 128 j j j b b b .
192 x 192 B B B B B B .
256 x 256 B B B B B B .
512 x 256 b b b b b b .
1024 x 256 b b b b b b .
2048 x 256 b b b b b . .
384 x 384 B B B B B . .
512 x 512 B B B B . . .
768 x 768 B B B . . . .
1024 x 1024 B B B . . . .
Inside a window of k = min(M, N), gk (Householder bidiagonalization and implicit QR, one threadgroup per matrix, k <= 83 on this device; on the matrix itself where it fits, else after a QR) instead of the Jacobi backend the split picks, fitted over the 295 points against the best GPU backend, gk included. Chosen: k = 8 .. 56.
| rule | geomean regret | worst | >10% | total time / oracle | est. picks |
|---|---|---|---|---|---|
| without gk (the split alone) | 1.1973 | 3.07x | 119 | 1.040 | 0 |
| with gk for k in (8, 56) | 1.0957 | 2.41x | 61 | 1.023 | 0 |
Refitted once stage 1c chose to share batches with the CPU (gk shared leads the Jacobi backends where gk alone did not): k = 8 .. 56 became k = 8 .. 80, the window in the row.
2 windows are within 0.5% of the best geomean: gk_min_k 4 .. 8, gk_max_k 56 .. 56.
Held out: fitted on 167 points (window (4, 56)), scored on the other 128: geomean 1.1115x, worst 2.20x, against 1.1885x, worst 3.07x without gk.
gk over the best Jacobi backend, M x N x batch: 4x4x1 0.88x, 4x4x4 0.85x, 4x4x16 0.76x, 4x4x64 0.88x, 4x4x256 0.92x, 4x4x1024 1.47x, 4x4x4096 1.45x, 8x8x1 0.72x, 8x8x4 1.19x, 8x8x16 1.07x, 8x8x64 1.43x, 8x8x256 1.20x, 8x8x1024 1.56x, 8x8x4096 1.32x, 16x8x1 0.98x, 16x8x4 1.43x, 16x8x16 1.47x, 16x8x64 1.49x, 16x8x256 2.25x, 16x8x1024 1.26x, 16x8x4096 1.25x, 32x8x1 1.20x, 32x8x4 0.88x, 32x8x16 1.41x, 32x8x64 1.52x, 32x8x256 1.54x, 32x8x1024 1.09x, 32x8x4096 1.07x, 64x8x1 1.29x, 64x8x4 1.20x, 64x8x16 1.45x, 64x8x64 1.58x, 64x8x256 1.82x, 64x8x1024 1.03x, 64x8x4096 1.01x, 128x8x1 1.27x, 128x8x4 1.51x, 128x8x16 1.68x, 128x8x64 1.75x, 128x8x256 2.03x, 128x8x1024 1.00x, 128x8x4096 0.96x, 256x8x1 1.49x, 256x8x4 1.88x, 256x8x16 1.89x, 256x8x64 1.72x, 256x8x256 1.20x, 256x8x1024 0.96x, 256x8x4096 0.90x, 16x16x1 1.48x, 16x16x4 0.72x, 16x16x16 1.50x, 16x16x64 1.17x, 16x16x256 1.48x, 16x16x1024 1.71x, 16x16x4096 2.10x, 32x16x1 1.27x, 32x16x4 1.39x, 32x16x16 1.37x, 32x16x64 1.62x, 32x16x256 1.37x, 32x16x1024 1.37x, 32x16x4096 1.54x, 64x16x1 1.20x, 64x16x4 1.27x, 64x16x16 1.40x, 64x16x64 0.88x, 64x16x256 1.31x, 64x16x1024 1.34x, 64x16x4096 1.35x, 128x16x1 1.22x, 128x16x4 1.31x, 128x16x16 1.26x, 128x16x64 1.74x, 128x16x256 1.26x, 128x16x1024 1.22x, 128x16x4096 1.19x, 256x16x1 1.26x, 256x16x4 1.26x, 256x16x16 1.10x, 256x16x64 1.00x, 256x16x256 1.09x, 256x16x1024 1.17x, 256x16x4096 1.19x, 512x16x1 0.69x, 512x16x4 0.80x, 512x16x16 0.53x, 512x16x64 0.60x, 512x16x256 1.02x, 512x16x1024 1.03x, 512x16x4096 1.06x, 24x24x1 0.66x, 24x24x4 1.28x, 24x24x16 0.67x, 24x24x64 1.06x, 24x24x256 1.96x, 24x24x1024 1.94x, 24x24x4096 2.14x, 32x32x1 0.67x, 32x32x4 0.48x, 32x32x16 0.53x, 32x32x64 0.99x, 32x32x256 2.01x, 32x32x1024 2.48x, 32x32x4096 2.40x, 64x32x1 0.58x, 64x32x4 0.51x, 64x32x16 0.53x, 64x32x64 1.00x, 64x32x256 1.41x, 64x32x1024 1.70x, 64x32x4096 1.63x, 128x32x1 0.72x, 128x32x4 0.49x, 128x32x16 0.80x, 128x32x64 1.04x, 128x32x256 1.19x, 128x32x1024 1.38x, 128x32x4096 1.23x, 256x32x1 0.44x, 256x32x4 0.41x, 256x32x16 0.45x, 256x32x64 0.75x, 256x32x256 1.20x, 256x32x1024 1.26x, 256x32x4096 1.25x, 512x32x1 0.57x, 512x32x4 0.51x, 512x32x16 0.46x, 512x32x64 0.90x, 512x32x256 1.14x, 512x32x1024 1.11x, 512x32x4096 1.13x, 1024x32x1 0.58x, 1024x32x4 0.66x, 1024x32x16 0.62x, 1024x32x64 0.95x, 1024x32x256 1.03x, 1024x32x1024 0.96x, 1024x32x4096 1.09x, 40x40x1 0.63x, 40x40x4 0.89x, 40x40x16 0.62x, 40x40x64 1.33x, 40x40x256 2.10x, 40x40x1024 2.55x, 40x40x4096 2.39x, 48x48x1 0.65x, 48x48x4 0.74x, 48x48x16 0.74x, 48x48x64 1.41x, 48x48x256 2.08x, 48x48x1024 2.86x, 48x48x4096 3.07x, 56x56x1 0.67x, 56x56x4 0.67x, 56x56x16 0.65x, 56x56x64 1.40x, 56x56x256 2.19x, 56x56x1024 2.45x, 56x56x4096 2.45x, 64x64x1 0.60x, 64x64x4 0.59x, 64x64x16 0.61x, 64x64x64 1.44x, 64x64x256 1.73x, 64x64x1024 1.74x, 64x64x4096 1.77x, 128x64x1 0.51x, 128x64x4 0.55x, 128x64x16 0.46x, 128x64x64 0.97x, 128x64x256 0.95x, 128x64x1024 1.23x, 128x64x4096 1.16x, 256x64x1 0.56x, 256x64x4 0.58x, 256x64x16 0.55x, 256x64x64 1.08x, 256x64x256 1.03x, 256x64x1024 1.19x, 256x64x4096 1.13x, 512x64x1 0.55x, 512x64x4 0.61x, 512x64x16 0.64x, 512x64x64 1.02x, 512x64x256 1.09x, 512x64x1024 1.06x, 512x64x4096 1.11x, 1024x64x1 0.57x, 1024x64x4 0.66x, 1024x64x16 0.75x, 1024x64x64 1.04x, 1024x64x256 1.10x, 1024x64x1024 1.04x, 1024x64x4096 1.03x, 2048x64x1 0.64x, 2048x64x4 0.76x, 2048x64x16 0.81x, 2048x64x64 0.98x, 2048x64x256 1.01x, 2048x64x1024 1.03x, 80x80x1 0.63x, 80x80x4 0.66x, 80x80x16 0.69x, 80x80x64 1.26x, 80x80x256 1.87x, 80x80x1024 2.05x, 80x80x4096 2.05x
gk over the CPU, M x N x batch: 4x4x1 0.02x, 4x4x4 0.09x, 4x4x16 0.14x, 4x4x64 0.51x, 4x4x256 0.41x, 4x4x1024 1.00x, 4x4x4096 1.77x, 8x8x1 0.02x, 8x8x4 0.08x, 8x8x16 0.35x, 8x8x64 0.53x, 8x8x256 0.74x, 8x8x1024 1.20x, 8x8x4096 1.50x, 16x8x1 0.04x, 16x8x4 0.11x, 16x8x16 0.45x, 16x8x64 0.63x, 16x8x256 0.91x, 16x8x1024 1.51x, 16x8x4096 1.77x, 32x8x1 0.04x, 32x8x4 0.12x, 32x8x16 0.33x, 32x8x64 0.69x, 32x8x256 0.98x, 32x8x1024 1.41x, 32x8x4096 1.57x, 64x8x1 0.05x, 64x8x4 0.13x, 64x8x16 0.31x, 64x8x64 0.67x, 64x8x256 0.98x, 64x8x1024 1.19x, 64x8x4096 1.60x, 128x8x1 0.06x, 128x8x4 0.18x, 128x8x16 0.38x, 128x8x64 0.72x, 128x8x256 0.90x, 128x8x1024 0.89x, 128x8x4096 1.21x, 256x8x1 0.07x, 256x8x4 0.21x, 256x8x16 0.48x, 256x8x64 0.75x, 256x8x256 0.85x, 256x8x1024 1.19x, 256x8x4096 0.89x, 16x16x1 0.06x, 16x16x4 0.13x, 16x16x16 0.31x, 16x16x64 0.49x, 16x16x256 1.03x, 16x16x1024 1.27x, 16x16x4096 1.92x, 32x16x1 0.09x, 32x16x4 0.15x, 32x16x16 0.35x, 32x16x64 0.63x, 32x16x256 1.30x, 32x16x1024 1.62x, 32x16x4096 2.13x, 64x16x1 0.11x, 64x16x4 0.18x, 64x16x16 0.39x, 64x16x64 0.71x, 64x16x256 1.17x, 64x16x1024 1.41x, 64x16x4096 1.72x, 128x16x1 0.11x, 128x16x4 0.18x, 128x16x16 0.45x, 128x16x64 0.71x, 128x16x256 0.97x, 128x16x1024 1.41x, 128x16x4096 1.23x, 256x16x1 0.15x, 256x16x4 0.21x, 256x16x16 0.50x, 256x16x64 0.70x, 256x16x256 0.82x, 256x16x1024 1.05x, 256x16x4096 0.97x, 512x16x1 0.13x, 512x16x4 0.18x, 512x16x16 0.31x, 512x16x64 0.58x, 512x16x256 0.85x, 512x16x1024 0.64x, 512x16x4096 0.55x, 24x24x1 0.07x, 24x24x4 0.12x, 24x24x16 0.31x, 24x24x64 0.55x, 24x24x256 1.21x, 24x24x1024 1.41x, 24x24x4096 1.66x, 32x32x1 0.09x, 32x32x4 0.12x, 32x32x16 0.27x, 32x32x64 0.55x, 32x32x256 1.15x, 32x32x1024 1.53x, 32x32x4096 1.66x, 64x32x1 0.10x, 64x32x4 0.15x, 64x32x16 0.27x, 64x32x64 0.56x, 64x32x256 0.94x, 64x32x1024 1.38x, 64x32x4096 1.38x, 128x32x1 0.12x, 128x32x4 0.16x, 128x32x16 0.31x, 128x32x64 0.56x, 128x32x256 0.77x, 128x32x1024 0.99x, 128x32x4096 0.98x, 256x32x1 0.11x, 256x32x4 0.14x, 256x32x16 0.24x, 256x32x64 0.42x, 256x32x256 0.69x, 256x32x1024 0.78x, 256x32x4096 0.88x, 512x32x1 0.15x, 512x32x4 0.20x, 512x32x16 0.26x, 512x32x64 0.48x, 512x32x256 0.66x, 512x32x1024 0.67x, 512x32x4096 0.78x, 1024x32x1 0.19x, 1024x32x4 0.30x, 1024x32x16 0.35x, 1024x32x64 0.60x, 1024x32x256 0.61x, 1024x32x1024 0.59x, 1024x32x4096 0.75x, 40x40x1 0.10x, 40x40x4 0.14x, 40x40x16 0.28x, 40x40x64 0.60x, 40x40x256 0.94x, 40x40x1024 1.35x, 40x40x4096 1.30x, 48x48x1 0.10x, 48x48x4 0.15x, 48x48x16 0.27x, 48x48x64 0.58x, 48x48x256 1.00x, 48x48x1024 1.25x, 48x48x4096 1.24x, 56x56x1 0.10x, 56x56x4 0.13x, 56x56x16 0.23x, 56x56x64 0.52x, 56x56x256 0.83x, 56x56x1024 0.91x, 56x56x4096 0.91x, 64x64x1 0.12x, 64x64x4 0.13x, 64x64x16 0.22x, 64x64x64 0.53x, 64x64x256 0.81x, 64x64x1024 0.81x, 64x64x4096 0.83x, 128x64x1 0.13x, 128x64x4 0.15x, 128x64x16 0.20x, 128x64x64 0.49x, 128x64x256 0.58x, 128x64x1024 0.76x, 128x64x4096 0.70x, 256x64x1 0.14x, 256x64x4 0.19x, 256x64x16 0.24x, 256x64x64 0.52x, 256x64x256 0.61x, 256x64x1024 0.73x, 256x64x4096 0.69x, 512x64x1 0.18x, 512x64x4 0.26x, 512x64x16 0.34x, 512x64x64 0.55x, 512x64x256 0.63x, 512x64x1024 0.74x, 512x64x4096 0.75x, 1024x64x1 0.24x, 1024x64x4 0.35x, 1024x64x16 0.38x, 1024x64x64 0.61x, 1024x64x256 0.67x, 1024x64x1024 0.80x, 1024x64x4096 0.77x, 2048x64x1 0.31x, 2048x64x4 0.44x, 2048x64x16 0.53x, 2048x64x64 0.65x, 2048x64x256 0.68x, 2048x64x1024 0.87x, 80x80x1 0.15x, 80x80x4 0.17x, 80x80x16 0.21x, 80x80x64 0.47x, 80x80x256 0.67x, 80x80x1024 0.68x, 80x80x4096 0.64x
From a batch on, gk_share: gk and the CPU path at once on one batch, the GPU taking chunks from the front and the CPU from the back. Fitted against the best GPU backend, the shared one included, on the 88 points where it was timed and gk is the GPU’s choice. Chosen: from batch 1024.
| rule | geomean regret | worst | >10% | total time / oracle | est. picks |
|---|---|---|---|---|---|
| gk alone | 1.1417 | 1.76x | 37 | 1.342 | 0 |
| shared from batch 1024 | 1.0710 | 1.67x | 19 | 1.020 | 0 |
gk_share over gk alone, M x N x batch: 8x8x64 0.78x, 8x8x256 0.70x, 8x8x1024 0.64x, 8x8x4096 0.93x, 16x8x64 0.76x, 16x8x256 0.65x, 16x8x1024 0.62x, 16x8x4096 0.90x, 32x8x64 0.73x, 32x8x256 0.66x, 32x8x1024 0.73x, 32x8x4096 0.70x, 64x8x64 0.71x, 64x8x256 0.72x, 64x8x1024 0.84x, 64x8x4096 0.90x, 128x8x64 0.76x, 128x8x256 0.85x, 128x8x1024 1.16x, 128x8x4096 1.05x, 256x8x64 0.83x, 256x8x256 0.96x, 256x8x1024 0.99x, 256x8x4096 1.39x, 16x16x64 0.80x, 16x16x256 0.77x, 16x16x1024 1.01x, 16x16x4096 0.78x, 32x16x64 0.78x, 32x16x256 0.78x, 32x16x1024 0.97x, 32x16x4096 0.85x, 64x16x64 0.78x, 64x16x256 0.90x, 64x16x1024 1.03x, 64x16x4096 1.16x, 128x16x64 0.81x, 128x16x256 1.15x, 128x16x1024 1.21x, 128x16x4096 1.06x, 256x16x64 0.83x, 256x16x256 1.29x, 256x16x1024 1.16x, 256x16x4096 1.46x, 512x16x64 0.95x, 512x16x256 1.07x, 512x16x1024 1.26x, 512x16x4096 1.69x, 24x24x64 0.84x, 24x24x256 0.89x, 24x24x1024 1.23x, 24x24x4096 1.02x, 32x32x64 0.82x, 32x32x256 1.10x, 32x32x1024 1.12x, 32x32x4096 1.29x, 64x32x64 0.86x, 64x32x256 1.52x, 64x32x1024 1.16x, 64x32x4096 1.39x, 128x32x64 0.93x, 128x32x256 1.29x, 128x32x1024 1.57x, 128x32x4096 1.63x, 256x32x64 0.95x, 256x32x256 1.07x, 256x32x1024 1.13x, 256x32x4096 1.13x, 512x32x64 0.99x, 512x32x256 1.09x, 512x32x1024 1.25x, 512x32x4096 1.34x, 1024x32x64 1.05x, 1024x32x256 1.07x, 1024x32x1024 1.66x, 1024x32x4096 1.33x, 40x40x64 0.86x, 40x40x256 1.52x, 40x40x1024 1.14x, 40x40x4096 1.42x, 48x48x64 0.91x, 48x48x256 1.54x, 48x48x1024 1.40x, 48x48x4096 1.51x, 56x56x64 0.94x, 56x56x256 1.09x, 56x56x1024 1.70x, 56x56x4096 1.76x
GPU iff k <= gpu_max_k, l <= gpu_max_l (l = max(M, N)), batch * k >= gpu_min_batch_times_k and batch >= gpu_min_batch, with k = min(M, N), scored against the best of the CPU and the GPU backends (gk included). worst is over the points where the chosen backend was timed; a pick the cost model had to guess is listed in the warnings instead.
| rule | geomean regret | worst | >10% | total time / oracle | est. picks |
|---|---|---|---|---|---|
| oracle (best per point) | 1.0000 | 1.00x | 0 | 1.000 | 0 |
| policy in effect (‘56’, ‘16384’, ‘1’, ‘256’) | 1.0196 | 1.55x | 22 | 1.007 | 0 |
| fitted (‘56’, ‘16384’, ‘1’, ‘256’) | 1.0196 | 1.55x | 22 | 1.007 | 0 |
93 of 6171 combinations are within 0.5% of the best geomean: gpu_max_k 56 .. 80, gpu_min_batch_times_k 8192 .. 16384, gpu_min_batch 1 .. 16, gpu_max_l 128 .. none.
Large batches: the GPU also for k above gpu_max_k up to 80 (and l <= gpu_max_l) in a batch of at least 1024, fitted with the product rule (per cap, the rule, the clause over it and the rule again given the clause, the best kept) (product rule alone 1.0277, worst 1.58x; chosen 1.0196, worst 1.55x).
xychart-beta
title "Regret by gpu_min_batch_times_k"
x-axis "gpu_min_batch_times_k" [0, 64, 128, 256, 512, 1024, 2048, 4096, 8192, 16384, none]
y-axis "geometric-mean regret" 1.0 --> 1.58
line [1.5699, 1.2491, 1.1962, 1.1197, 1.0958, 1.0531, 1.0403, 1.0226, 1.0228, 1.0196, 1.0671]
| gpu_min_batch_times_k | 0 | 64 | 128 | 256 | 512 | 1024 | 2048 | 4096 | 8192 | 16384 | none |
|---|---|---|---|---|---|---|---|---|---|---|---|
| geomean | 1.5699 | 1.2491 | 1.1962 | 1.1197 | 1.0958 | 1.0531 | 1.0403 | 1.0226 | 1.0228 | 1.0196 | 1.0671 |
| worst | 54.35x | 8.41x | 8.22x | 4.41x | 4.41x | 2.41x | 2.41x | 1.60x | 1.60x | 1.55x | 2.14x |
xychart-beta
title "Regret by gpu_max_l"
x-axis "gpu_max_l" [16, 24, 32, 40, 48, 56, 64, 80, 96, 128, 192, 256, 384, 512, 768, 1024, 2048, none]
y-axis "geometric-mean regret" 1.0 --> 1.08
line [1.0696, 1.0657, 1.0569, 1.0532, 1.0490, 1.0457, 1.0335, 1.0309, 1.0309, 1.0228, 1.0228, 1.0196, 1.0196, 1.0204, 1.0204, 1.0202, 1.0248, 1.0248]
| gpu_max_l | 16 | 24 | 32 | 40 | 48 | 56 | 64 | 80 | 96 | 128 | 192 | 256 | 384 | 512 | 768 | 1024 | 2048 | none |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| geomean | 1.0696 | 1.0657 | 1.0569 | 1.0532 | 1.0490 | 1.0457 | 1.0335 | 1.0309 | 1.0309 | 1.0228 | 1.0228 | 1.0196 | 1.0196 | 1.0204 | 1.0204 | 1.0202 | 1.0248 | 1.0248 |
| worst | 2.14x | 2.14x | 1.99x | 1.99x | 1.99x | 1.99x | 1.71x | 1.71x | 1.71x | 1.55x | 1.55x | 1.55x | 1.55x | 1.55x | 1.55x | 1.55x | 1.55x | 1.55x |
xychart-beta
title "Regret by gpu_min_batch"
x-axis "gpu_min_batch" [1, 4, 16]
y-axis "geometric-mean regret" 1.0 --> 1.03
line [1.0196, 1.0196, 1.0196]
| gpu_min_batch | 1 | 4 | 16 |
|---|---|---|---|
| geomean | 1.0196 | 1.0196 | 1.0196 |
| worst | 1.55x | 1.55x | 1.55x |
xychart-beta
title "Regret by gpu_max_k"
x-axis "gpu_max_k" [8, 16, 24, 32, 40, 48, 56, 64, 80, 96, 128, 192, 256, 384, 512, 768, 1024, none]
y-axis "geometric-mean regret" 1.0 --> 1.09
line [1.0196, 1.0196, 1.0196, 1.0196, 1.0196, 1.0196, 1.0196, 1.0240, 1.0253, 1.0335, 1.0492, 1.0594, 1.0757, 1.0757, 1.0757, 1.0757, 1.0757, 1.0757]
| gpu_max_k | 8 | 16 | 24 | 32 | 40 | 48 | 56 | 64 | 80 | 96 | 128 | 192 | 256 | 384 | 512 | 768 | 1024 | none |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| geomean | 1.0196 | 1.0196 | 1.0196 | 1.0196 | 1.0196 | 1.0196 | 1.0196 | 1.0240 | 1.0253 | 1.0335 | 1.0492 | 1.0594 | 1.0757 | 1.0757 | 1.0757 | 1.0757 | 1.0757 | 1.0757 |
| worst | 1.55x | 1.55x | 1.55x | 1.55x | 1.55x | 1.55x | 1.55x | 1.73x | 1.73x | 2.33x | 2.55x | 4.40x | 4.90x | 4.90x | 4.90x | 4.90x | 4.90x | 4.90x |
Held-out check of a per-k boundary against the product rule, fitted on 167 points and scored on the other 128; the verdict is a bootstrap.
| rule | fitted on train | train geomean | test geomean | test worst | verdict |
|---|---|---|---|---|---|
| product rule | [56, 16384, 1, 256] | 1.0189 | 1.0206 | 1.51x | baseline |
| per-k table | {“min_batch_by_k”: {“4”: null, “8”: 1024, “16”: 256, “24”: 1024, “32”: 256, “40”: 1024, “48”: 256, “56”: 1024, “64”: 1024, “80”: null, “96”: null, “128”: null, “192”: null, “256”: null, “384”: null, “512”: null, “768”: null, “1024”: null}} | 1.0158 | 1.0357 | 1.77x | rejected (better in 0% of resamples, median gain -1.4%) |
Best backend per point (c CPU, J whole-matrix kernel, B block kernel, j and b the same after QR, g gk, G gk shared with the CPU, . not measured), what the rule picks, and the speedup of the best GPU backend over the CPU:
M x N \ batch 1 4 16 64 256 1024 4096
4 x 4 c c c c c g g
8 x 8 c c c c c g g
16 x 8 c c c c c g g
32 x 8 c c c c c g g
64 x 8 c c c c c g g
128 x 8 c c c c c G G
256 x 8 c c c c c J G
16 x 16 c c c c g G g
32 x 16 c c c c g g g
64 x 16 c c c c g G G
128 x 16 c c c c G G G
256 x 16 c c c c G G G
512 x 16 c c c c c c c
24 x 24 c c c c g G G
32 x 32 c c c c G G G
64 x 32 c c c c G G G
128 x 32 c c c c c G G
256 x 32 c c c c c c c
512 x 32 c c c c c c G
1024 x 32 c c c c c c G
40 x 40 c c c c G G G
48 x 48 c c c c G G G
56 x 56 c c c c c G G
64 x 64 c c c c G G G
128 x 64 c c c c c G G
256 x 64 c c c c c G G
512 x 64 c c c c c G G
1024 x 64 c c c c c c G
2048 x 64 c c c c c G c
80 x 80 c c c c G G G
96 x 96 c c c c c c c
128 x 128 c c c c c c c
256 x 128 c c c c c c c
512 x 128 c c c c c c c
1024 x 128 c c c c c c c
2048 x 128 c c c c c c .
192 x 192 c c c c c c .
256 x 256 c c c c c c .
512 x 256 c c c c c c .
1024 x 256 c c c c c c .
2048 x 256 c c c c c . .
384 x 384 c c c c c . .
512 x 512 c c c c . . .
768 x 768 c c c . . . .
1024 x 1024 c c c . . . .
M x N \ batch 1 4 16 64 256 1024 4096
4 x 4 c c c c c c J
8 x 8 c c c c c c G
16 x 8 c c c c c c G
32 x 8 c c c c c c G
64 x 8 c c c c c c G
128 x 8 c c c c c c G
256 x 8 c c c c c c G
16 x 16 c c c c c G G
32 x 16 c c c c c G G
64 x 16 c c c c c G G
128 x 16 c c c c c G G
256 x 16 c c c c c G G
512 x 16 c c c c c c c
24 x 24 c c c c c G G
32 x 32 c c c c c G G
64 x 32 c c c c c G G
128 x 32 c c c c c G G
256 x 32 c c c c c G G
512 x 32 c c c c c c c
1024 x 32 c c c c c c c
40 x 40 c c c c c G G
48 x 48 c c c c c G G
56 x 56 c c c c c G G
64 x 64 c c c c c G G
128 x 64 c c c c c G G
256 x 64 c c c c c G G
512 x 64 c c c c c c c
1024 x 64 c c c c c c c
2048 x 64 c c c c c c c
80 x 80 c c c c c G G
96 x 96 c c c c c c c
128 x 128 c c c c c c c
256 x 128 c c c c c c c
512 x 128 c c c c c c c
1024 x 128 c c c c c c c
2048 x 128 c c c c c c .
192 x 192 c c c c c c .
256 x 256 c c c c c c .
512 x 256 c c c c c c .
1024 x 256 c c c c c c .
2048 x 256 c c c c c . .
384 x 384 c c c c c . .
512 x 512 c c c c . . .
768 x 768 c c c . . . .
1024 x 1024 c c c . . . .
M x N \ batch 1 4 16 64 256 1024 4096
4 x 4 0.02 0.10 0.19 0.58 0.45 1.00 1.77
8 x 8 0.03 0.08 0.35 0.53 0.74 1.20 1.50
16 x 8 0.04 0.11 0.45 0.63 0.91 1.51 1.77
32 x 8 0.04 0.13 0.33 0.69 0.98 1.41 1.57
64 x 8 0.05 0.13 0.31 0.67 0.98 1.19 1.60
128 x 8 0.06 0.18 0.38 0.72 0.90 0.89 1.25
256 x 8 0.07 0.21 0.48 0.75 0.85 1.25 0.99
16 x 16 0.06 0.18 0.31 0.49 1.03 1.27 1.92
32 x 16 0.09 0.15 0.35 0.63 1.30 1.62 2.13
64 x 16 0.11 0.18 0.39 0.81 1.17 1.41 1.72
128 x 16 0.11 0.18 0.45 0.71 0.97 1.41 1.23
256 x 16 0.15 0.21 0.50 0.70 0.82 1.05 0.97
512 x 16 0.18 0.22 0.57 0.96 0.85 0.64 0.55
24 x 24 0.11 0.12 0.46 0.55 1.21 1.41 1.66
32 x 32 0.13 0.25 0.52 0.55 1.15 1.53 1.66
64 x 32 0.17 0.30 0.51 0.57 0.94 1.38 1.38
128 x 32 0.17 0.32 0.38 0.56 0.77 0.99 0.98
256 x 32 0.24 0.35 0.53 0.56 0.69 0.78 0.88
512 x 32 0.26 0.40 0.56 0.54 0.66 0.67 0.78
1024 x 32 0.33 0.46 0.57 0.63 0.61 0.61 0.75
40 x 40 0.17 0.16 0.46 0.60 0.94 1.35 1.30
48 x 48 0.16 0.20 0.37 0.58 1.00 1.25 1.24
56 x 56 0.15 0.20 0.35 0.52 0.83 0.91 0.91
64 x 64 0.19 0.22 0.36 0.53 0.81 0.81 0.83
128 x 64 0.25 0.28 0.42 0.50 0.61 0.76 0.70
256 x 64 0.25 0.33 0.44 0.52 0.61 0.73 0.69
512 x 64 0.32 0.42 0.53 0.55 0.63 0.74 0.75
1024 x 64 0.41 0.53 0.51 0.61 0.67 0.80 0.77
2048 x 64 0.49 0.58 0.65 0.67 0.68 0.87 0.81
80 x 80 0.24 0.26 0.31 0.47 0.67 0.68 0.64
96 x 96 0.25 0.31 0.38 0.39 0.49 0.45 0.43
128 x 128 0.23 0.30 0.37 0.42 0.45 0.41 0.39
256 x 128 0.31 0.40 0.56 0.59 0.58 0.59 0.56
512 x 128 0.35 0.37 0.53 0.59 0.62 0.66 0.63
1024 x 128 0.42 0.43 0.54 0.63 0.63 0.75 0.74
2048 x 128 0.48 0.53 0.63 0.67 0.69 0.88 .
192 x 192 0.21 0.20 0.25 0.27 0.25 0.23 .
256 x 256 0.30 0.30 0.24 0.24 0.23 0.20 .
512 x 256 0.44 0.41 0.35 0.37 0.33 0.31 .
1024 x 256 0.47 0.48 0.43 0.44 0.40 0.38 .
2048 x 256 0.55 0.54 0.52 0.51 0.51 . .
384 x 384 0.34 0.30 0.19 0.16 0.15 . .
512 x 512 0.52 0.38 0.18 0.16 . . .
768 x 768 0.51 0.35 0.15 . . . .
1024 x 1024 0.67 0.32 0.25 . . . .
The rule of stage 2 again, with constants of its own, for svdvals: both sides skip the vectors, by different amounts. Fitted on the 202 points where gk and the CPU were timed for singular values alone and gk is the GPU’s choice. Chosen: GPU iff k <= 56, l <= 256, batch * k >= 16384 and batch >= 1.
| rule | geomean regret | worst | >10% | total time / oracle | est. picks |
|---|---|---|---|---|---|
| CPU always | 1.0814 | 1.98x | 39 | 1.095 | 0 |
| as with vectors (stage 2’s rule) | 1.0305 | 2.00x | 20 | 1.026 | 0 |
| fitted (‘56’, ‘16384’, ‘1’, ‘256’) | 1.0347 | 2.00x | 21 | 1.054 | 0 |
Held out: fitted on 115 points ((‘80’, ‘16384’, ‘1’, ‘128’)), scored on the other 87: geomean 1.0589x, worst 2.40x, against 1.0407x, worst 1.69x as with vectors.
gk over the CPU for singular values alone, M x N x batch: 8x8x1 0.03x, 8x8x4 0.09x, 8x8x16 0.23x, 8x8x64 0.50x, 8x8x256 0.71x, 8x8x1024 1.05x, 8x8x4096 1.65x, 16x8x1 0.03x, 16x8x4 0.10x, 16x8x16 0.30x, 16x8x64 0.64x, 16x8x256 0.89x, 16x8x1024 1.52x, 16x8x4096 1.92x, 32x8x1 0.03x, 32x8x4 0.12x, 32x8x16 0.36x, 32x8x64 0.55x, 32x8x256 0.97x, 32x8x1024 1.29x, 32x8x4096 1.94x, 64x8x1 0.03x, 64x8x4 0.10x, 64x8x16 0.37x, 64x8x64 0.64x, 64x8x256 0.90x, 64x8x1024 1.35x, 64x8x4096 1.82x, 128x8x1 0.03x, 128x8x4 0.10x, 128x8x16 0.46x, 128x8x64 0.73x, 128x8x256 0.83x, 128x8x1024 1.77x, 128x8x4096 1.72x, 256x8x1 0.03x, 256x8x4 0.16x, 256x8x16 0.49x, 256x8x64 0.82x, 256x8x256 0.84x, 256x8x1024 1.24x, 256x8x4096 1.27x, 16x16x1 0.03x, 16x16x4 0.09x, 16x16x16 0.32x, 16x16x64 0.55x, 16x16x256 0.87x, 16x16x1024 1.15x, 16x16x4096 1.44x, 32x16x1 0.04x, 32x16x4 0.12x, 32x16x16 0.34x, 32x16x64 0.51x, 32x16x256 0.84x, 32x16x1024 1.30x, 32x16x4096 1.98x, 64x16x1 0.04x, 64x16x4 0.16x, 64x16x16 0.30x, 64x16x64 0.64x, 64x16x256 0.87x, 64x16x1024 1.21x, 64x16x4096 1.64x, 128x16x1 0.05x, 128x16x4 0.15x, 128x16x16 0.32x, 128x16x64 0.63x, 128x16x256 0.75x, 128x16x1024 1.53x, 128x16x4096 1.27x, 256x16x1 0.07x, 256x16x4 0.18x, 256x16x16 0.44x, 256x16x64 0.65x, 256x16x256 0.97x, 256x16x1024 1.18x, 256x16x4096 0.94x, 512x16x1 0.12x, 512x16x4 0.20x, 512x16x16 0.35x, 512x16x64 0.42x, 512x16x256 0.44x, 512x16x1024 0.58x, 512x16x4096 0.44x, 24x24x1 0.04x, 24x24x4 0.09x, 24x24x16 0.23x, 24x24x64 0.45x, 24x24x256 0.82x, 24x24x1024 1.07x, 24x24x4096 1.51x, 32x32x1 0.04x, 32x32x4 0.10x, 32x32x16 0.23x, 32x32x64 0.41x, 32x32x256 0.92x, 32x32x1024 1.06x, 32x32x4096 1.26x, 64x32x1 0.05x, 64x32x4 0.10x, 64x32x16 0.26x, 64x32x64 0.45x, 64x32x256 0.69x, 64x32x1024 0.85x, 64x32x4096 0.94x, 128x32x1 0.06x, 128x32x4 0.12x, 128x32x16 0.32x, 128x32x64 0.49x, 128x32x256 0.53x, 128x32x1024 0.78x, 128x32x4096 0.67x, 256x32x1 0.09x, 256x32x4 0.14x, 256x32x16 0.27x, 256x32x64 0.40x, 256x32x256 0.56x, 256x32x1024 0.58x, 256x32x4096 0.76x, 512x32x1 0.12x, 512x32x4 0.19x, 512x32x16 0.30x, 512x32x64 0.54x, 512x32x256 0.50x, 512x32x1024 0.47x, 512x32x4096 0.66x, 1024x32x1 0.16x, 1024x32x4 0.25x, 1024x32x16 0.33x, 1024x32x64 0.66x, 1024x32x256 0.49x, 1024x32x1024 0.48x, 1024x32x4096 0.67x, 40x40x1 0.04x, 40x40x4 0.08x, 40x40x16 0.23x, 40x40x64 0.32x, 40x40x256 0.62x, 40x40x1024 0.91x, 40x40x4096 1.01x, 48x48x1 0.04x, 48x48x4 0.08x, 48x48x16 0.20x, 48x48x64 0.31x, 48x48x256 0.62x, 48x48x1024 0.97x, 48x48x4096 0.89x, 56x56x1 0.05x, 56x56x4 0.08x, 56x56x16 0.18x, 56x56x64 0.32x, 56x56x256 0.69x, 56x56x1024 0.74x, 56x56x4096 0.77x, 64x64x1 0.05x, 64x64x4 0.08x, 64x64x16 0.16x, 64x64x64 0.34x, 64x64x256 0.55x, 64x64x1024 0.65x, 64x64x4096 0.63x, 128x64x1 0.08x, 128x64x4 0.10x, 128x64x16 0.14x, 128x64x64 0.31x, 128x64x256 0.42x, 128x64x1024 0.57x, 128x64x4096 0.60x, 256x64x1 0.08x, 256x64x4 0.13x, 256x64x16 0.21x, 256x64x64 0.35x, 256x64x256 0.43x, 256x64x1024 0.56x, 256x64x4096 0.56x, 512x64x1 0.10x, 512x64x4 0.17x, 512x64x16 0.24x, 512x64x64 0.38x, 512x64x256 0.47x, 512x64x1024 0.65x, 512x64x4096 0.65x, 1024x64x1 0.14x, 1024x64x4 0.22x, 1024x64x16 0.28x, 1024x64x64 0.41x, 1024x64x256 0.46x, 1024x64x1024 0.70x, 1024x64x4096 0.70x, 2048x64x1 0.19x, 2048x64x4 0.27x, 2048x64x16 0.38x, 2048x64x64 0.44x, 2048x64x256 0.44x, 2048x64x1024 0.74x, 80x80x1 0.11x, 80x80x4 0.13x, 80x80x16 0.18x, 80x80x64 0.39x, 80x80x256 0.84x, 80x80x1024 0.84x, 80x80x4096 0.83x
Where the rule above chooses the CPU, the bidiag backend (GPU bidiagonalization, then LAPACK’s bidiagonal solve) from a threshold k = min(M, N) on (0: never), for batches up to a cap (0: any; it solves a batch one matrix after another, the CPU path spreads one over every core), fitted on the points where bidiag was timed (k >= 128, within the cost cap): the region the threshold decides. With vectors it is scored against the best of all backends, bidiag included; for singular values alone (svdvals), bidiag against the CPU at the points where the rule chooses the CPU.
| threshold | batch cap | geomean regret | worst | without bidiag: geomean | worst | held out (fitted on half) | |
|---|---|---|---|---|---|---|---|
| with vectors | 1024 | 4 | 1.0129 | 1.50x | 1.1692 | 4.09x | from 1024, batch <= 4: 1.0147 vs 1.0587 |
| singular values alone | 1024 | 2 | 1.0057 | 1.36x | 1.0861 | 2.68x | from 1024, batch <= 2: 1.0000 vs 1.0277 |
bidiag over the CPU (with vectors), M x N x batch: 128x128x1 0.38x, 128x128x4 0.14x, 128x128x16 0.05x, 128x128x64 0.04x, 128x128x256 0.04x, 256x128x1 0.44x, 256x128x4 0.18x, 256x128x16 0.07x, 256x128x64 0.05x, 256x128x256 0.05x, 512x128x1 0.48x, 512x128x4 0.16x, 512x128x16 0.08x, 512x128x64 0.06x, 512x128x256 0.06x, 1024x128x1 0.55x, 1024x128x4 0.19x, 1024x128x16 0.09x, 1024x128x64 0.08x, 1024x128x256 0.07x, 2048x128x1 0.57x, 2048x128x4 0.21x, 2048x128x16 0.11x, 2048x128x64 0.09x, 2048x128x256 0.08x, 192x192x1 0.47x, 192x192x4 0.14x, 192x192x16 0.06x, 192x192x64 0.05x, 192x192x256 0.04x, 256x256x1 0.70x, 256x256x4 0.22x, 256x256x16 0.08x, 256x256x64 0.07x, 512x256x1 0.71x, 512x256x4 0.22x, 512x256x16 0.08x, 512x256x64 0.07x, 1024x256x1 0.73x, 1024x256x4 0.24x, 1024x256x16 0.10x, 1024x256x64 0.08x, 2048x256x1 0.78x, 2048x256x4 0.25x, 2048x256x16 0.12x, 2048x256x64 0.11x, 384x384x1 0.79x, 384x384x4 0.30x, 384x384x16 0.11x, 384x384x64 0.10x, 512x512x1 1.27x, 512x512x4 0.46x, 512x512x16 0.18x, 512x512x64 0.16x, 768x768x1 1.50x, 768x768x4 0.60x, 768x768x16 0.30x, 1024x1024x1 2.30x, 1024x1024x4 0.90x, 1024x1024x16 0.75x, 1280x1280x1 2.09x, 1280x1280x2 1.52x, 1280x1280x4 0.82x, 1536x1536x1 2.65x, 1536x1536x2 1.80x, 1536x1536x4 1.02x, 1792x1792x1 2.46x, 1792x1792x2 1.82x, 1792x1792x4 1.17x, 2048x2048x1 3.65x, 2048x2048x2 2.81x, 2048x2048x4 1.90x, 3072x3072x1 3.76x, 4096x4096x1 4.09x
bidiag over the CPU (singular values alone), M x N x batch: 128x128x1 0.38x, 128x128x4 0.14x, 128x128x16 0.05x, 128x128x64 0.04x, 128x128x256 0.04x, 256x128x1 0.39x, 256x128x4 0.15x, 256x128x16 0.05x, 256x128x64 0.04x, 256x128x256 0.04x, 512x128x1 0.40x, 512x128x4 0.14x, 512x128x16 0.06x, 512x128x64 0.04x, 512x128x256 0.04x, 1024x128x1 0.41x, 1024x128x4 0.13x, 1024x128x16 0.06x, 1024x128x64 0.05x, 1024x128x256 0.04x, 2048x128x1 0.38x, 2048x128x4 0.14x, 2048x128x16 0.05x, 2048x128x64 0.05x, 2048x128x256 0.04x, 192x192x1 0.32x, 192x192x4 0.11x, 192x192x16 0.05x, 192x192x64 0.03x, 192x192x256 0.03x, 256x256x1 0.46x, 256x256x4 0.16x, 256x256x16 0.06x, 256x256x64 0.05x, 512x256x1 0.43x, 512x256x4 0.16x, 512x256x16 0.07x, 512x256x64 0.05x, 1024x256x1 0.44x, 1024x256x4 0.15x, 1024x256x16 0.07x, 1024x256x64 0.05x, 2048x256x1 0.44x, 2048x256x4 0.14x, 2048x256x16 0.07x, 2048x256x64 0.06x, 384x384x1 0.60x, 384x384x4 0.20x, 384x384x16 0.11x, 384x384x64 0.07x, 512x512x1 0.87x, 512x512x4 0.30x, 512x512x16 0.14x, 512x512x64 0.11x, 768x768x1 1.04x, 768x768x4 0.36x, 768x768x16 0.25x, 1024x1024x1 1.71x, 1024x1024x4 0.54x, 1024x1024x16 0.73x, 1280x1280x1 1.56x, 1280x1280x2 0.93x, 1280x1280x4 0.49x, 1536x1536x1 1.88x, 1536x1536x2 1.17x, 1536x1536x4 0.70x, 1792x1792x1 1.70x, 1792x1792x2 1.21x, 1792x1792x4 0.85x, 2048x2048x1 2.27x, 2048x2048x2 1.68x, 2048x2048x4 1.36x, 3072x3072x1 2.58x, 4096x4096x1 2.68x
For singular values alone, where the rules above choose the CPU or bidiag, the band backend (the two-stage reduction: A to a band on the GPU in blocks whose work is matrix products, the band to bidiagonal on the CPU’s cores, then bisection on the GPU) from a threshold k on (0: never), within bidiag’s batch cap, fitted on the 24 points where it was timed (k >= 512) against the CPU, bidiag and band.
Chosen: 768 (in effect: 1536): 1.0750 geometric-mean regret, worst 2.85x; without band 1.3690, worst 3.68x.
Its band’s width: 16 (in effect: 16); geometric mean of each width’s time over the best width’s at each point: 8 1.080, 16 1.017, 32 1.474. 16, the default, unless another is better by more than 1%.
Thresholds within the fit’s tolerance of the best, and within 3% of it on the points where the two choose differently: 768.
band over bidiag, M x N x batch: 512x512x1 1.01x, 512x512x4 1.20x, 512x512x16 1.27x, 512x512x64 1.29x, 768x768x1 1.03x, 768x768x4 1.22x, 768x768x16 1.29x, 1024x1024x1 1.11x, 1024x1024x4 1.31x, 1024x1024x16 1.38x, 1280x1280x1 1.23x, 1280x1280x2 1.37x, 1280x1280x4 1.47x, 1536x1536x1 1.41x, 1536x1536x2 1.58x, 1536x1536x4 1.68x, 1792x1792x1 1.45x, 1792x1792x2 1.66x, 1792x1792x4 1.84x, 2048x2048x1 1.66x, 2048x2048x2 1.86x, 2048x2048x4 2.09x, 3072x3072x1 2.67x, 4096x4096x1 3.68x
band over the CPU, M x N x batch: 512x512x1 0.88x, 512x512x4 0.36x, 512x512x16 0.17x, 512x512x64 0.15x, 768x768x1 1.08x, 768x768x4 0.43x, 768x768x16 0.33x, 1024x1024x1 1.90x, 1024x1024x4 0.71x, 1024x1024x16 1.01x, 1280x1280x1 1.92x, 1280x1280x2 1.28x, 1280x1280x4 0.72x, 1536x1536x1 2.66x, 1536x1536x2 1.85x, 1536x1536x4 1.18x, 1792x1792x1 2.46x, 1792x1792x2 2.01x, 1792x1792x4 1.57x, 2048x2048x1 3.76x, 2048x2048x2 3.14x, 2048x2048x4 2.85x, 3072x3072x1 6.88x, 4096x4096x1 9.87x
With singular vectors, where the rules above choose the CPU or bidiag, the band backend (the two-stage reduction, its band 16 wide; both stages’ reflectors applied on the GPU, Q1 Q2 and P1 P2 formed while the CPU chases the band and solves the bidiagonal problem) from a threshold k on (0: never), within bidiag’s batch cap, fitted on the 24 points where it was timed (k >= 512) against the CPU, bidiag and band.
Chosen: 1024 (in effect: never): 1.0559 geometric-mean regret, worst 1.74x; without band 1.2452, worst 2.45x.
Thresholds within the fit’s tolerance of the best, and within 3% of it on the points where the two choose differently: 1024.
band over bidiag, M x N x batch: 512x512x1 1.13x, 512x512x4 0.88x, 512x512x16 0.86x, 512x512x64 0.87x, 768x768x1 1.16x, 768x768x4 0.87x, 768x768x16 0.88x, 1024x1024x1 1.22x, 1024x1024x4 0.91x, 1024x1024x16 0.89x, 1280x1280x1 1.28x, 1280x1280x2 1.02x, 1280x1280x4 1.00x, 1536x1536x1 1.35x, 1536x1536x2 1.09x, 1536x1536x4 1.08x, 1792x1792x1 1.38x, 1792x1792x2 1.16x, 1792x1792x4 1.15x, 2048x2048x1 1.51x, 2048x2048x2 1.28x, 2048x2048x4 1.28x, 3072x3072x1 2.04x, 4096x4096x1 2.45x
band over the CPU, M x N x batch: 512x512x1 1.43x, 512x512x4 0.41x, 512x512x16 0.15x, 512x512x64 0.14x, 768x768x1 1.74x, 768x768x4 0.52x, 768x768x16 0.26x, 1024x1024x1 2.80x, 1024x1024x4 0.82x, 1024x1024x16 0.67x, 1280x1280x1 2.68x, 1280x1280x2 1.55x, 1280x1280x4 0.83x, 1536x1536x1 3.56x, 1536x1536x2 1.95x, 1536x1536x4 1.11x, 1792x1792x1 3.39x, 1792x1792x2 2.11x, 1792x1792x4 1.34x, 2048x2048x1 5.53x, 2048x2048x2 3.59x, 2048x2048x4 2.43x, 3072x3072x1 7.68x, 4096x4096x1 10.01x
Pass-to-pass ratio, 2266 measurements: median 1.008, p90 1.067, max 2.83.
| runtime | n | median | p90 | max |
|---|---|---|---|---|
| <1 ms | 739 | 1.020 | 1.207 | 2.83 |
| 1-3 ms | 375 | 1.007 | 1.046 | 1.53 |
| 3-10 ms | 350 | 1.007 | 1.030 | 1.26 |
| 10-30 ms | 240 | 1.006 | 1.028 | 1.30 |
| 30-100 ms | 247 | 1.005 | 1.024 | 1.10 |
| >100 ms | 315 | 1.003 | 1.018 | 1.10 |