Cross-backend notes

Where the same model and chip were recorded on more than one backend, we show them side by side. 117 comparable cases so far. For regressions and improvements between builds of the same backend, see the build A/B notes.

AREX-2 on M6 (Q4_K_M)

Splash: 22 tok/s (100%) community source
LM Studio: 6.8 tok/s (31%) community source

Auto-generated comparison; relative % vs the fastest recorded backend. Differences between backends on the same model+chip can reflect build/version, sampling, and context differences, not just the backend itself.

Bonsai 2 27B on Arc B580 (PTQ1_0)

llama.cpp (SYCL): 250 tok/s (100%) community source
llama.cpp (SYCL): 90 tok/s (36%) community source
llama.cpp (SYCL): 44 tok/s (18%) community source
llama.cpp (SYCL): 42 tok/s (17%) community source
llama.cpp (SYCL): 36 tok/s (14%) community source
llama.cpp (SYCL): 26.4 tok/s (11%) community source
llama.cpp (PrismML SYCL fork): 21 tok/s (8%) community source
llama.cpp (PrismML SYCL fork): 8 tok/s (3%) community source

deci 70B on RTX 5060 Ti (IQ4_XS - 4.25 bpw)

llama.cpp (CUDA): 14.6 tok/s (100%) community source
llama.cpp (CUDA): 14.59 tok/s (100%) community source
llama.cpp (CUDA): 14.41 tok/s (99%) community source
llama.cpp (CUDA): 14.35 tok/s (98%) community source
llama.cpp (Vulkan): 10.61 tok/s (73%) community source
llama.cpp (Vulkan): 9.76 tok/s (67%) community source
llama.cpp (Vulkan): 9.73 tok/s (67%) community source
llama.cpp (Vulkan): 8.48 tok/s (58%) community source
llama.cpp (Vulkan): 8.45 tok/s (58%) community source
llama.cpp (Vulkan): 6.73 tok/s (46%) community source

gemma3 27B on Radeon RX 7900 XTX (Q4_0)

llama.cpp (Vulkan): 35.25 tok/s (100%) community source
llama.cpp (ROCm): 32.73 tok/s (93%) community source

gemma3 27B on Radeon RX 7900 XTX (Q6_K)

llama.cpp (Vulkan): 7.05 tok/s (100%) community source
llama.cpp (ROCm): 6.81 tok/s (97%) community source

Gemma 4 12B on RX 6800 (Q4_K_M)

llama.cpp (Vulkan): 52.1 tok/s (100%) community source
llama.cpp (ROCm): 48.9 tok/s (94%) community source

Gemma 4 12B on Radeon RX 9060 XT (Q4_K_M)

llama.cpp (Vulkan): 39.0 tok/s (100%) community source
llama.cpp (ROCm): 37.6 tok/s (96%) community source

gemma-4-12b-it on Radeon RX 6700 XT (IQ4_NL)

llama.cpp (Vulkan): 40.92 tok/s (100%) community source
llama.cpp (ROCm): 34.6 tok/s (85%) community source

Gemma4 31B on Radeon RX 7900 XTX (Q8_0)

llama.cpp (ROCm): 32.35 tok/s (100%) community source
llama.cpp (Vulkan): 24.42 tok/s (75%) community source
llama.cpp (ROCm): 23.43 tok/s (72%) community source
llama.cpp (ROCm): 22.97 tok/s (71%) community source
llama.cpp (Vulkan): 21.03 tok/s (65%) community source

gemma-4-31B-it on Intel Arc B580 Limited Edition (UD-Q4_K_XL)

llama.cpp (SYCL): 28.44 tok/s (100%) community source
llama.cpp (SYCL): 23.79 tok/s (84%) community source
Vulkan: 23.11 tok/s (81%) community source
Vulkan: 20.87 tok/s (73%) community source

Gemma 4 E4B / gemma4 E4B on Arc B580 (Q4_0)

llama.cpp (Vulkan): 90.8 tok/s (100%) community source
llama.cpp (Vulkan): 90.7 tok/s (100%) community source
llama.cpp (Vulkan): 87.0 tok/s (96%) community source
llama.cpp (Vulkan): 86.2 tok/s (95%) community source
llama.cpp (SYCL): 82.3 tok/s (91%) community source
llama.cpp (SYCL): 81.08 tok/s (89%) community source
llama.cpp (Vulkan): tok/s (0%) community source
llama.cpp (Vulkan): tok/s (0%) community source

glm4moe 106B.A12B on Radeon RX 7900 XTX (Q3_K - Medium)

llama.cpp (ROCm): 13.61 tok/s (100%) community source
llama.cpp (Vulkan): 10.35 tok/s (76%) community source

gpt-oss 20B on RX 6800 (MXFP4)

llama.cpp (Vulkan): 145.9 tok/s (100%) community source
llama.cpp (ROCm): 121.2 tok/s (83%) community source

gpt-oss 20B on Radeon RX 9060 XT (MXFP4)

llama.cpp (Vulkan): 105.8 tok/s (100%) community source
llama.cpp (ROCm): 92.8 tok/s (88%) community source

Hy-MT2-30B-A3B-APEX-Imatrix-I-Balanced on Radeon 8060S (Q6_K)

llama.cpp (Vulkan): 74.28 tok/s (100%) community source
llama.cpp (ROCm): 59.19 tok/s (80%) community source

Llama-3.2-1B on Snapdragon 7 Gen 4 (SM7750) (Q4_0)

llama.cpp (CPU): 34.4 tok/s (100%) community source
llama.cpp (Hexagon): 29.9 tok/s (87%) community source
llama.cpp (Hexagon): 27.0 tok/s (78%) community source

llama 7B on A100 (Q4_0)

llama.cpp (CUDA): 215.11 tok/s (100%) community source
llama.cpp (CUDA): 200.9 tok/s (93%) community source
llama.cpp (CUDA): 200.05 tok/s (93%) community source
llama.cpp (CUDA): 196.26 tok/s (91%) community source
llama.cpp (CUDA): 190.88 tok/s (89%) community source
llama.cpp (CUDA): 185.91 tok/s (86%) community source
llama.cpp (CUDA): 176.88 tok/s (82%) community source
llama.cpp (CUDA): 168.23 tok/s (78%) community source
llama.cpp (CUDA): 155.72 tok/s (72%) community source
llama.cpp (CUDA): 141.32 tok/s (66%) community source
llama.cpp (CUDA): 128.41 tok/s (60%) community source
llama.cpp (Vulkan): 121.83 tok/s (57%) community source
llama.cpp (Vulkan): 120.53 tok/s (56%) community source
llama.cpp (Vulkan): 108.39 tok/s (50%) community source
llama.cpp (Vulkan): 102.46 tok/s (48%) community source
llama.cpp (CUDA): 65.91 tok/s (31%) community source
llama.cpp (CUDA): 65.48 tok/s (30%) community source

llama 7B on A4000 (Q4_0)

llama.cpp (CUDA): 85.17 tok/s (100%) community source
llama.cpp (CUDA): 83.77 tok/s (98%) community source
llama.cpp (CUDA): 80.39 tok/s (94%) community source
llama.cpp (CUDA): 78.12 tok/s (92%) community source
llama.cpp (Vulkan): 76.07 tok/s (89%) community source
llama.cpp (Vulkan): 73.74 tok/s (87%) community source
llama.cpp (CUDA): 23.76 tok/s (28%) community source
llama.cpp (CUDA): 23.63 tok/s (28%) community source

llama 7B on A5000 (Q4_0)

llama.cpp (Vulkan): 140.43 tok/s (100%) community source
llama.cpp (Vulkan): 139.89 tok/s (100%) community source
llama.cpp (CUDA): 135.83 tok/s (97%) community source
llama.cpp (CUDA): 130.07 tok/s (93%) community source

llama 7B on A770 (Q4_0)

llama.cpp (SYCL): 82.92 tok/s (100%) community source
llama.cpp (SYCL): 81.6 tok/s (98%) community source
llama.cpp (SYCL): 72.54 tok/s (87%) community source
llama.cpp (SYCL): 71.32 tok/s (86%) community source
llama.cpp (SYCL): 70.54 tok/s (85%) community source
llama.cpp (SYCL): 70.18 tok/s (85%) community source
llama.cpp (SYCL): 69.73 tok/s (84%) community source
llama.cpp (Vulkan): 53.07 tok/s (64%) community source
llama.cpp (SYCL): 52.98 tok/s (64%) community source
llama.cpp (SYCL): 52.76 tok/s (64%) community source
llama.cpp (SYCL): 52.62 tok/s (63%) community source
llama.cpp (Vulkan): 52.56 tok/s (63%) community source
llama.cpp (SYCL): 52.26 tok/s (63%) community source
llama.cpp (Vulkan): 52.04 tok/s (63%) community source
llama.cpp (Vulkan): 50.9 tok/s (61%) community source
llama.cpp (Vulkan): 49.66 tok/s (60%) community source
llama.cpp (Vulkan): 49.43 tok/s (60%) community source
llama.cpp (Vulkan): 49.02 tok/s (59%) community source
llama.cpp (Vulkan): 48.45 tok/s (58%) community source
llama.cpp (Vulkan): 48.3 tok/s (58%) community source
llama.cpp (Vulkan): 48.17 tok/s (58%) community source
llama.cpp (Vulkan): 45.22 tok/s (55%) community source
llama.cpp (SYCL): 38.1 tok/s (46%) community source
llama.cpp (SYCL): 16.57 tok/s (20%) community source
llama.cpp (SYCL): 13.23 tok/s (16%) community source
llama.cpp (SYCL): 11.14 tok/s (13%) community source

llama 7B on Arc A380 (Q4_0)

llama.cpp (SYCL): 26.57 tok/s (100%) community source
llama.cpp (SYCL): 25.81 tok/s (97%) community source
llama.cpp (SYCL): 25.63 tok/s (96%) community source
llama.cpp (SYCL): 25.57 tok/s (96%) community source
llama.cpp (SYCL): 24.04 tok/s (90%) community source
llama.cpp (SYCL): 23.97 tok/s (90%) community source
llama.cpp (SYCL): 23.84 tok/s (90%) community source
llama.cpp (SYCL): 23.81 tok/s (90%) community source
llama.cpp (SYCL): 20.26 tok/s (76%) community source
llama.cpp (SYCL): 19.76 tok/s (74%) community source
llama.cpp (SYCL): 19.7 tok/s (74%) community source
llama.cpp (OpenVINO): 19.21 tok/s (72%) community source
llama.cpp (SYCL): 19.12 tok/s (72%) community source
llama.cpp (SYCL): 18.62 tok/s (70%) community source
llama.cpp (SYCL): 18.54 tok/s (70%) community source
llama.cpp (SYCL): 18.19 tok/s (68%) community source
llama.cpp (OpenVINO): 18.18 tok/s (68%) community source
llama.cpp (SYCL): 18.16 tok/s (68%) community source
llama.cpp (Vulkan): 17.87 tok/s (67%) community source
llama.cpp (Vulkan): 17.57 tok/s (66%) community source
llama.cpp (Vulkan): 17.55 tok/s (66%) community source
llama.cpp (Vulkan): 17.25 tok/s (65%) community source
llama.cpp (SYCL): 16.17 tok/s (61%) community source
llama.cpp (SYCL): 15.97 tok/s (60%) community source
llama.cpp (SYCL): 14.96 tok/s (56%) community source
llama.cpp (SYCL): 14.95 tok/s (56%) community source

llama 7B on Arc A750 (Q4_0)

llama.cpp (Vulkan): 43.96 tok/s (100%) community source
llama.cpp (Vulkan): 42.66 tok/s (97%) community source
llama.cpp (Vulkan): 42.15 tok/s (96%) community source
llama.cpp (Vulkan): 41.43 tok/s (94%) community source
llama.cpp (SYCL): 36.64 tok/s (83%) community source
llama.cpp (SYCL): 36.2 tok/s (82%) community source
llama.cpp (Vulkan): 27.57 tok/s (63%) community source
llama.cpp (SYCL): 23.61 tok/s (54%) community source

llama 7B on Arc A770 (Q4_0)

llama.cpp (Vulkan): 75.75 tok/s (100%) community source
llama.cpp (Vulkan): 74.24 tok/s (98%) community source
llama.cpp (ROCm): 71.93 tok/s (95%) community source
llama.cpp (ROCm): 69.27 tok/s (91%) community source
llama.cpp (SYCL): 54.38 tok/s (72%) community source
llama.cpp (Vulkan): 52.48 tok/s (69%) community source
llama.cpp (Vulkan): 51.43 tok/s (68%) community source
llama.cpp (Vulkan): 51.36 tok/s (68%) community source
llama.cpp (Vulkan): 50.47 tok/s (67%) community source
llama.cpp (Vulkan): 50.39 tok/s (67%) community source
llama.cpp (Vulkan): 49.55 tok/s (65%) community source
llama.cpp (Vulkan): 49.34 tok/s (65%) community source
llama.cpp (Vulkan): 48.46 tok/s (64%) community source
llama.cpp (SYCL): 47.81 tok/s (63%) community source
llama.cpp (Vulkan): 44.49 tok/s (59%) community source
llama.cpp (SYCL): 43.18 tok/s (57%) community source
llama.cpp (SYCL): 41.48 tok/s (55%) community source
llama.cpp (Vulkan): 41.19 tok/s (54%) community source
llama.cpp (Vulkan): 40.62 tok/s (54%) community source
llama.cpp (Vulkan): 40.56 tok/s (54%) community source
llama.cpp (Vulkan): 39.41 tok/s (52%) community source
llama.cpp (Vulkan): 39.13 tok/s (52%) community source
llama.cpp (SYCL): 39.09 tok/s (52%) community source
llama.cpp (Vulkan): 38.8 tok/s (51%) community source
llama.cpp (Vulkan): 38.07 tok/s (50%) community source
llama.cpp (SYCL): 34.47 tok/s (46%) community source
llama.cpp (SYCL): 31.45 tok/s (42%) community source
llama.cpp (SYCL): 18.22 tok/s (24%) community source

llama 7B on Arc B580 (Q4_0)

llama.cpp (SYCL): 88.13 tok/s (100%) community source
llama.cpp (SYCL): 86.54 tok/s (98%) community source
llama.cpp (SYCL): 83.67 tok/s (95%) community source
llama.cpp (SYCL): 83.39 tok/s (95%) community source
llama.cpp (SYCL): 82.85 tok/s (94%) community source
llama.cpp (SYCL): 82.49 tok/s (94%) community source
llama.cpp (SYCL): 78.15 tok/s (89%) community source
llama.cpp (SYCL): 77.92 tok/s (88%) community source
llama.cpp (SYCL): 73.76 tok/s (84%) community source
llama.cpp (Vulkan): 72.0 tok/s (82%) community source
llama.cpp (Vulkan): 70.14 tok/s (80%) community source
llama.cpp (SYCL): 65.67 tok/s (75%) community source

llama 7B on Arc Pro B50 (Q4_0)

llama.cpp (SYCL): 47.8 tok/s (100%) community source
llama.cpp (Vulkan): 47.69 tok/s (100%) community source
llama.cpp (Vulkan): 46.3 tok/s (97%) community source
llama.cpp (SYCL): 45.92 tok/s (96%) community source
llama.cpp (SYCL): 45.74 tok/s (96%) community source
llama.cpp (Vulkan): 41.82 tok/s (87%) community source
llama.cpp (Vulkan): 41.82 tok/s (87%) community source
llama.cpp (Vulkan): 40.13 tok/s (84%) community source
llama.cpp (Vulkan): 40.13 tok/s (84%) community source

llama 7B on Arc Pro B60 (Q4_0)

llama.cpp (SYCL): 84.44 tok/s (100%) community source
llama.cpp (SYCL): 84.12 tok/s (100%) community source
llama.cpp (SYCL): 80.03 tok/s (95%) community source
llama.cpp (SYCL): 80.03 tok/s (95%) community source
llama.cpp (SYCL): 77.65 tok/s (92%) community source
llama.cpp (Vulkan): 76.83 tok/s (91%) community source
llama.cpp (Vulkan): 73.01 tok/s (86%) community source
llama.cpp (Vulkan): 71.42 tok/s (85%) community source
llama.cpp (Vulkan): 70.54 tok/s (84%) community source
llama.cpp (Vulkan): 69.46 tok/s (82%) community source
llama.cpp (Vulkan): 68.55 tok/s (81%) community source
llama.cpp (Vulkan): 68.07 tok/s (81%) community source
llama.cpp (Vulkan): 65.94 tok/s (78%) community source
llama.cpp (Vulkan): 53.56 tok/s (63%) community source
llama.cpp (SYCL): 43.93 tok/s (52%) community source

llama 7B on Arc Pro B70 (Q4_0)

llama.cpp (Vulkan): 116.67 tok/s (100%) community source
llama.cpp (Vulkan): 112.43 tok/s (96%) community source
llama.cpp (SYCL): 112.08 tok/s (96%) community source
llama.cpp (SYCL): 111.3 tok/s (95%) community source
llama.cpp (SYCL): 111.13 tok/s (95%) community source
llama.cpp (SYCL): 111.13 tok/s (95%) community source
llama.cpp (SYCL): 109.99 tok/s (94%) community source
llama.cpp (SYCL): 109.72 tok/s (94%) community source
llama.cpp (SYCL): 107.11 tok/s (92%) community source
llama.cpp (SYCL): 107.11 tok/s (92%) community source
llama.cpp (SYCL): 106.99 tok/s (92%) community source
llama.cpp (SYCL): 106.69 tok/s (91%) community source
llama.cpp (SYCL): 105.64 tok/s (91%) community source
llama.cpp (SYCL): 105.45 tok/s (90%) community source
llama.cpp (Vulkan): 82.19 tok/s (70%) community source
llama.cpp (Vulkan): 80.39 tok/s (69%) community source

llama 7B on GB10 (Q4_0)

llama.cpp (Vulkan): 60.39 tok/s (100%) community source
llama.cpp (CUDA): 60.24 tok/s (100%) community source
llama.cpp (CUDA): 59.62 tok/s (99%) community source
llama.cpp (CUDA): 59.06 tok/s (98%) community source
llama.cpp (CUDA): 59.05 tok/s (98%) community source
llama.cpp (Vulkan): 58.51 tok/s (97%) community source
llama.cpp (CUDA): 58.38 tok/s (97%) community source
llama.cpp (CUDA): 58.14 tok/s (96%) community source
llama.cpp (CUDA): 58.13 tok/s (96%) community source
llama.cpp (CUDA): 57.21 tok/s (95%) community source
llama.cpp (CUDA): 56.74 tok/s (94%) community source
llama.cpp (CUDA): 54.91 tok/s (91%) community source
llama.cpp (CUDA): 54.39 tok/s (90%) community source
llama.cpp (Vulkan): 53.64 tok/s (89%) community source
llama.cpp (Vulkan): 53.21 tok/s (88%) community source
llama.cpp (Vulkan): 52.28 tok/s (87%) community source
llama.cpp (Vulkan): 51.7 tok/s (86%) community source
llama.cpp (Vulkan): 50.8 tok/s (84%) community source
llama.cpp (Vulkan): 48.42 tok/s (80%) community source
llama.cpp (Vulkan): 47.55 tok/s (79%) community source
llama.cpp (Vulkan): 46.67 tok/s (77%) community source
llama.cpp (Vulkan): 46.61 tok/s (77%) community source
llama.cpp (CPU): 24.08 tok/s (40%) community source

llama 7B on GeForce RTX 5060 Ti (Q4_0)

llama.cpp (CUDA): 97.32 tok/s (100%) community source
llama.cpp (Vulkan): 95.84 tok/s (98%) community source
llama.cpp (CUDA): 95.69 tok/s (98%) community source
llama.cpp (CUDA): 95.1 tok/s (98%) community source
llama.cpp (CUDA): 93.9 tok/s (96%) community source
llama.cpp (CUDA): 93.46 tok/s (96%) community source
llama.cpp (CUDA): 92.75 tok/s (95%) community source
llama.cpp (CUDA): 92.52 tok/s (95%) community source
llama.cpp (CUDA): 90.94 tok/s (93%) community source
llama.cpp (Vulkan): 90.61 tok/s (93%) community source
llama.cpp (Vulkan): 66.59 tok/s (68%) community source
llama.cpp (Vulkan): 63.79 tok/s (66%) community source
llama.cpp (CUDA): 53.96 tok/s (55%) community source
llama.cpp (CUDA): 53.65 tok/s (55%) community source
llama.cpp (CUDA): 52.85 tok/s (54%) community source
llama.cpp (CUDA): 51.3 tok/s (53%) community source
llama.cpp (Vulkan): 34.71 tok/s (36%) community source
llama.cpp (Vulkan): 34.19 tok/s (35%) community source

llama 7B on M1 (Q4_0)

llama.cpp (Metal): 13.99 tok/s (100%) community source
llama.cpp (Vulkan): 12.85 tok/s (92%) community source
llama.cpp (Vulkan): 12.47 tok/s (89%) community source
llama.cpp (Vulkan): 12.41 tok/s (89%) community source
llama.cpp (Vulkan): 12.38 tok/s (88%) community source

llama 7B on M2 Max (F16)

llama.cpp (CPU): 8.19 tok/s (100%) community source
llama.cpp (Vulkan): 2.02 tok/s (25%) community source

llama 7B on M2 Max (Q4_0)

llama.cpp (Vulkan): 24.07 tok/s (100%) community source
llama.cpp (CPU): 22.86 tok/s (95%) community source

llama 7B on M2 Max (Q8_0)

llama.cpp (Vulkan): 19.79 tok/s (100%) community source
llama.cpp (CPU): 13.79 tok/s (70%) community source

llama 7B on M3 Pro (Q4_0)

llama.cpp (Metal): 31.54 tok/s (100%) community source
llama.cpp (Metal): 31.52 tok/s (100%) community source
llama.cpp (Vulkan): 26.39 tok/s (84%) community source
llama.cpp (Vulkan): 26.23 tok/s (83%) community source

llama 7B on M3 Ultra (Q4_0)

llama.cpp (Vulkan): 115.54 tok/s (100%) community source
llama.cpp (Vulkan): 114.55 tok/s (99%) community source
llama.cpp (Metal): 102.28 tok/s (89%) community source

llama 7B on M4 (Q4_0)

llama.cpp (Vulkan): 170.56 tok/s (100%) community source
llama.cpp (Vulkan): 160.78 tok/s (94%) community source
llama.cpp (Metal): 72.26 tok/s (42%) community source
llama.cpp (Metal): 23.29 tok/s (14%) community source

llama 7B on M4 Max (Q4_0)

llama.cpp (Metal): 96.62 tok/s (100%) community source
llama.cpp (Metal): 95.19 tok/s (99%) community source
llama.cpp (Vulkan): 75.02 tok/s (78%) community source
llama.cpp (Vulkan): 71.79 tok/s (74%) community source

llama 7B on RTX 3060 (Q4_0)

llama.cpp (Vulkan): 80.59 tok/s (100%) community source
llama.cpp (Vulkan): 75.94 tok/s (94%) community source
llama.cpp (Vulkan): 73.71 tok/s (91%) community source
llama.cpp (CUDA): 70.78 tok/s (88%) community source
llama.cpp (CUDA): 68.38 tok/s (85%) community source
llama.cpp (Vulkan): 68.34 tok/s (85%) community source
llama.cpp (Vulkan): 67.11 tok/s (83%) community source
llama.cpp (Vulkan): 66.12 tok/s (82%) community source
llama.cpp (Vulkan): 65.63 tok/s (81%) community source
llama.cpp (Vulkan): 64.76 tok/s (80%) community source
llama.cpp (Vulkan): 54.28 tok/s (67%) community source
llama.cpp (Vulkan): 52.66 tok/s (65%) community source
llama.cpp (Vulkan): 50.23 tok/s (62%) community source
llama.cpp (Vulkan): 49.03 tok/s (61%) community source
llama.cpp (Vulkan): 20.18 tok/s (25%) community source
llama.cpp (Vulkan): 10.81 tok/s (13%) community source
llama.cpp (Vulkan): 9.79 tok/s (12%) community source
llama.cpp (Vulkan): 9.48 tok/s (12%) community source
llama.cpp (Vulkan): 7.69 tok/s (10%) community source
llama.cpp (Vulkan): 7.33 tok/s (9%) community source

llama 7B on RTX 3070 (Q4_0)

llama.cpp (CUDA): 84.62 tok/s (100%) community source
llama.cpp (Vulkan): 78.71 tok/s (93%) community source
llama.cpp (Vulkan): 63.64 tok/s (75%) community source
llama.cpp (Vulkan): 62.92 tok/s (74%) community source

llama 7B on RTX 3080 (Q4_0)

llama.cpp (Vulkan): 143.44 tok/s (100%) community source
llama.cpp (CUDA): 131.01 tok/s (91%) community source
llama.cpp (Vulkan): 62.16 tok/s (43%) community source
llama.cpp (Vulkan): 18.32 tok/s (13%) community source

llama 7B on RTX 3090 (Q4_0)

llama.cpp (CUDA): 163.69 tok/s (100%) community source
llama.cpp (Vulkan): 162.28 tok/s (99%) community source
llama.cpp (Vulkan): 160.13 tok/s (98%) community source
llama.cpp (CUDA): 159.36 tok/s (97%) community source
llama.cpp (Vulkan): 157.82 tok/s (96%) community source
llama.cpp (Vulkan): 156.57 tok/s (96%) community source
llama.cpp (Vulkan): 151.65 tok/s (93%) community source
llama.cpp (Vulkan): 151.65 tok/s (93%) community source
llama.cpp (Vulkan): 143.46 tok/s (88%) community source
llama.cpp (Vulkan): 142.76 tok/s (87%) community source
llama.cpp (Vulkan): 136.81 tok/s (84%) community source
llama.cpp (Vulkan): 133.63 tok/s (82%) community source
llama.cpp (Vulkan): 126.06 tok/s (77%) community source
llama.cpp (Vulkan): 125.6 tok/s (77%) community source
llama.cpp (Vulkan): 123.72 tok/s (76%) community source
llama.cpp (Vulkan): 122.61 tok/s (75%) community source
llama.cpp (Vulkan): 120.44 tok/s (74%) community source
llama.cpp (Vulkan): 118.5 tok/s (72%) community source
llama.cpp (Vulkan): 108.45 tok/s (66%) community source
llama.cpp (Vulkan): 107.32 tok/s (66%) community source
llama.cpp (Vulkan): 106.03 tok/s (65%) community source
llama.cpp (ROCm): 83.55 tok/s (51%) community source
llama.cpp (ROCm): 73.58 tok/s (45%) community source
llama.cpp (Vulkan): 36.82 tok/s (22%) community source
llama.cpp (Vulkan): 34.16 tok/s (21%) community source
llama.cpp (Vulkan): 33.9 tok/s (21%) community source
llama.cpp (Vulkan): 22.8 tok/s (14%) community source
llama.cpp (Vulkan): 22.41 tok/s (14%) community source
llama.cpp (Vulkan): 16.56 tok/s (10%) community source
llama.cpp (Vulkan): 16.05 tok/s (10%) community source
llama.cpp (Vulkan): 10.09 tok/s (6%) community source
llama.cpp (Vulkan): 10.06 tok/s (6%) community source
llama.cpp (Vulkan): 8.55 tok/s (5%) community source
llama.cpp (Vulkan): 5.82 tok/s (4%) community source
llama.cpp (Vulkan): 5.71 tok/s (3%) community source
llama.cpp (Vulkan): 5.61 tok/s (3%) community source
llama.cpp (Vulkan): 5.54 tok/s (3%) community source
llama.cpp (Vulkan): 5.42 tok/s (3%) community source
llama.cpp (Vulkan): 5.35 tok/s (3%) community source
llama.cpp (Vulkan): 2.87 tok/s (2%) community source
llama.cpp (Vulkan): 2.85 tok/s (2%) community source
llama.cpp (Vulkan): 2.67 tok/s (2%) community source
llama.cpp (Vulkan): 2.63 tok/s (2%) community source

llama 7B on RTX 4050 (Q4_0)

llama.cpp (CUDA): 43.72 tok/s (100%) community source
llama.cpp (Vulkan): 41.89 tok/s (96%) community source

llama 7B on RTX 4090 (Q4_0)

llama.cpp (Vulkan): 190.1 tok/s (100%) community source
llama.cpp (CUDA): 189.07 tok/s (99%) community source
llama.cpp (Vulkan): 187.97 tok/s (99%) community source
llama.cpp (CUDA): 185.04 tok/s (97%) community source
llama.cpp (Vulkan): 179.62 tok/s (94%) community source
llama.cpp (Vulkan): 170.95 tok/s (90%) community source
llama.cpp (Vulkan): 166.45 tok/s (88%) community source

llama 7B on RTX 5060 Ti (Q4_0)

llama.cpp (Vulkan): 104.42 tok/s (100%) community source
llama.cpp (Vulkan): 102.66 tok/s (98%) community source
llama.cpp (Vulkan): 102.31 tok/s (98%) community source
llama.cpp (Vulkan): 101.35 tok/s (97%) community source
llama.cpp (Vulkan): 95.98 tok/s (92%) community source
llama.cpp (CUDA): 94.3 tok/s (90%) community source
llama.cpp (Vulkan): 93.76 tok/s (90%) community source
llama.cpp (CUDA): 93.3 tok/s (89%) community source
llama.cpp (Vulkan): 92.19 tok/s (88%) community source
llama.cpp (Vulkan): 91.85 tok/s (88%) community source
llama.cpp (CUDA): 91.56 tok/s (88%) community source
llama.cpp (Vulkan): 91.41 tok/s (88%) community source
llama.cpp (CUDA): 90.95 tok/s (87%) community source
llama.cpp (CUDA): 89.87 tok/s (86%) community source
llama.cpp (CUDA): 89.75 tok/s (86%) community source
llama.cpp (CUDA): 89.57 tok/s (86%) community source
llama.cpp (Vulkan): 87.05 tok/s (83%) community source
llama.cpp (Vulkan): 83.26 tok/s (80%) community source
llama.cpp (Vulkan): 81.48 tok/s (78%) community source
llama.cpp (Vulkan): 78.1 tok/s (75%) community source
llama.cpp (Vulkan): 74.8 tok/s (72%) community source
llama.cpp (Vulkan): 74.52 tok/s (71%) community source
llama.cpp (Vulkan): 74.47 tok/s (71%) community source
llama.cpp (Vulkan): 62.05 tok/s (59%) community source
llama.cpp (Vulkan): 60.02 tok/s (57%) community source
llama.cpp (Vulkan): 49.44 tok/s (47%) community source
llama.cpp (Vulkan): 47.35 tok/s (45%) community source
llama.cpp (Vulkan): 43.79 tok/s (42%) community source
llama.cpp (Vulkan): 42.47 tok/s (41%) community source

llama 7B on RTX 5070 Ti (Q4_0)

llama.cpp (CUDA): 168.2 tok/s (100%) community source
llama.cpp (CUDA): 153.55 tok/s (91%) community source
llama.cpp (CUDA): 149.28 tok/s (89%) community source
llama.cpp (Vulkan): 135.63 tok/s (81%) community source
llama.cpp (Vulkan): 133.94 tok/s (80%) community source
llama.cpp (Vulkan): 126.67 tok/s (75%) community source
llama.cpp (Vulkan): 125.89 tok/s (75%) community source

llama 7B on RX 6800 (Q4_0)

llama.cpp (Vulkan): 106.49 tok/s (100%) community source
llama.cpp (Vulkan): 101.47 tok/s (95%) community source
llama.cpp (ROCm): 93.6 tok/s (88%) community source
llama.cpp (ROCm): 86.03 tok/s (81%) community source

llama 7B on Radeon 660M (Q4_0)

llama.cpp (Vulkan): 16.64 tok/s (100%) community source
llama.cpp (Vulkan): 16.56 tok/s (100%) community source
llama.cpp (Vulkan): 16.2 tok/s (97%) community source
llama.cpp (Vulkan): 16.11 tok/s (97%) community source
llama.cpp (ROCm): 12.72 tok/s (76%) community source

llama 7B on Radeon 8060S (Q4_0)

llama.cpp (Vulkan): 176.18 tok/s (100%) community source
llama.cpp (Vulkan): 171.61 tok/s (97%) community source
llama.cpp (Vulkan): 164.05 tok/s (93%) community source
llama.cpp (ROCm): 97.98 tok/s (56%) community source
llama.cpp (ROCm): 93.84 tok/s (53%) community source
llama.cpp (ROCm): 68.67 tok/s (39%) community source
llama.cpp (ROCm): 66.27 tok/s (38%) community source
llama.cpp (Vulkan): 55.32 tok/s (31%) community source
llama.cpp (Vulkan): 54.76 tok/s (31%) community source
llama.cpp (Vulkan): 54.38 tok/s (31%) community source
llama.cpp (Vulkan): 53.59 tok/s (30%) community source
llama.cpp (Vulkan): 53.37 tok/s (30%) community source
llama.cpp (Vulkan): 53.21 tok/s (30%) community source
llama.cpp (Vulkan): 53.0 tok/s (30%) community source
llama.cpp (Vulkan): 52.69 tok/s (30%) community source
llama.cpp (Vulkan): 52.5 tok/s (30%) community source
llama.cpp (Vulkan): 52.28 tok/s (30%) community source
llama.cpp (Vulkan): 52.23 tok/s (30%) community source
llama.cpp (Vulkan): 52.03 tok/s (30%) community source
llama.cpp (Vulkan): 52.01 tok/s (30%) community source
llama.cpp (Vulkan): 51.89 tok/s (29%) community source
llama.cpp (Vulkan): 51.81 tok/s (29%) community source
llama.cpp (Vulkan): 51.71 tok/s (29%) community source
llama.cpp (Vulkan): 51.68 tok/s (29%) community source
llama.cpp (Vulkan): 51.67 tok/s (29%) community source
llama.cpp (Vulkan): 51.62 tok/s (29%) community source
llama.cpp (ROCm): 51.33 tok/s (29%) community source
llama.cpp (Vulkan): 51.14 tok/s (29%) community source
llama.cpp (ROCm): 50.75 tok/s (29%) community source
llama.cpp (ROCm): 50.69 tok/s (29%) community source
llama.cpp (ROCm): 50.59 tok/s (29%) community source
llama.cpp (ROCm): 50.57 tok/s (29%) community source
llama.cpp (ROCm): 50.4 tok/s (29%) community source
llama.cpp (ROCm): 50.3 tok/s (29%) community source
llama.cpp (ROCm): 50.29 tok/s (29%) community source
llama.cpp (ROCm): 49.5 tok/s (28%) community source
llama.cpp (ROCm): 49.1 tok/s (28%) community source
llama.cpp (ROCm): 48.97 tok/s (28%) community source
llama.cpp (ROCm): 47.97 tok/s (27%) community source
llama.cpp (ROCm): 47.36 tok/s (27%) community source
llama.cpp (Vulkan): 46.69 tok/s (27%) community source
llama.cpp (ROCm): 46.68 tok/s (26%) community source
llama.cpp (ROCm): 46.65 tok/s (26%) community source
llama.cpp (ROCm): 46.51 tok/s (26%) community source
llama.cpp (ROCm): 46.46 tok/s (26%) community source
llama.cpp (ROCm): 46.15 tok/s (26%) community source
llama.cpp (Vulkan): 46.09 tok/s (26%) community source
llama.cpp (ROCm): 46.01 tok/s (26%) community source
llama.cpp (ROCm): 45.53 tok/s (26%) community source
llama.cpp (ROCm): 44.15 tok/s (25%) community source
llama.cpp (ROCm): 43.79 tok/s (25%) community source
llama.cpp (CPU): 29.28 tok/s (17%) community source
llama.cpp (CPU): 27.58 tok/s (16%) community source

llama 7B on Radeon 840M (Q4_0)

llama.cpp (Vulkan): 19.03 tok/s (100%) community source
llama.cpp (Vulkan): 17.66 tok/s (93%) community source
llama.cpp (ROCm): 16.12 tok/s (85%) community source
llama.cpp (ROCm): 14.33 tok/s (75%) community source
llama.cpp (CPU): 13.18 tok/s (69%) community source
llama.cpp (CPU): 12.95 tok/s (68%) community source

llama 7B on Radeon 890M (Q4_0)

llama.cpp (Vulkan): 23.1 tok/s (100%) community source
llama.cpp (Vulkan): 23.06 tok/s (100%) community source
llama.cpp (ROCm): 22.37 tok/s (97%) community source
llama.cpp (Vulkan): 22.03 tok/s (95%) community source
llama.cpp (Vulkan): 21.99 tok/s (95%) community source
llama.cpp (ROCm): 21.18 tok/s (92%) community source
llama.cpp (Vulkan): 20.0 tok/s (87%) community source
llama.cpp (Vulkan): 19.93 tok/s (86%) community source
llama.cpp (Vulkan): 19.81 tok/s (86%) community source
llama.cpp (Vulkan): 19.78 tok/s (86%) community source
llama.cpp (Vulkan): 19.39 tok/s (84%) community source
llama.cpp (Vulkan): 19.29 tok/s (84%) community source
llama.cpp (Vulkan): 19.16 tok/s (83%) community source
llama.cpp (ROCm): 19.08 tok/s (83%) community source
llama.cpp (ROCm): 19.06 tok/s (83%) community source
llama.cpp (Vulkan): 18.76 tok/s (81%) community source
llama.cpp (ROCm): 18.24 tok/s (79%) community source
llama.cpp (ROCm): 18.02 tok/s (78%) community source
llama.cpp (ROCm): 16.73 tok/s (72%) community source
llama.cpp (ROCm): 16.16 tok/s (70%) community source
llama.cpp (ROCm): 16.12 tok/s (70%) community source
llama.cpp (Vulkan): 15.11 tok/s (65%) community source
llama.cpp (Vulkan): 14.94 tok/s (65%) community source
llama.cpp (ROCm): 12.97 tok/s (56%) community source
llama.cpp (Vulkan): 8.31 tok/s (36%) community source
llama.cpp (Vulkan): 8.3 tok/s (36%) community source

llama 7B on Radeon AI PRO R9700 (Q4_0)

llama.cpp (Vulkan): 131.19 tok/s (100%) community source
llama.cpp (Vulkan): 130.97 tok/s (100%) community source
llama.cpp (Vulkan): 125.96 tok/s (96%) community source
llama.cpp (ROCm): 121.62 tok/s (93%) community source
llama.cpp (ROCm): 121.29 tok/s (92%) community source
llama.cpp (ROCm): 112.72 tok/s (86%) community source

llama 7B on Radeon RX 5700 XT (Q4_0)

llama.cpp (Vulkan): 74.43 tok/s (100%) community source
llama.cpp (Vulkan): 71.66 tok/s (96%) community source
llama.cpp (Vulkan): 71.58 tok/s (96%) community source
llama.cpp (Vulkan): 70.73 tok/s (95%) community source
llama.cpp (Vulkan): 70.13 tok/s (94%) community source
llama.cpp (Vulkan): 69.14 tok/s (93%) community source
llama.cpp (ROCm): 67.55 tok/s (91%) community source
llama.cpp (Vulkan): 67.53 tok/s (91%) community source
llama.cpp (ROCm): 62.02 tok/s (83%) community source
llama.cpp (Vulkan): 23.21 tok/s (31%) community source

llama 7B on Radeon RX 6600 (Q4_0)

llama.cpp (Vulkan): 53.24 tok/s (100%) community source
llama.cpp (Vulkan): 53.17 tok/s (100%) community source
llama.cpp (Vulkan): 52.47 tok/s (99%) community source
llama.cpp (Vulkan): 50.63 tok/s (95%) community source
llama.cpp (Vulkan): 50.57 tok/s (95%) community source
llama.cpp (Vulkan): 50.11 tok/s (94%) community source
llama.cpp (Vulkan): 49.76 tok/s (93%) community source
llama.cpp (Vulkan): 48.52 tok/s (91%) community source
llama.cpp (Vulkan): 48.31 tok/s (91%) community source
llama.cpp (Vulkan): 47.49 tok/s (89%) community source
llama.cpp (Vulkan): 47.47 tok/s (89%) community source
llama.cpp (Vulkan): 34.34 tok/s (65%) community source
llama.cpp (Vulkan): 25.61 tok/s (48%) community source
llama.cpp (Vulkan): 12.16 tok/s (23%) community source
llama.cpp (CPU): 12.08 tok/s (23%) community source

llama 7B on Radeon RX 6800 (Q4_0)

llama.cpp (Vulkan): 98.11 tok/s (100%) community source
llama.cpp (Vulkan): 97.3 tok/s (99%) community source
llama.cpp (Vulkan): 96.65 tok/s (99%) community source
llama.cpp (Vulkan): 95.61 tok/s (97%) community source
llama.cpp (ROCm): 86.91 tok/s (89%) community source
llama.cpp (ROCm): 83.92 tok/s (86%) community source
llama.cpp (ROCm): 79.78 tok/s (81%) community source
llama.cpp (Vulkan): 9.87 tok/s (10%) community source
llama.cpp (Vulkan): 9.63 tok/s (10%) community source

llama 7B on Radeon RX 6800 XT (Q4_0)

llama.cpp (Vulkan): 100.5 tok/s (100%) community source
llama.cpp (Vulkan): 100.32 tok/s (100%) community source
llama.cpp (Vulkan): 95.56 tok/s (95%) community source
llama.cpp (Vulkan): 94.81 tok/s (94%) community source
llama.cpp (Vulkan): 91.59 tok/s (91%) community source
llama.cpp (ROCm): 88.31 tok/s (88%) community source
llama.cpp (Vulkan): 87.56 tok/s (87%) community source
llama.cpp (Vulkan): 86.47 tok/s (86%) community source
llama.cpp (Vulkan): 86.17 tok/s (86%) community source
llama.cpp (ROCm): 84.88 tok/s (84%) community source
llama.cpp (Vulkan): 79.6 tok/s (79%) community source
llama.cpp (Vulkan): 79.35 tok/s (79%) community source
llama.cpp (Vulkan): 78.3 tok/s (78%) community source
llama.cpp (Vulkan): 71.37 tok/s (71%) community source
llama.cpp (Vulkan): 70.17 tok/s (70%) community source
llama.cpp (Vulkan): 59.36 tok/s (59%) community source
llama.cpp (Vulkan): 59.22 tok/s (59%) community source

llama 7B on Radeon RX 6900 XT (Q4_0)

llama.cpp (Vulkan): 108.0 tok/s (100%) community source
llama.cpp (Vulkan): 108.0 tok/s (100%) community source
llama.cpp (Vulkan): 106.15 tok/s (98%) community source
llama.cpp (Vulkan): 106.15 tok/s (98%) community source
llama.cpp (Vulkan): 101.42 tok/s (94%) community source
llama.cpp (Vulkan): 98.89 tok/s (92%) community source
llama.cpp (Metal): 98.27 tok/s (91%) community source
llama.cpp (Vulkan): 95.7 tok/s (89%) community source
llama.cpp (ROCm): 92.96 tok/s (86%) community source
llama.cpp (ROCm): 88.49 tok/s (82%) community source
llama.cpp (ROCm): 85.6 tok/s (79%) community source
llama.cpp (ROCm): 85.6 tok/s (79%) community source
llama.cpp (ROCm): 85.48 tok/s (79%) community source
llama.cpp (ROCm): 85.42 tok/s (79%) community source
llama.cpp (ROCm): 85.04 tok/s (79%) community source
llama.cpp (Vulkan): 83.54 tok/s (77%) community source
llama.cpp (ROCm): 83.02 tok/s (77%) community source
llama.cpp (Vulkan): 82.26 tok/s (76%) community source
llama.cpp (ROCm): 80.45 tok/s (74%) community source
llama.cpp (ROCm): 77.84 tok/s (72%) community source
llama.cpp (ROCm): 75.24 tok/s (70%) community source
llama.cpp (Vulkan): 72.39 tok/s (67%) community source
llama.cpp (Vulkan): 72.31 tok/s (67%) community source
llama.cpp (Vulkan): 71.96 tok/s (67%) community source
llama.cpp (Vulkan): 66.81 tok/s (62%) community source
llama.cpp (Vulkan): 37.59 tok/s (35%) community source
llama.cpp (Vulkan): 37.33 tok/s (35%) community source
llama.cpp (Vulkan): 36.72 tok/s (34%) community source

llama 7B on Radeon RX 7600 (Q4_0)

llama.cpp (Vulkan): 60.88 tok/s (100%) community source
llama.cpp (ROCm): 58.73 tok/s (96%) community source
llama.cpp (Vulkan): 58.03 tok/s (95%) community source
llama.cpp (ROCm): 58.0 tok/s (95%) community source
llama.cpp (ROCm): 53.48 tok/s (88%) community source
llama.cpp (ROCm): 53.27 tok/s (88%) community source
llama.cpp (CPU): 9.72 tok/s (16%) community source
llama.cpp (CPU): 9.72 tok/s (16%) community source

llama 7B on Radeon RX 7600 XT (Q4_0)

llama.cpp (Vulkan): 67.98 tok/s (100%) community source
llama.cpp (Vulkan): 65.36 tok/s (96%) community source
llama.cpp (Vulkan): 59.02 tok/s (87%) community source
llama.cpp (Vulkan): 58.44 tok/s (86%) community source
llama.cpp (Vulkan): 56.11 tok/s (83%) community source
llama.cpp (Vulkan): 55.2 tok/s (81%) community source
llama.cpp (Vulkan): 53.7 tok/s (79%) community source
llama.cpp (Vulkan): 53.02 tok/s (78%) community source
llama.cpp (Vulkan): 52.84 tok/s (78%) community source
llama.cpp (Vulkan): 52.8 tok/s (78%) community source
llama.cpp (Vulkan): 52.69 tok/s (78%) community source
llama.cpp (Vulkan): 51.42 tok/s (76%) community source
llama.cpp (Vulkan): 50.63 tok/s (74%) community source
llama.cpp (ROCm): 48.58 tok/s (71%) community source
llama.cpp (Vulkan): 48.31 tok/s (71%) community source
llama.cpp (Vulkan): 47.94 tok/s (71%) community source
llama.cpp (ROCm): 47.65 tok/s (70%) community source
llama.cpp (Vulkan): 46.98 tok/s (69%) community source
llama.cpp (Vulkan): 37.94 tok/s (56%) community source
llama.cpp (Vulkan): 16.06 tok/s (24%) community source
llama.cpp (Vulkan): 8.17 tok/s (12%) community source

llama 7B on Radeon RX 7800 XT (Q4_0)

llama.cpp (Vulkan): 124.86 tok/s (100%) community source
llama.cpp (Vulkan): 123.89 tok/s (99%) community source
llama.cpp (Vulkan): 122.52 tok/s (98%) community source
llama.cpp (Vulkan): 121.58 tok/s (97%) community source
llama.cpp (Vulkan): 121.4 tok/s (97%) community source
llama.cpp (Vulkan): 121.38 tok/s (97%) community source
llama.cpp (Vulkan): 120.25 tok/s (96%) community source
llama.cpp (Vulkan): 120.12 tok/s (96%) community source
llama.cpp (Vulkan): 119.89 tok/s (96%) community source
llama.cpp (Vulkan): 118.27 tok/s (95%) community source
llama.cpp (Vulkan): 116.97 tok/s (94%) community source
llama.cpp (Vulkan): 116.61 tok/s (93%) community source
llama.cpp (Vulkan): 115.71 tok/s (93%) community source
llama.cpp (Vulkan): 115.55 tok/s (93%) community source
llama.cpp (Vulkan): 114.18 tok/s (91%) community source
llama.cpp (Vulkan): 114.12 tok/s (91%) community source
llama.cpp (Vulkan): 113.29 tok/s (91%) community source
llama.cpp (Vulkan): 112.01 tok/s (90%) community source
llama.cpp (Vulkan): 111.97 tok/s (90%) community source
llama.cpp (Vulkan): 111.97 tok/s (90%) community source
llama.cpp (Vulkan): 107.53 tok/s (86%) community source
llama.cpp (Vulkan): 103.98 tok/s (83%) community source
llama.cpp (ROCm): 100.94 tok/s (81%) community source
llama.cpp (ROCm): 100.94 tok/s (81%) community source
llama.cpp (ROCm): 99.55 tok/s (80%) community source
llama.cpp (Vulkan): 98.15 tok/s (79%) community source
llama.cpp (Vulkan): 98.13 tok/s (79%) community source
llama.cpp (Vulkan): 97.07 tok/s (78%) community source
llama.cpp (Vulkan): 96.89 tok/s (78%) community source
llama.cpp (Vulkan): 96.89 tok/s (78%) community source
llama.cpp (ROCm): 96.8 tok/s (78%) community source
llama.cpp (Vulkan): 96.61 tok/s (77%) community source
llama.cpp (Vulkan): 96.03 tok/s (77%) community source
llama.cpp (Vulkan): 96.03 tok/s (77%) community source
llama.cpp (ROCm): 95.99 tok/s (77%) community source
llama.cpp (ROCm): 95.99 tok/s (77%) community source
llama.cpp (Vulkan): 95.2 tok/s (76%) community source
llama.cpp (Vulkan): 95.17 tok/s (76%) community source

llama 7B on Radeon RX 7900 GRE (Q4_0)

llama.cpp (Vulkan): 116.7 tok/s (100%) community source
llama.cpp (Vulkan): 116.11 tok/s (99%) community source
llama.cpp (Vulkan): 112.59 tok/s (96%) community source
llama.cpp (Vulkan): 111.32 tok/s (95%) community source
llama.cpp (Vulkan): 111.31 tok/s (95%) community source
llama.cpp (Vulkan): 110.91 tok/s (95%) community source
llama.cpp (Vulkan): 110.8 tok/s (95%) community source
llama.cpp (Vulkan): 110.73 tok/s (95%) community source
llama.cpp (Vulkan): 110.5 tok/s (95%) community source
llama.cpp (Vulkan): 105.91 tok/s (91%) community source
llama.cpp (Vulkan): 104.75 tok/s (90%) community source
llama.cpp (ROCm): 102.38 tok/s (88%) community source
llama.cpp (Vulkan): 101.73 tok/s (87%) community source
llama.cpp (ROCm): 97.53 tok/s (84%) community source
llama.cpp (ROCm): 96.53 tok/s (83%) community source
llama.cpp (ROCm): 96.07 tok/s (82%) community source

llama 7B on Radeon RX 7900 XT (Q4_0)

llama.cpp (Vulkan): 137.78 tok/s (100%) community source
llama.cpp (Vulkan): 135.65 tok/s (98%) community source
llama.cpp (Vulkan): 123.18 tok/s (89%) community source
llama.cpp (ROCm): 122.86 tok/s (89%) community source
llama.cpp (Vulkan): 120.62 tok/s (88%) community source
llama.cpp (Vulkan): 117.65 tok/s (85%) community source
llama.cpp (Vulkan): 117.57 tok/s (85%) community source
llama.cpp (Vulkan): 117.45 tok/s (85%) community source
llama.cpp (ROCm): 117.11 tok/s (85%) community source
llama.cpp (Vulkan): 116.94 tok/s (85%) community source
llama.cpp (ROCm): 116.15 tok/s (84%) community source
llama.cpp (ROCm): 116.0 tok/s (84%) community source
llama.cpp (ROCm): 112.75 tok/s (82%) community source
llama.cpp (ROCm): 112.3 tok/s (82%) community source
llama.cpp (Vulkan): 77.0 tok/s (56%) community source
llama.cpp (Vulkan): 71.52 tok/s (52%) community source
llama.cpp (Vulkan): 60.18 tok/s (44%) community source
llama.cpp (Vulkan): 55.41 tok/s (40%) community source

llama 7B on Radeon RX 7900 XTX (Q4_0)

llama.cpp (Vulkan): 310.91 tok/s (100%) community source
llama.cpp (Vulkan): 308.88 tok/s (99%) community source
llama.cpp (Vulkan): 298.5 tok/s (96%) community source
llama.cpp (Vulkan): 278.22 tok/s (89%) community source
llama.cpp (Vulkan): 195.3 tok/s (63%) community source
llama.cpp (Vulkan): 191.28 tok/s (62%) community source
llama.cpp (Vulkan): 190.92 tok/s (61%) community source
llama.cpp (Vulkan): 182.63 tok/s (59%) community source
llama.cpp (Vulkan): 179.66 tok/s (58%) community source
llama.cpp (Vulkan): 173.81 tok/s (56%) community source
llama.cpp (Vulkan): 173.73 tok/s (56%) community source
llama.cpp (Vulkan): 173.36 tok/s (56%) community source
llama.cpp (Vulkan): 173.13 tok/s (56%) community source
llama.cpp (Vulkan): 172.86 tok/s (56%) community source
llama.cpp (Vulkan): 172.84 tok/s (56%) community source
llama.cpp (Vulkan): 172.79 tok/s (56%) community source
llama.cpp (Vulkan): 172.76 tok/s (56%) community source
llama.cpp (Vulkan): 172.71 tok/s (56%) community source
llama.cpp (Vulkan): 170.69 tok/s (55%) community source
llama.cpp (Vulkan): 170.44 tok/s (55%) community source
llama.cpp (Vulkan): 170.13 tok/s (55%) community source
llama.cpp (ROCm): 170.12 tok/s (55%) community source
llama.cpp (Vulkan): 169.38 tok/s (54%) community source
llama.cpp (Vulkan): 169.06 tok/s (54%) community source
llama.cpp (Vulkan): 168.77 tok/s (54%) community source
llama.cpp (Vulkan): 168.74 tok/s (54%) community source
llama.cpp (Vulkan): 168.63 tok/s (54%) community source
llama.cpp (Vulkan): 168.62 tok/s (54%) community source
llama.cpp (Vulkan): 168.53 tok/s (54%) community source
llama.cpp (Vulkan): 168.53 tok/s (54%) community source
llama.cpp (Vulkan): 168.04 tok/s (54%) community source
llama.cpp (Vulkan): 167.79 tok/s (54%) community source
llama.cpp (Vulkan): 167.78 tok/s (54%) community source
llama.cpp (Vulkan): 167.42 tok/s (54%) community source
llama.cpp (Vulkan): 167.31 tok/s (54%) community source
llama.cpp (Vulkan): 167.31 tok/s (54%) community source
llama.cpp (ROCm): 167.11 tok/s (54%) community source
llama.cpp (Vulkan): 166.25 tok/s (53%) community source
llama.cpp (Vulkan): 165.94 tok/s (53%) community source
llama.cpp (Vulkan): 165.57 tok/s (53%) community source
llama.cpp (Vulkan): 165.38 tok/s (53%) community source
llama.cpp (Vulkan): 165.38 tok/s (53%) community source
llama.cpp (Vulkan): 165.23 tok/s (53%) community source
llama.cpp (Vulkan): 165.08 tok/s (53%) community source
llama.cpp (Vulkan): 164.84 tok/s (53%) community source
llama.cpp (Vulkan): 164.32 tok/s (53%) community source
llama.cpp (Vulkan): 164.28 tok/s (53%) community source
llama.cpp (Vulkan): 164.22 tok/s (53%) community source
llama.cpp (Vulkan): 164.21 tok/s (53%) community source
llama.cpp (ROCm): 163.77 tok/s (53%) community source
llama.cpp (Vulkan): 163.0 tok/s (52%) community source
llama.cpp (Vulkan): 162.78 tok/s (52%) community source
llama.cpp (Vulkan): 162.78 tok/s (52%) community source
llama.cpp (Vulkan): 162.16 tok/s (52%) community source
llama.cpp (Vulkan): 162.02 tok/s (52%) community source
llama.cpp (Vulkan): 162.02 tok/s (52%) community source
llama.cpp (Vulkan): 161.63 tok/s (52%) community source
llama.cpp (Vulkan): 161.53 tok/s (52%) community source
llama.cpp (Vulkan): 161.48 tok/s (52%) community source
llama.cpp (Vulkan): 161.46 tok/s (52%) community source
llama.cpp (Vulkan): 161.39 tok/s (52%) community source
llama.cpp (Vulkan): 161.07 tok/s (52%) community source
llama.cpp (Vulkan): 160.78 tok/s (52%) community source
llama.cpp (Vulkan): 160.77 tok/s (52%) community source
llama.cpp (Vulkan): 160.77 tok/s (52%) community source
llama.cpp (Vulkan): 160.2 tok/s (52%) community source
llama.cpp (Vulkan): 159.86 tok/s (51%) community source
llama.cpp (Vulkan): 159.78 tok/s (51%) community source
llama.cpp (Vulkan): 159.46 tok/s (51%) community source
llama.cpp (ROCm): 157.91 tok/s (51%) community source
llama.cpp (Vulkan): 157.41 tok/s (51%) community source
llama.cpp (ROCm): 155.69 tok/s (50%) community source
llama.cpp (Vulkan): 154.32 tok/s (50%) community source
llama.cpp (ROCm): 153.91 tok/s (50%) community source
llama.cpp (ROCm): 153.88 tok/s (49%) community source
llama.cpp (ROCm): 153.72 tok/s (49%) community source
llama.cpp (Vulkan): 150.97 tok/s (49%) community source
llama.cpp (ROCm): 150.86 tok/s (49%) community source
llama.cpp (ROCm): 150.4 tok/s (48%) community source
llama.cpp (Vulkan): 150.34 tok/s (48%) community source
llama.cpp (Vulkan): 149.04 tok/s (48%) community source
llama.cpp (Vulkan): 148.34 tok/s (48%) community source
llama.cpp (Vulkan): 148.21 tok/s (48%) community source
llama.cpp (Vulkan): 148.01 tok/s (48%) community source
llama.cpp (Vulkan): 147.25 tok/s (47%) community source
llama.cpp (ROCm): 147.06 tok/s (47%) community source
llama.cpp (Vulkan): 146.66 tok/s (47%) community source
llama.cpp (Vulkan): 145.78 tok/s (47%) community source
llama.cpp (Vulkan): 145.7 tok/s (47%) community source
llama.cpp (Vulkan): 145.35 tok/s (47%) community source
llama.cpp (ROCm): 145.32 tok/s (47%) community source
llama.cpp (ROCm): 145.23 tok/s (47%) community source
llama.cpp (Vulkan): 145.0 tok/s (47%) community source
llama.cpp (Vulkan): 144.9 tok/s (47%) community source
llama.cpp (ROCm): 144.84 tok/s (47%) community source
llama.cpp (Vulkan): 143.99 tok/s (46%) community source
llama.cpp (ROCm): 143.98 tok/s (46%) community source
llama.cpp (Vulkan): 143.75 tok/s (46%) community source
llama.cpp (ROCm): 142.17 tok/s (46%) community source
llama.cpp (Vulkan): 141.52 tok/s (46%) community source
llama.cpp (ROCm): 140.96 tok/s (45%) community source
llama.cpp (ROCm): 140.94 tok/s (45%) community source
llama.cpp (ROCm): 140.94 tok/s (45%) community source
llama.cpp (ROCm): 140.83 tok/s (45%) community source
llama.cpp (Vulkan): 140.76 tok/s (45%) community source
llama.cpp (Vulkan): 140.68 tok/s (45%) community source
llama.cpp (ROCm): 140.27 tok/s (45%) community source
llama.cpp (ROCm): 140.27 tok/s (45%) community source
llama.cpp (ROCm): 139.89 tok/s (45%) community source
llama.cpp (Vulkan): 139.81 tok/s (45%) community source
llama.cpp (ROCm): 139.68 tok/s (45%) community source
llama.cpp (ROCm): 138.91 tok/s (45%) community source
llama.cpp (Vulkan): 136.98 tok/s (44%) community source
llama.cpp (Vulkan): 136.24 tok/s (44%) community source
llama.cpp (ROCm): 136.05 tok/s (44%) community source
llama.cpp (Vulkan): 134.87 tok/s (43%) community source
llama.cpp (ROCm): 134.46 tok/s (43%) community source
llama.cpp (Vulkan): 134.46 tok/s (43%) community source
llama.cpp (ROCm): 134.07 tok/s (43%) community source
llama.cpp (ROCm): 134.07 tok/s (43%) community source
llama.cpp (ROCm): 134.01 tok/s (43%) community source
llama.cpp (ROCm): 133.59 tok/s (43%) community source
llama.cpp (ROCm): 133.59 tok/s (43%) community source
llama.cpp (Vulkan): 133.42 tok/s (43%) community source
llama.cpp (Vulkan): 133.02 tok/s (43%) community source
llama.cpp (ROCm): 132.78 tok/s (43%) community source
llama.cpp (Vulkan): 132.75 tok/s (43%) community source
llama.cpp (ROCm): 132.75 tok/s (43%) community source
llama.cpp (ROCm): 132.17 tok/s (43%) community source
llama.cpp (Vulkan): 131.95 tok/s (42%) community source
llama.cpp (ROCm): 131.7 tok/s (42%) community source
llama.cpp (Vulkan): 130.9 tok/s (42%) community source
llama.cpp (Vulkan): 129.8 tok/s (42%) community source
llama.cpp (Vulkan): 129.69 tok/s (42%) community source
llama.cpp (Vulkan): 129.48 tok/s (42%) community source
llama.cpp (ROCm): 129.33 tok/s (42%) community source
llama.cpp (Vulkan): 129.3 tok/s (42%) community source
llama.cpp (Vulkan): 129.22 tok/s (42%) community source
llama.cpp (Vulkan): 129.2 tok/s (42%) community source
llama.cpp (Vulkan): 129.15 tok/s (42%) community source
llama.cpp (Vulkan): 129.07 tok/s (42%) community source
llama.cpp (Vulkan): 129.07 tok/s (42%) community source
llama.cpp (Vulkan): 128.95 tok/s (41%) community source
llama.cpp (Vulkan): 128.94 tok/s (41%) community source
llama.cpp (Vulkan): 128.79 tok/s (41%) community source
llama.cpp (ROCm): 128.25 tok/s (41%) community source
llama.cpp (ROCm): 127.43 tok/s (41%) community source
llama.cpp (Vulkan): 127.32 tok/s (41%) community source
llama.cpp (ROCm): 126.28 tok/s (41%) community source
llama.cpp (Vulkan): 126.06 tok/s (41%) community source
llama.cpp (Vulkan): 126.06 tok/s (41%) community source
llama.cpp (ROCm): 125.84 tok/s (40%) community source
llama.cpp (ROCm): 124.61 tok/s (40%) community source
llama.cpp (Vulkan): 123.7 tok/s (40%) community source
llama.cpp (ROCm): 123.69 tok/s (40%) community source
llama.cpp (Vulkan): 122.9 tok/s (40%) community source
llama.cpp (ROCm): 122.71 tok/s (39%) community source
llama.cpp (ROCm): 122.56 tok/s (39%) community source
llama.cpp (ROCm): 122.06 tok/s (39%) community source
llama.cpp (Vulkan): 122.02 tok/s (39%) community source
llama.cpp (ROCm): 121.87 tok/s (39%) community source
llama.cpp (Vulkan): 121.62 tok/s (39%) community source
llama.cpp (Vulkan): 121.61 tok/s (39%) community source
llama.cpp (Vulkan): 121.57 tok/s (39%) community source
llama.cpp (Vulkan): 121.56 tok/s (39%) community source
llama.cpp (Vulkan): 121.55 tok/s (39%) community source
llama.cpp (Vulkan): 121.5 tok/s (39%) community source
llama.cpp (Vulkan): 121.33 tok/s (39%) community source
llama.cpp (ROCm): 121.14 tok/s (39%) community source
llama.cpp (ROCm): 120.88 tok/s (39%) community source
llama.cpp (ROCm): 120.73 tok/s (39%) community source
llama.cpp (Vulkan): 119.98 tok/s (39%) community source
llama.cpp (Vulkan): 119.98 tok/s (39%) community source
llama.cpp (ROCm): 119.97 tok/s (39%) community source
llama.cpp (Vulkan): 119.69 tok/s (38%) community source
llama.cpp (ROCm): 119.0 tok/s (38%) community source
llama.cpp (ROCm): 118.71 tok/s (38%) community source
llama.cpp (ROCm): 118.23 tok/s (38%) community source
llama.cpp (ROCm): 117.61 tok/s (38%) community source
llama.cpp (ROCm): 117.12 tok/s (38%) community source
llama.cpp (ROCm): 116.83 tok/s (38%) community source
llama.cpp (ROCm): 116.83 tok/s (38%) community source
llama.cpp (ROCm): 114.02 tok/s (37%) community source
llama.cpp (ROCm): 112.18 tok/s (36%) community source
llama.cpp (ROCm): 110.7 tok/s (36%) community source
llama.cpp (ROCm): 109.92 tok/s (35%) community source
llama.cpp (ROCm): 109.92 tok/s (35%) community source
llama.cpp (ROCm): 107.16 tok/s (34%) community source
llama.cpp (ROCm): 107.02 tok/s (34%) community source
llama.cpp (Vulkan): 100.62 tok/s (32%) community source
llama.cpp (ROCm): 99.96 tok/s (32%) community source
llama.cpp (ROCm): 98.71 tok/s (32%) community source
llama.cpp (Vulkan): 97.96 tok/s (32%) community source
llama.cpp (ROCm): 95.04 tok/s (31%) community source
llama.cpp (ROCm): 92.28 tok/s (30%) community source
llama.cpp (Vulkan): 91.48 tok/s (29%) community source
llama.cpp (Vulkan): 88.12 tok/s (28%) community source
llama.cpp (ROCm): 74.66 tok/s (24%) community source
llama.cpp (ROCm): 70.83 tok/s (23%) community source
llama.cpp (Vulkan): 56.35 tok/s (18%) community source

llama 7B on Radeon RX 9060 XT (Q4_0)

llama.cpp (Vulkan): 97.01 tok/s (100%) community source
llama.cpp (Vulkan): 93.51 tok/s (96%) community source
llama.cpp (Vulkan): 74.01 tok/s (76%) community source
llama.cpp (Vulkan): 73.86 tok/s (76%) community source
llama.cpp (Vulkan): 72.24 tok/s (74%) community source
llama.cpp (Vulkan): 72.0 tok/s (74%) community source
llama.cpp (Vulkan): 71.95 tok/s (74%) community source
llama.cpp (Vulkan): 71.45 tok/s (74%) community source
llama.cpp (Vulkan): 71.07 tok/s (73%) community source
llama.cpp (Vulkan): 70.54 tok/s (73%) community source
llama.cpp (Vulkan): 70.52 tok/s (73%) community source
llama.cpp (ROCm): 70.18 tok/s (72%) community source
llama.cpp (ROCm): 67.58 tok/s (70%) community source
llama.cpp (ROCm): 67.27 tok/s (69%) community source
llama.cpp (Vulkan): 66.67 tok/s (69%) community source
llama.cpp (ROCm): 66.48 tok/s (69%) community source
llama.cpp (ROCm): 65.42 tok/s (67%) community source
llama.cpp (ROCm): 63.98 tok/s (66%) community source
llama.cpp (Vulkan): 54.96 tok/s (57%) community source
llama.cpp (Vulkan): 54.84 tok/s (57%) community source
llama.cpp (Vulkan): 44.1 tok/s (45%) community source
llama.cpp (Vulkan): 38.37 tok/s (40%) community source
llama.cpp (Vulkan): 38.31 tok/s (39%) community source
llama.cpp (Vulkan): 34.72 tok/s (36%) community source
llama.cpp (Vulkan): 28.37 tok/s (29%) community source
llama.cpp (Vulkan): 15.07 tok/s (16%) community source

llama 7B on Radeon RX 9070 (Q4_0)

llama.cpp (Vulkan): 119.71 tok/s (100%) community source
llama.cpp (Vulkan): 119.51 tok/s (100%) community source
llama.cpp (ROCm): 115.84 tok/s (97%) community source
llama.cpp (ROCm): 115.32 tok/s (96%) community source
llama.cpp (ROCm): 115.11 tok/s (96%) community source
llama.cpp (ROCm): 114.48 tok/s (96%) community source
llama.cpp (ROCm): 99.39 tok/s (83%) community source
llama.cpp (ROCm): 97.1 tok/s (81%) community source

llama 7B on Radeon RX 9070 XT (Q4_0)

llama.cpp (Vulkan): 142.88 tok/s (100%) community source
llama.cpp (Vulkan): 138.84 tok/s (97%) community source
llama.cpp (Vulkan): 138.45 tok/s (97%) community source
llama.cpp (Vulkan): 137.31 tok/s (96%) community source
llama.cpp (Vulkan): 137.11 tok/s (96%) community source
llama.cpp (Vulkan): 136.64 tok/s (96%) community source
llama.cpp (Vulkan): 132.27 tok/s (93%) community source
llama.cpp (Vulkan): 131.99 tok/s (92%) community source
llama.cpp (Vulkan): 131.54 tok/s (92%) community source
llama.cpp (Vulkan): 131.01 tok/s (92%) community source
llama.cpp (Vulkan): 130.57 tok/s (91%) community source
llama.cpp (Vulkan): 130.56 tok/s (91%) community source
llama.cpp (Vulkan): 130.12 tok/s (91%) community source
llama.cpp (Vulkan): 128.8 tok/s (90%) community source
llama.cpp (Vulkan): 128.53 tok/s (90%) community source
llama.cpp (Vulkan): 128.49 tok/s (90%) community source
llama.cpp (ROCm): 128.38 tok/s (90%) community source
llama.cpp (Vulkan): 127.69 tok/s (89%) community source
llama.cpp (Vulkan): 127.38 tok/s (89%) community source
llama.cpp (ROCm): 127.26 tok/s (89%) community source
llama.cpp (Vulkan): 126.81 tok/s (89%) community source
llama.cpp (Vulkan): 125.64 tok/s (88%) community source
llama.cpp (Vulkan): 125.45 tok/s (88%) community source
llama.cpp (Vulkan): 124.75 tok/s (87%) community source
llama.cpp (Vulkan): 124.33 tok/s (87%) community source
llama.cpp (Vulkan): 123.7 tok/s (87%) community source
llama.cpp (Vulkan): 122.54 tok/s (86%) community source
llama.cpp (Vulkan): 121.25 tok/s (85%) community source
llama.cpp (ROCm): 121.04 tok/s (85%) community source
llama.cpp (Vulkan): 120.52 tok/s (84%) community source
llama.cpp (Vulkan): 120.46 tok/s (84%) community source
llama.cpp (ROCm): 119.97 tok/s (84%) community source
llama.cpp (Vulkan): 117.72 tok/s (82%) community source
llama.cpp (Vulkan): 117.22 tok/s (82%) community source
llama.cpp (ROCm): 116.95 tok/s (82%) community source
llama.cpp (Vulkan): 116.82 tok/s (82%) community source
llama.cpp (ROCm): 116.5 tok/s (82%) community source
llama.cpp (ROCm): 116.46 tok/s (82%) community source
llama.cpp (ROCm): 109.81 tok/s (77%) community source
llama.cpp (ROCm): 108.45 tok/s (76%) community source
llama.cpp (ROCm): 106.7 tok/s (75%) community source
llama.cpp (ROCm): 106.63 tok/s (75%) community source
llama.cpp (ROCm): 102.55 tok/s (72%) community source
llama.cpp (ROCm): 102.44 tok/s (72%) community source
llama.cpp (ROCm): 101.61 tok/s (71%) community source
llama.cpp (ROCm): 101.27 tok/s (71%) community source
llama.cpp (ROCm): 98.13 tok/s (69%) community source
llama.cpp (ROCm): 98.11 tok/s (69%) community source
llama.cpp (ROCm): 97.28 tok/s (68%) community source
llama.cpp (ROCm): 97.27 tok/s (68%) community source
llama.cpp (ROCm): 97.21 tok/s (68%) community source
llama.cpp (ROCm): 94.32 tok/s (66%) community source
llama.cpp (ROCm): 93.26 tok/s (65%) community source
llama.cpp (ROCm): 91.86 tok/s (64%) community source
llama.cpp (ROCm): 90.08 tok/s (63%) community source
llama.cpp (ROCm): 89.9 tok/s (63%) community source
llama.cpp (ROCm): 89.51 tok/s (63%) community source
llama.cpp (ROCm): 89.17 tok/s (62%) community source
llama.cpp (ROCm): 88.96 tok/s (62%) community source
llama.cpp (ROCm): 87.57 tok/s (61%) community source
llama.cpp (ROCm): 87.43 tok/s (61%) community source
llama.cpp (ROCm): 86.16 tok/s (60%) community source
llama.cpp (ROCm): 86.1 tok/s (60%) community source
llama.cpp (ROCm): 85.43 tok/s (60%) community source
llama.cpp (ROCm): 85.12 tok/s (60%) community source
llama.cpp (ROCm): 84.34 tok/s (59%) community source
llama.cpp (ROCm): 83.63 tok/s (59%) community source
llama.cpp (ROCm): 80.81 tok/s (57%) community source
llama.cpp (ROCm): 32.78 tok/s (23%) community source

llama 7B on T4 (Q4_0)

llama.cpp (CUDA): 46.38 tok/s (100%) community source
llama.cpp (CUDA): 44.03 tok/s (95%) community source
llama.cpp (Vulkan): 34.66 tok/s (75%) community source
llama.cpp (Vulkan): 33.36 tok/s (72%) community source

llama 7B mostly on M1 Max (F16)

llama.cpp (Metal): 23.03 tok/s (100%) community source
llama.cpp (Metal): 22.96 tok/s (100%) community source
llama.cpp (Metal): 22.55 tok/s (98%) community source
llama.cpp (Metal): 22.41 tok/s (97%) community source
llama.cpp (CUDA): 19.45 tok/s (84%) community source
llama.cpp (CUDA): 19.1 tok/s (83%) community source

llama 7B mostly on M1 Max (Q4_0)

llama.cpp (CUDA): 61.39 tok/s (100%) community source
llama.cpp (Metal): 61.19 tok/s (100%) community source
llama.cpp (Metal): 60.96 tok/s (99%) community source
llama.cpp (CUDA): 57.87 tok/s (94%) community source
llama.cpp (Metal): 57.39 tok/s (93%) community source
llama.cpp (Metal): 54.61 tok/s (89%) community source

llama 7B mostly on M1 Max (Q8_0)

llama.cpp (Metal): 40.2 tok/s (100%) community source
llama.cpp (Metal): 40.17 tok/s (100%) community source
llama.cpp (Metal): 38.5 tok/s (96%) community source
llama.cpp (Metal): 37.81 tok/s (94%) community source
llama.cpp (CUDA): 35.02 tok/s (87%) community source
llama.cpp (CUDA): 33.86 tok/s (84%) community source

llama 7B mostly on M3 (Q4_0)

llama.cpp (CUDA): 73.38 tok/s (100%) community source
llama.cpp (CUDA): 69.83 tok/s (95%) community source
llama.cpp (CUDA): 69.49 tok/s (95%) community source
llama.cpp (Metal): 30.36 tok/s (41%) community source
llama.cpp (Metal): 21.34 tok/s (29%) community source
llama.cpp (CUDA): 19.23 tok/s (26%) community source

llama 7B mostly on M3 (Q8_0)

llama.cpp (CUDA): 50.75 tok/s (100%) community source
llama.cpp (CUDA): 50.47 tok/s (99%) community source
llama.cpp (CUDA): 42.43 tok/s (84%) community source
llama.cpp (CUDA): 19.11 tok/s (38%) community source
llama.cpp (Metal): 16.46 tok/s (32%) community source
llama.cpp (Metal): 12.27 tok/s (24%) community source

MiniMax-M2.7 on Radeon 8065S (UD-IQ4_XS)

llama.cpp (Vulkan): tok/s (0%) community source
llama.cpp (Vulkan): tok/s (0%) community source
llama.cpp (Vulkan): tok/s (0%) community source
llama.cpp (Vulkan): tok/s (0%) community source
llama.cpp (Vulkan): tok/s (0%) community source
llama.cpp (Vulkan): tok/s (0%) community source
llama.cpp (Vulkan): tok/s (0%) community source
llama.cpp (Vulkan): tok/s (0%) community source
llama.cpp (Vulkan): tok/s (0%) community source
llama.cpp (ROCm): tok/s (0%) community source
llama.cpp (ROCm): tok/s (0%) community source

nemotron_h_moe 31B.A3.5B on GB10 (Q4_K - Medium)

llama.cpp (Vulkan): 64.5 tok/s (100%) community source
llama.cpp (CUDA): 61.76 tok/s (96%) community source

Qwen2.5-7B on 2 A10 (AWQ)

SGLang: 1157.76 tok/s (100%) community source
SGLang: 1151.33 tok/s (99%) community source
vLLM: 1074.01 tok/s (93%) community source
vLLM: 1025.81 tok/s (89%) community source
SGLang: 853.94 tok/s (74%) community source
SGLang: 511.92 tok/s (44%) community source

Qwen2.5-7B on A10 (AWQ)

SGLang: 1544.94 tok/s (100%) community source
vLLM: 961.05 tok/s (62%) community source
SGLang: 333.00 tok/s (22%) community source
vLLM: 259.53 tok/s (17%) community source

Qwen 3 30B-A3B on M4 Max (4-bit)

MLX: 110 tok/s (100%) sourced source
llama.cpp: 90 tok/s (82%) sourced source

Qwen3 4B on A10 (AWQ)

vLLM: 656.60 tok/s (100%) community source
vLLM: 607.98 tok/s (93%) community source
vLLM: 469.62 tok/s (72%) community source
vLLM: 433.77 tok/s (66%) community source
SGLang: 424.58 tok/s (65%) community source
SGLang: 423.26 tok/s (64%) community source
SGLang: 413.54 tok/s (63%) community source
SGLang: 401.26 tok/s (61%) community source
vLLM: 395.12 tok/s (60%) community source
vLLM: 377.98 tok/s (58%) community source
SGLang: 369.19 tok/s (56%) community source
SGLang: 337.17 tok/s (51%) community source
vLLM: 278.73 tok/s (42%) community source
vLLM: 249.99 tok/s (38%) community source
SGLang: 239.82 tok/s (37%) community source
SGLang: 238.21 tok/s (36%) community source

Qwen3.5-0.8B on Dimensity 9400 (Q4_K_M)

llama.cpp (Vulkan): 36.88 tok/s (100%) community source
llama.cpp (Vulkan): 36.62 tok/s (99%) community source
llama.cpp (Vulkan): 36.4 tok/s (99%) community source
llama.cpp (Vulkan): 36.03 tok/s (98%) community source
llama.cpp (Vulkan): 35.8 tok/s (97%) community source
llama.cpp (CPU): 2.95 tok/s (8%) community source

Qwen3.5-0.8B-Japanese-SFT on M5 (Q4_K_M)

llama.cpp (Metal): 169 tok/s (100%) community source
llama.cpp (CPU): 92 tok/s (54%) community source

Qwen3.5-0.8B-Japanese-SFT on M5 (Q8_0)

llama.cpp (Metal): 123 tok/s (100%) community source
llama.cpp (CPU): 78 tok/s (63%) community source

qwen35 27B on Arc Pro B70 (Q8_0)

llama.cpp (Vulkan): 18.6 tok/s (100%) community source
llama.cpp (Vulkan): 18.56 tok/s (100%) community source
llama.cpp (SYCL): 16.66 tok/s (90%) community source
llama.cpp (SYCL): 16.64 tok/s (89%) community source
llama.cpp (SYCL): 16.63 tok/s (89%) community source
llama.cpp (SYCL): 16.6 tok/s (89%) community source

Qwen3.5-35B-A3B on M1 Max (64 GB) (4bit)

MLX: 57.6 tok/s (100%) sourced source
LM Studio (MLX): 57 tok/s (99%) sourced source

Qwen3.5-35B-A3B on RTX 4090 (Q4_K_S)

llama.cpp (CUDA): tok/s (0%) community source
llama.cpp (CUDA): tok/s (0%) community source
llama.cpp (CUDA): tok/s (0%) community source
llama.cpp (CUDA): tok/s (0%) community source
llama.cpp (CUDA): tok/s (0%) community source
llama.cpp (CUDA): tok/s (0%) community source
llama.cpp: tok/s (0%) sourced source
llama.cpp: tok/s (0%) sourced source
llama.cpp: tok/s (0%) sourced source
llama.cpp: tok/s (0%) sourced source

Qwen3.5-4B on RTX 2060 6 GB (Q4_K_M)

llama.cpp (CUDA): 56.1 tok/s (100%) community source
llama.cpp (Vulkan): 47.0 tok/s (84%) community source
llama.cpp (Vulkan): 15.4 tok/s (27%) community source
llama.cpp (CUDA): 3.09 tok/s (6%) community source

Qwen3.5-4B on Snapdragon 7 Gen 4 (SM7750) (Q4_0)

llama.cpp (CPU): 9.1 tok/s (100%) community source
llama.cpp (Hexagon): 8.1 tok/s (89%) community source
llama.cpp (OpenCL): 7.5 tok/s (82%) community source
llama.cpp (Hexagon): 6.2 tok/s (68%) community source

Qwen3.5-9B on Arc A770 (UD-Q4_K_XL)

llama.cpp (Vulkan): 26 tok/s (100%) community source
llama.cpp (SYCL): 21.19 tok/s (82%) community source

qwen35moe 35B.A3B on Arc Pro B60 (Q4_K - Medium)

llama.cpp (SYCL): 81.84 tok/s (100%) community source
llama.cpp: 81.68 tok/s (100%) community source
llama.cpp (SYCL): 81.67 tok/s (100%) community source
llama.cpp: 81.63 tok/s (100%) community source

Qwen3.6-35B-A3B-APEX-MTP-I-Balanced on Radeon 8060S (Q5_K_M)

llama.cpp (Vulkan): 80.89 tok/s (100%) community source
llama.cpp (ROCm): 69.47 tok/s (86%) community source
llama.cpp (Vulkan): 66.86 tok/s (83%) community source
llama.cpp (ROCm): 56.41 tok/s (70%) community source

Qwen3.8-27B on 2x RX 7900 XTX (Q4_K_XL)

llama.cpp: 47.06 tok/s (100%) community source
llama.cpp: 46.99 tok/s (100%) community source
llama.cpp: 46.9 tok/s (100%) community source
llama.cpp: 46.48 tok/s (99%) community source
llama.cpp: 46.28 tok/s (98%) community source
llama.cpp: 34.03 tok/s (72%) community source
Vulkan: 23.36 tok/s (50%) community source

Qwen3.8-27B on 2xR9700 (UD-Q4_K_XL)

llama.cpp (ROCm): 71.42 tok/s (100%) community source
llama.cpp (Vulkan): 55.78 tok/s (78%) community source
llama.cpp (Vulkan): 55.58 tok/s (78%) community source
llama.cpp (Vulkan): 52.65 tok/s (74%) community source
llama.cpp (ROCm): 50.49 tok/s (71%) community source
llama.cpp (ROCm): 44.56 tok/s (62%) community source
llama.cpp (Vulkan): 42.96 tok/s (60%) community source
llama.cpp (Vulkan): 41.12 tok/s (58%) community source
llama.cpp (Vulkan): 40.78 tok/s (57%) community source
llama.cpp (Vulkan): 38.52 tok/s (54%) community source
llama.cpp (Vulkan): 37.52 tok/s (53%) community source
llama.cpp (Vulkan): 35.95 tok/s (50%) community source
llama.cpp (Vulkan): 35.42 tok/s (50%) community source
llama.cpp (ROCm): 35.17 tok/s (49%) community source
llama.cpp (Vulkan): 35.11 tok/s (49%) community source
llama.cpp (Vulkan): 34.07 tok/s (48%) community source
llama.cpp (Vulkan): 33.87 tok/s (47%) community source
llama.cpp (Vulkan): 33.72 tok/s (47%) community source
llama.cpp (ROCm): 32.84 tok/s (46%) community source
llama.cpp (ROCm): 32.40 tok/s (45%) community source

Qwen3.8-27B on Arc Pro B70 (Q4_K_M)

llama.cpp (SYCL): 38.5 tok/s (100%) community source
llama.cpp (Vulkan): 35.6 tok/s (92%) community source
llama.cpp (Vulkan): 33.4 tok/s (87%) community source
llama.cpp (SYCL): 31.7 tok/s (82%) community source
llama.cpp (SYCL): 22.8 tok/s (59%) community source
llama.cpp (SYCL): 22.5 tok/s (58%) community source
llama.cpp (Vulkan): 21.4 tok/s (56%) community source
llama.cpp (SYCL): 18.8 tok/s (49%) community source
llama.cpp (SYCL): 15.9 tok/s (41%) community source
llama.cpp (SYCL): 15.8 tok/s (41%) community source
llama.cpp (Vulkan): 14.9 tok/s (39%) community source
llama.cpp (Vulkan): 13.7 tok/s (36%) community source
llama.cpp (Vulkan): 13.3 tok/s (35%) community source
llama.cpp (Vulkan): 9.8 tok/s (25%) community source
llama.cpp (Vulkan): 8.4 tok/s (22%) community source
llama.cpp (Vulkan): 8.2 tok/s (21%) community source
llama.cpp (Vulkan): 4.5 tok/s (12%) community source
llama.cpp (Vulkan): 4.3 tok/s (11%) community source

Qwen3.8-27B on Arc Pro B70 (Q8_0)

llama.cpp (SYCL): 50.92 tok/s (100%) community source
llama.cpp (SYCL): 46.11 tok/s (91%) community source
llama.cpp (SYCL): 41.35 tok/s (81%) community source
llama.cpp (SYCL): 33.53 tok/s (66%) community source
llama.cpp (SYCL): 33.52 tok/s (66%) community source
llama.cpp (SYCL): 33.23 tok/s (65%) community source
llama.cpp (SYCL): 32.5 tok/s (64%) community source
llama.cpp (SYCL): 30.62 tok/s (60%) community source
llama.cpp (SYCL): 26.69 tok/s (52%) community source
llama.cpp (SYCL): 25.99 tok/s (51%) community source
llama.cpp (SYCL): 16.59 tok/s (33%) community source
llama.cpp (SYCL): 16.34 tok/s (32%) community source
llama.cpp (SYCL): 16.01 tok/s (31%) community source
llama.cpp (SYCL): 15.82 tok/s (31%) community source
llama.cpp (SYCL): 15.81 tok/s (31%) community source
llama.cpp (SYCL): 15.8 tok/s (31%) community source
llama.cpp (SYCL): 15.8 tok/s (31%) community source
llama.cpp (SYCL): 15.79 tok/s (31%) community source
llama.cpp (SYCL): 15.79 tok/s (31%) community source
llama.cpp (SYCL): 15.79 tok/s (31%) community source
llama.cpp (SYCL): 15.79 tok/s (31%) community source
llama.cpp (SYCL): 15.79 tok/s (31%) community source
llama.cpp (SYCL): 15.79 tok/s (31%) community source
llama.cpp (SYCL): 15.79 tok/s (31%) community source
llama.cpp (SYCL): 15.79 tok/s (31%) community source
llama.cpp (SYCL): 15.79 tok/s (31%) community source
llama.cpp (SYCL): 15.79 tok/s (31%) community source
llama.cpp (SYCL): 15.73 tok/s (31%) community source
llama.cpp (SYCL): 15.72 tok/s (31%) community source
llama.cpp (SYCL): 15.69 tok/s (31%) community source
llama.cpp (SYCL): 15.67 tok/s (31%) community source
llama.cpp (SYCL): 15.52 tok/s (30%) community source
llama.cpp (SYCL): 13.86 tok/s (27%) community source
llama.cpp (SYCL): 5.07 tok/s (10%) community source
llama.cpp: tok/s (0%) sourced source
llama.cpp: tok/s (0%) sourced source
llama.cpp: tok/s (0%) sourced source
llama.cpp: tok/s (0%) sourced source
llama.cpp: tok/s (0%) sourced source
llama.cpp: tok/s (0%) sourced source
llama.cpp (SYCL): tok/s (0%) community source
llama.cpp (SYCL): tok/s (0%) community source
llama.cpp (SYCL): tok/s (0%) community source
llama.cpp (SYCL): tok/s (0%) community source
llama.cpp (SYCL): tok/s (0%) community source
llama.cpp (SYCL): tok/s (0%) community source
llama.cpp (SYCL): tok/s (0%) community source

Qwen3.8-27B on M5 Pro (20-core GPU) (Q8_0)

llama.cpp (Metal): 9.25 tok/s (100%) community source
llama.cpp: 9.18 tok/s (99%) community source
llama.cpp (Metal): 9.09 tok/s (98%) community source
llama.cpp: 9.02 tok/s (98%) community source
llama.cpp: tok/s (0%) community source
llama.cpp: tok/s (0%) community source
llama.cpp: tok/s (0%) community source
llama.cpp: tok/s (0%) community source
llama.cpp: tok/s (0%) community source
llama.cpp: tok/s (0%) community source
llama.cpp: tok/s (0%) community source
llama.cpp: tok/s (0%) community source
llama.cpp (Metal): tok/s (0%) community source
llama.cpp (Metal): tok/s (0%) community source
llama.cpp (Metal): tok/s (0%) community source
llama.cpp (Metal): tok/s (0%) community source

Qwen3.8-27B on RTX 5060 Ti (UD-IQ3_S)

llama.cpp (CUDA): 38.8 tok/s (100%) community source
llama.cpp: 38.8 tok/s (100%) community source
llama.cpp (CUDA): 26.55 tok/s (68%) community source
llama.cpp: 26.55 tok/s (68%) community source
llama.cpp: 19 tok/s (49%) community source
llama.cpp (CUDA): 2.01 tok/s (5%) community source
llama.cpp (CUDA): 2.01 tok/s (5%) community source
llama.cpp: 2.01 tok/s (5%) community source
llama.cpp: 2.01 tok/s (5%) community source
llama.cpp (CUDA): 1.67 tok/s (4%) community source
llama.cpp: 1.67 tok/s (4%) community source

Qwen3.8-27B on RTX 5080 (IQ4_XS)

llama.cpp (Vulkan): 35.59 tok/s (100%) community source
llama.cpp (CUDA): 16 tok/s (45%) community source

Qwen3.8-27B on RTX 5090 (NVFP4)

llama.cpp (CUDA): 152 tok/s (100%) community source
llama.cpp (CUDA): 87.45 tok/s (58%) community source
llama.cpp (CUDA): 82.8 tok/s (54%) community source
llama.cpp (CUDA): 78.9 tok/s (52%) community source
llama.cpp (CUDA): tok/s (0%) community source
llama.cpp (CUDA): tok/s (0%) community source
NInfer: tok/s (0%) community source
llama.cpp (CUDA): tok/s (0%) community source
llama.cpp (CUDA): tok/s (0%) community source
llama.cpp (CUDA): tok/s (0%) community source
llama.cpp (CUDA): tok/s (0%) community source

Qwen3.8-27B on RTX 5090 (Q6_K)

llama.cpp (focus-llama): 102.7 tok/s (100%) community source
llama.cpp: 76.2 tok/s (74%) community source

Qwen3.8-27B on RTX PRO 6000 (Q8_0)

llama.cpp (CUDA): 102.08 tok/s (100%) community source
llama.cpp (CUDA): 90.29 tok/s (88%) community source
llama.cpp: tok/s (0%) sourced source
llama.cpp: tok/s (0%) sourced source
llama.cpp: tok/s (0%) sourced source
llama.cpp: tok/s (0%) sourced source

Qwen3.8-27B on RX 6800 (IQ4_XS)

llama.cpp (Vulkan): 28.4 tok/s (100%) community source
llama.cpp (ROCm): 26.1 tok/s (92%) community source

Qwen3.8-27B on RX 7900 XTX (Q4_K_XL)

Vulkan: 39.07 tok/s (100%) community source
llama.cpp: 36.05 tok/s (92%) community source
llama.cpp: 36.02 tok/s (92%) community source
llama.cpp: 35.54 tok/s (91%) community source

Qwen3.8-27B on Radeon RX 9060 XT (UD-IQ4_XS)

llama.cpp (Vulkan): 20.1 tok/s (100%) community source
llama.cpp (ROCm): 19.5 tok/s (97%) community source

Qwen3.8-27B-AD on Radeon 8060S (Q5_K_M)

llama.cpp (Vulkan): 16.98 tok/s (100%) community source
llama.cpp (ROCm): 14.09 tok/s (83%) community source
llama.cpp (Vulkan): 10.76 tok/s (63%) community source
llama.cpp (ROCm): 10.52 tok/s (62%) community source

Qwen3.8-27B-Coder390-EfficientThink-Opus5.5-GPT6Astra-Grok4.7-DSV4Pro-K3-SFT-RLOO on RTX Pro 6000 Blackwell (Q2 LynnStyle)

NInfer: 521.0 tok/s (100%) community source
llama.cpp (CUDA): 174.8 tok/s (34%) community source

Qwen3.8-27B-Coder390-EfficientThink-Opus5.5-GPT6Astra-Grok4.7-DSV4Pro-K3-SFT-RLOO on RTX Pro 6000 Blackwell (Q3 LynnStyle)

NInfer: 512.3 tok/s (100%) community source
llama.cpp (CUDA): 191.8 tok/s (37%) community source
llama.cpp (CUDA): 112.2 tok/s (22%) community source
llama.cpp (CUDA): 110.2 tok/s (22%) community source
llama.cpp (CUDA): 95.6 tok/s (19%) community source
llama.cpp (CUDA): 94.4 tok/s (18%) community source

Qwen3.8-27B-Coder390-EfficientThink-Opus5.5-GPT6Astra-Grok4.7-DSV4Pro-K3-SFT-RLOO on RTX Pro 6000 Blackwell (Q4 LynnStyle)

NInfer: 436.4 tok/s (100%) community source
llama.cpp (CUDA): 81.1 tok/s (19%) community source

Qwen3.8-27B-Coder390-EfficientThink-Opus5.5-GPT6Astra-Grok4.7-DSV4Pro-K3-SFT-RLOO on RTX Pro 6000 Blackwell (Q6_K)

NInfer: 388.1 tok/s (100%) community source
NInfer: 235.5 tok/s (61%) community source
llama.cpp (CUDA): 189.0 tok/s (49%) community source
NInfer: 131.9 tok/s (34%) community source
llama.cpp (CUDA): 119.0 tok/s (31%) community source
llama.cpp (CUDA): 97.2 tok/s (25%) community source
llama.cpp (CUDA): 53.4 tok/s (14%) community source

Qwen3.8-27B-Coder390-EfficientThink-Opus5.5-GPT6Astra-Grok4.7-DSV4Pro-K3-SFT-RLOO on RTX Pro 6000 Blackwell (Q8_0)

NInfer: 486.3 tok/s (100%) community source
llama.cpp (CUDA): 190.8 tok/s (39%) community source

Qwen3.8-9B-Distill on M1 Max (oQ4e)

oMLX: 56.5 tok/s (100%) community source
oMLX: 46.6 tok/s (82%) community source
MLX: 43.0 tok/s (76%) community source

Qwen3.8-Flash-Next on 6x RTX 3090 (UD-Q4_K_XL)

llama.cpp (multigpu fork): 41.9 tok/s (100%) community source
llama.cpp (multigpu fork): 33.7 tok/s (80%) community source
llama.cpp (multigpu fork): 27.3 tok/s (65%) community source
llama.cpp (CUDA): 25.7 tok/s (61%) community source
llama.cpp (CUDA): 10.2 tok/s (24%) community source
llama.cpp (CUDA): 2.3 tok/s (5%) community source
llama.cpp (CUDA): tok/s (0%) community source
llama.cpp (multigpu fork): tok/s (0%) community source
llama.cpp (CUDA): tok/s (0%) community source
llama.cpp (multigpu fork): tok/s (0%) community source
llama.cpp (CUDA): tok/s (0%) community source
llama.cpp (multigpu fork): tok/s (0%) community source

Qwen3.8-Flash-Next on 6x RTX 4090 (UD-Q4_K_XL)

llama.cpp (multigpu fork): 57.5 tok/s (100%) community source
llama.cpp (multigpu fork): 48.7 tok/s (85%) community source
llama.cpp (CUDA): 46.2 tok/s (80%) community source
llama.cpp (multigpu fork): 30.8 tok/s (54%) community source
llama.cpp (CUDA): 21.0 tok/s (37%) community source
llama.cpp (CUDA): tok/s (0%) community source
llama.cpp (multigpu fork): tok/s (0%) community source
llama.cpp (CUDA): tok/s (0%) community source
llama.cpp (multigpu fork): tok/s (0%) community source
llama.cpp (multigpu fork): tok/s (0%) community source

Qwen3.8-Flash-Next on RTX 5090 (Gyro-S)

Strata: 221 tok/s (100%) community source
llama.cpp (CUDA): 218 tok/s (99%) community source
llama.cpp (CUDA): 210 tok/s (95%) community source
Strata: 209 tok/s (95%) community source
llama.cpp (CUDA): 180 tok/s (81%) community source
Strata: 140 tok/s (63%) community source
llama.cpp (CUDA): 113 tok/s (51%) community source
Strata: tok/s (0%) community source

Qwen7B-awq on 2 A10 (AWQ)

SGLang: 1822.21 tok/s (100%) community source
vLLM: 1816.33 tok/s (100%) community source
SGLang: 1811.35 tok/s (99%) community source
vLLM: 1757.90 tok/s (96%) community source
vLLM: 1534.72 tok/s (84%) community source
vLLM: 1519.86 tok/s (83%) community source
vLLM: 1217.92 tok/s (67%) community source
vLLM: 1131.43 tok/s (62%) community source
SGLang: 1117.99 tok/s (61%) community source
SGLang: 1115.89 tok/s (61%) community source
SGLang: 422.97 tok/s (23%) community source

Qwen7B-awq on 4 A10 (AWQ)

SGLang: 2560.58 tok/s (100%) community source
SGLang: 2540.23 tok/s (99%) community source
SGLang: 2526.98 tok/s (99%) community source
SGLang: 2123.63 tok/s (83%) community source
SGLang: 2119.20 tok/s (83%) community source
SGLang: 2086.65 tok/s (81%) community source
vLLM: 1914.26 tok/s (75%) community source
vLLM: 1887.62 tok/s (74%) community source
vLLM: 1566.10 tok/s (61%) community source
vLLM: 1534.41 tok/s (60%) community source
vLLM: 1318.94 tok/s (52%) community source
vLLM: 1309.40 tok/s (51%) community source

Qwopus3.6-27B-Coder-Compat-MTP on Radeon 8060S (Q5_K_M)

llama.cpp (Vulkan): 22.38 tok/s (100%) community source
llama.cpp (ROCm): 19.38 tok/s (87%) community source
llama.cpp (Vulkan): 11.23 tok/s (50%) community source
llama.cpp (ROCm): 10.94 tok/s (49%) community source

Sakura-Qwen3.8-Flash-Next on Radeon 8060S (IQ3_XXS)

llama.cpp (Vulkan): 22.7 tok/s (100%) community source
llama.cpp (ROCm): 4.9 tok/s (22%) community source

Sakura-Qwen3.8-Flash-Next-Swift on Radeon 8060S (IQ2_XS)

llama.cpp (Vulkan): 37.6 tok/s (100%) community source
llama.cpp (Vulkan): 35.2 tok/s (94%) community source
llama.cpp (Vulkan): 24 tok/s (64%) community source
llama.cpp (ROCm): 21.6 tok/s (57%) community source

Sakura-Qwen3.8-Flash-Next-Swift on Radeon 8060S (IQ3_XXS)

llama.cpp (Vulkan): 34.5 tok/s (100%) community source
llama.cpp (Vulkan): 33.6 tok/s (97%) community source
llama.cpp (Vulkan): 33.4 tok/s (97%) community source
llama.cpp (Vulkan): 32.8 tok/s (95%) community source
llama.cpp (Vulkan): 21 tok/s (61%) community source
llama.cpp (ROCm): 5 tok/s (14%) community source