Llama 3.1 8B
76 records across 10 hardware.
| hardware | model | quant | backend | tok/s | pp tok/s | W | ttft s | ctx/tg | date | class | flags | source |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| H100 | Llama 3.1 8B | FP8 | vLLM | 5005 | – | – | – | – | 2026-09-30 | community | – | source |
| H100 | Llama 3.1 8B | FP8 | vLLM | 4977 | – | – | – | – | 2026-09-30 | community | – | source |
| H100 | Llama 3.1 8B | FP8 | vLLM | 4821 | – | – | – | – | 2026-09-30 | community | – | source |
| RTX 5090 | Llama 3.1 8B | Q4_K_M | llama.cpp (CUDA) | 509.6 | – | – | – | – | 2026-10-03 | community | – | source |
| RTX 5090 | Llama 3.1 8B | Q4_K_M | llama.cpp (CUDA) | 500.7 | – | – | – | – | 2026-10-03 | community | – | source |
| RTX 5090 | Llama 3.1 8B | Q4_K_M | llama.cpp (CUDA) | 375.1 | – | – | – | – | 2026-10-03 | community | – | source |
| RTX 5090 | Llama 3.1 8B | Q4_K_M | llama.cpp (CUDA) | 231.93 | 12761.21 | – | – | tg128 | 2026-04-04 | community | – | source |
| RTX 5090 | Llama 3.1 8B | Q4_K_M | llama.cpp (CUDA) | 230.70 | 12850.64 | – | – | tg128 | 2026-04-04 | community | – | source |
| RTX 5090 | Llama 3.1 8B | Q4_K_M | llama.cpp (CUDA) | 173.43 | 12352.17 | – | – | tg128 | 2026-04-04 | community | – | source |
| RTX 5090 | Llama 3.1 8B | Q4_K_M | llama.cpp (CUDA) | 162.95 | 12388.47 | – | – | tg128 | 2026-04-04 | community | – | source |
| RTX 5090 | Llama 3.1 8B | Q4_K_M | llama.cpp (CUDA) | 161.79 | 12420.45 | – | – | tg128 | 2026-04-04 | community | – | source |
| RTX 5090 | Llama 3.1 8B | Q4_K_M | llama.cpp (CUDA) | 153.54 | 12369.44 | – | – | tg128 | 2026-04-04 | community | – | source |
| RTX 4090 | Llama 3.1 8B | Q4_K_M | llama.cpp (CUDA) | 150.59 | 10112.60 | – | – | tg128 | 2026-04-04 | community | – | source |
| RTX 4090 | Llama 3.1 8B | Q4_K_M | llama.cpp (CUDA) | 150.4 | 9365 | – | – | tg128 | 2026-04-03 | community | – | source |
| RTX 4090 | Llama 3.1 8B | Q4_K_M | llama.cpp (CUDA) | 150.17 | 10076.76 | – | – | tg128 | 2026-04-04 | community | – | source |
| RTX 4090 | Llama 3.1 8B | Q4_K_M | llama.cpp (CUDA) | 149.3 | 10060 | – | – | tg128 | 2026-04-03 | community | – | source |
| RTX 5090 | Llama 3.1 8B | Q4_K_M | llama.cpp (CUDA) | 144.22 | 11739.74 | – | – | tg128 | 2026-04-04 | community | – | source |
| RTX 4090 | Llama 3.1 8B | Q4_K_M | llama.cpp (CUDA) | 140.0 | 10903 | – | – | tg128 | 2026-04-03 | community | – | source |
| RTX 5090 | Llama 3.1 8B | Q4_K_M | llama.cpp (CUDA) | 139.90 | 11799.41 | – | – | tg128 | 2026-04-04 | community | – | source |
| RTX 4090 | Llama 3.1 8B | Q4_K_M | llama.cpp (CUDA) | 139.83 | 10957.29 | – | – | tg128 | 2026-04-04 | community | – | source |
| RTX 4090 | Llama 3.1 8B | Q4_K_M | llama.cpp (CUDA) | 138.4 | 10925 | – | – | tg128 | 2026-04-03 | community | – | source |
| RTX 4090 | Llama 3.1 8B | Q4_K_M | llama.cpp (CUDA) | 137.81 | 10936.64 | – | – | tg128 | 2026-04-04 | community | – | source |
| RTX 4090 | Llama 3.1 8B | Q4_K_M | llama.cpp (CUDA) | 136.37 | 10808.36 | – | – | tg128 | 2026-04-04 | community | – | source |
| RTX 4090 | Llama 3.1 8B | Q4_K_M | llama.cpp (CUDA) | 134.73 | 10794.80 | – | – | tg128 | 2026-04-04 | community | – | source |
| RTX 5090 | Llama 3.1 8B | Q4_K_M | llama.cpp (CUDA) | 134.12 | 11914.27 | – | – | tg128 | 2026-04-04 | community | – | source |
| RTX 4090 | Llama 3.1 8B | Q4_K_XL | llama.cpp llama-bench | 131 | – | – | – | 4096 | 2026-08-17 | sourced | – | source |
| RTX 4090 | Llama 3.1 8B | Q4_K_M | llama.cpp (CUDA) | 129.26 | 10571.92 | – | – | tg128 | 2026-04-04 | community | – | source |
| RTX 4090 | Llama 3.1 8B | Q4_K_M | llama.cpp (CUDA) | 128.3 | 10476 | – | – | tg128 | 2026-04-03 | community | – | source |
| RTX 4090 | Llama 3.1 8B | Q4_K_M | llama.cpp (CUDA) | 127.79 | 10456.24 | – | – | tg128 | 2026-04-04 | community | – | source |
| RTX 5090 | Llama 3.1 8B | Q4_K_M | llama.cpp (CUDA) | 125.44 | 11309.70 | – | – | tg128 | 2026-04-04 | community | – | source |
| RTX 4090 | Llama 3.1 8B | Q4_K_M | llama.cpp (CUDA) | 124.3 | 10405 | – | – | tg128 | 2026-04-03 | community | – | source |
| RTX 4090 | Llama 3.1 8B | Q4_K_M | llama.cpp (CUDA) | 123.95 | 10384.88 | – | – | tg128 | 2026-04-04 | community | – | source |
| RTX 4090 | Llama 3.1 8B | Q4_K_M | llama.cpp (CUDA) | 122.68 | 10154.79 | – | – | tg128 | 2026-04-04 | community | – | source |
| M2 Max | Llama 3.1 8B | 4bit | oMLX | 70.3 | 563.0 | – | – | 1024 | 2026-10-08 | community | – | source |
| M2 Max | Llama 3.1 8B | 4bit | oMLX | 61.9 | 535.5 | – | – | 4096 | 2026-10-08 | community | – | source |
| Tesla P40 | Llama 3.1 8B | Q4_0 | llama.cpp (CUDA) | 54.75 | – | – | – | tg128 | 2026-10 | community | – | source |
| Tesla P40 | Llama 3.1 8B | Q4_0 | llama.cpp (CUDA) | 53.69 | – | – | – | 496 | 2026-10 | community | – | source |
| RTX 4090 | Llama 3.1 8B | Q4_K_XL | llama.cpp llama-bench | 53.07 | – | – | – | 65536 | 2026-08-17 | sourced | – | source |
| Tesla P40 | Llama 3.1 8B | Q4_0 | llama.cpp (CUDA) | 52.28 | – | – | – | tg128 | 2026-10 | community | – | source |
| Tesla P40 | Llama 3.1 8B | Q4_0 | llama.cpp (CUDA) | 51.87 | – | – | – | 2016 | 2026-10 | community | – | source |
| Tesla P40 | Llama 3.1 8B | Q4_0 | llama.cpp (CUDA) | 51.39 | – | – | – | 496 | 2026-10 | community | – | source |
| 2x RTX 5060 Ti | Llama 3.1 8B | Q5_K_M | llama.cpp (CUDA) | 50 | – | – | – | 8192 | 2026-03-31 | community | – | source |
| Tesla P40 | Llama 3.1 8B | Q4_0 | llama.cpp (CUDA) | 49.72 | – | – | – | 2016 | 2026-10 | community | – | source |
| Tesla P40 | Llama 3.1 8B | Q4_K_M | llama.cpp (CUDA) | 48.94 | – | – | – | tg128 | 2026-10 | community | – | source |
| Tesla P40 | Llama 3.1 8B | Q4_K_M | llama.cpp (CUDA) | 48.34 | – | – | – | tg128 | 2026-10 | community | – | source |
| Tesla P40 | Llama 3.1 8B | Q4_K_M | llama.cpp (CUDA) | 48.04 | – | – | – | 496 | 2026-10 | community | – | source |
| Tesla P40 | Llama 3.1 8B | Q4_K_M | llama.cpp (CUDA) | 47.47 | – | – | – | 496 | 2026-10 | community | – | source |
| Tesla P40 | Llama 3.1 8B | Q4_K_M | llama.cpp (CUDA) | 46.53 | – | – | – | 2016 | 2026-10 | community | – | source |
| Tesla P40 | Llama 3.1 8B | Q4_K_M | llama.cpp (CUDA) | 46.05 | – | – | – | 2016 | 2026-10 | community | – | source |
| DGX Spark | Llama 3.1 8B | unspecified | vLLM | 15.83 | – | – | – | – | 2026-10-07 | community | – | source |
| M4 Max (128 GB) | Llama 3.1 8B | Q4_K_M | llama.cpp | 12.2 | – | – | – | 32768 | 2026-10-17 | sourced | – | source |
| gfx1151 | Llama 3.1 8B | AWQ-INT4 | vLLM | 9.2 | – | – | – | 128 | 2026-10-13 | sourced | – | source |
| gfx1151 | Llama 3.1 8B | AWQ-INT4 | vLLM | 9.1 | – | – | – | 128 | 2026-10-13 | sourced | – | source |
| gfx1151 | Llama 3.1 8B | AWQ-INT4 | vLLM | 8.6 | – | – | – | 128 | 2026-10-13 | sourced | – | source |
| gfx1151 | Llama 3.1 8B | AWQ-INT4 | vLLM | 8.6 | – | – | – | 128 | 2026-10-13 | sourced | – | source |
| 2x RTX 5060 Ti | Llama 3.1 8B | Q5_K_M | llama.cpp (CUDA) | 8.4 | – | – | – | 8192 | 2026-03-31 | community | – | source |
| gfx1151 | Llama 3.1 8B | AWQ-INT4 | vLLM | 8.4 | – | – | – | 128 | 2026-10-13 | sourced | – | source |
| gfx1151 | Llama 3.1 8B | AWQ-INT4 | vLLM | 8.4 | – | – | – | 128 | 2026-10-13 | sourced | – | source |
| gfx1151 | Llama 3.1 8B | AWQ-INT4 | vLLM | 8.2 | – | – | – | 8192 | 2026-10-13 | sourced | – | source |
| gfx1151 | Llama 3.1 8B | AWQ-INT4 | vLLM | 8.1 | – | – | – | 8192 | 2026-10-13 | sourced | – | source |
| M4 Max (128 GB) | Llama 3.1 8B | Q4_K_M | llama.cpp | 7.9 | – | – | – | 66560 | 2026-10-17 | sourced | – | source |
| gfx1151 | Llama 3.1 8B | AWQ-INT4 | vLLM | 7.7 | – | – | – | 128 | 2026-10-13 | sourced | contradiction | source |
| gfx1151 | Llama 3.1 8B | AWQ-INT4 | vLLM | 7.7 | – | – | – | 128 | 2026-10-13 | sourced | contradiction | source |
| gfx1151 | Llama 3.1 8B | AWQ-INT4 | vLLM | 7.2 | – | – | – | 8192 | 2026-10-13 | sourced | contradiction | source |
| gfx1151 | Llama 3.1 8B | AWQ-INT4 | vLLM | 7.1 | – | – | – | 8192 | 2026-10-13 | sourced | contradiction | source |
| gfx1151 | Llama 3.1 8B | AWQ-INT4 | vLLM | 6.4 | – | – | – | 8192 | 2026-10-13 | sourced | contradiction | source |
| gfx1151 | Llama 3.1 8B | AWQ-INT4 | vLLM | 6.4 | – | – | – | 8192 | 2026-10-13 | sourced | contradiction | source |
| gfx1151 | Llama 3.1 8B | AWQ-INT4 | vLLM | 6.0 | – | – | – | 8192 | 2026-10-13 | sourced | – | source |
| gfx1151 | Llama 3.1 8B | AWQ-INT4 | vLLM | 5.1 | – | – | – | 8192 | 2026-10-13 | sourced | contradiction | source |
| gfx1151 | Llama 3.1 8B | AWQ-INT4 | vLLM | 4.6 | – | – | – | 8192 | 2026-10-13 | sourced | contradiction | source |
| TU106 | Llama 3.1 8B | Q4_0 | llama.cpp (CUDA) | – | 2052 | – | – | 8000 | 2026-09 | community | – | source |
| TU106 | Llama 3.1 8B | Q4_0 | llama.cpp (CUDA) | – | 2127 | – | – | 8000 | 2026-09 | community | – | source |
| DGX Spark | Llama 3.1 8B | Q6_K | llama.cpp (Vulkan) | – | 2238.9 | – | – | – | 2026-10 | community | – | source |
| DGX Spark | Llama 3.1 8B | Q6_K | llama.cpp (Vulkan) | – | 2419.3 | – | – | – | 2026-10 | community | – | source |
| DGX Spark | Llama 3.1 8B | Q6_K | llama.cpp (Vulkan) | – | 2316.6 | – | – | – | 2026-10 | community | – | source |
| DGX Spark | Llama 3.1 8B | Q6_K | llama.cpp (Vulkan) | – | 2688.4 | – | – | – | 2026-10 | community | – | source |