Llama.cpp
High-performance LLM inference service based on llama.cpp, supporting GGUF models and multimodal vision models
The main goal of llama.cpp is to implement LLM reasoning on a wide range of hardware (on-premises and in the cloud) with minimal settings and the most advanced performance. performance. Ordinary C/C++ implementation without any dependencies Apple chips are first-class citizens-optimized through the ARM NEON, Accelerate and Metal frameworks AVX, AVX2, AVX512 and AMX supporting x86 architecture RVV, ZVFH, ZFH, ZICBOP and ZIHINTPAUSE supporting RISC-V architecture 1.5 Bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, and 8-bit integer quantization to speed up reasoning and reduce memory usage Running LLM's custom CUDA core on NVIDIA GPUs (support for AMD GPUs via HIP, support for Moore Threads GPUs via MUSA Support Vulkan and SYCL backend CPU+GPU hybrid reasoning can partially accelerate models larger than the total VRAM capacity. By default, you can download the gguf file directly to the models in the data directory. You can also use model.ini to perform refined control. You can refer to my configuration. # ───────────────────────────────────────────── # llama.cpp models.ini -Global Default # ───────────────────────────────────────────── [*] ctx-size = 4096 fit = off flash-attn = true #Plain text model acceleration; automatically skipped when multimodal is not supported # ───────────────────────────────────────────── # MiniCPM-V 4.6 Q8_0 -Multimodal, with the heaviest nuclear pressure #Strategy: compress KV cache + extremely small ubatch to prevent bandwidth from being full # ───────────────────────────────────────────── [MiniCPM-V-4.6-Q8_0] model = /models/MiniCPM-V-4.6-Q8_0.gguf mmproj = /models/mmproj-MiniCPM-V-4.6-Q8_0.gguf n-gpu-layers = 99 kv-offload = true ctx-size = 2048 batch-size = 512 ubatch-size = 64 #Prevent the nuclear display from being stuck instantly during large image transmission cache-type-k = q8_0 # KV cache quantization, saving 30 - 40% video memory cache-type-v = q8_0 threads = 4 #Leave margin for GPU scheduling and don't run all over the physical cores jinja = true # ───────────────────────────────────────────── #Hy-MT2 1.8B Q4_K_M -Small pure text model, very light #Strategy: Full context + large ubatch to improve throughput # ───────────────────────────────────────────── [Hy-MT2-1.8B-Q4_K_M] model = /models/Hy-MT2-1.8B-Q4_K_M.gguf n-gpu-layers = 99 ctx-size = 8192 # 1.8B is very light and can be pulled directly without pressure batch-size = 512 ubatch-size = 256 #Small model amplifies ubatch for higher throughput cache-type-k = q4_0 #Extreme memory saving, negligible 1.8B accuracy loss cache-type-v = q4_0 flash-attn = true jinja = true # ───────────────────────────────────────────── # Gemma 4E2B Q4_K_M -Multimodal, 8K Dessert Spot #Policy: flash-attn is temporarily closed (mproj compatibility to be verified) # ───────────────────────────────────────────── [Gemma-4-E2B-Q4] model = /models/gemma-4-E2B-it-Q4_K_M.gguf mmproj = /models/mmproj-F16.gguf #Confirm that this file is a dedicated projector for Gemma 4️ n-gpu-layers = 99 kv-offload = true ctx-size = 8192 batch-size = 512 ubatch-size = 128 cache-type-k = q8_0 cache-type-v = q8_0 temp = 1.0 flash-attn = false # Gemma 4mproj will not be opened until compatibility is confirmed jinja = true
