LLM Inference

5 articles

AI Security • Nov 5, 2026

Fix a KV Cache Leak: 155 Responses Exposed Other Users' Data

Check your LLM server for a KV cache leak: compare pool size to context length and send a canary prompt. Cloudflare's GLM-5.3 bug exposed 155 responses.

9 min read

LLM Engineering • Aug 15, 2026

Qwen3.8 27B VRAM: how to fit 262K context in 16 GiB, not 64

Qwen3.8 27B VRAM math: 25.9 GiB of FP8 weights plus 16 GiB of KV cache at 262,144 tokens, not 64. The arithmetic, and where a 48 GB card breaks.

10 min read

LLM Engineering • Aug 11, 2026

Fix slow LLM inference in macOS VMs: 12.6 → 207 tok/s

LLM inference in macOS VMs collapses to 12.63 tok/s because the guest reports GPU family 5 and llama.cpp drops its matrix kernels. The check, and its limits.

9 min read

LLM Engineering • Aug 8, 2026

vLLM throughput tuning: configure these four flags, not a bigger GPU

vLLM throughput tuning starts with KV cache blocks, not a bigger GPU. The four flags that decide your tokens/sec, and the one that quietly backfires.

9 min read

LLM Engineering • Aug 4, 2026

Run 70B LLM on 4GB GPU: AirLLM's Real Tradeoff

Run 70B LLM on 4GB GPU hardware with AirLLM's layer-by-layer inference. The VRAM math is real — you just pay for it in disk bandwidth. The honest tradeoff.

11 min read