Smaller, faster, safer: running Kimi and GLM at scale
Points and comments are a snapshot, not live.
Cloudflare details KV cache quantization, weight compression, and cache integrity for large models.
Cloudflare's Workers AI runs Kimi K2.6 and GLM 5.2 on GPUs. To fit these mixture-of-experts models into memory, they quantize the KV cache from BF16 to FP8, doubling context capacity to ~1.37 million tokens with no accuracy loss across benchmarks like GSM8K and MMLU. For GLM, they compress weights from FP8 to INT4, shrinking the checkpoint from 705 GB to 421 GB and boosting decode speed up to 55%. They also implement a KV cache integrity check with tags, costing under 1% on throughput and latency. These optimizations are deployed via SGLang, with different precision used for prefill (FP8) and decode (INT4).
What commenters are saying
Many commenters criticized the article's prose as AI-generated slop, with some saying Cloudflare blogs are no longer meant for human reading. Some praised the transparency around KV cache quantization but wished for more detailed benchmarks, noting the absence of coding benchmarks and the need to test sensitivity across different model families. Others questioned whether quantizing without explicit warning on model pages is fraudulent, while a few defended the practice if quality is preserved. Practical discussion touched on job titles for inference optimization and pricing availability.