Inference

I run Qwen 3.5 and 3.6 (35B MoE, 3B active parameters) in production across three nodes — two DGX Spark (FP8, vLLM) and one RTX 5090 (Q4, llama.cpp). After 100+ benchmark scenarios and thousands of inference calls, three problems dominated my debugging time: Quantization loss is not uniform — MoE models at Q4 lose 16% on CJK tasks vLLM is non-deterministic under speculative decoding — identical prompts produce different outputs Thinking tokens consume 60–90% of the budget on tasks where they provide zero benefit None of these show up in standard benchmarks. All of them break production workflows. ...

I run Qwen 3.6 35B on three machines. The RTX 5090 generates at 204 tok/s. The DGX Spark pair generates at 65 tok/s. By every benchmark leaderboard metric, the 5090 is 3x faster. But for multi-step coding tasks with thinking enabled, the DGX pair completes the job faster. And for single-turn questions, the 5090 delivers the answer in under 2 seconds while the DGX takes 8–12 seconds. tok/s alone told me nothing useful about actual user experience. Here’s what I learned building benchmarks for all three nodes. ...