framework
SGLang is an open-source inference-serving framework for large language models, optimized for high-throughput deployment via efficient KV-cache reuse.
No evidence-backed relationships are recorded.
Moonshot AI released open weights for Kimi K2.7-Code, a trillion-parameter coding model that activates 32 billion parameters per token. Its headline gains come from Moonshot's own benchmarks, so teams paying for proprietary agents have to measure it on their own code.
Reality
Moonshot published 2.8 trillion open weights. At four bits per parameter that is about 1.4TB resident before any cache, which rules out the eight-way H100 node most teams assume.
Reality
IBM's 3B, 8B and 30B dense models all get a thinking switch and native tool calling, but only the two larger ones get agentic RL, and the tuning mixture leans hard on software engineering.
Perspective Coverage
3 publishersReality
Liquid AI shipped a 279.5-million-parameter draft model for LFM2.5-VL-3B on September 24, reporting 2.30x to 3.13x faster decoding on an Apple M5 Max. Image encoding and prompt prefill are unchanged, and the company's tests did not cover quantized deployments.
Reality
NVIDIA's SWE-Serve scores the same 627 patches twice on 19 SGLang tasks, once with the live-serving tests and once without. The pass rate falls from 69.4% to 45.9%, and 242 of the 276 live tests came from SGLang itself.
Reality
DigitalOcean's serving guide puts the cost of a long request on two mechanics, the prefill pass and the KV cache, and says the real choice is which mix of caching, prefix reuse and retrieval fits the budget.
Reality
Halo adds expert and tensor parallelism to Hugging Face models and still saves SafeTensors that from_pretrained can load. Its best number, 9,009 tokens per second per GPU against TRL's 3,885, came from synthetic fixed-length sequences.
Reality
AWS counted 13 SageMaker inference launches so far in 2026. The one that changes production behaviour most is a prioritized list of up to five instance types, each allowed its own model optimization settings.
Reality
The designated successor to GenAI-Perf splits load generation from record processing so the benchmark client stops hitting Python's GIL. Teams holding a GenAI-Perf baseline inherit a port and a re-run.
Reality
The number came out of one harness on OpenRouter at temperature 0.9 with a 64,000-token output cap, and it holds for a bounded translation task while the same model sits mid-table on open-ended terminal work.
Reality
Periodic Labs has published the infrastructure account behind Neon, its trillion-parameter model, and the figures in it measure the training and serving harness down to a single 1.90-millisecond routing payload.
Reality
A September 3 playbook traces speculative decoding's draft architectures from EAGLE-3 to DFlash, and grounds the case in a 70B model that decodes at 15 to 20 tokens a second on eight H100s.
Reality
Ao Qu and collaborators open-sourced Reef on September 15th, an OpenAI-format inference server that stamps each response with a record ID, matches later feedback to that ID, and publishes a retrained artifact only if its evaluation stage accepts it.
Reality
SemiAnalysis says its AgentX benchmark clocked Nvidia's Vera Rubin NVL72 at up to seven times Blackwell's token throughput per megawatt on pre-release software, and puts the profit gain at over two times per gigawatt.
Reality
ROCm sits under PyTorch, vLLM and SGLang; Vulkan is what llama.cpp-class engines compile shaders against. The engine you already run narrows the choice to one, and the rest of the work is reading logs.
Reality
A custom SGLang framework feeds teacher outputs into the loss while training runs. The reported eightfold speedup comes from a stack of separate optimizations, and the named multipliers reach about 5.5x on their own.
Reality
Thinking Machines Lab argues that temperature-zero inference varies run to run not because GPUs are chaotic but because kernels change their reduction order with load, which makes bit-identical output an engineering target.
Reality
Alibaba's 27B model fits a 32GB card with 15GB to spare. Tom's Hardware still had to pick between llama.cpp's full 262K window at half-hour prefill and a supported vLLM deployment capped at 32K.
Reality
Qwen3.8-Flash-Next puts 36 Gated DeltaNet layers and 12 sparse-attention layers on Hugging Face, which means the retrieval budget Qwen4 will inherit is something you can measure against your own traces now.
Perspective Coverage
5 publishersReality
A cache hit answers 13 to 31 percent faster than a miss, and an ICML 2025 audit found seven of 17 live providers sharing that cache across users, though the one demonstrated 100 percent reconstruction ran against self-hosted vLLM and SGLang.
Reality
Leadership · September 5, 2026 · 1 publisher
Build · August 31, 2026 · 2 publishers
Leadership · August 30, 2026 · 1 publisher
Build · August 29, 2026 · 1 publisher
Build · August 28, 2026 · 1 publisher
Build · August 28, 2026 · 1 publisher
Build · August 27, 2026 · 1 publisher
Leadership · August 27, 2026 · 1 publisher
Build · August 25, 2026 · 1 publisher
Build · August 25, 2026 · 1 publisher
Leadership · August 25, 2026 · 1 publisher
Build · August 24, 2026 · 1 publisher
Build · August 24, 2026 · 1 publisher
Build · August 23, 2026 · 1 publisher
Build · August 22, 2026 · 1 publisher
Build · August 21, 2026 · 1 publisher
Science · August 20, 2026 · 1 publisher
Build · August 19, 2026 · 1 publisher
Build · August 18, 2026 · 1 publisher
Build · August 18, 2026 · 1 publisher
Build · August 14, 2026 · 1 publisher