Loading today’s stories
Build1 distinct publisher3 min readUpdated
A week-long failure log on two RTX 3090s under WSL2 lands on one config at 170-210 tok/s. Everything before it died in dependency resolution, not in the math.
The Engineer · Build desk
Compiled by The EngineerSomething wrong?How this is made
A developer published a full week's failure log from putting Qwen3.8-27B onto a dual RTX 3090 box with no NVLink, PCIe Gen4, on Windows 10 plus WSL2 (Ubuntu 24.04) [1]. The interesting part is not the 170-210 tok/s the final config reached on code and JSON output [2], it is that almost every dead end was in the serving stack, the checkpoint, or the CUDA toolchain, and not in the model or the GPUs.
vLLM went first. Version 0.25.1 in Docker loaded and ran, but the model's built-in multi-token prediction head bought nothing across two GPUs without NVLink: 56 tok/s with speculation against 58 without, and first-token latency rising from 8.7s to 15s [3]. That is a 2 tok/s regression paid for with 6.3 extra seconds of TTFT [1]. Version 0.27.1 never initialised at all: nvidia-cuda-nvrtc had installed as a 0.0.0a0 placeholder so libnvrtc.so.13 was missing, and after a force reinstall flashinfer's sampling JIT demanded a CUDA-13-only nvcc flag against a local 12.8 toolchain [4]. The author's verdict is to skip vLLM 0.27+ for this model on 3090s [5].
SGLang was worse, hanging during weight loading with the port never listening [6]. Dozens of parameter combinations changed nothing; comparing keys in model.safetensors.index.json showed the checkpoint was half-quantized, with the MTP head and some linear-attention layers stored as plain `weight` rather than `weight_packed`, which is why the equivalent Qwen3.6 checkpoint loads and this one dies [7]. Three flags plus a source patch were needed to get past the no-NVLink and WSL failure modes at all [8][9], and the result ran at roughly 10 tok/s [10].
The throughput came from abandoning the quantized MTP head. Because the INT4 quantisation had also quantised a head that ships in BF16, acceptance sat at 1.07-1.27, which the author frames as a quantisation problem rather than a tuning one [12]. An independent 1.4B draft model proposing 7-token blocks reached acceptance of 3.7-4.2, around four tokens per verification pass [11].
Two hard limits are worth writing down. GDN kernels need CUDA 13 to compile, where 12.8 hangs on load and 13.3 fails to build, leaving 13.0 as the only working version [13]. And flashinfer's GDN kernels require SM90+, so on SM86 silicon only the Triton linear-attention backend is available [14]; `--enable-torch-compile` also crashes on GDN [15].
Two officially recommended settings cost real throughput here. The `extra_buffer` mamba radix cache strategy took decode from 62 to 30 tok/s, roughly 52 percent, as a permanent charge against a rare failure [16][2]. A 2048 chunked-prefill size, sensible for concurrency, quadrupled prefill iterations on single-request 50-90K contexts [17]. The most banal finding is the most reusable: a dropped trailing backslash made bash truncate the launch command and silently discard every later argument, booting the server on defaults [18].
These are self-reported numbers from one machine, translated with LLM help, and unverified [22]. Watch whether vLLM 0.27+ stops requiring a matching local nvcc, and whether quantisers start shipping W4A16 checkpoints that leave the MTP head in BF16, which is the difference between speculation working and needing a second model in VRAM.
Follow any of these and your For You feed starts watching them — no settings page required.
Ranked by verification strength, evidence, and original report placement.
The author spent a full week getting Qwen3.8-27B (hybrid GDN architecture, 48 linear-attention plus 16 full-attention layers, built-in MTP head) running on a dual RTX 3090 (24GB x 2) box with no NVLink, PCIe Gen4, Windows 10 plus WSL2 (Ubuntu-24.04).
The configuration the author finally arrived at delivered 170-210 tok/s on code and JSON workloads.
vLLM 0.25.1 (Docker) loads, but MTP speculation gives zero benefit on two GPUs without NVLink (56 vs 58 tok/s) and pushes first-token latency from 8.7s to 15s.
vLLM 0.27.1 hits a missing libnvrtc.so.13 because nvidia-cuda-nvrtc installed as a 0.0.0a0 placeholder package; after force-reinstalling, flashinfer's sampling JIT demands the CUDA-13-only --host-stub-linkage-explicit while local nvcc is 12.8, and engine init crashes every time.
The author's verdict: skip vLLM 0.27+ for Qwen3.8 on 3090s; the old Docker 0.25.1 works but gives no speedup.
Switching to SGLang was worse: the model froze during weight loading, the process hung and the port never listened.
Evidence-backed comparisons of source perspectives and observed adoption signals. Read the methodology
Which Builder, Operator, and Investor concerns the observed source mix emphasized—not a truth score.
Evidence, demonstrated adoption, hype gap, incentives, and confidence are assessed independently, each on its own current evidence. How these are measured.
Detailed but single-source and unreproduced
The account is unusually specific for a personal post: named package versions, error strings, tensor-key differences, flag-level fixes and a complete launch script, which makes most failure claims checkable in principle. But every datapoint comes from one author on one machine, with no independent reproduction, no upstream issue references, and no standardised benchmark methodology behind the throughput and acceptance figures. The author also discloses LLM-assisted translation, which does not affect the measurements but leaves wording precision partly mediated.
One documented hobbyist deployment
Observable adoption is a single self-hosted deployment plus its self-measured benchmarks. The post shows real use of SGLang, DSpark speculative decoding and a community AWQ-INT4 checkpoint in a working configuration, which is more than a demo, but there is no evidence of anyone else running this stack, no download or usage figures, and no organisational deployment.
Deflationary framing, slightly overreaching generalisation
Framing runs against hype: the piece is titled and structured as a pitfall log, reports its own dead ends (10 tok/s, no MTP benefit, a hung SGLang load), and contradicts official parameter recommendations with measured penalties. The mild overreach is generalising one machine's results into stack-wide verdicts and the unsupported claim that silent command truncation explains 80% of 'my parameters don't work' reports, but the dominant direction is understatement relative to the practical value of the documented failures.
Individual author, no disclosed vendor stake
The post is self-published by an individual on a developer blogging platform, with no vendor sponsorship, product, pricing or affiliate content, and no framework it appears to represent; it criticises vLLM, SGLang defaults and a community checkpoint alike. Residual incentive is the ordinary one of platform visibility for a long how-I-fixed-it post, plus a disclosed LLM translation step that shapes presentation. Nothing in the material indicates commercial pressure on the findings.
Credible mechanics, weak external verification
The mechanism-level claims (SM86 versus SM90+ kernels, cu-version toolchain matching, bash truncation semantics, quantized MTP head hurting acceptance) are internally coherent and specific enough to be trusted as one operator's experience. Confidence is capped by the absence of any second source, the n=1 deployment, and unstandardised performance numbers, so the failure modes are more credible than the exact throughput figures or the stack-wide verdicts drawn from them.
build
Inco AI's DFlash 2: 21% longer accepted drafts for 1.3% latency and 18.5M parameters1 distinct publisher
build
Qwen3.8's 27B dense checkpoint is the one operators can actually host1 distinct publisher
build
1,500 submissions in 14 days: what a 12th-place GPU kernel says about agent loops1 distinct publisher
build
Your vLLM Manifest Would Boot SGLang Too, And That Is the Problem1 distinct publisher
Distinct publishers with included, body-backed reporting in this cluster.
dev.to
1 article · August 18, 2026