jitLLM claims 90% of llama.cpp speed by compiling Java bytecode to CUDA kernels

On NVIDIA hardware with the newer toolchains, TornadoVM's JIT emits CUDA C and cuTile code at runtime. It can also bind existing CUDA libraries directly from Java ^[8]. It handles device transfers itself and reuses Java memory where it can ^[9]. According to the dev.to write-up, that removes most of the boilerplate a team would otherwise write by hand against the CUDA API ^[9].

We think the strongest design choice here is also the plainest one. The backend is picked at runtime, so the same Java code can target CUDA, OpenCL or SPIR-V, and Apple Silicon is in scope ^[10]. TornadoVM has been in development at Manchester for years ^[7]. jitLLM started as GPULlama3.java, a GPU-accelerated fork of the pure-Java Llama3.java, and it keeps a CPU path for machines without a GPU ^[11].

The speed claim spread after the project reached r/LocalLLaMA on October 8 ^[5]. The number comes from the project itself. It was shown on an RTX 5090 in the project's own recordings, and no independent benchmark has confirmed it ^[2]. The baseline is hard to beat. llama.cpp is hand-tuned C++ and CUDA with years of kernel optimization behind it ^[15], so a 10% gap ^[18] from compiled Java would be a strong result. Before the figure means anything for a given shop, it has to hold on smaller cards, on the quantization formats services actually run, and across the models jitLLM lists. The author of the dev.to analysis, who has not run jitLLM on their own hardware ^[14], wrote: "Treat it as promising until someone replicates it on a plain RTX 4060 with a Q4 model, which is what most of us actually run." ^[3]

That replication is cheap to set up. jitLLM reads standard GGUF files, the same format llama.cpp uses ^[11], so both engines can load one model file on one card. Wiring it into a service takes one dependency and one builder call: add `langchain4j-jitllm`, then construct a `JitLLMChatModel` with `onGPU(true)` ^[17]. Quarkus users get the model as a CDI bean through `quarkus-langchain4j-jitllm` ^[13].

In the dev.to write-up's account, the sidecar jitLLM is aimed at brings PyTorch and a CUDA toolkit with it, plus its own Dockerfile, its own set of CVEs and a REST call between two halves of one application ^[16]. jitLLM drops the Python process and ships no C++ code ^[1]. That describes the repository. On the host, the kernels still run through NVIDIA's CUDA stack, and jitLLM can call existing CUDA libraries from Java ^[8]. The write-up does not say which NVIDIA toolchain components a deployment still needs when kernels are generated as CUDA C at runtime ^[8].