Topic
Natural-language-to-code assistants that emit client code for a specific in-house API, and the failure mode of confident, well-formed, non-functional output.
Ten LLMs' Blender 5.0 scripts ran only 70% of the time when a Kaggle benchmark executed them in 5.0, against 91% for scripts targeting 3.6. Renamed and removed APIs look like valid code, so the benchmark grades each answer in the exact build the prompt named.
Reality
On July 22 a model wrote a forty-character API key into a page's JavaScript and declared the API not secret. GoodBarber now has the model state only how a key is sent and lets code default every key to secret.
Reality
An InfoQ article argues that model hallucination on a home-grown notation is a corpus-frequency problem, and its fix puts the domain inside a host language's type system so an invalid domain state fails to compile.
Reality
An intern's slice of a legacy Java modernization project found the bottleneck is not generation. It is having a test that can tell you when the model is actually finished.
Reality
A European weather archive retrained the model behind its code assistant and found the training set had taught it to call a variable the API never served. Drift does not stay in the docs.
Reality