Build1 publisher3 min readPublished
Kestra 2.0 takes the database credential out of the worker
Four years after shipping an orchestrator that wanted a Kafka cluster and an Elasticsearch cluster to schedule a job, Kestra has rebuilt the engine around that complaint, keeping the Apache 2.0 licence in place.
The Engineer · Build desk

What happened
- Kestra 2.0 shipped this month with most of the engine replaced, four years and one engine rewrite after the February 2022 post that introduced the project on Kafka and Elasticsearch.
- The engine now separates a control plane holding the executor, scheduler, webserver, indexer and a new worker controller, none of which run user code, from workers that form the data plane.
- Each worker opens one persistent gRPC stream to the worker controller and always initiates it, with jobs travelling out and results, logs and metrics coming back on that same stream.
- 2.0 ships one executor, one scheduler and one worker, lets you pick the queue and the repository independently, and deletes the Kafka Streams engine outright.
- Loop replaces ForEach and ForEachItem, running each iteration as its own sub-execution so a runaway loop can no longer take the executor down, which the post says had happened.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- capability Tasks can now run inside networks that will never route to the control plane's database, which removes the excuse for the unorchestrated cron box that teams settled for instead.
- cost Collapsing two engine implementations into one takes a maintenance bill off Kestra, and it also stops operators having to work out which of two code paths their behaviour came from.
- decision Anyone holding 1.x flows built on ForEach and ForEachItem has to price a syntax migration the post leaves unspecified before scheduling the upgrade.
- constraint The worker split only pays if the direction your security team blocked was inbound; egress-restricted networks and single-region installs whose workers already sit beside the database get nothing from it.
In 1.x every worker needed a connection to the central database, and that single design decision dictated where you were allowed to run tasks [10]. When a security team would not open a route from one network to the Postgres in another, the options were a full Kestra deployment per site or scripts left on a local box under cron with no orchestration at all; the author of the dev.to post writes that he watched teams pick the second more than once [11].
Because the worker now initiates, it holds no database credentials and nothing has to connect inbound to it, so it can sit in another cloud, another region, or a network that only permits outbound traffic [15]. Each worker can be required to present a client certificate or a JWT before any job is dispatched to it, over a channel that can carry TLS [14]. The trust boundary moves from a database ACL to a per-worker identity check, which is the version a security review can approve without a network exception. The post leaves failure behaviour largely undocumented: what happens to in-flight jobs when the stream drops, whether 1.x ForEach syntax still parses, and which queue implementations ship in 2.0 [25].
The deployment saving is older than this release. The 2022 architecture used Kafka as both queue and database with Elasticsearch as the repository behind the UI [1], and it ran 350,000 executions a month at Leroy Merlin [2]. Divide by 30 days and then by 1,440 minutes and that is about 8 executions a minute [3], for two clusters. The post says the most common reply was that requiring both to schedule a job was absurd, and calls that fair [4]. A JDBC backend followed four months later, with a single Postgres or MySQL holding queue and state [5], which dates the cheap deployment story to roughly June 2022 [6], about three years and three months before the September 2025 1.0 long-term-support release [7][9].
The engine work in 2.0 settles a maintenance bill run up since that 2022 split, rather than delivering a first cheapening. In 1.x the queue and the repository came as a fixed pair, JDBC for both or Kafka plus Elasticsearch, which meant two engine implementations and every bug fixed twice [16]. That is the debt being paid. The licence stays where it was, with the post's headline still saying Apache 2.0 [23]. All of it is the vendor's own account, published on dev.to, and it carries no throughput figures for 2.0 [26]. It does retire the word "infinitely" from the 2022 title [24], which is more than most release copy gives back.
The quota and Copilot defaults matter more for day-to-day operations than the new canvas does. Quotas cap executions per flow or namespace, so one team's runaway stops being everyone's incident [19]. Setting kestra.ai.enabled: false switches the Copilot off, after which no AI endpoint exists on the server, and the AI Agent task can point at Ollama on your own hardware [22]. Any flow can be exposed as a tool through MCP, and the resulting execution carries the same permissions as a person clicking Run, labelled system.from: mcp, with steps you cannot undo able to wait for a human [21]. Underneath, the post says it is still YAML in your Git repo with no code that imports Kestra [28], which keeps the exit cost roughly where it was.
For the topology change to be worth an upgrade in your shop, the blocked direction has to have been inbound. A network that also refuses outbound gRPC to your control plane gains nothing from the split, and neither does a single-region install whose workers already sit next to the database.
What to watch
- Whether 2.0 gets a long-term-support designation as 1.0 did, which the post does not state.
- Which queue and repository implementations actually ship in 2.0 now that the pair is decoupled and the Kafka Streams engine is gone.
- Documentation of worker stream behaviour under partition, specifically what happens to in-flight jobs when the gRPC connection drops.