Skip to content

Build1 publisher2 min readPublished

ClickHouse's own log tables usually fill the disk on self-hosted Langfuse and SigNoz

ClickHouse's own system logs grow unchecked by default under self-hosted Langfuse and SigNoz, with one Langfuse text_log reaching 59.20 GiB. A dev.to guide confirms the cause with one read-only query and caps regrowth with per-log TTLs.

The Engineer · Build desk

Illustration accompanying ClickHouse's own log tables usually fill the disk on self-hosted Langfuse and SigNoz

What happened

  • A dev.to guide says most public reports of full disks on self-hosted Langfuse and SigNoz trace back to ClickHouse's own log tables, such as system.trace_log and system.text_log.
  • ClickHouse's documentation says of its system log tables: "By default, table growth is unlimited."
  • The guide's check is one read-only query on system.parts that ranks tables by bytes on disk, so ClickHouse's logs can be set against trace data.
  • By default ClickHouse refuses to drop or truncate any table larger than 50,000,000,000 bytes, failing with error code 359.
  • The lasting fix is a config.d file that sets a TTL on each log table, followed by a ClickHouse restart.

Compiled by The EngineerSomething wrong?How this is made

Why it matters

  • constraint Operators with the biggest logs cannot clear them with a plain TRUNCATE; the statement that would free the most space is the one ClickHouse blocks by default.
  • decision Langfuse operators can empty system.query_log but should keep it switched on, because Langfuse reads it to follow some of its own queries.
  • cost Applying the TTLs costs a second cleanup pass: the restart leaves *_log_0 and *_log_1 copies holding disk until someone drops them.
  • decision Langfuse installs can no longer upgrade ClickHouse first; the guide says to update Langfuse before moving ClickHouse to 26.8 or later.

ClickHouse writes its own diagnostics into tables in the system database: query_log, trace_log, text_log, metric_log and more [13]. Recent default configs set a TTL on only a few small logs. Since 25.9, processors_profile_log keeps 30 days [13]. The big ones get none [13]. In the reports the guide draws on, the stack people installed to keep their traces spends most of its disk on the database's notes about itself [1].

The check sums bytes_on_disk from system.parts over active parts, grouped by database and table, and returns the top 15 [3]. The guide runs it with `--readonly=1`, and in that session ClickHouse refuses any change [4]. I like that detail. It makes the query safe to paste into a production shell. Langfuse's compose file already sets CLICKHOUSE_USER and CLICKHOUSE_PASSWORD inside the container, and SigNoz's users.xml leaves the default user without a password [19]. To read the output, compare the system rows with your own data: default on Langfuse, the signoz_* databases on SigNoz [5].

The cause claim rests on "most public reports", in the guide's words, and the guide says ClickStack has the same problem [1]. The commands were tested on stock clickhouse/clickhouse-server images 24.8, 25.12 and 26.9, with the SigNoz commands on 25.5 and 25.12, using the ClickHouse service and config from the two projects' compose files [2]. For the diagnosis to carry over, your ClickHouse needs to sit close to those stock defaults. If your own tables top the list, the guide's next suspects are Docker's container logs, deleted rows and stuck parts [17].

Freeing the space has a trap. The drop guard sits at about 46.6 GiB and covers TRUNCATE as well as DROP [8]. The text_log in Langfuse discussion #15024 was 59.20 GiB [9]. It was about 12.6 GiB over the line [1]. On ClickHouse 23.12 and later, the guide lifts the limit for one statement [10]:

``` TRUNCATE TABLE system.text_log SETTINGS max_table_size_to_drop = 0; ```

The other route is touching /var/lib/clickhouse/flags/force_drop_table. ClickHouse deletes that flag after the first DROP or TRUNCATE that needs it, so a second oversized table needs a new flag [11]. I'd use the SETTINGS form, because its scope is the one statement it is written on. Either route removes only ClickHouse's diagnostics, not your traces [6]. No restart is needed [6].

Truncation is temporary. According to the guide, the logs start growing again right away [20]. The TTL config has one placement exception: for opentelemetry_span_log the TTL goes inside the <engine> element [15].

What to watch

  • Whether a future ClickHouse default config adds TTLs to trace_log and text_log, as 25.9 did for processors_profile_log.
  • Whether Langfuse or SigNoz ship system-log TTLs in their own compose files, removing the manual config.d step.
Loading claim ledger
Loading source directory links
Loading share composer
Loading topic controls
Loading related stories