Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

237 points · 180 comments on HN · read original →

Points and comments are a snapshot, not live.

A single systemd-journald log line can trigger 49KB+ of disk writes on ext4 and 110KB+ on btrfs.

The issue reports that systemd-journald causes excessive disk I/O, with a single log entry resulting in 49KB+ writes on ext4 and 110KB+ on btrfs. A VM writing 2 lines of logs per second produces ~50 IOPS. The reporter links this to a previously closed issue (#15292) and criticizes journald's format as inefficient, noting files are multiple times larger than the actual log content. They also mention corruption on unclean reboots. Suggestions to disable compression made no meaningful difference, as short log entries may not trigger compression.

One experienced contributor notes the on-disk format disperses information at small, discontiguous offsets, causing write amplification when combined with mmap'd I/O, as the kernel must write back entire blocks. They had previously focused on read performance and stability, not the on-disk structure.

What commenters are saying

Commenters widely concur that journald's disk I/O and format are problematic. Several report high write amplification, especially on COW filesystems like btrfs. One commenter notes mmap'd writes aggravate the issue, as the OS can commit changes asynchronously without control over sync order. Others advocate replacing journald entirely: using it only as a router, switching to syslog-ng, or adopting Devuan (which omits systemd). Several share anecdotes of systemd's log format being slower than grepping gzipped text files.

One contributor who improved journald's read performance and stability states the on-disk format lacked consideration for block-oriented storage, with small datums written at discontiguous offsets forcing full-block writeback. Another points out journald has poor scaling with many journal files, though a fix improved this from 'nearly unusable to slow-as-usual.'