Skip to content

Default RocksDB WAL log size settings #837

@iadmytro

Description

@iadmytro

With current settings rocksdb produces a lot of WAL log files that are not removed in timely manner. It leads to substantial memory consumption.
Behavior governed by two settings set_max_total_wal_size and set_wal_ttl_seconds

  1. If both set to 0, logs will be deleted asap and will not get into the archive.
  2. If wal_ttl_seconds is 0 and wal_size_limit_mb is not 0, WAL files will be checked every 10 min and if total size is greater then wal_size_limit_mb, they will be deleted starting with the earliest until size_limit is met. All empty files will be deleted.
  3. If wal_ttl_seconds is not 0 and wall_size_limit_mb is 0, then WAL files will be checked every wal_ttl_seconds / 2 and those that are older than wal_ttl_seconds will be deleted.
  4. If both are not 0, WAL files will be checked every 10 min and both checks will be performed with ttl being first.

Source

By default it's option 1

set_max_total_wal_size(0)
set_wal_ttl_seconds(0)

Suggestion.
Switch to option 4

set_max_total_wal_size(some_value)
set_wal_ttl_seconds(some_value)

which gives more predictable cleanup, it will happen according to set_wal_ttl_seconds value, and potentially smaller memory consumption, garbage will not accumulate.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions