v1.27.0

Dragonfly v1.27.0 - a Valentine Edition 💘

This marks our first release for the year, featuring numerous stability fixes and performance enhancements.

Important features and changes:

1. Fixes around JSON+SEARCH: #4232 #4314
2. Support for memcached meta protocol #4362 #4398 #4366
3. Fixes in tiering code #4395 #4388 #4416
4. Added LMPOP #4504 and ZMPOP #4385

What's Changed

• fix: test_network_disconnect_during_migration by @BorysTheDev in #4345
• refactor(rdb_saver): Add SnapshotDataConsumer to SliceSnapshot by @BagritsevichStepan in #4287
• chore(rax_tree): Introduce raxFreeWithCallback call in RaxTreeMap destructor by @BagritsevichStepan in #4255
• refactor(search_family): Add Aggregator class by @BagritsevichStepan in #4290
• fix: avoid on stack allocation of lz4 compression context by @romange in #4322
• chore: update helio and improve our stack overflow resiliency by @romange in #4349
• fix: configure jsonconst double parser to use std::from_chars by @romange in #4360
• fix(search_family): Support boolean and nullable types in indexes by @BagritsevichStepan in #4314
• fix(search_family): Support multiple fields in SORTBY option in the FT.AGGREGATE command. SECOND PR by @BagritsevichStepan in #4232
• test: skip test_network_disconnect_during_migration by @BorysTheDev in #4359
• fix(json_family): Remove std::endl during logging in ParseJsonPath method by @BagritsevichStepan in #4363
• feat: add support for meta memcache commands by @romange in #4362
• chore(test): disable failing test untill fixed by @adiholden in #4367
• chore(deps): bump the actions group across 1 directory with 2 updates by @dependabot in #4361
• feat: support memcache meta responses by @romange in #4366
• fix(pytest): fix failing test test_replication_timeout_on_full_sync by @adiholden in #4370
• chore(json_family): Remove redundant JSON value copying by @BagritsevichStepan in #4375
• feat(rdb_saver): Support big value serialization for stream by @BagritsevichStepan in #4376
• chore(snapshot): Small cleanup in Snapshot code by @BagritsevichStepan in #4377
• fix: stream memory counting durin snapshot loading by @BorysTheDev in #4346
• fix: return value for DflyMigrateAck by @BorysTheDev in #4379
• chore(server): cleanup replication shard sync execution by @adiholden in #4374
• test: fix test_network_disconnect_during_migration by @BorysTheDev in #4378
• fix(snapshot_test): Fix test_big_value_serialization_memory_limit after adding streams support by @BagritsevichStepan in #4383
• fix: ExternalAllocator::Free with large sizes by @romange in #4388
• fix(server): debug populate consume less memory by @adiholden in #4384
• refactor: slot_set don't use stack memory anymore by @BorysTheDev in #4386
• feat:Adding support for ZMPOP command by @guyzilla in #4385
• fix(replication): do not log to journal on callback fail by @adiholden in #4392
• chore: minor refactorings around dense_set deletions by @romange in #4390
• feat(server): refactor allow preempt on journal record by @adiholden in #4393
• fix: allow cluster node load snapshot bigger than maxmemory by @BorysTheDev in #4394
• fix: properly clear tiered stashes upon expirations by @romange in #4395
• feat: support deletions with meta protocol by @romange in #4398
• fix(tests): check cluster big snapshot in unit test by @adiholden in #4403
• chore(pytest): add timeout per test by @adiholden in #4404
• fix: provide resp3 option to CapturingReplyBuilder by @BorysTheDev in #4400
• chore: deprecate unneeded runtime flags by @romange in #4405
• feat: Yield inside huge values migration serialization by @chakaz in #4197
• chore(pytest): add check for rss grow in replicaiton big values by @adiholden in #4406
• feat: allow querying of json objects stored as strings by @romange in #4399
• test: move ReplicaofRejectOnLoad test from pytest into unit tests by @BorysTheDev in #4410
• chore: Increase migration wait time in test_cluster_migration_huge_container by @chakaz in #4409
• chore: Introduce small buffer in redis parser by @romange in #4076
• fix: invalid iterator case in Qlist::Erase by @romange in #4414
• test: add test migration in cache mode by @BorysTheDev in #4413
• fix: ExtentTree use of invalidated iterator by @kostasrim in #4416
• chore: fix build on FreeBSD by @chakaz in #4412
• chore: reduce number of logs in cluster tests by @BorysTheDev in #4419
• fix: geosearch and georadius response format by @kostasrim in #4420
• chore: split geo and zset families by @kostasrim in #4421
• fix: switch to SHUTTING_DOWN state unconditionally by @romange in #4408
• chore: split zset and geo family tests by @kostasrim in #4425
• refactor: remove transaction lib on cluster code dependency by @BorysTheDev in #4417
• chore: Increase migration wait time in test_cluster_migration_huge_container by @chakaz in #4427
• chore: update jsoncons version to 0.178 by @romange in #4368
• fix: improve error propagation with RESTORE commands by @romange in #4428
• fix: show managed stuff on privileged port by @romange in #4433
• chore: dragonfly connection refactorings by @romange in #4434
• fix: reply_builder should properly serialize bulk strings endings. by @romange in #4441
• chore: pull latest helio by @romange in #4446
• chore: Remove template Callbacks by @BagritsevichStepan in #4431
• fix: resp inline parsing correctly resets itself by @romange in #4458
• fix(ci): sanitizers build by @kostasrim in #4457
• chore: Add stats print for slot migrations by @chakaz in #4456
• fix: slot calculation during transaction squashing by @BorysTheDev in #4460
• fix(sanitizers): Set stack size to 64 KB during tests for Sanitizers by @BagritsevichStepan in #4454
• Fix test hypothesis by @cunla in #4380
• feat(dev): allow monitoring a valkey server on localhost by @romange in #4467
• chore: add Dash::Prefetch function by @romange in #4476
• chore: dfly_bench now accepts data and score template names by @romange in #4477
• chore: replace deprecated HELIO_STACK_CHECK by @kostasrim in #4465
• chore: Fix all clang build warnings by @chakaz in #4475
• fix: hiredis requires df to report version >7.4 by @kostasrim in #4474
• chore: make per-thread QueueBackpressure objects global by @romange in #4482
• chore: OpManagerTest fixes by @romange in #4480
• chore: run replication reg tests with epoll by @kostasrim in #4426
• chore: remove pubsub semantics from Connection::WeakRef by @romange in #4483
• Fix typo in outgoing_slot_migration.cc by @moredure in #4484
• feat: allow finish and start migration in the same config by @BorysTheDev in #4486
• feat: add support for noreply and great options by @romange in #4485
• feat(json_family): Add json_nesting_depth_limit flag by @BagritsevichStepan in #4444
• fix (stream): XRANGE incorrectly interprets the end parameter by @adiholden in #4443
• feat: implement connection timeout functionality by @romange in #4407
• feat: add 'testing_time' limit option to dfly_bench by @romange in #4487
• fix: test_acl_cat_commands_multi_exec_squash by @kostasrim in #4492
• chore: add sanitizers to ci workflow by @kostasrim in #4462
• chore: improve test_timeout robustness by @romange in #4494
• fix(cluster): crash in cluster migration by @adiholden in #4495
• skip test_migration_one_after_another on epoll by @adiholden in #4499
• fix: several bugs by @romange in #4500
• chore: new docker build pipeline by @romange in #4503
• fix: rpm package setup by @romange in #4506
• fix: Do not bump elements during RDB load by @chakaz in #4507
• fix cluster: migration crash fix by @adiholden in #4508
• chore: split epoll/uring reg test flows by @kostasrim in #4512
• fix(test): fix rdb random fail on OOM by @adiholden in #4509
• chore: add defrag for StringSet by @kostasrim in #4308
• chore: add benchmarking of ScanOpts::Matches by @romange in #4511
• fix: crash for delconsumer during stream reading by @BorysTheDev in #4513
• chore(deps): bump github/codeql-action from 3.28.0 to 3.28.5 in the actions group across 1 directory by @dependabot in #4514
• chore: remove unused code by @romange in #4516
• fix: LMPOP implementation was added by @vyavdoshenko in #4504
• chore: add reflex matcher to the benchmarks by @romange in #4520
• chore: introduce GlobMatcher by @romange in #4521
• chore: Add Lua force atomicity flag by @chakaz in #4523
• chore: add more logs to the op_manager code by @romange in #4527
• fix(search_family): Add options test for the FT.SEARCH command by @BagritsevichStepan in #4478
• chore(ci): report proactor kind on reg test failure by @kostasrim in #4531
• chore: update release build for arm by @BorysTheDev in #4519
• chore: refactoring around glob matching by @romange in #4540
• chore: clean up of deprecated flags by @romange in #4545
• chore: move QList::Node definition into dragonfly codebase by @romange in #4547
• chore: Add benchmarks of qlist compression paths by @romange in #4543
• chore: remove clang install step from ci workflow by @kostasrim in #4550
• fix: test_snapshoting_during_migration by @BorysTheDev in #4555
• fix: Bug GEQ range query bug by @romange in #4557
• fix: consume parser error on dfly load by @kostasrim in #4556
• chore: GlobMatcher uses now reflex::Matcher regex engine by @romange in #4528
• chore: Add FakeRedis capture/compare to more cluster tests and seed during migration by @chakaz in #4542
• Fix BullMQ tests workflow in GitHub actions by @chakaz in #4565
• fix(stream_family): Fix journaling in the XADD and XTRIM commands by @BagritsevichStepan in #4448
• refactor: rename Context into ExecutionState by @BorysTheDev in #4562
• chore: pull helio epoll fix and tests by @kostasrim in #4559
• fix server: fix deadlock blmove does not conclude on error by @adiholden in #4561
• feat: introduce more options for traffic logger by @romange in #4571
• fix: a regex bug with escaping inside of character classes by @romange in #4575
• docs: the broken link was fixed by @vyavdoshenko in #4579
• fix: a rare deadlock with blmove and multi-db by @romange in #4568
• server: enable big value serialization by default by @adiholden in #4581
• chore: minor simplification around WaitWatch by @romange in #4580
• fix: ascii logo was added by @vyavdoshenko in #4524
• fix: migration finalization process by @BorysTheDev in #4576
• fix: incorrect passing of db_index in EVAL transactions by @romange in #4586
• test: add ClusterEvalCrossslot by @BorysTheDev in #4587
• fix: huge TTL in RESTORE cmd is rounded to kMaxExpireDeadlineMs by @BorysTheDev in #4589

Huge thanks to all the contributors! ❤️

New Contributors

• @guyzilla made their first contribution in #4385
• @moredure made their first contribution in #4484
• @vyavdoshenko made their first contribution in #4504

Full Changelog: v1.26.0...v1.27.0