KeyValueStore Tools 0.2 Adds a Cluster Rebalancing Planner and Sizes Redis and Valkey Streams
Adding a primary to a Redis or Valkey cluster is the easy part. Then it sits there with no slots, and somebody has to decide which of the other primaries’ 16,384 slots go to it, move them, and not lose a key on the way. redis-cli --cluster rebalance will do it, but it won’t show you the plan first, it can’t balance by bytes, and it gives you nothing to read before it starts.
Version 0.2 of the tools adds a fifteenth one for that job, the Cluster Rebalancing Planner. Paste the output of CLUSTER NODES. It works out a target for each primary, by slot count or by the bytes your keys take, with weights if one machine should carry twice the share and a drain button for one you want to empty. You see slots or bytes per primary before and after, every move as a slot range, and a shell script of the standard SETSLOT and MIGRATE commands. The page never connects to your cluster. The script does, when you run it.
We ran it on real clusters of seven nodes on Redis 6.2, 7.4 and 8.10 and Valkey 8.0 and 9.1, each holding 4,000 keys, the biggest under a few shared hash tags. Three plans in a row on each: fill the empty primary, balance by MEMORY USAGE, drain one of the originals. Every time, every node ended up agreeing on exactly the owners in the plan, and all 4,000 keys read back with their values. Balancing by bytes took the fullest primary from 26 percent over its share to within 0.1 percent.
What the Review Caught
Before a release, an independent review goes through the code. This one found a nasty one. A key with an empty name, which Redis allows, comes back from CLUSTER GETKEYSINSLOT as an empty line. The first version of the script took that to mean the slot was empty, handed the slot over, and the old owner dropped the key. The script still said “Done”.
The script now moves keys until the old owner counts none left in the slot, checks every slot of a range is empty before it changes owner, and stops the moment a round of MIGRATE moves nothing. On a real cluster, the empty name and a key with a newline in it now both stop the script at their slot, with the key still where it was. A second review round found nothing to fix.
Real clusters taught it one more thing. A primary that gives away its last slot turns into a replica of the one that took it, and refuses the last commands. Redis, Valkey 8 and Valkey 9 refuse in three different ways. The script accepts a refusal only once that primary really holds nothing.
Streams and Hash Field TTLs in the Memory Calculator
The Memory Calculator now covers streams, with consumer groups, consumers and pending entries, on all 15 versions. A million entries of three 30-byte values take about 104 MB on every version. Pending entries are where it gets expensive: 100,000 of them in one group add 24 MB on Redis 8.10 and 12 MB on Valkey 9.1.
It also covers hash fields with a TTL of their own, the HEXPIRE family, on Redis 7.4 and later and Valkey 9.0 and later. Here the two servers part ways. Give 5 of 20 fields a TTL in 100,000 small hashes and Redis 8.10 grows by about 3 percent, keeping each hash packed. Valkey 9.1 turns each hash into a hash table and grows by about 60 percent.
Both were checked the usual way, against used_memory on the 15 built servers: 2,205 stream cases and 945 field-TTL cases, which the calculator matches to the byte, apart from a few Valkey cases where entries are placed by memory address and only the expected value can be given.
Keeping Up With Patch Releases
Redis and Valkey ship patches often, and the recorded answers behind three of the tools come from pinned versions. A new monthly job builds the newest patch of every tracked line, records it, replays it through the tools and commits a report. It never changes a tool by itself; when a patch behaves differently, the report says where.
The code and the 0.2.0 package are on GitHub, and every tool is on the tools page.