- Allow replicas to track master's migrate task state
Previously, we only propagate import task info to replicas, but now we
also support propagating migrate task info, so the new master can
initiate slots trimming again if needed after failover, this can avoid
data redundancy.
- Prevent replicas from initiating slot trimming actively
Lack of data cleaning mechanism on source side, so we allow replicas to
continue pending slot trimming, but it is not good idea to let replicas
trim actively. As we introduce above feature, we can delete this logic
MSETEX doesn't properly check ACL key permissions for all keys - only
the first key is validated.
MSETEX arguments look like: MSETEX <numkeys> key1 val1 key2 val2 ... EX
seconds
Keys are at every 2nd position (step=2). When Redis extracts keys for
ACL checking, it calculates where the last key is:
last = first + numkeys - 1; => calculation ignores step
last = first + (numkeys-1) * step;
With 2 keys starting at position 2:
Bug: last = 2 + 2 - 1 = 3 → only checks position 2
Fix: last = 2 + (2-1)*2 = 4 → checks positions 2 and 4
Fixes#14657
**Disclaimer: this patch was created with the help of AI**
My experience with the Redis test not passing on older hardware didn't
stop just with the other PR opened with the same problem. There was
another deadlock happening when the test was writing a lot of commands
without reading it back, and the cause seems related to the fact that
such tests have something in common. They create a deferred client (that
does not read replies at all, if not asked to), flood the server with 1
million of requests without reading anything back. This results in a
networking issue where the TCP socket stops accepting more data, and the
test hangs forever.
To read those replies from time to time allows to run the test on such
older hardware.
Ping oranagra that introduced at least one of the bulk writes tests.
AFAIK there is no problem in the test, if we change it in this way,
since the slave buffer is going to be filled anyway. But better to be
sure that it was not intentional to write all those data without reading
back for some reason I can't see.
IMPORTANT NOTE: **I am NOT sure at all** that the TCP socket senses
congestion in one side and also stops the other side, but anyway this
fix works well and is likely a good idea in general. At the same time, I
doubt there is a pending bug in Redis that makes it hang if the output
buffer is too large, or we are flooding the system with too many
commands without reading anything back. So the actual cause remains
cloudy. I remember that Redis, when the output limit is reached, could
kill the client, and not lower the priority of command processing. Maybe
Oran knows more about this.
## LLM commit message.
The test "slave buffer are counted correctly" was hanging indefinitely
on slow machines. The test sends 1M pipelined commands without reading
responses, which triggers a TCP-level deadlock.
Root cause: When the test client sends commands without reading
responses:
1. Server processes commands and sends responses
2. Client's TCP receive buffer fills (client not reading)
3. Server's TCP send buffer fills
4. Packets get dropped due to buffer pressure
5. TCP congestion control interprets this as network congestion
6. cwnd (congestion window) drops to 1, RTO increases exponentially
7. After multiple backoffs, RTO reaches ~100 seconds
8. Connection becomes effectively frozen
This was confirmed by examining TCP socket state showing cwnd:1,
backoff:9, rto:102912ms, and rwnd_limited:100% on the client side.
The fix interleaves reads with writes by processing responses every
10,000 commands. This prevents TCP buffers from filling to the point
where congestion control triggers the pathological backoff behavior.
The test still validates the same functionality (slave buffer memory
accounting) since the measurement happens after all commands complete.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
### Summary
This PR introduces two new maxmemory eviction policies: `volatile-lrm`
and `allkeys-lrm`.
LRM (Least Recently Modified) is similar to LRU but only updates the
timestamp on write operations, not read operations. This makes it useful
for evicting keys that haven't been modified recently, regardless of how
frequently they are read.
### Core Implementation
The LRM implementation reuses the existing LRU infrastructure but with a
key difference in when timestamps are updated:
- **LRU**: Updates timestamp on both read and write operations
- **LRM**: Updates timestamp only on write operations via `updateLRM()`
### Key changes:
Add `keyModified()` to accept an optional `robj *val` parameter and call
`updateLRM()` when a value is provided. Since `keyModified()` serves as
the unified entry point for all key modifications, placing the LRM
update here ensures timestamps are consistently updated across all write
operations
---------
Co-authored-by: oranagra <oran@redislabs.com>
Co-authored-by: Yuan Wang <yuan.wang@redis.com>
This bug was introduced by #14130 and found by guybe7
When using XTRIM/XADD with approx mode (~) and DELREF/ACKED delete
strategies, if a node was eligible for removal but couldn't be removed
directly (because consumer group references need to be checked), the
code would incorrectly break out of the loop instead of continuing to
process entries within the node. This fix allows the per-entry deletion
logic to execute for eligible nodes when using non-KEEPREF strategies.
Follow https://github.com/redis/redis/pull/14423
In https://github.com/redis/redis/pull/14423,
I thought the last lpNext operation of the iterator occurred at the end
of streamIteratorGetID.
However, I overlooked the fact that after calling
`streamIteratorGetID()`, we might still use `streamIteratorGetField()`
to continue moving within the current entry.
This means that during reverse iteration, the iterator could move back
to a previous entry position.
To fix this, in this PR I record the current position at the beginning
of streamIteratorGetID().
When we enter it again next time, we ensure that the entry position does
not exceed the previous one,
that is, during forward iteration the entry must be greater than the
last entry position,
and during reverse iteration it must be smaller than the last entry
position.
Note that the fix for https://github.com/redis/redis/pull/14423 has been
replaced by this fix.
This PR implements support for automatic client authentication based on
a field in the client's TLS certificate.
We adopt ValKey’s PR: https://github.com/valkey-io/valkey/pull/1920
API Changes:
Add New configuration tls-auth-clients-user
- Allowed values: `off` (default), `CN`.
- `off` – disable TLS certificate–based auto-authentication.
- `CN` – derive the ACL username from the Common Name (CN) field of the
client certificate.
New INFO stat
- `acl_access_denied_tls_cert`
- Counts failed TLS certificate–based authentication attempts, i.e. TLS
connections where a client certificate was presented, a username was
derived from it, but no matching ACL user was found.
New ACL LOG reason
- Reason string: `"tls-cert"`
- Emitted when a client certificate’s Common Name fails to match any
existing ACL user.
Implementation Details:
- Added getCertFieldByName() utility to extract fields from peer
certificates.
- Added autoAuthenticateClientFromCert() to handle automatic login logic
post-handshake.
- Integrated automatic authentication into the TLSAccept function after
handshake completion.
- Updated test suite (tests/integration/tls.tcl) to validate the
feature.
This PR containts a few changes for ASM:
**Bug fix:**
- Fixes an issue in ASM when adjacent slot ranges are provided in
CLUSTER MIGRATION IMPORT command (e.g. 0-10 11-100). ASM task keeps the
original slot ranges as given, but later the source node reconstructs
the slot ranges from the config update as a single range (e.g. 0-100).
This causes asmLookupTaskBySlotRangeArray() to fail to match the task,
and the source node incorrectly marks the ASM task as failed. Although
the migration completes successfully, the source node performs a
blocking trim operation for these keys, assuming the slot ownership
changed outside of an ASM operation. With this PR, redis merges adjacent
slot ranges in a slot range array to avoid this problem.
**Other improvements:**
- Indicates imported/migrated key count in the log once asm operation is
completed.
- Use error return value instead of assert in parseSlotRangesOrReply()
- Validate slot range array that is given by cluster implementation on
ASM_EVENT_IMPORT_START.
---------
Co-authored-by: Yuan Wang <yuan.wang@redis.com>
This PR introduces a mechanism that allows a module to temporarily
disable trimming after an ASM migration operation so it can safely
finish ongoing asynchronous jobs that depend on keys in migrating (and
about to be trimmed) slots.
1. **ClusterDisableTrim/ClusterEnableTrim**
We introduce `ClusterDisableTrim/ClusterEnableTrim` Module APIs to allow
module to disable/enable slot migration
```
/* Disable automatic slot trimming. */
int RM_ClusterDisableTrim(RedisModuleCtx *ctx)
/* Enable automatic slot trimming */
int RM_ClusterEnableTrim(RedisModuleCtx *ctx)
```
**Please notice**: Redis will not start any subsequent import or migrate
ASM operations while slot trimming is disabled, so modules must
re-enable trimming immediately after completing their pending work.
The only valid and meaningful time for a module to disable trimming
appears to be after the MIGRATE_COMPLETED event.
2. **REDISMODULE_OPEN_KEY_ACCESS_TRIMMED**
Added REDISMODULE_OPEN_KEY_ACCESS_TRIMMED to RM_OpenKey() so that module
can operate with these keys in the unowned slots after trim is paused.
And now we don't delete the key if it is in trim job when we access it.
And `expireIfNeeded` returns `KEY_VALID` if
`EXPIRE_ALLOW_ACCESS_TRIMMED` is set, otherwise, returns `KEY_TRIMMED`
without deleting key.
3. **REDISMODULE_CTX_FLAGS_TRIM_IN_PROGRESS**
We also extend RM_GetContextFlags() to include a flag
REDISMODULE_CTX_FLAGS_TRIM_IN_PROGRESS indicating whether a trimming job
is pending (due to trim pause) or in progress. Modules could
periodically poll this flag to synchronize their internal state, e.g.,
if a trim job was delayed or if the module incorrectly assumed trimming
was still active.
Bugfix: RM_SetClusterFlags could not clear a flag after enabling it first.
---------
Co-authored-by: Ozan Tezcan <ozantezcan@gmail.com>
`repl-diskless-load` feature can effectively reduce the time of full
synchronization, but maybe it is not widely used.
`swapdb` option needs double `maxmemory`, and `on-empty-db` only works
on the first full sync (the replica must have no data).
This PR introduce a new option: `flushdb` - Always flush the entire
dataset before diskless load. If the diskless load fails, the replica
will lose all existing data.
Of course, it brings the risk of data loss, but it provides a choice if
you want to reduce full sync time and accept this risk.
This PR implements flexible keyword-based argument parsing for all 12
hash field expiration commands, allowing users to specify arguments in
any logical order rather than being constrained by rigid positional
requirements.
This enhancement follows Redis's modern design of keyword-based flexible
argument ordering and significantly improves user experience.
Commands with Flexible Parsing
HEXPIRE, HPEXPIRE, HEXPIREAT, HPEXPIREAT, HGETEX, HSETEX
some examples:
HEXPIRE:
* All these are equivalent and valid:
HEXPIRE key EX 60 NX FIELDS 2 f1 f2
HEXPIRE key NX EX 60 FIELDS 2 f1 f2
HEXPIRE key FIELDS 2 f1 f2 EX 60 NX
HEXPIRE key FIELDS 2 f1 f2 NX EX 60
HEXPIRE key NX FIELDS 2 f1 f2 EX 60
HGETEX:
* All these are equivalent and valid:
HGETEX key EX 60 FIELDS 2 f1 f2
HGETEX key FIELDS 2 f1 f2 EX 60
HSETEX:
* All these are equivalent and valid:
HSETEX key FNX EX 60 FIELDS 2 f1 v1 f2 v2
HSETEX key EX 60 FNX FIELDS 2 f1 v1 f2 v2
HSETEX key FIELDS 2 f1 v1 f2 v2 FNX EX 60
HSETEX key FIELDS 2 f1 v1 f2 v2 EX 60 FNX
HSETEX key FNX FIELDS 2 f1 v1 f2 v2 EX 60
Decouple kvstore from its metadata by introducing `kvstoreType` structure of
callbacks. This resolves the abstraction layer violation of having kvstore
include `server.h` directly.
Move (again) cluster slot statistics to per slot dicts' metadata. The callback
`canFreeDict` is used to prevent freeing empty per slot dicts from losing per
slot statistics.
Co-authored-by: Ran Tidhar <ran.tidhar@redis.com>
Fixes#14538
If the master uses diskless synchronization and the replica uses
diskless load, we can disable RDB compression to reduce full sync time.
I tested on AWS and found we could reduce time by 20-40%.
In terms of implementation, when the replica can use diskless load, the
replica will send `replconf rdb-no-compress 1` to master to deliver a
RDB without compression.
If your network is slow, please disable repl-diskless-load, and maybe
even repl-diskless-sync
---------
Co-authored-by: Ozan Tezcan <ozantezcan@gmail.com>
The cluster implementation may be temporarily unavailable and return an
error to the `ASM_EVENT_MIGRATE_PREP` event to prevent starting a new
migration. Although this is most likely a transient condition, the
source node has no way to distinguish it from a real error, so it must
fail the import attempt and start a new one.
In Redis, failing an attempt is cheap, but in other cluster
implementations it may require cleaning up resources and can cause
unnecessary disruption.
This PR introduces a new `-NOTREADY` error reply for the `CLUSTER
SYNCSLOTS SYNC` command. When the source replies with `-NOTREADY`, the
destination can recognize the condition as transient and retry sending
`CLUSTER SYNCSLOTS SYNC` step periodically instead of failing the
attempt.
some error handling paths didn't remove the tags they added, but most
importantly, if the start_server proc is given the "tags" argument more
than once, on exit, it only removed the last one.
this problem exists in start_cluster in list.tcl, and the result was
that the "external:skip cluster modules" were not removed
In #14121, the SCAN filters order was changed, before #14121the order
was - pattern, expiration and type, after #14121pattern became last,
this break change broke the original behavior, which will cause scan
with pattern also to remove the expired keys.
This PR reorders the filters to be consistent with the original behavior
and extends a test to cover this scenario.
This PR fixes an issue(#14541) where EXEC’s ACL recheck was still being
performed during AOF loading, that may cause AOF loading failed, if ACL
rules are changed and don't allow some commands in MULTI-EXEC.
## Problem
When destroying a consumer group with `XGROUP DESTROY`, the cached
`min_cgroup_last_id` was not being invalidated. This caused incorrect
behavior when using `XDELEX` with the `ACKED` option, as the cache still
referenced the destroyed group's `last_id`.
## Solution
Invalidate the `min_cgroup_last_id` cache when the destroyed group's
`last_id` equals the cached minimum. The cache will be recalculated on
the next call to `streamEntryIsReferenced()`.
---------
Co-authored-by: guybe7 <guy.benoish@redislabs.com>
After https://github.com/redis/redis/pull/14226 module tests started
running with ASan enabled.
`auth.c` blocks the user on auth and spawns a thread that sleeps for
0.5s before unblocking the client and returning.
A tcl tests unloads the module which may happen just after the spawned
thread unblocks the client. In that case if the unloading finishes fast
enough the spawned thread may try to execute code from the module's
dynamic library that is already unloaded resulting in sefault.
Fix: just wait on the thread during module's OnUnload method.
These commands behave as DEL and SET (blindly Remove or Overwrite) when
they don't get IF* flags, and require the value of the key when they do
run with these flags.
Making sure they have the VARIABLE_FLAGS flag, and getKeysProc that can
provide the right flags depending on the arguments used. (the plain
flags when arguments are unknown are the common denominator ones)
Move lookupKey call in DELEX to avoid double lookup, which also means
(some, namely arity) syntax errors are checked (and reported) before
checking the existence of the key.
### Problem
The XREADGROUP command with CLAIM parameter incorrectly returns delivery
metadata (idle time and delivery count) as strings instead of integers,
contradicting the Redis specification.
### Solution
Updated the XREADGROUP CLAIM implementation to return delivery metadata
fields as integers, aligning with the documented specification and
maintaining consistency with Redis response conventions.
---------
Co-authored-by: debing.sun <debing.sun@redis.com>
Verify that following RDB load fields keep their expiration time.
Verify that hashes that had HFEs not counted following rdb load in
subexpiry (by command `info keyspace`)
This issue was introduced by redis/redis#14130.
The problem is that when the number of IDs exceeds STREAMID_STATIC_VECTOR_LEN (8), the code forgot to reallocate memory for the IDs array, which causes a stack overflow.
When the HGETEX command is used with the FIELDS option but without the required
numfields argument, the server would attempt to access an out-of-bounds argv index.
This PR adds a check to ensure numfields is present before accessing it,
returning an error if it is missing. Also includes a test case to cover this scenario.
The MurmurHash64A function in hyperloglog.c used an int parameter for length,
causing integer overflow when processing PFADD entries larger than 2GB.
This could lead to server crashes.
Changed the len parameter from int to size_t to properly handle
large inputs up to SIZE_MAX in HyperLogLog operations.
Refer to the implementation in facebook/mcrouter@2dbee3d/mcrouter/lib/fbi/hash.c#L54
Fix https://github.com/redis/redis/issues/14496
This PR makes the following changes:
- DIGEST: Always return 16 hex characters with leading zeros
Example: "00006c38adf31777" instead of "6c38adf31777"
- IFDEQ/IFDNE: Validate the digest must be exactly 16 characters
- IFDEQ/IFDNE: Use strcasecmp for case-insensitive hex comparison
Both uppercase and lowercase hex digits now work identically
---------
Co-authored-by: Marc Gravell <marc.gravell@gmail.com>
Co-authored-by: Yuan Wang <yuan.wang@redis.com>
Currently, it is difficult to determine whether an RDB persistence
failure is caused by a persistent issue in the process or by a temporary
error.
Tracking the number of consecutive RDB persistence failures would help
clarify this distinction. Multiple failures in a row could indicate an
ongoing problem, while isolated cases may point to transient or
environmental issues.
Make CLUSTER SLOT-STATS command report memory-bytes if accounting is turned on
disregarding cluster-slot-stats-enabled.
There is currently slight disparity between cluster-slot-stats-enabled and
whether per slot memory accounting is on:
Per slot memory accounting is turned on only iff cluster-slot-stats-enabled
was enabled on startup.
This PR reconciles this in CLUSTER SLOT-STATS output.
New behavior enables scenario where if one wants per slot memory accounting but
no other per slot stats, then redis can start with cluster-slot-stats-enabled and
then have it turned off through CONFIG SET avoiding stats collection overhead
other than memory-bytes per slot.
Revert a breaking change introduced in #14051 described in this comment
https://github.com/redis/redis/pull/14051#discussion_r2281765769
The non-negative check inside `checkNumericBoundaries` was ignoring that
passing a big unsigned long long value (> 2^63-1) will be passed as
negative value and will never reach the lower/upper boundary check.
The check is removed, reverting the breaking change.
This allows for RedisModule_ConfigSetNumeric to pass big unsigned number, albeit via `long long` parameter. Added comments about this behaviour.
Added tests for https://github.com/redis/redis/pull/14051#discussion_r2281765769
`mem_cluster_slot_migration_output_buffer` and
`mem_cluster_slot_migration_input_buffer` is transient, it will be reset
on disconnect when ASM task is failed or done. Actually for these
conditions, we just want to verify the metrics are accessible.
Failed CI job:
https://github.com/redis/redis/actions/runs/18859697064/job/53815311358
Also fix test "SLOT-ALLOCSIZE - Test DEBUG ALLOCSIZE-SLOTS-ASSERT
command"
which needs `cluster-enabled` and `cluster-slot-stats-enabled` configs
set to be effective.
In PR https://github.com/redis/redis/pull/14434, we made the keys
parameter flexible, meaning it could appear anywhere among the command
arguments. However, this also made key parsing more complex, since we
could no longer determine the fixed position of key arguments.
Therefore, in this PR, we reverted it back to using fixed positions for
the keys.
And also fix this
[comment](https://github.com/redis/redis/pull/14434#discussion_r2459282563).
---------
Co-authored-by: Yuan Wang <yuan.wang@redis.com>
Introduce a new command MSETEX to set multiple string keys with a shared
expiration in a single atomic operation. Also with flexible argument
parsing.
Syntax:
MSETEX KEYS numkeys key value [key value …] [XX | NX] [EX seconds | PX
milliseconds | EXAT unix-time-seconds | PXAT unix-time-milliseconds |
KEEPTTL]
Sets the given keys to their respective values.
This command is an extension of the MSETNX that adds expiration and XX
options.
Options:
EX seconds - Set the specified expiration time, in seconds
PX milliseconds - Set the specified expiration time, in milliseconds
EXAT timestamp-seconds - Set the specified Unix time at which the keys
will expire, in seconds
PXAT timestamp-milliseconds - Set the specified Unix time at which the
keys will expire, in milliseconds
KEEPTTL - Retain the time to live associated with the keys
XX - Only set the keys and their expiration if all already exist
NX - Only set the keys and their expiration if none exist
Flexible Argument Parsing examples:
- MSETEX EX 10 KEYS 2 k1 v1 k2 v2
- MSETEX KEYS 2 k1 v1 k2 v2 NX PX 5000
- MSETEX NX EX 10 KEYS 2 k1 v1 k2 v2
Return Values:
Integer reply: 1 - All keys were set successfully
Integer reply: 0 - No keys were set (due to NX/XX conditions)
Error reply - Syntax error or invalid arguments
Add O(1) memory accounting per key for all robj types.
This is part 1 of adding slot level memory accounting #10472 (discussed
in #10411 and #14119) for redis.
Currently the tracked allocation sizes are used in the `MEMORY USAGE`
command and replace previous approximation in `kvobjComputeSize()`. In
part 2, this is extended to provide per slot memory accounting.
Note that we're accounting for usable memory size (not requested memory
size). This is consistent with current `MEMORY USAGE` semantics and is
also provides more realistic metric than requested memory size.
Since asking for usable size of an allocation with `zmalloc_usable_size()`
is not free, care is taken to avoid it when possible and rely on usable
sizes reported from `zmalloc_usable()`/`zmalloc_free_usable()` et al.
For efficiency reasons for listpacks and small strings we currently
track requested memory. Since listpacks and small strings don't
store allocation size, we need `zmalloc_usable_size()` to get it and
in previous iterations of this PR the added overhead for some
commands was around 2-3%.
Quicklists, skiplists and streams track allocated size in `alloc_size`
field costing additional 8 bytes per instance.
Rax users that want memory accounting can provide `size_t *` pointer
where this should be done on rax creation.
Since dicts are generic data structures where most uses don't care
about memory accounting, having additional 8 bytes per instance
would be overkill and instead allocated size is tracked in metadata only
where it's necessary (currently only for hashes and sets with hashtable
encoding).
Dict's allocation size is easily approximated by `dictMemUsage()` and
the metadata `alloc_size` accumulates allocation sizes of associated
keys/values only.
Correctness is enforced in unit tests that compare O(1) accounted
allocations sizes with the result of O(n) memory accounted version (e.g.
`_rax_verify_alloc_size()`, `_ql_verify_alloc_size()`). Additionally for
REDIS_TEST builds correct `alloc_size` is asserted on robj deallocation.
This PR follows https://github.com/redis/redis/pull/14058
When Redis is shut down uncleanly (e.g., due to power loss or system
crashes), invalid bytes may remain at the end of AOF files.
This PR refactors the AOF corruption handling system and introduces
automatic recovery capabilities to improve resilience and reduce
downtime. The feature only handles corruption at the end of files,
respects the configured size limit to prevent excessive data loss,
maintains all valid commands before the corruption point.
Removed aof-load-broken configuration option
Rename aof-load-broken-max-size to aof-load-corrupt-tail-max-size.
New Config: aof-load-corrupt-tail-max-size - enables recovery for
corruption up to specified size
## Problem and Motivation
Currently, the client only parses one command, then executes it, then
parses new commands until the querybuf is consumed. Doing it this way
means we cannot perform memory prefetch when IO threads are not enabled,
and when IO threads are enabled, we can only parse the first command in
the IO thread, while the remaining command parsing still needs to be
done in the main thread.
This describes a limitation in the current Redis command processing
pipeline where:
Without IO threads: Commands are parsed and executed one by one
sequentially, preventing memory prefetching optimizations
With IO threads: Only the first command gets parsed in the IO thread,
but subsequent commands from the same client's query buffer must still
be parsed in the main thread
## Solution Overview
**Core Innovation**: Parse multiple user commands in advance through a
lookahead pipeline.
**Key Insight**: Since Redis already parses commands to extract keys, we
can do this parsing earlier and memory prefetch operations before the
command reaches execution, allowing multiple I/O operations to run in
parallel.
The bulk of the PR is a redesign of the command processing flow for both
standalone commands and transactional commands.
### High Level Command Processing Flow
#### Before This PR (processInputBuffer())
- While there is data in the client's query buffer:
- Read the data and try to parse a complete command
(processInlineBuffer() or processMultibulkBuffer()).
- If the command is incomplete, exit and wait for more data.
- The Command is complete. Process and potentially execute it
(processCommandAndResetClient(), processCommand()):
- Prepare for the next command (commandProcessed()).
### Major Changes in the Client's Structure
To support the new command processing flow:
- **New pendingCommand structure**: Since the previous flow processed
commands one at a time, it used the client structure to hold the current
(and only) parsed command arguments (argv/argc) and other metadata. In
the new design, multiple commands are processed, waiting for execution.
So, a new pendingCommand structure is introduced to hold a parsed
command's arguments and its metadata.
- **New pendingCommandList structure (pending_cmds)** that contains all
the pending commands with maintained order and includes a ready_len
counter that tracks the number of fully parsed commands ready for
execution. All commands are fully parsed except possibly the last one
(client's command order is maintained).
- **New pendingCommandPool structure (cmd_pool)** that manages a shared
pool for reusing pendingCommand objects to reduce memory allocation
overhead.
There is a configurable lookahead limit (server.lookahead) that controls
how many fully parsed commands (ready pending commands) to process ahead
of time.
#### New High Level Flow for Standalone Commands (processInputBuffer())
- While there is data in the client's query buffer or there are ready
pending commands:
- While there is data in the client's query buffer and we haven't
reached the lookahead limit:
- Read the data and try to parse a complete command
(processInlineBuffer() or processMultibulkBuffer()). Allocate a new
pending command if needed, store the command's metadata in the pending
command, and add the pending command to the client's pending commands
list.
- If the command is incomplete, exit and wait for more data.
- The command is complete, we have a new ready pending command,
preprocess it (preprocessCommand()):
- Extract the keys of the command and store the results in the pending
command (extractKeysAndSlot()).
- If there are pending commands, continue executing them until the queue
is empty.
## Transaction Support
### Major Changes in Structures
- The multiState structure now contains an array of pendingCommand
pointers instead of multiCmd pointers.
- The multiCmd structure was deleted (no longer needed).
### New Transaction Support
- queueMultiCommand():
- The pending commands are moved from the client's pending_cmds list to
the multiState's commands array.
## Detailed Changes
### Additional Client Structure Changes
- Replaced argv_len_sum with all_argv_len_sum to reflect the total
memory consumed by all pending commands.
### Clients and Pending Commands Management
- Clients using pending commands now manage the command arguments via
the pendingCommand. Specifically, the memory occupied by argv.
- **Pending commands management functions**:
- `initPendingCommand()` initializes a newly allocated pending command.
- `freeClientPendingCommand()` frees a pending command of a client and
its associated resources.
- `freeClientPendingCommands()` receives the number of pending commands
to free and calls freeClientPendingCommand() to free them.
### Buffer Processing Changes
- `processInlineBuffer()`, once a full command is read, used to populate
the client's command fields (argc, argv, etc.). Now it creates and
populates a pendingCommand, and adds it to the client's pending_cmds
list.
- `processMultibulkBuffer()`: Similar changes to processInlineBuffer().
The difference is that a pending command may already exist from a
previous call to the function, so parsing will continue populating it
instead of creating a new one.
- `resetClientInternal()` used to receive a free_argv parameter and pass
it to freeClientArgvInternal(), which freed the client's argv if set,
and also reset client's command fields. It now receives the number of
pending commands to free and handles two cases:
- The client uses pending commands so they are freed by calling
freeClientPendingCommands().
- The client doesn't use pending commands (e.g., LUA client) so the
client's argv is freed by calling freeClientArgvInternal().
It then frees the client's command fields that freeClientArgvInternal()
doesn't free now.
### Other Changes
- Simulate lookahead command preprocessing when loading an AOF and
queuing transaction commands; This is necessary since
queueMultiCommand() now requires a pending command.
- The INVALID_CLUSTER_SLOT constant was defined to indicate an invalid
cluster slot. It is used to signal a cross-slot error in
preprocessCommand().
- getNodeByQuery() no longer performs cross-slot checks, relying instead
on the checks already performed in preprocessCommand(). It also no
longer calls getKeysFromCommand() as this was also done in
preprocessCommand().
### Debugging
- Added "debug lookahead" command to print the size of the lookahead
pipeline for each client.
## New Configuration
- **lookahead**: Runtime-configurable lookahead depth (default: 16)
## Security
- **Limit lookahead for unauthenticated clients to 1**. This is both to
reduce memory overhead, and to prevent errors; AUTH can affect the
handling of succeeding commands.
---------
Co-authored-by: Slava Koyfman <slava.koyfman@redis.com>
Co-authored-by: Oran Agra <oran@redis.com>
Co-authored-by: Udi Ron <udi.ron@redis.com>
Co-authored-by: moticless <moticless@github.com>
Co-authored-by: Yuan Wang <yuan.wang@redis.com>
Enhancing #13069, by adding a more direct API for getting the slot of a
string
Introduces new module API:
```c
/* Like `RM_ClusterKeySlot`, but gets a char pointer and a length.
* Returns the cluster slot of a key, similar to the `CLUSTER KEYSLOT` command.
* This function works even if cluster mode is not enabled. */
unsigned int RM_ClusterKeySlotC(const char *keystr, size_t keylen)
```
