The bug was introduced by #5021 which only attempted avoid EXIST on an
already expired key from returning 1 on a replica.
Before that commit, dbExists was used instead of
lookupKeyRead (which had an undesired effect to "touch" the LRU/LFU)
Other than that, this commit fixes OBJECT to also come empty handed on
expired keys in replica.
And DEBUG DIGEST-VALUE to behave like DEBUG OBJECT (get the data from
the key regardless of it's expired state)
Blocking command should not be used with MULTI, LUA, and RM_Call. This is because,
the caller, who executes the command in this context, expects a reply.
Today, LUA and MULTI have a special (and different) treatment to blocking commands:
LUA - Most commands are marked with no-script flag which are checked when executing
and command from LUA, commands that are not marked (like XREAD) verify that their
blocking mode is not used inside LUA (by checking the CLIENT_LUA client flag).
MULTI - Command that is going to block, first verify that the client is not inside
multi (by checking the CLIENT_MULTI client flag). If the client is inside multi, they
return a result which is a match to the empty key with no timeout (for example blpop
inside MULTI will act as lpop)
For modules that perform RM_Call with blocking command, the returned results type is
REDISMODULE_REPLY_UNKNOWN and the caller can not really know what happened.
Disadvantages of the current state are:
No unified approach, LUA, MULTI, and RM_Call, each has a different treatment
Module can not safely execute blocking command (and get reply or error).
Though It is true that modules are not like LUA or MULTI and should be smarter not
to execute blocking commands on RM_Call, sometimes you want to execute a command base
on client input (for example if you create a module that provides a new scripting
language like javascript or python).
While modules (on modules command) can check for REDISMODULE_CTX_FLAGS_LUA or
REDISMODULE_CTX_FLAGS_MULTI to know not to block the client, there is no way to
check if the command came from another module using RM_Call. So there is no way
for a module to know not to block another module RM_Call execution.
This commit adds a way to unify the treatment for blocking clients by introducing
a new CLIENT_DENY_BLOCKING client flag. On LUA, MULTI, and RM_Call the new flag
turned on to signify that the client should not be blocked. A blocking command
verifies that the flag is turned off before blocking. If a blocking command sees
that the CLIENT_DENY_BLOCKING flag is on, it's not blocking and return results
which are matches to empty key with no timeout (as MULTI does today).
The new flag is checked on the following commands:
List blocking commands: BLPOP, BRPOP, BRPOPLPUSH, BLMOVE,
Zset blocking commands: BZPOPMIN, BZPOPMAX
Stream blocking commands: XREAD, XREADGROUP
SUBSCRIBE, PSUBSCRIBE, MONITOR
In addition, the new flag is turned on inside the AOF client, we do not want to
block the AOF client to prevent deadlocks and commands ordering issues (and there
is also an existing assert in the code that verifies it).
To keep backward compatibility on LUA, all the no-script flags on existing commands
were kept untouched. In addition, a LUA special treatment on XREAD and XREADGROUP was kept.
To keep backward compatibility on MULTI (which today allows SUBSCRIBE, and PSUBSCRIBE).
We added a special treatment on those commands to allow executing them on MULTI.
The only backward compatibility issue that this PR introduces is that now MONITOR
is not allowed inside MULTI.
Tests were added to verify blocking commands are not blocking the client on LUA, MULTI,
or RM_Call. Tests were added to verify the module can check for CLIENT_DENY_BLOCKING flag.
Co-authored-by: Oran Agra <oran@redislabs.com>
Co-authored-by: Itamar Haber <itamar@redislabs.com>
ZREVRANGEBYSCORE key max min [WITHSCORES] [LIMIT offset count]
When the offset is too large, the query is very slow. Especially when the offset is greater than the length of zset it is easy to determine whether the offset is greater than the length of zset at first, and If it exceed the length of zset, then return directly.
Co-authored-by: Oran Agra <oran@redislabs.com>
* Configuration file default should now be "auto".
* Expose "process_supervised" as an info field.
* Log messages improvements (clarify required systemd config, report
auto-detected supervision mode, etc.)
* Set server.supervised properly, so it can take precedence of
"daemonize" configuration.
* Produce clear warning if systemd is detected/requested but executable
is compiled without support for it, instead of silently ignoring.
* Handle systemd notification error on startup, and turn off supervised
mode if it failed.
Syntax:
COPY <key> <new-key> [DB <dest-db>] [REPLACE]
No support for module keys yet.
Co-authored-by: tmgauss
Co-authored-by: Itamar Haber <itamar@redislabs.com>
Co-authored-by: Oran Agra <oran@redislabs.com>
Add two optional callbacks to the RedisModuleTypeMethods structure, which is `free_effort`
and `unlink`. the `free_effort` callback indicates the effort required to free a module memory.
Currently, if the effort exceeds LAZYFREE_THRESHOLD, the module memory may be released
asynchronously. the `unlink` callback indicates the key has been removed from the DB by redis, and
may soon be freed by a background thread.
Add `lazyfreed_objects` info field, which represents the number of objects that have been
lazyfreed since redis was started.
Add `RM_GetTypeMethodVersion` API, which return the current redis-server runtime value of
`REDISMODULE_TYPE_METHOD_VERSION`. You can use that when calling `RM_CreateDataType` to know
which fields of RedisModuleTypeMethods are gonna be supported and which will be ignored.
- Add ZDIFF and ZDIFFSTORE which work similarly to SDIFF and SDIFFSTORE
- Make sure the new WITHSCORES argument that was added for ZUNION isn't considered valid for ZUNIONSTORE
Co-authored-by: Oran Agra <oran@redislabs.com>
BLPOP when there are elements in the list works in the same way as LPOP
does. Due to this they also does the same repetitive action and logic
for the same is written at two different places. This is a bad code
practice as the one needs the context to change the BLPOP list pop code
as well when the LPOP code gets changed.
Separated the generic logic from LPOP to a function that is being used
by the BLPOP code as well.
Test support for the new map, null and push message types. Map objects are parsed as a list of lists of key value pairs.
for instance: user => john password => 123
will be parsed to the following TCL list:
{{user john} {password 123}}
Also added the following tests:
Redirection still works with RESP3
Able to use a RESP3 client as a redirection client
No duplicate invalidation messages when turning BCAST mode on after normal tracking
Server is able to evacuate enough keys when num of keys surpasses limit by more than defined initial effort
Different clients using different protocols can track the same key
OPTOUT tests
OPTIN tests
Clients can redirect to the same connection
tracking-redir-broken test
HELLO 3 checks
Invalidation messages still work when using RESP3, with and without redirection
Switching to RESP3 doesn't disturb previous tracked keys
Tracking info is correct
Flushall and flushdb produce invalidation messages
These tests achieve 100% line coverage for tracking.c using lcov.
* Debug Populate: Add checks for count and keysize to avoid crash
* provide getRangeLongFromObjectOrReply and getPositiveLongFromObjectOrReply for range check
Expose new `loading_rdb_used_mem` showing the used memory of the server
that saved the RDB file we're currently using.
This is useful in diskless replication when the total size of the rdb is
unkown, and can be used as a rought estimation of progres.
Use that new field to calculate the "user friendly"
`loading_loaded_perc` and `loading_eta_seconds`.
Expose `master_sync_total_bytes` and `master_sync_total_bytes` to complement
on the existing `master_sync_total_bytes` (which cannot be used on its own
to calculate progress).
Add "user friendly" field for `master_sync_perc`
Perform full reset of all client connection states, is if the client was
disconnected and re-connected. This affects:
* MULTI state
* Watched keys
* MONITOR mode
* Pub/Sub subscription
* ACL/Authenticated state
* Client tracking state
* Cluster read-only/asking state
* RESP version (reset to 2)
* Selected database
* CLIENT REPLY state
The response is +RESET to make it easily distinguishable from other
responses.
Co-authored-by: Oran Agra <oran@redislabs.com>
Co-authored-by: Itamar Haber <itamar@redislabs.com>
Not disabling save, slower systems begun background save that did not
complete in time, resulting with SAVE failing with "ERR Background save
already in progress".
- rdbSaveSingleModuleAux() used RedisModuleIO's "bytes" field for
tracking written bytes before calling moduleInitIOContext() which sets
"bytes" to zero
- rdbSaveObject() re-initialized RedisModuleIO too late
This return value is not used at the moment since it's only tested
against -1, and the actual byte count isn't used yet.
Co-authored-by: Tomasz Poradowski <tomasz.poradowski@generiscorp.com>
This will allow to use: RedisModule_CreateStringPrintf(ctx, "%s %c %s", "string1", 0, "string2");
On large string, the previous code would incrementally retry to double the output buffer.
now it uses the the return value of snprintf and grows to the right size in one step.
and also avoids an excessive strlen in sdscat at the end.
Make it possible for redis-cli cluster import to work with source and
target that require AUTH.
Adding two different flags --cluster-from-user, --cluster-from-pass
and --cluster-askpass for source node authentication.
Also for target authentication, using existing --user and --pass flag.
Example:
./redis-cli --cluster import 127.0.0.1:7000 --cluster-from 127.0.0.1:6379 --pass 1234 --user default --cluster-from-user default --cluster-from-pass 123456
./redis-cli --cluster import 127.0.0.1:7000 --cluster-from 127.0.0.1:6379 --askpass --cluster-from-user default --cluster-from-askpass
Few config settings are also reflected by the INFO command.
these are mainly ones that are important for either an instant view of
the server status (to compare a metric to it's limit config),
Important configurations that are necessary in the crash log (which
currently doesn't print the config),
And things that are important for monitoring solutions (such as
Prometheus), which rely on INFO to collect their data.
Add cluster_connections to INFO CLUSTER:
This makes it possible to be combined together with connected_clients
and connected_slaves and be matched against maxclients
In redisFork(), we don't set child pid, so updateDictResizePolicy()
doesn't take effect, that isn't friendly for copy-on-write.
The bug was introduced this in redis 6.0: 56258c6
The reason that we want to get a full crash report on SIGABRT
is that the jmalloc, when detecting a corruption, calls abort().
This will cause the Redis to exist silently without any report
and without any way to analyze what happened.
The bug occurs when 'callback' re-registers itself to a point
in the future and the execution time in non-negligible:
'now' refers to time BEFORE callback was executed and is used
to calculate 'next_period'.
We must get the actual current time when calculating 'next_period'
Useful when you want to know through which bind address the client connected to
the server in case of multiple bind addresses.
- Adding `laddr` field to CLIENT list showing the local (bind) address.
- Adding `LADDR` option to CLIENT KILL to kill all the clients connected
to a specific local address.
- Refactoring to share code.
Background:
#3467 (redis 4.0.0), started ignoring ENOPROTOOPT, but did that only for
the default bind (in case bind config wasn't explicitly set).
#5598 (redis 5.0.3), added that for bind addresses explicitly set
(following bug reports in Debian for redis 4.0.9 and 5.0.1), it
also ignored a bunch of other errors like EPROTONOSUPPORT which was
requested in #3894, and also added EADDRNOTAVAIL (wasn't clear why).
This (ignoring EADDRNOTAVAIL) makes redis start successfully, even if a
certain network interface isn't up yet , in which case we rather redis
fail and will be re-tried when the NIC is up, see #7933.
However, it turns out that when IPv6 is disabled (supported but unused),
the error we're getting is EADDRNOTAVAIL. and in many systems the
default config file tries to bind to localhost for both v4 and v6 and
would like to silently ignore the error on v6 if disabled.
This means that we sometimes want to ignore EADDRNOTAVAIL and other times
we wanna fail.
So this commit changes these main things:
1. Ignore all the errors we ignore for both explicitly requested bind
address and a default implicit one.
2. Add a '-' prefix to allow EADDRNOTAVAIL be ignored (by default that's
different than the previous behavior).
3. Restructure that function in a more readable and maintainable way see
below.
4. Make the default behavior of listening to all achievable by setting
a bind config directive to * (previously only possible by omitting
it)
5. document everything.
The old structure of this function was that even if there are no bind
addresses requested, the loop that runs though the bind addresses runs
at least once anyway!
In that one iteration of the loop it binds to both v4 and v6 addresses,
handles errors for each of them separately, and then eventually at the
if-else chain, handles the error of the last bind attempt again!
This was very hard to read and very error prone to maintain, instead now
when the bind info is missing we create one with two entries, and run
the simple loop twice.
