Commit Graph

7128 Commits

Author SHA1 Message Date
sundb
7993780dda
Fix some wrong server.dirty increments (#8140)
Fix wrong server dirty increment in
* spopWithCountCommand
* hsetCommand
* ltrimCommand
* pfaddCommand

Some didn't increment the amount of fields (just one per command).
Others had excessive increments.
2020-12-15 09:30:24 +02:00
Itamar Haber
9acd40d97b
GEOSEARCH: change 'FROMLOC' to 'FROMLONLAT' (#8190)
And formats style a tiniee-winiee bit
2020-12-14 17:15:12 +02:00
Oran Agra
cfb449cc80
Sanitize dump payload: excessive free on dup zset fields (#8189) 2020-12-14 17:10:31 +02:00
filipe oliveira
19d46f8f2f
Expose Redis main thread cpu time, and scrape system time via INFO command. (#8132)
Exposes the main thread CPU info via info modules ( linux specific only )
(used_cpu_sys_main_thread and used_cpu_user_main_thread). This is important for:

- distinguish between main thread and io-threads cpu time total cpu time consumed ( check
  what is the first bottleneck on the used config )
- distinguish between main thread and modules threads total cpu time consumed

Apart from it, this commit also exposes the server_time_usec within the Server section so that we can
properly differentiate consecutive collection and calculate for example the CPU% and or / cpu time vs
wall time, etc...
2020-12-13 19:14:46 +02:00
Yossi Gottlieb
86e3395c11
Several (mostly Solaris-related) cleanups (#8171)
* Allow runtest-moduleapi use a different 'make', for systems where GNU Make is 'gmake'.
* Fix issue with builds on Solaris re-building everything from scratch due to CFLAGS/LDFLAGS not stored.
* Fix compile failure on Solaris due to atomicvar and a bunch of warnings.
* Fix garbled log timestamps on Solaris.
2020-12-13 17:09:54 +02:00
Wen Hui
f74c32cad2
redis-cli prompt: show transaction state, and fix db number on aborted EXEC (#6728) 2020-12-13 14:40:54 +02:00
Oran Agra
ab60dcf564
Add module event for repl-diskless-load swapdb (#8153)
When a replica uses the diskless-load swapdb approach, it backs up the old database,
then attempts to load a new one, and in case of failure, it restores the backup.

this means that modules with global out of keyspace data, must have an option to
subscribe to events and backup/restore/discard their global data too.
2020-12-13 14:36:06 +02:00
Wen Hui
8a23ca0b28
add missing raxStop calls in aof stream rewrite (#8162) 2020-12-13 11:47:24 +02:00
Madelyn Olson
5abdf9a556
Removed usage of bool from tls.c (#8175) 2020-12-13 11:11:29 +02:00
Wen Hui
bde33501c4
redis-cli -e option to exit with error code immediately when command fails (#8136)
without this option, redis-cli returns 0 even if command fails. kept this for backwards compatibility.
2020-12-13 10:04:45 +02:00
Wang Yuan
e3ff414513
Add total_forks to INFO STATS (#8155) 2020-12-13 10:01:18 +02:00
Yossi Gottlieb
63c1303cfb
Modules: add defrag API support. (#8149)
Add a new set of defrag functions that take a defrag context and allow
defragmenting memory blocks and RedisModuleStrings.

Modules can register a defrag callback which will be invoked when the
defrag process handles globals.

Modules with custom data types can also register a datatype-specific
defrag callback which is invoked for keys that require defragmentation.
The callback and associated functions support both one-step and
multi-step options, depending on the complexity of the key as exposed by
the free_effort callback.
2020-12-13 09:56:01 +02:00
gourav
ddd43b6bc3
Randomize the random number generator's seed used in redis-benchmark (#8174)
The pid of the benchmark process is used to randomize the random number generator's
seed. This ensures that when multiple benchmark processes are started at the same time
to generate load on a server, they use different seeds. This will ensure randomness in
the keys generated by different benchmark processes.
2020-12-12 17:27:35 -08:00
杨博东
4d06d99bf8
Add GEOSEARCH / GEOSEARCHSTORE commands (#8094)
Add commands to query geospatial data with bounding box.

Two new commands that replace the existing 4 GEORADIUS* commands.

GEOSEARCH key [FROMMEMBER member] [FROMLOC long lat] [BYRADIUS radius
unit] [BYBOX width height unit] [WITHCORD] [WITHDIST] [WITHASH] [COUNT
count] [ASC|DESC]

GEOSEARCHSTORE dest_key src_key [FROMMEMBER member] [FROMLOC long lat]
[BYRADIUS radius unit] [BYBOX width height unit] [WITHCORD] [WITHDIST]
[WITHASH] [COUNT count] [ASC|DESC] [STOREDIST]

- Add two types of CIRCULAR_TYPE and RECTANGLE_TYPE to achieve different searches
- Judge whether the point is within the rectangle, refer to:
geohashGetDistanceIfInRectangle
2020-12-12 02:21:05 +02:00
Yossi Gottlieb
8c291b97b9
TLS: Add different client cert support. (#8076)
This adds a new `tls-client-cert-file` and `tls-client-key-file`
configuration directives which make it possible to use different
certificates for the TLS-server and TLS-client functions of Redis.

This is an optional directive. If it is not specified the `tls-cert-file`
and `tls-key-file` directives are used for TLS client functions as well.

Also, `utils/gen-test-certs.sh` now creates additional server-only and client-only certs and will skip intensive operations if target files already exist.
2020-12-11 18:31:40 +02:00
Yossi Gottlieb
4e064fbab4
Add module data-type support for COPY. (#8112)
This adds a copy callback for module data types, in order to make
modules compatible with the new COPY command.

The callback is optional and COPY will fail for keys with data types
that do not implement it.
2020-12-09 20:22:45 +02:00
Yossi Gottlieb
cf3d79d4c1
Fix double close on IPv6 setup error. (#8168)
Fixes #8165.
2020-12-09 20:20:55 +02:00
Oran Agra
6a3c4ac50a
zset full dump sanitization bug (dup score instead of field) (#8167) 2020-12-09 17:05:05 +02:00
Oran Agra
48efc25f74
Handle output buffer limits for Module blocked clients (#8141)
Module blocked clients cache the response in a temporary client,
the reply list in this client would be affected by the recent fix
in #7202, but when the reply is later copied into the real client,
it would have bypassed all the checks for output buffer limit, which
would have resulted in both: responding with a partial response to
the client, and also not disconnecting it at all.
2020-12-08 16:41:20 +02:00
Itamar Haber
37f45d9e56
Adds exclusive range query intervals to XPENDING (#8130) 2020-12-08 11:43:00 +02:00
Yossi Gottlieb
ec02c761aa
Fix setproctitle related crashes. (#8150)
Makes spt_init more careful with assumptions about what memory regions
may be overwritten. It will now only consider a contiguous block of argv
and envp elements and mind any gaps.
2020-12-08 11:27:30 +02:00
Wen Hui
09b7f6c2a1
use getPositiveLongFromObjectOrReply for positive check in spop (#8146)
This is a cleanup commit that doesn't fix any bug, however it does change the error response text in case the user provided a negative number.
2020-12-08 11:15:39 +02:00
guybe7
6bb5503524
More efficient self-XCLAIM (#8098)
when the same consumer re-claim an entry that it already has, there's
no need to remove-and-insert if it's the same rax.
we do need to update the idle time though.
this commit only improves efficiency (doesn't change behavior).
2020-12-07 21:31:35 +02:00
filipe oliveira
b5e99bd064
Enable configuring OpenSSL using the standard openssl.cnf (#8143) 2020-12-07 14:30:12 +02:00
Yossi Gottlieb
bccbc5509a
Add CLIENT INFO and CLIENT LIST [id]. (#8113)
* Add CLIENT INFO subcommand.

The output is identical to CLIENT LIST but provides a single line for
the current client only.

* Add CLIENT LIST ID [id...].

Co-authored-by: Itamar Haber <itamar@redislabs.com>
2020-12-07 14:24:05 +02:00
David CARLIER
ec951cdc15
Solaris based system rss size report. (#8138) 2020-12-06 15:30:29 +02:00
Oran Agra
e288430c05 Sanitize dump payload: performance optimizations and tuning
First, if the ziplist header is surely inside the ziplist, do fast path
decoding rather than the careful one.

In that case, streamline the encoding if-else chain to be executed only
once, and the encoding validity tested at the end.

encourage inlining

likely / unlikely hints for speculative execution

Assertion used _exit(1) to tell the compiler that the code after them is
not reachable and get rid of warnings.

But in some cases assertions are placed inside tight loops, and any
piece of code in them can slow down execution (code cache and other
reasons), instead using either abort() or better yet, unreachable
builtin.
2020-12-06 14:54:34 +02:00
Oran Agra
7ca00d694d Sanitize dump payload: fail RESTORE if memory allocation fails
When RDB input attempts to make a huge memory allocation that fails,
RESTORE should fail gracefully rather than die with panic
2020-12-06 14:54:34 +02:00
Oran Agra
3716950cfc Sanitize dump payload: validate no duplicate records in hash/zset/intset
If RESTORE passes successfully with full sanitization, we can't affort
to crash later on assertion due to duplicate records in a hash when
converting it form ziplist to dict.
This means that when doing full sanitization, we must make sure there
are no duplicate records in any of the collections.
2020-12-06 14:54:34 +02:00
Oran Agra
c31055db61 Sanitize dump payload: fuzz tester and fixes for segfaults and leaks it exposed
The test creates keys with various encodings, DUMP them, corrupt the payload
and RESTORES it.
It utilizes the recently added use-exit-on-panic config to distinguish between
 asserts and segfaults.
If the restore succeeds, it runs random commands on the key to attempt to
trigger a crash.

It runs in two modes, one with deep sanitation enabled and one without.
In the first one we don't expect any assertions or segfaults, in the second one
we expect assertions, but no segfaults.
We also check for leaks and invalid reads using valgrind, and if we find them
we print the commands that lead to that issue.

Changes in the code (other than the test):
- Replace a few NPD (null pointer deference) flows and division by zero with an
  assertion, so that it doesn't fail the test. (since we set the server to use
  `exit` rather than `abort` on assertion).
- Fix quite a lot of flows in rdb.c that could have lead to memory leaks in
  RESTORE command (since it now responds with an error rather than panic)
- Add a DEBUG flag for SET-SKIP-CHECKSUM-VALIDATION so that the test don't need
  to bother with faking a valid checksum
- Remove a pile of code in serverLogObjectDebugInfo which is actually unsafe to
  run in the crash report (see comments in the code)
- fix a missing boundary check in lzf_decompress

test suite infra improvements:
- be able to run valgrind checks before the process terminates
- rotate log files when restarting servers
2020-12-06 14:54:34 +02:00
Oran Agra
ca1c182567 Sanitize dump payload: ziplist, listpack, zipmap, intset, stream
When loading an encoded payload we will at least do a shallow validation to
check that the size that's encoded in the payload matches the size of the
allocation.
This let's us later use this encoded size to make sure the various offsets
inside encoded payload don't reach outside the allocation, if they do, we'll
assert/panic, but at least we won't segfault or smear memory.

We can also do 'deep' validation which runs on all the records of the encoded
payload and validates that they don't contain invalid offsets. This lets us
detect corruptions early and reject a RESTORE command rather than accepting
it and asserting (crashing) later when accessing that payload via some command.

configuration:
- adding ACL flag skip-sanitize-payload
- adding config sanitize-dump-payload [yes/no/clients]

For now, we don't have a good way to ensure MIGRATE in cluster resharding isn't
being slowed down by these sanitation, so i'm setting the default value to `no`,
but later on it should be set to `clients` by default.

changes:
- changing rdbReportError not to `exit` in RESTORE command
- adding a new stat to be able to later check if cluster MIGRATE isn't being
  slowed down by sanitation.
2020-12-06 14:54:34 +02:00
Oran Agra
c4fdf09c05
prevent client tracking from causing feedback loop in performEvictions (#8100)
When client tracking is enabled signalModifiedKey can increase memory usage,
this can cause the loop in performEvictions to keep running since it was measuring
the memory usage impact of signalModifiedKey.

The section that measures the memory impact of the eviction should be just on dbDelete,
excluding keyspace notification, client tracking, and propagation to AOF and replicas.

This resolves part of the problem described in #8069
p.s. fix took 1 minute, test took about 3 hours to write.
2020-12-06 14:51:22 +02:00
guybe7
1df5bb5687
Make sure we do not propagate nested MULTI/EXEC (#8097)
One way this was happening is when a module issued an RM_Call which would inject MULTI.
If the module command that does that was itself issued by something else that already did
added MULTI (e.g. another module, or a Lua script), it would have caused nested MULTI.

In fact the MULTI state in the client or the MULTI_EMITTED flag in the context isn't
the right indication that we need to propagate MULTI or not, because on a nested calls
(possibly a module action called by a keyspace event of another module action), these
flags aren't retained / reflected.

instead there's now a global propagate_in_transaction flag for that.

in addition to that, we now have a global in_eval and in_exec flags, to serve the flags
of RM_GetContextFlags, since their dependence on the current client is wrong for the same
reasons mentioned above.
2020-12-06 13:14:18 +02:00
Wang Yuan
75f9dec644
Limit the main db and expires dictionaries to expand (#7954)
As we know, redis may reject user's requests or evict some keys if
used memory is over maxmemory. Dictionaries expanding may make
things worse, some big dictionaries, such as main db and expires dict,
may eat huge memory at once for allocating a new big hash table and be
far more than maxmemory after expanding.
There are related issues: #4213 #4583

More details, when expand dict in redis, we will allocate a new big
ht[1] that generally is double of ht[0], The size of ht[1] will be
very big if ht[0] already is big. For db dict, if we have more than
64 million keys, we need to cost 1GB for ht[1] when dict expands.

If the sum of used memory and new hash table of dict needed exceeds
maxmemory, we shouldn't allow the dict to expand. Because, if we
enable keys eviction, we still couldn't add much more keys after
eviction and rehashing, what's worse, redis will keep less keys when
redis only remains a little memory for storing new hash table instead
of users' data. Moreover users can't write data in redis if disable
keys eviction.

What this commit changed ?

Add a new member function expandAllowed for dict type, it provide a way
for caller to allow expand or not. We expose two parameters for this
function: more memory needed for expanding and dict current load factor,
users can implement a function to make a decision by them.
For main db dict and expires dict type, these dictionaries may be very
big and cost huge memory for expanding, so we implement a judgement
function: we can stop dict to expand provisionally if used memory will
be over maxmemory after dict expands, but to guarantee the performance
of redis, we still allow dict to expand if dict load factor exceeds the
safe load factor.
Add test cases to verify we don't allow main db to expand when left
memory is not enough, so that avoid keys eviction.

Other changes:

For new hash table size when expand. Before this commit, the size is
that double used of dict and later _dictNextPower. Actually we aim to
control a dict load factor between 0.5 and 1.0. Now we replace *2 with
+1, since the first check is that used >= size, the outcome of before
will usually be the same as _dictNextPower(used+1). The only case where
it'll differ is when dict_can_resize is false during fork, so that later
the _dictNextPower(used*2) will cause the dict to jump to *4 (i.e.
_dictNextPower(1025*2) will return 4096).
Fix rehash test cases due to changing algorithm of new hash table size
when expand.
2020-12-06 11:53:04 +02:00
guybe7
2f41a38568
Modules: Fix an integer sign bug in moduleTimerHandler (#8131)
bug was introduced in 1a91a2700b
2020-12-03 20:36:48 +02:00
Itamar Haber
441c490024
Adds exclusive ranges to X[REV]RANGE (#8072)
Adds the ability to use exclusive (open) start and end query intervals in XRANGE and XREVRANGE queries.

Fixes #6562
2020-12-03 14:36:48 +02:00
Felipe Machado
4cd1fb1f40
Iterate backwards on zdiff/zinter/zunion to optimize for zslInsert (#8105)
In the iterator for these functions, we'll traverse the sorted sets
in a reversed way so that largest elements come first. We prefer
this order because it's optimized for insertion in a skiplist, which
is the destination of the elements being iterated in there functions.
2020-12-03 10:12:07 +02:00
Wang Yuan
b55a827ea2
Backup keys to slots map and restore when fail to sync if diskless-load type is swapdb in cluster mode (#8108)
When replica diskless-load type is swapdb in cluster mode, we didn't backup
keys to slots map, so we will lose keys to slots map if fail to sync.
Now we backup keys to slots map at first, and restore it properly when fail.

This commit includes a refactory/cleanup of the backups mechanism (moving it to db.c and re-structuring it a bit).

Co-authored-by: Oran Agra <oran@redislabs.com>
2020-12-02 13:56:11 +02:00
luhuachao
7885faf18b
Modify help msg PING_BULK to PING_MBULK in benchmark (#8109)
As described in redis-benchamrk help message 'The test names are the same as the ones produced as output.', In redis-benchmark output, we can only see PING_BULK, but the cmd `redis-benchmark -t ping_bulk` is not supported. We  have to run it with ping_mbulk which is not user friendly.
2020-12-02 13:17:25 +02:00
Madelyn Olson
69b7113bb5
Getset fix (#8118)
* Fixed SET GET executing on wrong type

Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
2020-12-01 11:46:45 -08:00
sundb
3ba2281f96
Improve dbid range check for SELECT, MOVE, COPY (#8085)
SELECT used to read the index into a `long` variable, and then pass it to a function
that takes an `int`, possibly causing an overflow before the range check.

Now all these commands use better and cleaner range check, and that also results in
a slight change of the error response in case of an invalid database index.

SELECT:
in the past it would have returned either `-ERR invalid DB index` (if not a number),
or `-ERR DB index is out of range` (if not between 1..16 or alike).
now it'll return either `-ERR value is out of range` (if not a number), or
`-ERR value is out of range, value must between -2147483648 and 2147483647`
(if not in the range for an int), or `-ERR DB index is out of range`
(if not between 0..16 or alike)


MOVE:
in the past it would only fail with `-ERR index out of range` no matter the reason.
now return the same errors as the new ones for SELECT mentioned above.
(i.e. unlike for SELECT even for a value like 17 we changed the error message)

COPY:
doesn't really matter how it behaved in the past (new command), new behavior is
like the above two.
2020-12-01 21:41:26 +02:00
Itamar Haber
c1b1e8c329
Adds pub/sub channel patterns to ACL (#7993)
Fixes #7923.

This PR appropriates the special `&` symbol (because `@` and `*` are taken),
followed by a literal value or pattern for describing the Pub/Sub patterns that
an ACL user can interact with. It is similar to the existing key patterns
mechanism in function (additive) and implementation (copy-pasta). It also adds
the allchannels and resetchannels ACL keywords, naturally.

The default user is given allchannels permissions, whereas new users get
whatever is defined by the acl-pubsub-default configuration directive. For
backward compatibility in 6.2, the default of this directive is allchannels but
this is likely to be changed to resetchannels in the next major version for
stronger default security settings.

Unless allchannels is set for the user, channel access permissions are checked
as follows :
* Calls to both PUBLISH and SUBSCRIBE will fail unless a pattern matching the
  argumentative channel name(s) exists for the user.
* Calls to PSUBSCRIBE will fail unless the pattern(s) provided as an argument
  literally exist(s) in the user's list.

Such failures are logged to the ACL log.

Runtime changes to channel permissions for a user with existing subscribing
clients cause said clients to disconnect unless the new permissions permit the
connections to continue. Note, however, that PSUBSCRIBErs' patterns are matched
literally, so given the change bar:* -> b*, pattern subscribers to bar:* will be
disconnected.

Notes/questions:
* UNSUBSCRIBE, PUNSUBSCRIBE and PUBSUB remain unprotected due to lack of reasons
  for touching them.
2020-12-01 14:21:39 +02:00
Wang Yuan
c85bf2352d
Reset average ttl when empty databases (#8106)
On FLUSHDB or full sync, reset old average TTL stat.
This Stat is incrementally collected by the master over time when it searches for expired keys.
2020-11-30 23:15:14 +02:00
sundb
04056b767f
BITOP speedup when or/and output is 0/255, stop processing further keys (#8110)
when performing the and operation, if the output is 0, we can jump out of the loop.
when performing an or operation, if the output is 0xff, we can jump out of the loop.
2020-11-30 23:03:53 +02:00
guybe7
ada2ac9ae2
XPENDING with IDLE (#7972)
Used to filter stream pending entries by their idle-time,
useful for XCLAIMing entries that have not been processed
for some time
2020-11-29 12:08:47 +02:00
Oran Agra
cb5eadb33b
bitops limited to proto_max_bulk_len rather than 512MB (#8096)
we recently did that for SETRANGE and APPEND
2020-11-26 10:58:01 +02:00
Oran Agra
bbc2c44541
INFO client_recent_max_input_buffer includes argv array (#8065)
this metric already includes the argv bytes, like what clientsCronTrackClientsMemUsage does, but it's missing the array itself.

p.s. For the purpose of tracking expensive clients we don't need to include the size of the client struct and the static reply buffer in it.
2020-11-25 23:39:01 +02:00
kukey
cf88779527
Merge two aeDeleteFileEvent refs into one (#7521)
Merge two aeDeleteFileEvent refs into one
2020-11-25 13:37:54 -08:00
Dipankar Achinta
79a7c17176
Fix typo in ae.c file (#7895) 2020-11-25 13:36:21 -08:00
David CARLIER
0719388cfb
raspberry build fix. (#8095)
__ILP32__ is 32 bits ABI and does not imply x86, this patch resolves this.
2020-11-25 12:15:32 -08:00
sundb
25f457c7f6
Avoid excessive malloc and free in copyCommand robj creation (#8067)
Avoid multiple conditional judgments
Avoid allocating robj->ptr when we're gonna replace it right after.
2020-11-24 21:40:58 +02:00
Yossi Gottlieb
7e5a6313f0
Fix use-after-free issue in spt_copyenv. (#8088)
Seems to have gone unnoticed for a long time, because at least with
glibc it will only be triggered if setenv() was called before spt_init,
which Redis doesn't.

Fixes #8064.
2020-11-24 17:58:10 +02:00
David CARLIER
f16b52cb7d
redis_set_thread_title support for Haiku. (#8060) 2020-11-23 16:14:33 +02:00
xindoo
0b1d89d7ae
fix comment error about zslDeleteRangeByScore range (#8075)
Co-authored-by: Oran Agra <oran@redislabs.com>
2020-11-22 14:56:45 +02:00
Yossi Gottlieb
08d3e929e5
Clean up building with USE_SYSTEMD. (#8073)
When USE_SYSTEMD=yes is specified, try to use pkg-config to determine
libsystemd linker flags. If not found, silently fall back to simply
using "-lsystemd".

We now use a LIBSYSTEMD_LIBS variable so users can explicitly override
it and specify their own library.

If USE_SYSTEMD is unspecified the old behavior of auto-enabling it if
both pkg-config and libsystemd are available is retained.
2020-11-22 14:40:38 +02:00
Wang Yuan
f207e1682f
Fix diskless replication failure when has non-rdb child process (#8070)
If we enable diskless replication, set repl-diskless-sync-delay to 0,
and master has non-rdb child process such as rewrite aof child, master
will try to start to a new BGSAVE but fails immediately (before fork)
when replicas ask for full synchronization, and master always fails
to start a new BGSAVE and disconnects with replicas until non-rdb
child process exists.

this bug was introduced in #6271 (not yet released in 6.0.x)
2020-11-22 14:12:45 +02:00
Oran Agra
e6fa47380a
Fix bug with module GIL being released prematurely (#8061)
This is hopefully usually harmles.
The server.ready_keys will usually be empty so the code after releasing
the GIL will soon be done.
The only case where it'll actually process things is when a module
releases a client (or module) blocked on a key, by triggering this NOT
from within a command (e.g. a timer event).

This bug was introduced in redis 6.0.9, see #7903
2020-11-22 14:00:51 +02:00
Oran Agra
61954951ed
Fix oom-score-adj-values range, abs options, and bug when used in config file (#8046)
Fix: When oom-score-adj-values is provided in the config file after
oom-score-adj yes, it'll take an immediate action, before
readOOMScoreAdj was acquired, resulting in an error (out of range score
due to uninitialized value. delay the reaction the real call is made by
main().

Since the values are clamped to -1000..1000, and they're
applied as an offset from the value at startup (which may be -1000), we
need to allow the offsets to reach to +2000 so that a value of +1000 is
achievable in case the value at startup was -1000.

Adding an option for absolute values rather than relative ones.
2020-11-22 13:57:56 +02:00
Rosen Penev
a221a313ae
fix compilation with uClibc-ng (#8054)
backtrace can be compile time disabled.
2020-11-20 18:13:11 +02:00
guybe7
f8ae991717
EXISTS should not alter LRU, OBJECT should not reveal expired keys on replica (#8016)
The bug was introduced by #5021 which only attempted avoid EXIST on an
already expired key from returning 1 on a replica.

Before that commit, dbExists was used instead of
lookupKeyRead (which had an undesired effect to "touch" the LRU/LFU)

Other than that, this commit fixes OBJECT to also come empty handed on
expired keys in replica.

And DEBUG DIGEST-VALUE to behave like DEBUG OBJECT (get the data from
the key regardless of it's expired state)
2020-11-18 11:16:21 +02:00
Meir Shpilraien (Spielrein)
d87a0d0286
Unified MULTI, LUA, and RM_Call with respect to blocking commands (#8025)
Blocking command should not be used with MULTI, LUA, and RM_Call. This is because,
the caller, who executes the command in this context, expects a reply.

Today, LUA and MULTI have a special (and different) treatment to blocking commands:

LUA   - Most commands are marked with no-script flag which are checked when executing
and command from LUA, commands that are not marked (like XREAD) verify that their
blocking mode is not used inside LUA (by checking the CLIENT_LUA client flag).
MULTI - Command that is going to block, first verify that the client is not inside
multi (by checking the CLIENT_MULTI client flag). If the client is inside multi, they
return a result which is a match to the empty key with no timeout (for example blpop
inside MULTI will act as lpop)
For modules that perform RM_Call with blocking command, the returned results type is
REDISMODULE_REPLY_UNKNOWN and the caller can not really know what happened.

Disadvantages of the current state are:

No unified approach, LUA, MULTI, and RM_Call, each has a different treatment
Module can not safely execute blocking command (and get reply or error).
Though It is true that modules are not like LUA or MULTI and should be smarter not
to execute blocking commands on RM_Call, sometimes you want to execute a command base
on client input (for example if you create a module that provides a new scripting
language like javascript or python).
While modules (on modules command) can check for REDISMODULE_CTX_FLAGS_LUA or
REDISMODULE_CTX_FLAGS_MULTI to know not to block the client, there is no way to
check if the command came from another module using RM_Call. So there is no way
for a module to know not to block another module RM_Call execution.

This commit adds a way to unify the treatment for blocking clients by introducing
a new CLIENT_DENY_BLOCKING client flag. On LUA, MULTI, and RM_Call the new flag
turned on to signify that the client should not be blocked. A blocking command
verifies that the flag is turned off before blocking. If a blocking command sees
that the CLIENT_DENY_BLOCKING flag is on, it's not blocking and return results
which are matches to empty key with no timeout (as MULTI does today).

The new flag is checked on the following commands:

List blocking commands: BLPOP, BRPOP, BRPOPLPUSH, BLMOVE,
Zset blocking commands: BZPOPMIN, BZPOPMAX
Stream blocking commands: XREAD, XREADGROUP
SUBSCRIBE, PSUBSCRIBE, MONITOR
In addition, the new flag is turned on inside the AOF client, we do not want to
block the AOF client to prevent deadlocks and commands ordering issues (and there
is also an existing assert in the code that verifies it).

To keep backward compatibility on LUA, all the no-script flags on existing commands
were kept untouched. In addition, a LUA special treatment on XREAD and XREADGROUP was kept.

To keep backward compatibility on MULTI (which today allows SUBSCRIBE, and PSUBSCRIBE).
We added a special treatment on those commands to allow executing them on MULTI.

The only backward compatibility issue that this PR introduces is that now MONITOR
is not allowed inside MULTI.

Tests were added to verify blocking commands are not blocking the client on LUA, MULTI,
or RM_Call. Tests were added to verify the module can check for CLIENT_DENY_BLOCKING flag.

Co-authored-by: Oran Agra <oran@redislabs.com>
Co-authored-by: Itamar Haber <itamar@redislabs.com>
2020-11-17 18:58:55 +02:00
thomaston
39f716a121
ZREVRANGEBYSCORE Optimization for out of range offset (#5773)
ZREVRANGEBYSCORE key max min [WITHSCORES] [LIMIT offset count]
When the offset is too large, the query is very slow. Especially when the offset is greater than the length of zset it is easy to determine whether the offset is greater than the length of zset at first, and If it exceed the length of zset, then return directly.

Co-authored-by: Oran Agra <oran@redislabs.com>
2020-11-17 18:35:56 +02:00
Yossi Gottlieb
d638b05834
Improve and clean up supervised process support. (#8036)
* Configuration file default should now be "auto".
* Expose "process_supervised" as an info field.
* Log messages improvements (clarify required systemd config, report
  auto-detected supervision mode, etc.)
* Set server.supervised properly, so it can take precedence of
  "daemonize" configuration.
* Produce clear warning if systemd is detected/requested but executable
  is compiled without support for it, instead of silently ignoring.
* Handle systemd notification error on startup, and turn off supervised
  mode if it failed.
2020-11-17 12:52:49 +02:00
swamp0407
ea7cf737a1
Add COPY command (#7953)
Syntax:
COPY <key> <new-key> [DB <dest-db>] [REPLACE]

No support for module keys yet.

Co-authored-by: tmgauss
Co-authored-by: Itamar Haber <itamar@redislabs.com>
Co-authored-by: Oran Agra <oran@redislabs.com>
2020-11-17 12:03:05 +02:00
Oran Agra
9812e88959
Fix memory leaks in newly added ZDIFF (#8056) 2020-11-16 16:37:15 +02:00
chenyangyang
c1aaad06d8
Modules callbacks for lazy free effort, and unlink (#7912)
Add two optional callbacks to the RedisModuleTypeMethods structure, which is `free_effort`
and `unlink`. the `free_effort` callback indicates the effort required to free a module memory.
Currently, if the effort exceeds LAZYFREE_THRESHOLD, the module memory may be released
asynchronously. the `unlink` callback indicates the key has been removed from the DB by redis, and
may soon be freed by a background thread.

Add `lazyfreed_objects` info field, which represents the number of objects that have been
lazyfreed since redis was started.

Add `RM_GetTypeMethodVersion` API, which return the current redis-server runtime value of
`REDISMODULE_TYPE_METHOD_VERSION`. You can use that when calling `RM_CreateDataType` to know
which fields of RedisModuleTypeMethods are gonna be supported and which will be ignored.
2020-11-16 10:34:04 +02:00
Felipe Machado
d8fd48c436
Add new commands ZDIFF and ZDIFFSTORE (#7961)
- Add ZDIFF and ZDIFFSTORE which work similarly to SDIFF and SDIFFSTORE
- Make sure the new WITHSCORES argument that was added for ZUNION isn't considered valid for ZUNIONSTORE

Co-authored-by: Oran Agra <oran@redislabs.com>
2020-11-15 14:14:25 +02:00
sundb
204a14b8cf
Fix make warning in hellohook,c and testmodule.c (#8044)
* Fix build warning in hellohook,c and testmodule.c

* Change ci->id type to (unsigned long long)
2020-11-13 15:16:40 +02:00
Itamar Haber
92ec592520
Adds user parsing to redis-cli URIs (#8048) 2020-11-12 19:19:18 +02:00
Yash Ladha
c170365dcf
cleanup: move list pop logic to single function (#7997)
BLPOP when there are elements in the list works in the same way as LPOP
does. Due to this they also does the same repetitive action and logic
for the same is written at two different places. This is a bad code
practice as the one needs the context to change the BLPOP list pop code
as well when the LPOP code gets changed.

Separated the generic logic from LPOP to a function that is being used
by the BLPOP code as well.
2020-11-12 10:55:51 +02:00
Madelyn Olson
a0576bdeea
Initialize original client argv for aof (#8042)
(this is a fixup for #8006)

Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
2020-11-11 23:58:05 +02:00
tzongw
d5059ba5e1
Invalidate aeTimer when returning AE_NOMORE (#8022) 2020-11-11 12:10:54 +02:00
Wen Hui
dd1f20edc5
add tracking bcast flag and client redirection in client list (#7995) 2020-11-11 08:22:17 +02:00
Madelyn Olson
3feff7d78a
Rewritten commands are logged as their original command (#8006)
* Rewritten commands are logged as their original command

Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
2020-11-10 13:50:03 -08:00
Itamar Haber
5d0c6b0868
Adds help for '--cluster-yes' (#8035) 2020-11-10 22:02:09 +02:00
sundb
c09b5941ce optimization src/adlist.c:listJoin() 2020-11-09 21:48:56 -08:00
Meir Shpilraien (Spielrein)
97d647a139
Moved RMAPI_FUNC_SUPPORTED location such that it will be visible to modules (#8037)
The RMAPI_FUNC_SUPPORTED was defined in the wrong place on redismodule.h
and was not visible to modules.
2020-11-09 10:46:23 +02:00
bugwz
bcc46a2f25
Fix the init value for repl_id of rdbSaveInfo struct (#8026)
use 40 zeros rather than 30 zeros to match CONFIG_RUN_ID_SIZE.
this doesn't have any real implications.
2020-11-08 09:26:25 +02:00
David CARLIER
d428de590f
DragonFlyBSD resident memory amount (almost) similar as FreeBSD. (#8023) 2020-11-08 09:16:14 +02:00
sundb
cd1c600548
Typo fix: entires -> entries (#8031) 2020-11-08 08:32:38 +02:00
Wen Hui
254367788a
Debug Populate: Avoid server crash when passing negative value for key and value size (#8018)
* Debug Populate: Add checks for count and keysize to avoid crash

* provide getRangeLongFromObjectOrReply and getPositiveLongFromObjectOrReply for range check
2020-11-05 19:58:54 +02:00
Yossi Gottlieb
7e4325cbc9
Fix crash log output on ARM. (#8020) 2020-11-05 15:43:53 +02:00
Oran Agra
7ace7231c6
Better INFO fields to track diskless and disk-based replication progress (#7981)
Expose new `loading_rdb_used_mem` showing the used memory of the server
that saved the RDB file we're currently using.
This is useful in diskless replication when the total size of the rdb is
unkown, and can be used as a rought estimation of progres.

Use that new field to calculate the "user friendly"
`loading_loaded_perc` and `loading_eta_seconds`.

Expose `master_sync_total_bytes` and `master_sync_total_bytes` to complement
on the existing `master_sync_total_bytes` (which cannot be used on its own
to calculate progress).

Add "user friendly" field for `master_sync_perc`
2020-11-05 11:46:16 +02:00
Yossi Gottlieb
1fd456f91a
Add RESET command. (#7982)
Perform full reset of all client connection states, is if the client was
disconnected and re-connected. This affects:

* MULTI state
* Watched keys
* MONITOR mode
* Pub/Sub subscription
* ACL/Authenticated state
* Client tracking state
* Cluster read-only/asking state
* RESP version (reset to 2)
* Selected database
* CLIENT REPLY state

The response is +RESET to make it easily distinguishable from other
responses.

Co-authored-by: Oran Agra <oran@redislabs.com>
Co-authored-by: Itamar Haber <itamar@redislabs.com>
2020-11-05 10:51:26 +02:00
Tomasz Poradowski
d8fbd3a8ba
rdb: corrected RedisModuleIO initialization point (#8014)
- rdbSaveSingleModuleAux() used RedisModuleIO's "bytes" field for
  tracking written bytes before calling moduleInitIOContext() which sets
  "bytes" to zero
- rdbSaveObject() re-initialized RedisModuleIO too late

This return value is not used at the moment since it's only tested
against -1, and the actual byte count isn't used yet.

Co-authored-by: Tomasz Poradowski <tomasz.poradowski@generiscorp.com>
2020-11-04 16:19:18 +02:00
filipe oliveira
10b5006934
Enable specifying TLS ciphers(suites) in redis-cli/redis-benchmark (#8005)
Enable specifying the preferred ciphers and/or ciphersuites for redis-cli/redis-benchmark.

Co-authored-by: Yossi Gottlieb <yossigo@gmail.com>
2020-11-04 14:49:15 +02:00
Egor Seredin
f4ca3d8757
Allow '\0' inside of result of sdscatvprintf, and efficiency improvements (#6260)
This will allow to use: RedisModule_CreateStringPrintf(ctx, "%s %c %s", "string1", 0, "string2");

On large string, the previous code would incrementally retry to double the output buffer.
now it uses the the return value of snprintf and grows to the right size in one step.

and also avoids an excessive strlen in sdscat at the end.
2020-11-04 13:38:46 +02:00
Wen Hui
639b73cd2a
redis-cli cluster import support source and target that need auth (#7994)
Make it possible for redis-cli cluster import to work with source and
target that require AUTH.

Adding two different flags --cluster-from-user, --cluster-from-pass
and --cluster-askpass for source node authentication.
Also for target authentication, using existing --user and --pass flag.

Example:

./redis-cli --cluster import 127.0.0.1:7000 --cluster-from 127.0.0.1:6379 --pass 1234 --user default --cluster-from-user default --cluster-from-pass 123456

./redis-cli --cluster import 127.0.0.1:7000 --cluster-from 127.0.0.1:6379 --askpass --cluster-from-user default --cluster-from-askpass
2020-11-04 10:00:18 +02:00
Oran Agra
a698a6391a
Add maxclients and cluster_connections to INFO CLIENTS (#7979)
Few config settings are also reflected by the INFO command.
these are mainly ones that are important for either an instant view of
the server status (to compare a metric to it's limit config),
Important configurations that are necessary in the crash log (which
currently doesn't print the config),
And things that are important for monitoring solutions (such as
Prometheus), which rely on INFO to collect their data.

Add cluster_connections to INFO CLUSTER:
This makes it possible to be combined together with connected_clients
and connected_slaves and be matched against maxclients
2020-11-04 09:53:43 +02:00
Wang Yuan
89c78a9808
Disable rehash when redis has child process (#8007)
In redisFork(), we don't set child pid, so updateDictResizePolicy()
doesn't take effect, that isn't friendly for copy-on-write.

The bug was introduced this in redis 6.0: 56258c6
2020-11-03 17:16:11 +02:00
Meir Shpilraien (Spielrein)
f210e197f3
Added crash report on SIGABRT (#8004)
The reason that we want to get a full crash report on SIGABRT
is that the jmalloc, when detecting a corruption, calls abort().
This will cause the Redis to exist silently without any report
and without any way to analyze what happened.
2020-11-03 14:59:21 +02:00
Oran Agra
9122379abc
Propagate GETSET and SET-GET as SET (#7957)
- Generates a more backwards compatible command stream
- Slightly more efficient execution in replica/AOF
- Add a test for coverage
2020-11-03 14:56:57 +02:00
guybe7
1a91a2700b
Modules: Improve timer accuracy (#7987)
The bug occurs when 'callback' re-registers itself to a point
in the future and the execution time in non-negligible:
'now' refers to time BEFORE callback was executed and is used
to calculate 'next_period'.
We must get the actual current time when calculating 'next_period'
2020-11-02 18:18:42 +02:00
yoav-steinberg
84b3c18f71
Add local address to CLIENT LIST, and a CLIENT KILL filter. (#7913)
Useful when you want to know through which bind address the client connected to
the server in case of multiple bind addresses.

- Adding `laddr` field to CLIENT list showing the local (bind) address.
- Adding `LADDR` option to CLIENT KILL to kill all the clients connected
  to a specific local address.
- Refactoring to share code.
2020-10-28 21:13:44 +02:00
Oran Agra
441bfa2dfb
Optionally (default) fail to start if requested bind address is not available (#7936)
Background:
#3467 (redis 4.0.0), started ignoring ENOPROTOOPT, but did that only for
the default bind (in case bind config wasn't explicitly set).
#5598 (redis 5.0.3), added that for bind addresses explicitly set
(following bug reports in Debian for redis 4.0.9 and 5.0.1), it
also ignored a bunch of other errors like EPROTONOSUPPORT which was
requested in #3894, and also added EADDRNOTAVAIL (wasn't clear why).

This (ignoring EADDRNOTAVAIL) makes redis start successfully, even if a
certain network interface isn't up yet , in which case we rather redis
fail and will be re-tried when the NIC is up, see #7933.

However, it turns out that when IPv6 is disabled (supported but unused),
the error we're getting is EADDRNOTAVAIL. and in many systems the
default config file tries to bind to localhost for both v4 and v6 and
would like to silently ignore the error on v6 if disabled.
This means that we sometimes want to ignore EADDRNOTAVAIL and other times
we wanna fail.

So this commit changes these main things:
1. Ignore all the errors we ignore for both explicitly requested bind
   address and a default implicit one.
2. Add a '-' prefix to allow EADDRNOTAVAIL be ignored (by default that's
   different than the previous behavior).
3. Restructure that function in a more readable and maintainable way see
   below.
4. Make the default behavior of listening to all achievable by setting
  a bind config directive to * (previously only possible by omitting
  it)
5. document everything.

The old structure of this function was that even if there are no bind
addresses requested, the loop that runs though the bind addresses runs
at least once anyway!
In that one iteration of the loop it binds to both v4 and v6 addresses,
handles errors for each of them separately, and then eventually at the
if-else chain, handles the error of the last bind attempt again!
This was very hard to read and very error prone to maintain, instead now
when the bind info is missing we create one with two entries, and run
the simple loop twice.
2020-10-28 21:09:15 +02:00
Madelyn Olson
d310beb417 White space tweaks and skip categories already applied 2020-10-28 10:01:20 -07:00
Madelyn Olson
411bcf1a41 Further improved ACL algorithm for picking categories 2020-10-28 10:01:20 -07:00
Wen Hui
4342703743
refactor aof rewrite code to avoid memory leaks in error handling (#7976) 2020-10-28 12:35:28 +02:00
sundb
6987176059
docs: Fix some typos in comments and log messge (#7975) 2020-10-28 08:51:35 +02:00
filipe oliveira
39436b2152
TLS Support for redis-benchmark (#7959) 2020-10-28 08:00:54 +02:00