Commit Graph

9919 Commits

Author SHA1 Message Date
Yossi Gottlieb
84e0489562
Remove end-of-life CentOS 6 workflows. (#8121) 2020-12-02 13:47:25 +02:00
luhuachao
7885faf18b
Modify help msg PING_BULK to PING_MBULK in benchmark (#8109)
As described in redis-benchamrk help message 'The test names are the same as the ones produced as output.', In redis-benchmark output, we can only see PING_BULK, but the cmd `redis-benchmark -t ping_bulk` is not supported. We  have to run it with ping_mbulk which is not user friendly.
2020-12-02 13:17:25 +02:00
Madelyn Olson
69b7113bb5
Getset fix (#8118)
* Fixed SET GET executing on wrong type

Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
2020-12-01 11:46:45 -08:00
sundb
3ba2281f96
Improve dbid range check for SELECT, MOVE, COPY (#8085)
SELECT used to read the index into a `long` variable, and then pass it to a function
that takes an `int`, possibly causing an overflow before the range check.

Now all these commands use better and cleaner range check, and that also results in
a slight change of the error response in case of an invalid database index.

SELECT:
in the past it would have returned either `-ERR invalid DB index` (if not a number),
or `-ERR DB index is out of range` (if not between 1..16 or alike).
now it'll return either `-ERR value is out of range` (if not a number), or
`-ERR value is out of range, value must between -2147483648 and 2147483647`
(if not in the range for an int), or `-ERR DB index is out of range`
(if not between 0..16 or alike)


MOVE:
in the past it would only fail with `-ERR index out of range` no matter the reason.
now return the same errors as the new ones for SELECT mentioned above.
(i.e. unlike for SELECT even for a value like 17 we changed the error message)

COPY:
doesn't really matter how it behaved in the past (new command), new behavior is
like the above two.
2020-12-01 21:41:26 +02:00
Itamar Haber
c1b1e8c329
Adds pub/sub channel patterns to ACL (#7993)
Fixes #7923.

This PR appropriates the special `&` symbol (because `@` and `*` are taken),
followed by a literal value or pattern for describing the Pub/Sub patterns that
an ACL user can interact with. It is similar to the existing key patterns
mechanism in function (additive) and implementation (copy-pasta). It also adds
the allchannels and resetchannels ACL keywords, naturally.

The default user is given allchannels permissions, whereas new users get
whatever is defined by the acl-pubsub-default configuration directive. For
backward compatibility in 6.2, the default of this directive is allchannels but
this is likely to be changed to resetchannels in the next major version for
stronger default security settings.

Unless allchannels is set for the user, channel access permissions are checked
as follows :
* Calls to both PUBLISH and SUBSCRIBE will fail unless a pattern matching the
  argumentative channel name(s) exists for the user.
* Calls to PSUBSCRIBE will fail unless the pattern(s) provided as an argument
  literally exist(s) in the user's list.

Such failures are logged to the ACL log.

Runtime changes to channel permissions for a user with existing subscribing
clients cause said clients to disconnect unless the new permissions permit the
connections to continue. Note, however, that PSUBSCRIBErs' patterns are matched
literally, so given the change bar:* -> b*, pattern subscribers to bar:* will be
disconnected.

Notes/questions:
* UNSUBSCRIBE, PUNSUBSCRIBE and PUBSUB remain unprotected due to lack of reasons
  for touching them.
2020-12-01 14:21:39 +02:00
Wang Yuan
c85bf2352d
Reset average ttl when empty databases (#8106)
On FLUSHDB or full sync, reset old average TTL stat.
This Stat is incrementally collected by the master over time when it searches for expired keys.
2020-11-30 23:15:14 +02:00
sundb
04056b767f
BITOP speedup when or/and output is 0/255, stop processing further keys (#8110)
when performing the and operation, if the output is 0, we can jump out of the loop.
when performing an or operation, if the output is 0xff, we can jump out of the loop.
2020-11-30 23:03:53 +02:00
Itamar Haber
d322e7baba
Adds support for nested args in cli's help (#8074)
See https://github.com/redis/redis-doc/pull/1443

Also allows nameless commands.
2020-11-30 15:00:16 +02:00
guybe7
ada2ac9ae2
XPENDING with IDLE (#7972)
Used to filter stream pending entries by their idle-time,
useful for XCLAIMing entries that have not been processed
for some time
2020-11-29 12:08:47 +02:00
Oran Agra
cb5eadb33b
bitops limited to proto_max_bulk_len rather than 512MB (#8096)
we recently did that for SETRANGE and APPEND
2020-11-26 10:58:01 +02:00
Oran Agra
bbc2c44541
INFO client_recent_max_input_buffer includes argv array (#8065)
this metric already includes the argv bytes, like what clientsCronTrackClientsMemUsage does, but it's missing the array itself.

p.s. For the purpose of tracking expensive clients we don't need to include the size of the client struct and the static reply buffer in it.
2020-11-25 23:39:01 +02:00
kukey
cf88779527
Merge two aeDeleteFileEvent refs into one (#7521)
Merge two aeDeleteFileEvent refs into one
2020-11-25 13:37:54 -08:00
Dipankar Achinta
79a7c17176
Fix typo in ae.c file (#7895) 2020-11-25 13:36:21 -08:00
David CARLIER
0719388cfb
raspberry build fix. (#8095)
__ILP32__ is 32 bits ABI and does not imply x86, this patch resolves this.
2020-11-25 12:15:32 -08:00
sundb
25f457c7f6
Avoid excessive malloc and free in copyCommand robj creation (#8067)
Avoid multiple conditional judgments
Avoid allocating robj->ptr when we're gonna replace it right after.
2020-11-24 21:40:58 +02:00
Yossi Gottlieb
7e5a6313f0
Fix use-after-free issue in spt_copyenv. (#8088)
Seems to have gone unnoticed for a long time, because at least with
glibc it will only be triggered if setenv() was called before spt_init,
which Redis doesn't.

Fixes #8064.
2020-11-24 17:58:10 +02:00
David CARLIER
f16b52cb7d
redis_set_thread_title support for Haiku. (#8060) 2020-11-23 16:14:33 +02:00
Itamar Haber
0963edbc65
Adds tests for XADD/XTRIM's MAXLEN arguments (#8083) 2020-11-23 14:37:58 +02:00
xindoo
0b1d89d7ae
fix comment error about zslDeleteRangeByScore range (#8075)
Co-authored-by: Oran Agra <oran@redislabs.com>
2020-11-22 14:56:45 +02:00
Yossi Gottlieb
08d3e929e5
Clean up building with USE_SYSTEMD. (#8073)
When USE_SYSTEMD=yes is specified, try to use pkg-config to determine
libsystemd linker flags. If not found, silently fall back to simply
using "-lsystemd".

We now use a LIBSYSTEMD_LIBS variable so users can explicitly override
it and specify their own library.

If USE_SYSTEMD is unspecified the old behavior of auto-enabling it if
both pkg-config and libsystemd are available is retained.
2020-11-22 14:40:38 +02:00
Wang Yuan
f207e1682f
Fix diskless replication failure when has non-rdb child process (#8070)
If we enable diskless replication, set repl-diskless-sync-delay to 0,
and master has non-rdb child process such as rewrite aof child, master
will try to start to a new BGSAVE but fails immediately (before fork)
when replicas ask for full synchronization, and master always fails
to start a new BGSAVE and disconnects with replicas until non-rdb
child process exists.

this bug was introduced in #6271 (not yet released in 6.0.x)
2020-11-22 14:12:45 +02:00
Oran Agra
e6fa47380a
Fix bug with module GIL being released prematurely (#8061)
This is hopefully usually harmles.
The server.ready_keys will usually be empty so the code after releasing
the GIL will soon be done.
The only case where it'll actually process things is when a module
releases a client (or module) blocked on a key, by triggering this NOT
from within a command (e.g. a timer event).

This bug was introduced in redis 6.0.9, see #7903
2020-11-22 14:00:51 +02:00
Oran Agra
61954951ed
Fix oom-score-adj-values range, abs options, and bug when used in config file (#8046)
Fix: When oom-score-adj-values is provided in the config file after
oom-score-adj yes, it'll take an immediate action, before
readOOMScoreAdj was acquired, resulting in an error (out of range score
due to uninitialized value. delay the reaction the real call is made by
main().

Since the values are clamped to -1000..1000, and they're
applied as an offset from the value at startup (which may be -1000), we
need to allow the offsets to reach to +2000 so that a value of +1000 is
achievable in case the value at startup was -1000.

Adding an option for absolute values rather than relative ones.
2020-11-22 13:57:56 +02:00
Rosen Penev
a221a313ae
fix compilation with uClibc-ng (#8054)
backtrace can be compile time disabled.
2020-11-20 18:13:11 +02:00
guybe7
f8ae991717
EXISTS should not alter LRU, OBJECT should not reveal expired keys on replica (#8016)
The bug was introduced by #5021 which only attempted avoid EXIST on an
already expired key from returning 1 on a replica.

Before that commit, dbExists was used instead of
lookupKeyRead (which had an undesired effect to "touch" the LRU/LFU)

Other than that, this commit fixes OBJECT to also come empty handed on
expired keys in replica.

And DEBUG DIGEST-VALUE to behave like DEBUG OBJECT (get the data from
the key regardless of it's expired state)
2020-11-18 11:16:21 +02:00
Meir Shpilraien (Spielrein)
d87a0d0286
Unified MULTI, LUA, and RM_Call with respect to blocking commands (#8025)
Blocking command should not be used with MULTI, LUA, and RM_Call. This is because,
the caller, who executes the command in this context, expects a reply.

Today, LUA and MULTI have a special (and different) treatment to blocking commands:

LUA   - Most commands are marked with no-script flag which are checked when executing
and command from LUA, commands that are not marked (like XREAD) verify that their
blocking mode is not used inside LUA (by checking the CLIENT_LUA client flag).
MULTI - Command that is going to block, first verify that the client is not inside
multi (by checking the CLIENT_MULTI client flag). If the client is inside multi, they
return a result which is a match to the empty key with no timeout (for example blpop
inside MULTI will act as lpop)
For modules that perform RM_Call with blocking command, the returned results type is
REDISMODULE_REPLY_UNKNOWN and the caller can not really know what happened.

Disadvantages of the current state are:

No unified approach, LUA, MULTI, and RM_Call, each has a different treatment
Module can not safely execute blocking command (and get reply or error).
Though It is true that modules are not like LUA or MULTI and should be smarter not
to execute blocking commands on RM_Call, sometimes you want to execute a command base
on client input (for example if you create a module that provides a new scripting
language like javascript or python).
While modules (on modules command) can check for REDISMODULE_CTX_FLAGS_LUA or
REDISMODULE_CTX_FLAGS_MULTI to know not to block the client, there is no way to
check if the command came from another module using RM_Call. So there is no way
for a module to know not to block another module RM_Call execution.

This commit adds a way to unify the treatment for blocking clients by introducing
a new CLIENT_DENY_BLOCKING client flag. On LUA, MULTI, and RM_Call the new flag
turned on to signify that the client should not be blocked. A blocking command
verifies that the flag is turned off before blocking. If a blocking command sees
that the CLIENT_DENY_BLOCKING flag is on, it's not blocking and return results
which are matches to empty key with no timeout (as MULTI does today).

The new flag is checked on the following commands:

List blocking commands: BLPOP, BRPOP, BRPOPLPUSH, BLMOVE,
Zset blocking commands: BZPOPMIN, BZPOPMAX
Stream blocking commands: XREAD, XREADGROUP
SUBSCRIBE, PSUBSCRIBE, MONITOR
In addition, the new flag is turned on inside the AOF client, we do not want to
block the AOF client to prevent deadlocks and commands ordering issues (and there
is also an existing assert in the code that verifies it).

To keep backward compatibility on LUA, all the no-script flags on existing commands
were kept untouched. In addition, a LUA special treatment on XREAD and XREADGROUP was kept.

To keep backward compatibility on MULTI (which today allows SUBSCRIBE, and PSUBSCRIBE).
We added a special treatment on those commands to allow executing them on MULTI.

The only backward compatibility issue that this PR introduces is that now MONITOR
is not allowed inside MULTI.

Tests were added to verify blocking commands are not blocking the client on LUA, MULTI,
or RM_Call. Tests were added to verify the module can check for CLIENT_DENY_BLOCKING flag.

Co-authored-by: Oran Agra <oran@redislabs.com>
Co-authored-by: Itamar Haber <itamar@redislabs.com>
2020-11-17 18:58:55 +02:00
thomaston
39f716a121
ZREVRANGEBYSCORE Optimization for out of range offset (#5773)
ZREVRANGEBYSCORE key max min [WITHSCORES] [LIMIT offset count]
When the offset is too large, the query is very slow. Especially when the offset is greater than the length of zset it is easy to determine whether the offset is greater than the length of zset at first, and If it exceed the length of zset, then return directly.

Co-authored-by: Oran Agra <oran@redislabs.com>
2020-11-17 18:35:56 +02:00
Yossi Gottlieb
d638b05834
Improve and clean up supervised process support. (#8036)
* Configuration file default should now be "auto".
* Expose "process_supervised" as an info field.
* Log messages improvements (clarify required systemd config, report
  auto-detected supervision mode, etc.)
* Set server.supervised properly, so it can take precedence of
  "daemonize" configuration.
* Produce clear warning if systemd is detected/requested but executable
  is compiled without support for it, instead of silently ignoring.
* Handle systemd notification error on startup, and turn off supervised
  mode if it failed.
2020-11-17 12:52:49 +02:00
swamp0407
ea7cf737a1
Add COPY command (#7953)
Syntax:
COPY <key> <new-key> [DB <dest-db>] [REPLACE]

No support for module keys yet.

Co-authored-by: tmgauss
Co-authored-by: Itamar Haber <itamar@redislabs.com>
Co-authored-by: Oran Agra <oran@redislabs.com>
2020-11-17 12:03:05 +02:00
Oran Agra
9812e88959
Fix memory leaks in newly added ZDIFF (#8056) 2020-11-16 16:37:15 +02:00
chenyangyang
c1aaad06d8
Modules callbacks for lazy free effort, and unlink (#7912)
Add two optional callbacks to the RedisModuleTypeMethods structure, which is `free_effort`
and `unlink`. the `free_effort` callback indicates the effort required to free a module memory.
Currently, if the effort exceeds LAZYFREE_THRESHOLD, the module memory may be released
asynchronously. the `unlink` callback indicates the key has been removed from the DB by redis, and
may soon be freed by a background thread.

Add `lazyfreed_objects` info field, which represents the number of objects that have been
lazyfreed since redis was started.

Add `RM_GetTypeMethodVersion` API, which return the current redis-server runtime value of
`REDISMODULE_TYPE_METHOD_VERSION`. You can use that when calling `RM_CreateDataType` to know
which fields of RedisModuleTypeMethods are gonna be supported and which will be ignored.
2020-11-16 10:34:04 +02:00
Felipe Machado
d8fd48c436
Add new commands ZDIFF and ZDIFFSTORE (#7961)
- Add ZDIFF and ZDIFFSTORE which work similarly to SDIFF and SDIFFSTORE
- Make sure the new WITHSCORES argument that was added for ZUNION isn't considered valid for ZUNIONSTORE

Co-authored-by: Oran Agra <oran@redislabs.com>
2020-11-15 14:14:25 +02:00
sundb
204a14b8cf
Fix make warning in hellohook,c and testmodule.c (#8044)
* Fix build warning in hellohook,c and testmodule.c

* Change ci->id type to (unsigned long long)
2020-11-13 15:16:40 +02:00
Itamar Haber
92ec592520
Adds user parsing to redis-cli URIs (#8048) 2020-11-12 19:19:18 +02:00
Yash Ladha
c170365dcf
cleanup: move list pop logic to single function (#7997)
BLPOP when there are elements in the list works in the same way as LPOP
does. Due to this they also does the same repetitive action and logic
for the same is written at two different places. This is a bad code
practice as the one needs the context to change the BLPOP list pop code
as well when the LPOP code gets changed.

Separated the generic logic from LPOP to a function that is being used
by the BLPOP code as well.
2020-11-12 10:55:51 +02:00
Madelyn Olson
a0576bdeea
Initialize original client argv for aof (#8042)
(this is a fixup for #8006)

Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
2020-11-11 23:58:05 +02:00
Yossi Gottlieb
48a9d63af3
Add timer module API tests. (#8041) 2020-11-11 22:57:33 +02:00
tzongw
d5059ba5e1
Invalidate aeTimer when returning AE_NOMORE (#8022) 2020-11-11 12:10:54 +02:00
Wen Hui
dd1f20edc5
add tracking bcast flag and client redirection in client list (#7995) 2020-11-11 08:22:17 +02:00
Madelyn Olson
3feff7d78a
Rewritten commands are logged as their original command (#8006)
* Rewritten commands are logged as their original command

Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
2020-11-10 13:50:03 -08:00
Itamar Haber
5d0c6b0868
Adds help for '--cluster-yes' (#8035) 2020-11-10 22:02:09 +02:00
sundb
c09b5941ce optimization src/adlist.c:listJoin() 2020-11-09 21:48:56 -08:00
nitaicaro
19c29b6007
Extend client tracking tests (#7998)
Test support for the new map, null and push message types. Map objects are parsed as a list of lists of key value pairs.
for instance: user => john password => 123

will be parsed to the following TCL list:

{{user john} {password 123}}

Also added the following tests:

Redirection still works with RESP3

Able to use a RESP3 client as a redirection client

No duplicate invalidation messages when turning BCAST mode on after normal tracking

Server is able to evacuate enough keys when num of keys surpasses limit by more than defined initial effort

Different clients using different protocols can track the same key

OPTOUT tests

OPTIN tests

Clients can redirect to the same connection

tracking-redir-broken test

HELLO 3 checks

Invalidation messages still work when using RESP3, with and without redirection

Switching to RESP3 doesn't disturb previous tracked keys

Tracking info is correct

Flushall and flushdb produce invalidation messages

These tests achieve 100% line coverage for tracking.c using lcov.
2020-11-09 22:54:47 +02:00
Meir Shpilraien (Spielrein)
97d647a139
Moved RMAPI_FUNC_SUPPORTED location such that it will be visible to modules (#8037)
The RMAPI_FUNC_SUPPORTED was defined in the wrong place on redismodule.h
and was not visible to modules.
2020-11-09 10:46:23 +02:00
bugwz
bcc46a2f25
Fix the init value for repl_id of rdbSaveInfo struct (#8026)
use 40 zeros rather than 30 zeros to match CONFIG_RUN_ID_SIZE.
this doesn't have any real implications.
2020-11-08 09:26:25 +02:00
David CARLIER
d428de590f
DragonFlyBSD resident memory amount (almost) similar as FreeBSD. (#8023) 2020-11-08 09:16:14 +02:00
sundb
cd1c600548
Typo fix: entires -> entries (#8031) 2020-11-08 08:32:38 +02:00
Wen Hui
254367788a
Debug Populate: Avoid server crash when passing negative value for key and value size (#8018)
* Debug Populate: Add checks for count and keysize to avoid crash

* provide getRangeLongFromObjectOrReply and getPositiveLongFromObjectOrReply for range check
2020-11-05 19:58:54 +02:00
Yossi Gottlieb
7e4325cbc9
Fix crash log output on ARM. (#8020) 2020-11-05 15:43:53 +02:00
Oran Agra
7ace7231c6
Better INFO fields to track diskless and disk-based replication progress (#7981)
Expose new `loading_rdb_used_mem` showing the used memory of the server
that saved the RDB file we're currently using.
This is useful in diskless replication when the total size of the rdb is
unkown, and can be used as a rought estimation of progres.

Use that new field to calculate the "user friendly"
`loading_loaded_perc` and `loading_eta_seconds`.

Expose `master_sync_total_bytes` and `master_sync_total_bytes` to complement
on the existing `master_sync_total_bytes` (which cannot be used on its own
to calculate progress).

Add "user friendly" field for `master_sync_perc`
2020-11-05 11:46:16 +02:00