Commit Graph

11039 Commits

Author SHA1 Message Date
Oran Agra
75a950cb93
doc improvement about acl first-arg (#10199)
We recently removed capabilities from the first-arg feature of ACL and added a warning.
but we didn't document it.
ref: #10147 and redis/redis-doc#1761
2022-01-29 18:34:59 +02:00
Binbin
93d951556b
commands arguments improvement about unix-time type (#10203)
Change the name to unix-time-seconds or unix-time-milliseconds
to be consistent. Change the type from INTEGER to UNIX_TIME.

SET (EXAT and PXAT) was already ok.
and naming aside, both PXAT and EXAT everywhere used unit-time (for both milliseconds and seconds).
the only ones that where wrong are GETEX and XCLAIM (using "integer" for both seconds and milliseconds)
2022-01-29 15:21:59 +02:00
weiguo
76784e4ece
Use object size then pointer size when malloc (#10201) (#10202)
By a happy coincidence, sizeof(sds *) is equal to sizeof(sds) here,
while it's logically consistent to use sizeof(sds) instead.
2022-01-29 14:43:16 +02:00
Madelyn Olson
7a89e49859
Added missing documentation about the ACL GETUSER changes in acl-v2 (#10200) 2022-01-28 12:13:44 -08:00
Yossi Gottlieb
068190a802
Update generate-command-help.rb and help.h. (#10196)
There are some inevitable changes between the old and new output, as a
result of the different `commands.json` semantics.
2022-01-27 21:48:06 +02:00
Binbin
d616925835
Allow SET without GET arg on write-only ACL. Allow BITFIELD GET on read-only ACL (#10148)
SET is a R+W command, because it can also do `GET` on the data.
SET without GET is a write-only command.
SET with GET is a read+write command.

In #9974, we added ACL to let users define write-only access.
So when the user uses SET with GET option, and the user doesn't
have the READ permission on the key, we need to reject it,
but we rather not reject users with write-only permissions from using
the SET command when they don't use GET.

In this commit, we add a `getkeys_proc` function to control key
flags in SET command. We also add a new key spec flag (VARIABLE_FLAGS)
means that some keys might have different flags depending on arguments.

We also handle BITFIELD command, add a `bitfieldGetKeys` function.
BITFIELD GET is a READ ONLY command.
BITFIELD SET or BITFIELD INCR are READ WRITE commands.

Other changes:
1. SET GET was added in 6.2, add the missing since in set.json
2. Added tests to cover the changes in acl-v2.tcl
3. Fix some typos in server.h and cleanups in acl-v2.tcl

Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
Co-authored-by: Oran Agra <oran@redislabs.com>
2022-01-26 21:03:21 +02:00
Oran Agra
795ea011ba
Solve race in a BGSAVE test (#10190)
This PR attempts to solve two problems that happen sometime in valgrind:
`ERR Background save already in progress`
and
`not bgsave not aborted`

the test used to populate the database with DEBUG, which didn't
increment the dirty counter, so couldn't trigger an automatic bgsave.
then it used a manual bgsave, and aborted it (when it got aborted it
populated the dirty counter), and then it tried to do another bgsave.
that other bgsave could have failed if the automatic one already
started.
2022-01-26 19:46:02 +02:00
Oran Agra
da48a81290
solve race in expiration test (#10192)
Failed on a non-valgrind run. on this line:
```
assert_equal 0 [$slave exists k]
```
the condition in `keyIsExpired` is `now > when`.
so if the test is really fast, maybe it can get to EXISTS exactly 1000 milliseconds after the
expiration was set, and the key isn't yet gone)
2022-01-26 19:45:31 +02:00
guybe7
c79389f032
Reply for command args should be an array, not a set (#10188) 2022-01-26 12:49:24 +02:00
Binbin
7fdcada67b
Fix unused variable warning in subcommand.c (#10184)
Forgot to handle it in #10135.
2022-01-26 10:21:51 +02:00
Madelyn Olson
f6b76e50ad
Change expression to look for at least one limit exceeded (#10173)
This is an attempt to fix some of the issues with the cluster mode tests we are seeing in the daily run.

The test is trying to incrementally adds a bunch of publish messages, expecting that eventually one
of them will overflow. The tests stops one of the processes, so it expects that just that one Redis node
will overflow. I think the test is flaky because under certain circumstances multiple links are getting
disconnected, not just the one that is stalled.
2022-01-26 09:59:53 +02:00
Viktor Söderqvist
4491ee1805
redis-cli: Aligned RESP3 maps with multiline value in TTY (#10170)
Before:

```
127.0.0.1:6379> command info get
1)  1) "get"
    2) (integer) 2
    3) 1~ readonly
       2~ fast
    4) (integer) 1
    5) (integer) 1
    6) (integer) 1
    7) 1~ @read
       2~ @string
       3~ @fast
    8) (empty set)
    9) 1~ 1# "flags" => 1~ RO
             2~ access
          2# "begin_search" => 1# "type" => "index"
             2# "spec" => 1# "index" => (integer) 1
          3# "find_keys" => 1# "type" => "range"
             2# "spec" => 1# "lastkey" => (integer) 0
                2# "keystep" => (integer) 1
                3# "limit" => (integer) 0
   10) (empty set)
```

After:

```
127.0.0.1:6379> command info get
1)  1) "get"
    2) (integer) 2
    3) 1~ readonly
       2~ fast
    4) (integer) 1
    5) (integer) 1
    6) (integer) 1
    7) 1~ @read
       2~ @string
       3~ @fast
    8) (empty set)
    9) 1~ 1# "flags" =>
             1~ RO
             2~ access
          2# "begin_search" =>
             1# "type" => "index"
             2# "spec" => 1# "index" => (integer) 1
          3# "find_keys" =>
             1# "type" => "range"
             2# "spec" =>
                1# "lastkey" => (integer) 0
                2# "keystep" => (integer) 1
                3# "limit" => (integer) 0
   10) (empty set)
```
2022-01-25 17:15:57 +02:00
Meir Shpilraien (Spielrein)
5a38ccc253
Added engine stats to FUNCTION STATS command. (#10179)
Added the following statistics (per engine) to FUNCTION STATS command:
* number of functions
* number of libraries

Output example:
```
> FUNCTION stats
1) "running_script"
2) (nil)
3) "engines"
4) 1) "LUA"
   2) 1) "libraries_count"
      2) (integer) 1
      3) "functions_count"
      4) (integer) 1
```

To collect the stats, added a new dictionary to libraries_ctx that contains
for each engine, the engine statistics representing the current libraries_ctx.
Update the stats on:
1. Link library to libraries_ctx
2. Unlink library from libraries_ctx
3. Flushing libraries_ctx
2022-01-25 15:50:14 +02:00
yoav-steinberg
d26453a3f8
Fix commands.c build issue on merge (#10172)
Set commands.c's merge driver to binary so when it conflicts during a merge git will leave the local version unmodified.
This way our Makefile will rebuild it based on src/commands/*.json before trying to compile it.
Otherwise the file gets modified with merge conflict markers and gets the same timestamp as the *.json files,
so the Makefile doesn't attempt to rebuild it before compiling and we get a compilation error.
2022-01-25 12:24:06 +02:00
chenyang8094
ac011ebd7f
Delete the residual code related to aof rewrite buf (#10176) 2022-01-25 11:37:18 +02:00
Madelyn Olson
823da54361
Improve testing and update flags around commands without ACL keyspec flags (#10167)
This PR aims to improve the flags associated with some commands and adds various tests around
these cases. Specifically, it's concerned with commands which declare keys but have no ACL
flags (think `EXISTS`), the user needs either read or write permission to access this type of key.

This change is primarily concerned around commands in three categories:

# General keyspace commands
These commands are agnostic to the underlying data outside of side channel attacks, so they are not
marked as ACCESS.
* TOUCH
* EXISTS
* TYPE
* OBJECT 'all subcommands'

Note that TOUCH is not a write command, it could be a side effect of either a read or a write command.

# Length and cardinality commands
These commands are marked as NOT marked as ACCESS since they don't return actual user strings,
just metadata.
* LLEN
* STRLEN
* SCARD
* HSTRLEN

# Container has member commands
These commands return information about the existence or metadata about the key. These commands
are NOT marked as ACCESS since the check of membership is used widely in write commands
e.g. the response of HSET. 
* SISMEMBER
* HEXISTS

# Intersection cardinality commands
These commands are marked as ACCESS since they process data to compute the result.
* PFCOUNT
* ZCOUNT
* ZINTERCARD
* SINTERCARD
2022-01-25 09:55:30 +02:00
Madelyn Olson
c275010fff
Correctly handle minimum arity checks in scripts (#10171)
Correctly handle variable arity checks in scripts
2022-01-24 22:08:57 -08:00
Oran Agra
0343fe9fa5
re-fix EVAL timeout test (#10169)
The edit i made to #10098 before merging was broken.
(INFO is forbidden in LOADING)
2022-01-24 23:01:08 +02:00
chenyang8094
fa60049648
Fix EVAL timeout test failed on freebsd (#10098)
* Refactor EVAL timeout test
* since the test used r config set appendonly yes which generates a rewrite, it missed it's purpose
* Fix the bug that start_server returns before redis starts ready, which affects when multiple tests share the same dir.
* Elapsed time tracking no loner needed

Co-authored-by: Oran Agra <oran@redislabs.com>
2022-01-24 22:31:35 +02:00
Binbin
495ac8b79a
Fix outdated save key word in redis.conf (#10166)
For some complex data types, server.dirty actually counts
the number of elements that have been changed.
And in FLUSHDB or FLUSHALL, we count the number of keys.

So the word "key" is not strictly correct and is outdated.
Some discussion can be seen at #8140.
2022-01-24 22:02:42 +02:00
yoav-steinberg
7eadc5ee70
Support function flags in script EVAL via shebang header (#10126)
In #10025 we added a mechanism for flagging certain properties for Redis Functions.
This lead us to think we'd like to "port" this mechanism to Redis Scripts (`EVAL`) as well. 

One good reason for this, other than the added functionality is because it addresses the
poor behavior we currently have in `EVAL` in case the script performs a (non DENY_OOM) write operation
during OOM state. See #8478 (And a previous attempt to handle it via #10093) for details.
Note that in Redis Functions **all** write operations (including DEL) will return an error during OOM state
unless the function is flagged as `allow-oom` in which case no OOM checking is performed at all.

This PR:
- Enables setting `EVAL` (and `SCRIPT LOAD`) script flags as defined in #10025.
- Provides a syntactical framework via [shebang](https://en.wikipedia.org/wiki/Shebang_(Unix)) for
  additional script annotations and even engine selection (instead of just lua) for scripts.
- Provides backwards compatibility so scripts without the new annotations will behave as they did before.
- Appropriate tests.
- Changes `EVAL[SHA]/_RO` to be flagged as `STALE` commands. This makes it possible to flag individual
  scripts as `allow-stale` or not flag them as such. In backwards compatibility mode these commands will
  return the `MASTERDOWN` error as before.
- Changes `SCRIPT LOAD` to be flagged as a `STALE` command. This is mainly to make it logically
  compatible with the change to `EVAL` in the previous point. It enables loading a script on a stale server
  which is technically okay it doesn't relate directly to the server's dataset. Running the script does, but that
  won't work unless the script is explicitly marked as `allow-stale`.

Note that even though the LUA syntax doesn't support hash tag comments `.lua` files do support a shebang
tag on the top so they can be executed on Unix systems like any shell script. LUA's `luaL_loadfile` handles
this as part of the LUA library. In the case of `luaL_loadbuffer`, which is what Redis uses, I needed to fix the
input script in case of a shebang manually. I did this the same way `luaL_loadfile` does, by replacing the
first line with a single line feed character.
2022-01-24 16:50:02 +02:00
Viktor Söderqvist
857dc5bacd
Disable keyspec module API in 7.0 RC1 (#10135)
The keyspec API is not yet released and there is a plan to change it
in #10108, which is going to be included in RC2. Therefore, we hide
it in RC1 to avoid introducing a breaking change in RC2.

Co-authored-by: Oran Agra <oran@redislabs.com>
2022-01-24 15:02:07 +02:00
chenyang8094
6dc3f09cb9
Fix AOFRW limit test occasional failures on slower machines (#10164) 2022-01-24 14:55:24 +02:00
ny0312
b40a9ba5fd
Fix flaky cluster tests in 24-links.tcl (#10157)
* Fix flaky cluster test "Disconnect link when send buffer limit reached"
* Fix flaky cluster test "Each node has two links with each peer"

Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
2022-01-23 17:28:32 -08:00
Binbin
7e5ded2ad0
Fix timing issue in sentinel CKQUORUM test (#10036)
A test failure was reported in Daily CI (test-centos7-tls).
`CKQUORUM detects failover authorization cannot be reached`.

```
CKQUORUM detects failover authorization cannot be reached: FAILED:
Expected 'invalid command name "OK 4 usable Sentinels. Quorum and failover authorization can be reached"' to match '*NOQUORUM*'
```

It seems that current sentinel does not confirm that the other
sentinels are actually `down`, and then check the quorum.
It at least take 3 seconds on my machine, and we can see there
will be a timing issue with the hard code `after 5000`.

In this commit, we check the response of `SENTINEL SENTINELS mymaster`
to ensure that other sentinels are actually `down` in the view the
current sentinel. Solve the timing issue due to sentinel monitor mechanism.
2022-01-23 13:54:50 +02:00
Binbin
23325c135f
sub-command support for ACL CAT and COMMAND LIST. redisCommand always stores fullname (#10127)
Summary of changes:
1. Rename `redisCommand->name` to `redisCommand->declared_name`, it is a
  const char * for native commands and SDS for module commands.
2. Store the [sub]command fullname in `redisCommand->fullname` (sds).
3. List subcommands in `ACL CAT`
4. List subcommands in `COMMAND LIST`
5. `moduleUnregisterCommands` now will also free the module subcommands.
6. RM_GetCurrentCommandName returns full command name

Other changes:
1. Add `addReplyErrorArity` and `addReplyErrorExpireTime`
2. Remove `getFullCommandName` function that now is useless.
3. Some cleanups about `fullname` since now it is SDS.
4. Delete `populateSingleCommand` function from server.h that is useless.
5. Added tests to cover this change.
6. Add some module unload tests and fix the leaks
7. Make error messages uniform, make sure they always contain the full command
  name and that it's quoted.
7. Fixes some typos

see the history in #9504, fixes #10124

Co-authored-by: Oran Agra <oran@redislabs.com>
Co-authored-by: guybe7 <guy.benoish@redislabs.com>
2022-01-23 10:05:06 +02:00
guybe7
a6fd2a46d1
Improved handling of subcommands (don't allow ACL on first-arg of a sub-command) (#10147)
Recently we added extensive support for sub-commands in for redis 7.0,
this meant that the old ACL mechanism for
sub-commands wasn't needed, or actually was improved (to handle both include
and exclude control, like for commands), but only for real sub-commands.
The old mechanism in ACL was renamed to first-arg, and was able to match the
first argument of any command (including sub-commands).
We now realized that we might wanna completely delete that first-arg feature some
day, so the first step was not to give it new capabilities in 7.0 and it didn't have before.

Changes:
1. ACL: Block the first-arg mechanism on subcommands (we keep if in non-subcommands
  for backward compatibility)
2. COMMAND: When looking up a command, insist the command name doesn't contain
  extra words. Example: When a user issues `GET key` we want `lookupCommand` to return
  `getCommand` but when if COMMAND calls `lookupCommand` with `get|key` we want it to fail.

Other changes:
1. ACLSetUser: prevent a redundant command lookup
2022-01-22 14:09:40 +02:00
Madelyn Olson
55c81f2cd3
ACL V2 - Selectors and key based permissions (#9974)
* Implemented selectors which provide multiple different sets of permissions to users
* Implemented key based permissions 
* Added a new ACL dry-run command to test permissions before execution
* Updated module APIs to support checking key based permissions

Co-authored-by: Oran Agra <oran@redislabs.com>
2022-01-20 13:05:27 -08:00
guybe7
10bbeb6837
Add command tips to COMMAND DOCS (#10104)
Adding command tips (see https://redis.io/topics/command-tips) to commands.

Breaking changes:
1. Removed the "random" and "sort_for_script" flags. They are now command tips.
(this isn't affecting redis behavior since #9812, but could affect some client applications
that's relying on COMMAND command flags)

Summary of changes:
1. add BLOCKING flag (new flag) for all commands that could block. The ACL category with
  the same name is now implicit.
2. move RANDOM flag to a `nondeterministic_output` tip
3. move SORT_FOR_SCRIPT flag to `nondeterministic_output_order` tip
3. add REQUEST_POLICY and RESPONSE_POLICY where appropriate as documented in the tips
4. deprecate (ignore) the `random` flag for RM_CreateCommand

Other notes:
1. Proxies need to send `RANDOMKEY` to all shards and then select one key randomly.
  The other option is to pick a random shard and transfer `RANDOMKEY `to it, but that scheme
  fails if this specific shard is empty
2. Remove CMD_RANDOM from `XACK` (i.e. XACK does not have RANDOM_OUTPUT)
   It was added in 9e4fb96ca1, I guess by mistake.
   Also from `(P)EXPIRETIME` (new command, was flagged "random" by mistake).
3. Add `nondeterministic_output` to `OBJECT ENCODING` (for the same reason `XTRIM` has it:
   the reply may differ depending on the internal representation in memory)
4. RANDOM on `HGETALL` was wrong (there due to a limitation of the old script sorting logic), now
  it's `nondeterministic_output_order`
5. Unrelated: Hide CMD_PROTECTED from COMMAND
2022-01-20 11:32:11 +02:00
Meir Shpilraien (Spielrein)
9c60292250
Added functions support to redis-check-rdb (#10154)
The PR added the missing verification for functions on redis-check-rdb.
The verification only verifies the rdb structure and does not try to load the functions code and
verify more advance checks (like compilation of the function code).
2022-01-20 11:10:33 +02:00
perryitay
c4b788230c
Adding module api for processing commands during busy jobs and allow flagging the commands that should be handled at this status (#9963)
Some modules might perform a long-running logic in different stages of Redis lifetime, for example:
* command execution
* RDB loading
* thread safe context

During this long-running logic Redis is not responsive.

This PR offers 
1. An API to process events while a busy command is running (`RM_Yield`)
2. A new flag (`ALLOW_BUSY`) to mark the commands that should be handled during busy
  jobs which can also be used by modules (`allow-busy`)
3. In slow commands and thread safe contexts, this flag will start rejecting commands with -BUSY only
  after `busy-reply-threshold`
4. During loading (`rdb_load` callback), it'll process events right away (not wait for `busy-reply-threshold`),
  but either way, the processing is throttled to the server hz rate.
5. Allow modules to Yield to redis background tasks, but not to client commands

* rename `script-time-limit` to `busy-reply-threshold` (an alias to the pre-7.0 `lua-time-limit`)

Co-authored-by: Oran Agra <oran@redislabs.com>
2022-01-20 09:05:53 +02:00
Madelyn Olson
22172a4aa6
Fix double key declaration for renamenx and change flag to INSERT (#10137)
* Fix double key declaration for renamenx
* Change the flag from UPDATE to INSERT, since it can not affect the data in the key
2022-01-19 19:30:32 -08:00
Meir Shpilraien (Spielrein)
c556c57e5e
Added AOF rewrite support for functions. (#10141)
Function PR was merged without AOF rw support because we thought this feature was going
to be removed on Redis 7.

Tests was added on aofrw.tcl
Other existing aofrw tests where slow due to unwanted rdb-key-save-delay

Co-authored-by: Oran Agra <oran@redislabs.com>
2022-01-19 21:21:42 +02:00
David CARLIER
834fa5870c
zmalloc_get_rss openbsd implementation (#10149)
Add support for getting the RSS in OpenBSD
2022-01-19 20:56:12 +02:00
Binbin
3720de78c9
Fix fcall_ro key spec flags in commands.c (#10145)
In #10122, we modify the key spec flags to `RO` and `ACCESS`.
But forgot to call generate-command-code.py. Also formatted
it to follow the Python PEP8.
2022-01-19 16:07:28 +02:00
Wen Hui
68a8d0b46d
Add sentinel config set test case (#10114) 2022-01-19 11:57:51 +02:00
Ozan Tezcan
72e1b5de4d
Fix replica count check in migration tests. (#10140)
Tests were not using loop index as node id, checking replica count of the same node over and over.
2022-01-19 11:36:24 +02:00
Ozan Tezcan
1af0a2c5ae
Fix eventloop module test for valgrind (#10139)
was eating too much memory, and taking too long with valgrind
2022-01-19 09:13:51 +02:00
Oran Agra
eef9c6b0ee
New detailed key-spec flags (RO, RW, OW, RM, ACCESS, UPDATE, INSERT, DELETE) (#10122)
The new ACL key based permissions in #9974 require the key-specs (#8324) to have more
explicit flags rather than just READ and WRITE. See discussion in #10040

This PR defines two groups of flags:
One about how redis internally handles the key (mutually-exclusive).
The other is about the logical operation done from the user's point of view (3 mutually exclusive
write flags, and one read flag, all optional).
In both groups, if we can't explicitly flag something as explicit read-only, delete-only, or
insert-only, we flag it as `RW` or `UPDATE`.
here's the definition from the code:
```
/* Key-spec flags *
 * -------------- */
/* The following refer what the command actually does with the value or metadata
 * of the key, and not necessarily the user data or how it affects it.
 * Each key-spec may must have exaclty one of these. Any operation that's not
 * distinctly deletion, overwrite or read-only would be marked as RW. */
#define CMD_KEY_RO (1ULL<<0)     /* Read-Only - Reads the value of the key, but
                                  * doesn't necessarily returns it. */
#define CMD_KEY_RW (1ULL<<1)     /* Read-Write - Modifies the data stored in the
                                  * value of the key or its metadata. */
#define CMD_KEY_OW (1ULL<<2)     /* Overwrite - Overwrites the data stored in
                                  * the value of the key. */
#define CMD_KEY_RM (1ULL<<3)     /* Deletes the key. */
/* The follwing refer to user data inside the value of the key, not the metadata
 * like LRU, type, cardinality. It refers to the logical operation on the user's
 * data (actual input strings / TTL), being used / returned / copied / changed,
 * It doesn't refer to modification or returning of metadata (like type, count,
 * presence of data). Any write that's not INSERT or DELETE, would be an UPADTE.
 * Each key-spec may have one of the writes with or without access, or none: */
#define CMD_KEY_ACCESS (1ULL<<4) /* Returns, copies or uses the user data from
                                  * the value of the key. */
#define CMD_KEY_UPDATE (1ULL<<5) /* Updates data to the value, new value may
                                  * depend on the old value. */
#define CMD_KEY_INSERT (1ULL<<6) /* Adds data to the value with no chance of,
                                  * modification or deletion of existing data. */
#define CMD_KEY_DELETE (1ULL<<7) /* Explicitly deletes some content
                                  * from the value of the key. */
```

Unrelated changes:
- generate-command-code.py is only compatible with python3 (modified the shabang)
- generate-command-code.py print file on json parsing error
- rename `shard_channel` key-spec flag to just `channel`.
- add INCOMPLETE flag in input spec of SORT and SORT_RO
2022-01-18 16:00:00 +02:00
Wang Yuan
d697daa7a5
Use const char pointer in redismodule.h as far as possible (#10064)
When I used C++ to develop a redis module. i  used `string.data()` as the second parameter `ele`
of  `RedisModule_DigestAddStringBuffer`, but there is a warning, since we never change the `ele`,
i think we should use `const char` for it.

This PR adds const to just a handful of module APIs that required it, all not very widely used.
The implication is a breaking change in terms of compilation error that's easy to resolve, and no ABI impact.
The affected APIs are around Digest, Info injection, and Cluster bus messages.
2022-01-18 15:55:20 +02:00
Ozan Tezcan
99ab4236af
Add event loop support to the module API (#10001)
Modules can now register sockets/pipe to the Redis main thread event loop and do network operations asynchronously. Previously, modules had to maintain an event loop and another thread for asynchronous network operations.

Also, if a module is calling API functions after doing some network operations, it had to synchronize its event loop thread's access with Redis main thread by locking the GIL, causing contention on the lock. After this commit, no synchronization is needed as module can operate in Redis main thread context. So, this commit may improve the performance for some use cases.

Added three functions to the module API:

* RedisModule_EventLoopAdd(int fd, int mask, RedisModuleEventLoopFunc func, void *user_data)
* RedisModule_EventLoopDel(int fd, int mask)
* RedisModule_EventLoopAddOneShot(RedisModuleEventLoopOneShotFunc func, void *user_data) - This function can be called from other threads to trigger callback on Redis main thread. Callback will be triggered only once. If Redis main thread is sleeping, this call will wake up the Redis main thread.
Event loop callbacks are called by Redis main thread after locking the GIL. Inside callbacks, modules can operate as if they are holding the GIL.

Added REDISMODULE_EVENT_EVENTLOOP event with two subevents:

* REDISMODULE_SUBEVENT_EVENTLOOP_BEFORE_SLEEP
* REDISMODULE_SUBEVENT_EVENTLOOP_AFTER_SLEEP

These events are for modules that want to participate in the before and after sleep action. e.g It might be useful to implement batching : Read data from the network, write all to a file in one go on BEFORE_SLEEP event.
2022-01-18 13:10:07 +02:00
Yossi Gottlieb
25e6d4d459
Fix additional AOF filename issues. (#10110)
This extends the previous fix (#10049) to address any form of
non-printable or whitespace character (including newlines, quotes,
non-printables, etc.)

Also, removes the limitation on appenddirname, to align with the way
filenames are handled elsewhere in Redis.
2022-01-18 12:52:27 +02:00
Meir Shpilraien (Spielrein)
51f9bed3dd
Fix FUNCTION LOAD ignores unknown parameter. (#10131)
Following discussion on: https://github.com/redis/redis/issues/9899#issuecomment-1014689385
Raise error if unknows parameter is given to `FUNCTION LOAD`.

Before the fix:
```
127.0.0.1:6379> function load LUA lib2 foo bar "local function test1() return 5 end redis.register_function('test1', test1)"
OK
```

After the fix:
```
127.0.0.1:6379> function load LUA lib2 foo bar "local function test1() return 5 end redis.register_function('test1', test1)"
(error) ERR Unkowns option given: foo
```
2022-01-18 10:29:52 +02:00
Oran Agra
ae89958972
Set repl-diskless-sync to yes by default, add repl-diskless-sync-max-replicas (#10092)
1. enable diskless replication by default
2. add a new config named repl-diskless-sync-max-replicas that enables
   replication to start before the full repl-diskless-sync-delay was
   reached.
3. put replica online sooner on the master (see below)
4. test suite uses repl-diskless-sync-delay of 0 to be faster
5. a few tests that use multiple replica on a pre-populated master, are
   now using the new repl-diskless-sync-max-replicas
6. fix possible timing issues in a few cluster tests (see below)

put replica online sooner on the master 
----------------------------------------------------
there were two tests that failed because they needed for the master to
realize that the replica is online, but the test code was actually only
waiting for the replica to realize it's online, and in diskless it could
have been before the master realized it.

changes include two things:
1. the tests wait on the right thing
2. issues in the master, putting the replica online in two steps.

the master used to put the replica as online in 2 steps. the first
step was to mark it as online, and the second step was to enable the
write event (only after getting ACK), but in fact the first step didn't
contains some of the tasks to put it online (like updating good slave
count, and sending the module event). this meant that if a test was
waiting to see that the replica is online form the point of view of the
master, and then confirm that the module got an event, or that the
master has enough good replicas, it could fail due to timing issues.

so now the full effect of putting the replica online, happens at once,
and only the part about enabling the writes is delayed till the ACK.

fix cluster tests 
--------------------
I added some code to wait for the replica to sync and avoid race
conditions.
later realized the sentinel and cluster tests where using the original 5
seconds delay, so changed it to 0.

this means the other changes are probably not needed, but i suppose
they're still better (avoid race conditions)
2022-01-17 14:11:11 +02:00
zhaozhao.zz
90916f16a5
show subcommands latencystats (#10103)
since `info commandstats` already shows sub-commands, we should do the same in `info latencystats`.
similarly, the LATENCY HISTOGRAM command now shows sub-commands (with their full name) when:
* asking for all commands
* asking for a specific container command
* asking for a specific sub-command)

Co-authored-by: Oran Agra <oran@redislabs.com>
2022-01-17 12:32:32 +02:00
Binbin
26ef5132a6
Fix timing issue in PSETEX/PEXPIRE sub-second expire tests (#10121)
These two tests have a high probability of failure
on MacOS. Or it takes many retries to succeed.
Keys often expire before we can access them.

So this time we try to avoid this by reducing the time
of the first `after`, or removeing the first `after`.

The results of doing `20/81` and `0/101` are:
- PEXPIRE (20/81): 1069/1949
- PEXPIREAT (20/81): 1093/1949

- PEXPIRE (0/101): 31936 / 31936
- PEXPIREAT (0/101): 31936 / 31936

The first number is the number of times that the
test succeeded without any retries.
The second number is the total number of executions.

And we can see that `0/101` doesn't even need an extra
retries. Also reduces the time required for testing.
So in the end we chose `0/100`, i.e. remove the first `after`.

As for `PEXPIREAT`, there is no failure, but we still changed
it together, using `0/201`, after 2W tests, none of them failed.
2022-01-17 10:42:13 +02:00
chenyang8094
d7aeb2ce5f
Use am instead of using server.aof_manifest directly to call getBaseAndIncrAppendOnlyFilesSize (#10123) 2022-01-17 09:02:09 +02:00
David CARLIER
50fa627b90
zmalloc_get_rss netbsd impl fix proposal. (#10116)
Seems like the previous implementation was broken (always returning 0)

since kinfo_proc2 is used the KERN_PROC2 sysctl oid is more appropriate
and also the query's length was not necessarily accurate (6 here).
2022-01-16 10:03:09 +02:00
sundb
32e7b46a17
Fix quicklist node not being recompressed correctly after inserting a new node before or after it (#10120)
### Describe
Fix crash found by CI, Introduced by #9849.
When we do any operation on the quicklist, we should make sure that all nodes
of the quicklist should not be in the recompressed state.

### Issues
This PR fixes two issues with incorrect recompression.
1. The current quicklist node is full and the previous node isn't full,
    the current node is not recompressed correctly after inserting elements into the previous node.
2. The current quicklist node is full and the next node isn't full,
    the current node is not recompressed correctly after inserting elements into the next node.

### Test
Add two tests to cover incorrect compression issues.

### Other
Fix unittest test failure caused by assertion introduced by #9849.
2022-01-16 08:54:40 +02:00
David CARLIER
7da7d2aa8e
checkTcpBacklogSettings check for solaris based systems. (#10109) 2022-01-15 20:39:05 +02:00