Commit Graph

11039 Commits

Author SHA1 Message Date
小令童鞋
3bcc67d14f
Add appendonlydir-* in utils/create-cluster/.gitignore (#10471) 2022-03-26 09:53:10 +03:00
zhaozhao.zz
78bef6e1fe
optimize(remove) usage of client's pending_querybuf (#10413)
To remove `pending_querybuf`, the key point is reusing `querybuf`, it means master client's `querybuf` is not only used to parse command, but also proxy to sub-replicas.

1. add a new variable `repl_applied` for master client to record how many data applied (propagated via `replicationFeedStreamFromMasterStream()`) but not trimmed in `querybuf`.

2. don't sdsrange `querybuf` in `commandProcessed()`, we trim it to `repl_applied` after the whole replication pipeline processed to avoid fragmented `sdsrange`. And here are some scenarios we cannot trim to `qb_pos`:
    * we don't receive complete command from master
    * master client blocked because of client pause
    * IO threads operate read, master client flagged with CLIENT_PENDING_COMMAND

    In these scenarios, `qb_pos` points to the part of the current command or the beginning of next command, and the current command is not applied yet, so the `repl_applied` is not equal to `qb_pos`.

Some other notes:
* Do not do big arg optimization on master client, since we can only sdsrange `querybuf` after data sent to replicas.
* Set `qb_pos` and `repl_applied` to 0 when `freeClient` in `replicationCacheMaster`.
* Rewrite `processPendingCommandsAndResetClient` to `processPendingCommandAndInputBuffer`, let `processInputBuffer` to be called successively after `processCommandAndResetClient`.
2022-03-25 10:45:40 +08:00
Oran Agra
1a57af629c
Split daily CI into smaller chunks (#10469)
this should aid find the CI issues with freebsd and macos runs, and also
get faster results from valgrind and tls
2022-03-22 17:38:01 +02:00
Oran Agra
6761d10cc3
crash log, print killer pid only when si_code is SI_USER (#10454)
Avoid printing "Killed by PID" when si_code != SI_USER.
Apparently SI_USER isn't always set to 0. e.g. on Mac it's 0x10001 and the check that did <= was wrong.
2022-03-22 17:37:34 +02:00
Meir Shpilraien (Spielrein)
f3855a0930
Add new RM_Call flags for script mode, no writes, and error replies. (#10372)
The PR extends RM_Call with 3 new capabilities using new flags that
are given to RM_Call as part of the `fmt` argument.
It aims to assist modules that are getting a list of commands to be
executed from the user (not hard coded as part of the module logic),
think of a module that implements a new scripting language...

* `S` - Run the command in a script mode, this means that it will raise an
  error if a command which are not allowed inside a script (flaged with the
  `deny-script` flag) is invoked (like SHUTDOWN). In addition, on script mode,
  write commands are not allowed if there is not enough good replicas (as
  configured with `min-replicas-to-write`) and/or a disk error happened.

* `W` - no writes mode, Redis will reject any command that is marked with `write`
  flag. Again can be useful to modules that implement a new scripting language
  and wants to prevent any write commands.

* `E` - Return errors as RedisModuleCallReply. Today the errors that happened
  before the command was invoked (like unknown commands or acl error) return
  a NULL reply and set errno. This might be missing important information about
  the failure and it is also impossible to just pass the error to the user using
  RM_ReplyWithCallReply. This new flag allows you to get a RedisModuleCallReply
  object with the relevant error message and treat it as if it was an error that was
  raised by the command invocation.

Tests were added to verify the new code paths.

In addition small refactoring was done to share some code between modules,
scripts, and `processCommand` function:
1. `getAclErrorMessage` was added to `acl.c` to unified to log message extraction
  from the acl result
2. `checkGoodReplicasStatus` was added to `replication.c` to check the status of
  good replicas. It is used on `scriptVerifyWriteCommandAllow`, `RM_Call`, and
  `processCommand`.
3. `writeCommandsGetDiskErrorMessage` was added to `server.c` to get the error
  message on persistence failure. Again it is used on `scriptVerifyWriteCommandAllow`,
  `RM_Call`, and `processCommand`.
2022-03-22 14:13:28 +02:00
yiyuaner
08aed7e7dd
Fix an off by one error in zzlStrtod (#10465)
When vlen = sizeof(buf), the statement buf[vlen] = '\0' accessing the buffer buf is an off by one error.
2022-03-22 10:46:16 +02:00
zhaozhao.zz
79db037a4f
config rewrite enhancement, in case of line longer than 1024 (#8009)
When rewrite the config file, we need read the old config file first,
but the CONFIG_MAX_LEN is 1024, so if some lines are longer than it,
it will generate a wrong config file, and redis cannot reboot from
the new config file.

Rename CONFIG_MAX_LINE to CONFIG_READ_LEN
2022-03-22 16:34:01 +08:00
郭伟光
93dda65354
config rewrite failed save errno in case of being tainted (#10461)
errno would be potentially tainted during the serverLog() by the file io functions, such as fopen and fflush.
2022-03-21 14:20:23 +02:00
Ozan Tezcan
4517fadb59
Use exit code 1 if redis-cli fails to connect (#10438)
Use exit code 1 if redis-cli fails to connect.

Before https://github.com/redis/redis/pull/10382/, on a connection failure,
exit code would be 1.  After this PR, whether connection is established or not,
`noninteractive()` return value is used as the exit code. On a failure, this function
returns `REDIS_ERR` which is `-1`. It becomes `255` as exit codes are between `0-255`.

There is nothing wrong by returning 1 or 255 on failure as far as I know but it'll break
things that expect to see 1 as exit code on a connection failure. This is also how we
realized the issue. With this PR, changing behavior back to using 1 as exit code to
preserve backward compatibility.
2022-03-21 13:40:02 +02:00
guybe7
e82c1aedea
BITSET and BITFIELD SET should propagate even if just length changed (#10459)
Bug introduced in #9403, caused inconsistency between master and
replica in case just the length (i.e. set a high-index bit to 0)
changed.
2022-03-21 11:33:27 +02:00
Meir Shpilraien (Spielrein)
2f9cdcd733
Increase function tests timeout (#10458)
Increase function tests timeout to avoid false failures on
slow systems.
2022-03-21 11:00:27 +02:00
Madelyn Olson
557222d1e0
Fix timing issue in shards test and fix displayed TLS port (#10450) 2022-03-20 22:08:40 -07:00
郭伟光
fae5b1a19d
unblockClient: avoid to reset client when the client was shutdown-blocked (#10440)
fix #10439. see https://github.com/redis/redis/pull/9872
When executing SHUTDOWN we pause the client so we can un-pause it
if the shutdown fails.
this could happen during the timeout, if the shutdown is aborted, but could
also happen from withing the initial `call()` to shutdown, if the rdb save fails.
in that case when we return to `call()`, we'll crash if `c->cmd` has been set to NULL.

The call stack is:
```
unblockClient(c)
replyToClientsBlockedOnShutdown()
cancelShutdown()
finishShutdown()
prepareForShutdown()
shutdownCommand()
```

what's special about SHUTDOWN in that respect is that it can be paused,
and then un-paused before the original `call()` returns.
tests where added for both failed shutdown, and a followup successful one.
2022-03-20 15:18:53 +02:00
sundb
b9656adbd9
Restore ::singledb after cluster test (#10441)
When ::singledb is 0, we will use db 9 for the test db.
Since ::singledb is set to 1 in the cluster-related tests, but not restored, some subsequent
tests associated with db 9 will fail.
2022-03-18 14:10:24 +02:00
carlosfu
b69636d377
fix wrong comment about cluster_announce_hostname (#10274) 2022-03-16 20:26:30 -07:00
Madelyn Olson
e8771efda9
Fixed incorrect parsing of hostname information from nodes.conf (#10435) 2022-03-16 14:07:24 -07:00
Viktor Söderqvist
69017fa232
Fix redis-cli CLUSTER SETSLOT race conditions (#10381)
After migrating a slot, send CLUSTER SETSLOT NODE to the destination
node first to make sure the slot isn't left without an owner in case
the destination node crashes before it is set as new owner.

When informing the source node, it can happen that the destination
node has already informed it and if the source node has lost its
last slot, it has already turned itself into a replica. Redis-cli
should ignore this error in this case.
2022-03-16 10:11:38 -07:00
Binbin
61b7e5916d
Fix module redact test for valgrind (#10432)
The new module redact test will fail with valgrind:
```
[err]: modules can redact arguments in tests/unit/moduleapi/auth.tcl
Expected 'slowlog reset' to be equal to 'auth.redact 1 (redacted) 3 (redacted)' (context: type eval line 12 cmd {assert_equal {slowlog reset} [lindex [lindex [r slowlog get] 2] 3]} proc ::test)
```

The reason is that with `slowlog-log-slower-than 10000`,
`slowlog get` will have a chance to exceed 10ms.

Made two changes to avoid failure:
1. change `slowlog-log-slower-than` from 10000 to -1, distable it.
2. assert to use the previous execution result.

In theory, the second one can actually be left unchanged, but i
think it will be better if it is changed.
2022-03-16 08:53:57 +02:00
Harkrishn Patro
45ccae89bb
Add new cluster shards command (#10293)
Implement a new cluster shards command, which provides a flexible and extensible API for topology discovery.

Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
2022-03-15 18:24:40 -07:00
Madelyn Olson
416c9ac2ef
Add module API for redacting command arguments (#10425)
Add module API for redacting client commands
2022-03-15 18:21:13 -07:00
Wen Hui
c30de7073c
Sentinel: update command json files (#10374) 2022-03-15 18:38:07 +02:00
ranshid
1078e30c5f
make sort/ro commands validate external keys access patterns (#10106) (#10340)
Currently the sort and sort_ro can access external keys via `GET` and `BY`
in order to make sure the user cannot violate the authorization ACL
rules, the decision is to reject external keys access patterns unless ACL allows
SORT full access to all keys.
I.e. for backwards compatibility, SORT with GET/BY keeps working, but
if ACL has restrictions to certain keys, these features get permission denied.

### Implemented solution
We have discussed several potential solutions and decided to only allow the GET and BY
arguments when the user has all key permissions with the SORT command. The reasons
being that SORT with GET or BY is problematic anyway, for instance it is not supported in
cluster mode since it doesn't declare keys, and we're not sure the combination of that feature
with ACL key restriction is really required.
**HOWEVER** If in the fullness of time we will identify a real need for fine grain access
support for SORT, we would implement the complete solution which is the alternative
described below.

### Alternative (Completion solution):
Check sort ACL rules after executing it and before committing output (either via store or
to COB). it would require making several changes to the sort command itself. and would
potentially cause performance degradation since we will have to collect all the get keys
instead of just applying them to a temp array and then scan the access keys against the
ACL selectors. This solution can include an optimization to avoid the overheads of collecting
the key names, in case the ACL rules grant SORT full key-access, or if the ACL key pattern
literal matches the one used in GET/BY. It would also mean that authorization would be
O(nlogn) since we will have to complete most of the command execution before we can
perform verification

Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
Co-authored-by: Oran Agra <oran@redislabs.com>
2022-03-15 17:14:53 +02:00
yoav-steinberg
cf6dcb7bf1
Optimization: remove updateClientMemUsage from i/o threads. (#10401)
In a benchmark we noticed we spend a relatively long time updating the client
memory usage leading to performance degradation.
Before #8687 this was performed in the client's cron and didn't affect performance.
But since introducing client eviction we need to perform this after filling the input
buffers and after processing commands. This also lead me to write this code to be
thread safe and perform it in the i/o threads.

It turns out that the main performance issue here is related to atomic operations
being performed while updating the total clients memory usage stats used for client
eviction (`server.stat_clients_type_memory[]`). This update needed to be atomic
because `updateClientMemUsage()` was called from the IO threads.

In this commit I make sure to call `updateClientMemUsage()` only from the main thread.
In case of threaded IO I call it for each client during the "fan-in" phase of the read/write
operation. This also means I could chuck the `updateClientMemUsageBucket()` function
which was called during this phase and embed it into `updateClientMemUsage()`.

Profiling shows this makes `updateClientMemUsage()` (on my x86_64 linux) roughly x4 faster.
2022-03-15 14:18:23 +02:00
Binbin
871fa12fec
Sentinel: fix reconnect test timing issue (#10424)
We need to wait for `sentinelTimer` to kick in, and then trigger the reconnect.

As for another change, we should better call `server_set_password` before calling SENTINEL SET auth-pass.

Fixes problem introeuced in #10400
2022-03-14 11:13:14 +02:00
DarrenJiang13
38ed6c6007
improve string2ll() to avoid extra conversion for long integer string. (#10408)
For an integer string like "123456789012345678901" which could cause
overflow-failure in string2ll() conversion, we could compare its length at
the beginning to avoid extra work.

* move LONG_STR_SIZE to be in declared in util.h, next to MAX_LONG_DOUBLE_CHARS
2022-03-14 08:22:57 +02:00
郭伟光
dc7a9d3a31
Cleanup: replicationFeedMonitors use the monitor list arg it got (#10417)
Better check the monitors list argument instead of server.monitors in the function,
although they are basically the same in the context, so this doesn't have any
impact on the current code.
2022-03-13 16:19:42 +02:00
Moti Cohen
a6bf509810
Sentinel: fix no reconnect after auth-pass is changed (#10400)
When updating SENTINEL with master’s new password (command:
`SENTINEL SET mymaster auth-pass some-new-password`), 
sentinel might still keep the old connection and avoid reconnecting 
with the new password. This is because of wrong logic that traces 
the last ping (pong) time to servers. In fact it worked fine until 8631e64 
changed the condition to send ping. To resolve it with minimal risk, 
let’s disconnect master and replicas once changing password/user. 

Based on earlier work of yz1509.
2022-03-13 10:13:47 +02:00
Wen Hui
b576fbc474
Remove redundancy GETNAME in client help command message (#10418)
probably a copy paste error.
2022-03-13 08:27:41 +02:00
Binbin
1797330e2e
Initialize help when using redis-cli help or redis-cli ? (#10382)
The following usage will output an empty newline:
```
> redis-cli help set
empty line
```

The reason is that in interactive mode, we have called
`cliInitHelp`, which initializes help.

When using `redis-cli help xxx` or `redis-cli help ? xxx`,
we can't match the command due to empty `helpEntries`,
so we output an empty newline.

In this commit, we will call `cliInitHelp` to init the help.
Note that in this case, we need to call `cliInitHelp` (COMMAND DOCS)
every time, which i think is acceptable.

So now the output will look like:
```
[redis]# src/redis-cli help get

  GET key
  summary: Get the value of a key
  since: 1.0.0
  group: string

[redis]#
```

Fixes #10378

This PR also fix a redis-cli crash when using `--ldb --eval`:
```
[root]# src/redis-cli --ldb --eval test.lua test 1
Lua debugging session started, please use:
quit    -- End the session.
restart -- Restart the script in debug mode again.
help    -- Show Lua script debugging commands.

* Stopped at 1, stop reason = step over
-> 1   local num = redis.call('GET', KEYS[1]);
redis-cli: redis-cli.c:718: cliCountCommands: Assertion
`commandTable->element[i]->type == 1' failed.
Aborted
```
Because in ldb mode, `COMMAND DOCS` or `COMMAND` will
return an array, only with one element, and the type
is `REDIS_REPLY_STATUS`, the result is `<error> Unknown
Redis Lua debugger command or wrong number of arguments`.

So if we are in the ldb mode, and init the Redis HELP, we
will get the wrong response and crash the redis-cli.
In ldb mode we don't initialize HELP, help is only initialized
after the lua debugging session ends.

It was broken in #10043
2022-03-10 18:20:01 +02:00
ranshid
11b071a22b
ACL DRYRUN does not validate the verified command args. (#10405)
As a result we segfault when parsing and matching the command keys.
2022-03-10 10:08:41 +02:00
zhugezy
a26cab9dd6
set "disable-thp" config immutable (#10409)
It's confusing for this config to be modifiable since it only takes effect on startup
2022-03-10 09:52:49 +02:00
rangerzhang
4e012daee9
Fix outdated comments on updateSlavesWaitingBgsave (#10394)
* fix-replication-comments

The described capacity
 `and to schedule a new BGSAVE if there are slaves that attached while a BGSAVE was in progress`
was moved to `checkChildrenDone()`  named by `replicationStartPendingFork`

But the comment was not changed, may misleading others.

* remove-misleading-comments

The described capacity
 `to schedule a new BGSAVE if there are slaves that attached while a BGSAVE was in progress` 
and 
`or when the replication RDB transfer strategy is modified from disk to socket or the other way around` 
were not correct now.
2022-03-10 09:51:55 +02:00
Oran Agra
311a757cb1
Some adjustments to command hints. (#10375)
* stats and latency commands have non-deterministic output.
* the ones about latency should be sent to ALL_NODES (considering
  reads from replicas)
* the ones about running scripts and memory usage only to masters.
* stats aggregation is SPECIAL (like in INFO)
2022-03-09 16:40:27 +02:00
a2tt
86ca9b25e2
fix typos (#10402) 2022-03-09 13:58:23 +02:00
蔡相跃
24da71e507
Fix typo "the the" (#10399) 2022-03-09 13:55:17 +02:00
sundb
adc5a3217c
Use dismissMemory to dismiss COW of client output buffer (#10403)
c->buf is not sds, so we should use dismissMemory instead of dismissSds to dismiss it.
This is a recent regression from #10371
2022-03-09 13:32:03 +02:00
Ronald Petty
b104f3cabc
Update redis.conf (#10396)
Typo in conf file comment.
2022-03-08 12:52:54 -08:00
guybe7
2a2954086a
XREADGROUP: Unblock client if stream is deleted (#10306)
Deleting a stream while a client is blocked XREADGROUP should unblock the client.

The idea is that if a client is blocked via XREADGROUP is different from
any other blocking type in the sense that it depends on the existence of both
the key and the group. Even if the key is deleted and then revived with XADD
it won't help any clients blocked on XREADGROUP because the group no longer
exist, so they would fail with -NOGROUP anyway.
The conclusion is that it's better to unblock these clients (with error) upon
the deletion of the key, rather than waiting for the first XADD. 

Other changes:
1. Slightly optimize all `serveClientsBlockedOn*` functions by checking `server.blocked_clients_by_type`
2. All `serveClientsBlockedOn*` functions now use a list iterator rather than looking at `listFirst`, relying
  on `unblockClient` to delete the head of the list. Before this commit, only `serveClientsBlockedOnStreams`
  used to work like that.
3. bugfix: CLIENT UNBLOCK ERROR should work even if the command doesn't have a timeout_callback
  (only relevant to module commands)
2022-03-08 17:10:36 +02:00
zhaozhao.zz
728e62523e
script should not allow may-replicate commands when client pause write (#10364)
In some special commands like eval_ro / fcall_ro we allow no-writes commands.
But may-replicate commands are no-writes too, that leads crash when client pause write:
2022-03-08 16:53:11 +02:00
Oran Agra
b3fe4f31a2
dismiss COW of client output buffer now that it's dynamic (#10371)
since #9822, the static reply buffer is no longer part of the client structure, so we need to dismiss it.
2022-03-08 15:17:15 +02:00
zhugezy
4f19b4d0c1
remove a piece of redundant comment (#10392)
introduced in #10147 since we blocked the first-arg mechanism on subcommands
2022-03-08 13:35:37 +02:00
Yossi Gottlieb
38052fd702
Fix redis-benchmark --cluster with IPv6. (#10393)
Currently, CLUSTER NODES is parsed and was not done correctly for IPv6
addresses.
2022-03-08 13:10:24 +02:00
Binbin
7ef68cd5ec
Add missing doc_flags to cluster-slaves (#10387)
Add `DEPRECATED` doc_flag.
2022-03-07 17:59:51 +02:00
Shaya Potter
23f03e7965
Modules: Add REDISMODULE_EVENT_CONFIG (#10311)
Add a new REDISMODULE_EVENT_CONFIG event type for notifying modules when Redis configuration changes.
2022-03-07 17:37:57 +02:00
Binbin
45d83fb2d4
Fix timing issue in rehash test (#10388)
`Expected '*table size: 4096*' to match '*table size: 8192*'`

This test failed once on daily macOS, the reason is because
the bgsave has not stopped after the kill and `after 200`.
So there is a child process and no rehash triggered.

This commit use `waitForBgsave` to wait for it to finish.
2022-03-07 13:44:07 +02:00
Yossi Gottlieb
6740e1753d
Fix redis-cli test issues on tcl8.5. (#10386)
Apparently using `\x` produces different results between tclsh 8.5 and
8.6, whereas `\u` is more consistent.
2022-03-06 13:02:35 +02:00
Yuta Hongo
e3ef73dc2a
redis-cli: Better --json Unicode support and --quoted-json (#10286)
Normally, `redis-cli` escapes non-printable data received from Redis, using a custom scheme (which is also used to handle quoted input). When using `--json` this is not desired as it is not compatible with RFC 7159, which specifies JSON strings are assumed to be Unicode and how they should be escaped.

This commit changes `--json` to follow RFC 7159, which means that properly encoded Unicode strings in Redis will result with a valid Unicode JSON.

However, this introduces a new problem with `--json` and data that is not valid Unicode (e.g., random binary data, text that follows other encoding, etc.). To address this, we add `--quoted-json` which produces JSON strings that follow the original redis-cli quoting scheme.

For example, a value that consists of only null (0x00) bytes will show up as:
* `"\u0000\u0000\u0000"` when using `--json`
* `"\\x00\\x00\\x00"` when using `--quoted-json`
2022-03-05 21:25:52 +02:00
Binbin
af6d5c5932
Constrain cluster node name logging (#10376)
Cluster node name is not null terminated, so need to be constrained.
2022-03-02 22:41:31 -08:00
Henry
feb032fd42
A faster and more robust code of zslRandomLevel using RAND_MAX (#5539)
1. since ZSKIPLIST_P is float, using it directly inside the condition used to causes floating point code to be used (gcc/x86)
2. In some operating system(eg.Windows), the largest value returned from random() is 0x7FFF(15bit), so after bitwise AND with 0xFFFF, the probability of the less operation returning true in the while loop's condition is no more equal to ZSKIPLIST_P.
3. In case some library has random() returning int in range [0~ZSKIPLIST_P*65535], the while loop will be an infinite loop.
4. on Linux where RAND_MAX is higher than 0xFFFF, this change actually improves precision (despite not matching the result against a float value)
2022-03-02 10:40:39 +02:00
ranshid
9b15dd288e
Introduce debug command to disable reply buffer resizing (#10360)
In order to resolve some flaky tests which hard rely on examine memory footprint.
we introduce the following fixes:

# Fix in client-eviction test - by @yoav-steinberg 
Sometime the libc allocator can use different size client struct allocations.
this may cause unexpected memory calculations to fail the test.

# Introduce new DEBUG command for disabling reply buffer resizing
In order to eliminate reply buffer resizing during specific tests.
we introduced the ability to disable (and enable) the resizing cron job

Co-authored-by: yoav-steinberg yoav@redislabs.com
2022-03-01 14:40:29 +02:00