7342 Commits

Author SHA1 Message Date
Wen Hui
c69a9b2f61
fix leak in error handling of debug populate command (#7062)
valsize was not modified during the for loop below instead of getting from c->argv[4], therefore there is no need to put inside the for loop.. Moreover, putting the check outside loop will also avoid memory leaking, decrRefCount(key) should be called in the original code if we put the check in for loop
2020-07-28 22:05:48 +03:00
Yossi Gottlieb
784ceeb90d
TLS: Propagate and handle SSL_new() failures. (#7576)
The connection API may create an accepted connection object in an error
state, and callers are expected to check it before attempting to use it.

Co-authored-by: mrpre <mrpre@163.com>
2020-07-28 11:32:47 +03:00
Jiayuan Chen
f31260b044
Add optional tls verification (#7502)
Adds an `optional` value to the previously boolean `tls-auth-clients` configuration keyword.

Co-authored-by: Yossi Gottlieb <yossigo@gmail.com>
2020-07-28 10:45:21 +03:00
Yossi Gottlieb
c75512d89d TLS: support cluster/replication without tls-port.
Initialize and configure OpenSSL even when tls-port is not used, because
we may still have tls-cluster or tls-replication.

Also, make sure to reconfigure OpenSSL when these parameters are changed
as TLS could have been enabled for the first time.
2020-07-27 13:26:02 +03:00
grishaf
4126ca466f
Fix prepareForShutdown function declaration (#7566) 2020-07-26 08:27:30 +03:00
zhaozhao.zz
da840e9851
more strict check in rioConnRead (#7564) 2020-07-24 14:40:19 +08:00
Meir Shpilraien (Spielrein)
8d82639319
This PR introduces a new loaded keyspace event (#7536)
Co-authored-by: Oran Agra <oran@redislabs.com>
Co-authored-by: Itamar Haber <itamar@redislabs.com>
2020-07-23 12:38:51 +03:00
Oran Agra
40d7fca368
Fix harmless bug in rioConnRead (#7557)
this code is in use only if the master is disk-based, and the replica is
diskless. In this case we use a buffered reader, but we must avoid reading
past the rdb file, into the command stream. which Luckly rdb.c doesn't
really attempt to do (it knows how much it should read).

When rioConnRead detects that the extra buffering attempt reaches beyond
the read limit it should read less, but if the caller actually requested
more, then it should return with an error rather than a short read. the
bug would have resulted in short read.

in order to fix it, the code must consider the real requested size, and
not the extra buffering size.
2020-07-23 12:37:43 +03:00
Madelyn Olson
818dc3a089
Properly reset errno for rdbLoad (#7542) 2020-07-21 17:00:13 -07:00
WuYunlong
f7f77a746a
Clarification on the bug that was fixed in PR #7539. (#7541)
Before that PR, processCommand() did not notice that cmd could be a module
command in which case getkeys_proc member has a different meaning.

The outcome was that a module command which doesn't take any key names in its
arguments (similar to SLOWLOG) would be handled as if it might have key name arguments
(similar to MEMORY), would consider cluster redirect but will end up with 0 keys
after an excessive call to getKeysFromCommand, and eventually do the right thing.
2020-07-21 09:41:44 +03:00
Wen Hui
4e8f2d6881
Add missing calls to raxStop (#7532)
Since the dynamic allocations in raxIterator are only used for deep walks, memory
leak due to missing call to raxStop can only happen for rax with key names longer
than 32 bytes.

Out of all the missing calls, the only ones that may lead to a leak are the rax
for consumer groups and consumers, and these were only in AOFRW and rdbSave, which
normally only happen in fork or at shutdown.
2020-07-21 08:13:05 +03:00
Wen Hui
2fbd0271f6
add missing caching command in client help (#7399) 2020-07-20 18:53:03 -07:00
zhaozhao.zz
13e50935a8
replication: need handle -NOPERM error after send ping (#7538) 2020-07-20 22:21:55 +08:00
WuYunlong
e4d7de608c
Fix cluster redirect for module command with no firstkey. (#7539)
Before this commit, processCommand() did not notice that cmd could be a module command
which declared `getkeys-api` and handled it for the purpose of cluster redirect it
as if it doesn't use any keys.

This commit fixed it by reusing the codes in addReplyCommand().
2020-07-20 15:33:06 +03:00
WuYunlong
86fed3fe09
Refactor streamAppendItem() by deleting redundancy condition. (#7487)
It will never happen that "lp != NULL && lp_bytes >= server.stream_node_max_bytes".
Assume that "lp != NULL && lp_bytes >= server.stream_node_max_bytes",
we got the following conditions:
a. lp != NULL
b. lp_bytes >= server.stream_node_max_bytes

If server.stream_node_max_bytes is 0, given condition a, condition b is always satisfied
If server.stream_node_max_bytes is not 0, given condition a and condition b, the codes just a
	few lines above set lp to NULL, a controdiction with condition a

So that condition b is recundant. We could delete it safely.
2020-07-20 13:14:27 +03:00
yoav-steinberg
d484b8a04e
Support passing stack allocated module strings to moduleCreateArgvFromUserFormat (#7528)
Specifically, the key passed to the module aof_rewrite callback is a stack allocated robj. When passing it to RedisModule_EmitAOF (with appropriate "s" fmt string) redis used to panic when trying to inc the ref count of the stack allocated robj. Now support such robjs by coying them to a new heap robj. This doesn't affect performance because using the alternative "c" or "b" format strings also copies the input to a new heap robj.
2020-07-16 20:59:38 +03:00
杨博东
8596d483bc
Stream avoid duplicate parse id (#7450) 2020-07-16 08:57:27 +03:00
Luke Palmer
5f716ea467
Send null for invalidate on flush (#7469) 2020-07-15 10:53:41 -07:00
dmurnane
9242ccf238
Notify systemd on sentinel startup (#7168)
Co-authored-by: Daniel Murnane <dmurnane@eitccorp.com>
2020-07-15 13:29:26 +03:00
Developer-Ecosystem-Engineering
c2b5f1c15b
Add registers dump support for Apple silicon (#7453)
Export following environment variables before building on macOS on Apple silicon

export ARCH_FLAGS="-arch arm64"
export SDK_NAME=macosx
export SDK_PATH=$(xcrun --show-sdk-path --sdk $SDK_NAME)
export CFLAGS="$ARCH_FLAGS -isysroot $SDK_PATH -I$SDK_PATH/usr/include"
export CXXFLAGS=$CFLAGS
export LDFLAGS="$ARCH_FLAGS"
export CC="$(xcrun -sdk $SDK_PATH --find clang) $CFLAGS"
export CXX="$(xcrun -sdk $SDK_PATH --find clang++) $CXXFLAGS"
export LD="$(xcrun -sdk $SDK_PATH --find ld) $LDFLAGS"

make
make test
..
All tests passed without errors!

Backtrack logging assumes x86 and required updating
2020-07-15 12:44:03 +03:00
Wen Hui
d85af4d6f5
correct error msg for num connections reaching maxclients in cluster mode (#7444) 2020-07-15 12:38:47 +03:00
WuYunlong
93bdbf5aa4
Fix command help for unexpected options (#7476) 2020-07-15 12:38:22 +03:00
WuYunlong
dc690161d5
Refactor RM_KeyType() by using macro. (#7486) 2020-07-15 12:37:44 +03:00
Oran Agra
a176cb56a3
diskless master disconnect replicas when rdb child failed (#7518)
in case the rdb child failed, crashed or terminated unexpectedly redis
would have marked the replica clients with repl_put_online_on_ack and
then kill them only after a minute when no ack was received.

it would not stream anything to these connections, so the only effect of
this bug is a delay of 1 minute in the replicas attempt to re-connect.
2020-07-14 20:21:59 +03:00
Qu Chen
938c35302f
Replica always reports master's config epoch in CLUSTER NODES output. (#7235) 2020-07-13 07:16:06 -07:00
Oran Agra
6a81450144
RESTORE ABSTTL skip expired keys - leak (#7511) 2020-07-13 16:40:19 +03:00
jimgreen2013
67660881ed
fix description about ziplist, the code is ok (#6318)
* fix description about ZIP_BIG_PREVLEN(the code is ok), it's similar to
antirez#4705

* fix description about ziplist entry encoding field (the code is ok),
the max length should be 2^32 - 1 when encoding is 5 bytes
2020-07-11 14:51:44 -05:00
杨博东
e9aba28932
STORE variants: SINTER,SUNION,SDIFF,ZUNION use setKey instead of dbDelete+dbAdd (#7489)
one of the differences (other than consistent code with SORT, GEORADIUS), is that the LFU of the old key is retained.
2020-07-11 15:52:41 +03:00
马永泽
279b4a1464
fix benchmark in cluster mode fails to authenticate (#7488)
Co-authored-by: Oran Agra <oran@redislabs.com> (styling)
2020-07-10 16:37:11 +03:00
Yossi Gottlieb
3e6f2b1a45
TLS: Session caching configuration support. (#7420)
* TLS: Session caching configuration support.
* TLS: Remove redundant config initialization.
2020-07-10 11:33:47 +03:00
Yossi Gottlieb
5266293a0f
TLS: Ignore client cert when tls-auth-clients off. (#7457) 2020-07-10 10:32:21 +03:00
James Hilliard
6a014af79a
Use pkg-config to properly detect libssl and libcrypto libraries (#7452) 2020-07-10 10:30:09 +03:00
Yossi Gottlieb
d9f970d8d3
TLS: Add missing redis-cli options. (#7456)
* Tests: fix and reintroduce redis-cli tests.

These tests have been broken and disabled for 10 years now!

* TLS: add remaining redis-cli support.

This adds support for the redis-cli --pipe, --rdb and --replica options
previously unsupported in --tls mode.

* Fix writeConn().
2020-07-10 10:25:55 +03:00
Oran Agra
b23e251036 redis-cli --hotkeys fixed to handle non-printable key names 2020-07-10 10:07:47 +03:00
Oran Agra
6f8a8647de redis-cli --bigkeys fixed to handle non-printable key names 2020-07-10 10:07:47 +03:00
Oran Agra
5977a94842
RESTORE ABSTTL won't store expired keys into the db (#7472)
Similarly to EXPIREAT with TTL in the past, which implicitly deletes the
key and return success, RESTORE should not store key that are already
expired into the db.
When used together with REPLACE it should emit a DEL to keyspace
notification and replication stream.
2020-07-10 10:02:37 +03:00
huangzhw
d6180c8c86
defrag.c activeDefragSdsListAndDict when defrag sdsele, We can't use (#7492)
it to calculate hash, we should use newsds.
2020-07-10 08:29:44 +03:00
Oran Agra
69ade87325
tests/valgrind: don't use debug restart (#7404)
* tests/valgrind: don't use debug restart

DEBUG REATART causes two issues:
1. it uses execve which replaces the original process and valgrind doesn't
   have a chance to check for errors, so leaks go unreported.
2. valgrind report invalid calls to close() which we're unable to resolve.

So now the tests use restart_server mechanism in the tests, that terminates
the old server and starts a new one, new PID, but same stdout, stderr.

since the stderr can contain two or more valgrind report, it is not enough
to just check for the absence of leaks, we also need to check for some known
errors, we do both, and fail if we either find an error, or can't find a
report saying there are no leaks.

other changes:
- when killing a server that was already terminated we check for leaks too.
- adding DEBUG LEAK which was used to test it.
- adding --trace-children to valgrind, although no longer needed.
- since the stdout contains two or more runs, we need slightly different way
  of checking if the new process is up (explicitly looking for the new PID)
- move the code that handles --wait-server to happen earlier (before
  watching the startup message in the log), and serve the restarted server too.

* squashme - CR fixes
2020-07-10 08:26:52 +03:00
Oran Agra
9bbf768d3c
change references to the github repo location (#7479) 2020-07-10 08:25:26 +03:00
zhaozhao.zz
1978f996d8
BITOP: propagate only when it really SET or DEL targetkey (#5783)
For example:
    BITOP not targetkey sourcekey

If targetkey and sourcekey doesn't exist, BITOP has no effect,
we do not propagate it, thus can save aof and replica flow.
2020-07-10 08:20:27 +03:00
antirez
ad0a9df77a Update comment to clarify change in #7398. 2020-06-25 12:58:21 +02:00
Salvatore Sanfilippo
760021e677
Merge pull request #7398 from caiyuxinggg/work
cluster.c remove "if (nodeIsMaster(myself))" judgement before clusterSendFail in markNodeAsFailingIfNeeded, avoiding slave failover requires twice vote requests
2020-06-25 12:56:26 +02:00
antirez
959cdb358b Merge branch 'unstable' of github.com:/antirez/redis into unstable 2020-06-24 09:09:59 +02:00
antirez
a5a3a7bbc6 LPOS: option FIRST renamed RANK. 2020-06-24 09:09:43 +02:00
Salvatore Sanfilippo
6bbbdd26f4
Merge pull request #7390 from oranagra/exec_fails_abort
EXEC always fails with EXECABORT and multi-state is cleared
2020-06-23 13:12:52 +02:00
Oran Agra
65a3307bc9 EXEC always fails with EXECABORT and multi-state is cleared
In order to support the use of multi-exec in pipeline, it is important that
MULTI and EXEC are never rejected and it is easy for the client to know if the
connection is still in multi state.

It was easy to make sure MULTI and DISCARD never fail (done by previous
commits) since these only change the client state and don't do any actual
change in the server, but EXEC is a different story.

Since in the past, it was possible for clients to handle some EXEC errors and
retry the EXEC, we now can't affort to return any error on EXEC other than
EXECABORT, which now carries with it the real reason for the abort too.

Other fixes in this commit:
- Some checks that where performed at the time of queuing need to be re-
  validated when EXEC runs, for instance if the transaction contains writes
  commands, it needs to be aborted. there was one check that was already done
  in execCommand (-READONLY), but other checks where missing: -OOM, -MISCONF,
  -NOREPLICAS, -MASTERDOWN
- When a command is rejected by processCommand it was rejected with addReply,
  which was not recognized as an error in case the bad command came from the
  master. this will enable to count or MONITOR these errors in the future.
- make it easier for tests to create additional (non deferred) clients.
- add tests for the fixes of this commit.
2020-06-23 12:01:33 +03:00
antirez
21f62c3346 Include cluster.h for getClusterConnectionsCount(). 2020-06-22 11:44:11 +02:00
antirez
746297314f Fix BITFIELD i64 type handling, see #7417. 2020-06-22 11:41:19 +02:00
antirez
59fd178014 Clarify maxclients and cluster in conf. Remove myself too. 2020-06-22 11:21:21 +02:00
hwware
1bfa2d27a6 fix memory leak in sentinel connection sharing 2020-06-21 23:04:28 -04:00