Commit Graph

9694 Commits

Author SHA1 Message Date
Oran Agra
eb6241a3dd
Include internal sds fragmentation in MEMORY reporting (#7864)
The MEMORY command is used for debugging memory usage, so it should include internal
fragmentation, same as used_memory
2020-10-01 11:30:22 +03:00
Oran Agra
dc803d25a6
Fix crash in script timeout during AOF loading (#7870) 2020-10-01 11:27:45 +03:00
Rafi Einstein
b8187d39fb
Makefile: enable program suffixes via PROG_SUFFIX (#7868) 2020-10-01 10:56:23 +03:00
nitaicaro
8fb89a5728
Fixed Tracking test “The other connection is able to get invalidations” (#7871)
PROBLEM:

[$rd1 read] reads invalidation messages one by one, so it's never going to see the second invalidation message produced after INCR b, whether or not it exists. Adding another read will block incase no invalidation message is produced.

FIX:

We switch the order of "INCR a" and "INCR b" - now "INCR b" comes first. We still only read the first invalidation message produces. If an invalidation message is wrongly produces for b - then it will be produced before that of a, since "INCR b" comes before "INCR a".

Co-authored-by: Nitai Caro <caronita@amazon.com>
2020-09-30 19:52:01 +03:00
Yossi Gottlieb
67b43f75e2 Modules: expose real client on conn events.
When REDISMODULE_EVENT_CLIENT_CHANGE events are delivered, modules may
want to mutate the client state (e.g. perform authentication).

This change links the module context with the real client rather than a
fake client for these events.
2020-09-30 17:06:57 +03:00
Yossi Gottlieb
cfccfbd6f4 Module API: Fail ineffective auth calls.
The client pointed to by the module context may in some cases be a fake
client. RM_Authenticate*() calls in this case would be ineffective but
appear to succeed, and this change fails them to make it easier to catch
such cases.
2020-09-30 17:06:57 +03:00
Yossi Gottlieb
1591e3479d
TLS: Do not require CA config if not used. (#7862)
The tls-ca-cert or tls-ca-cert-dir configuration parameters are only
used when Redis needs to authenticate peer certificates, in one of these
scenarios:

1. Incoming clients or replicas, with `tls-auth-clients` enabled.
2. A replica authenticating the master's peer certificate.
3. Cluster nodes authenticating other nodes when establishing the bus
   protocol connection.
2020-09-29 20:48:21 +03:00
YoongHM
448c435b1b
Fix compilation warnings in Lua and jemalloc dependencies (#7785)
- The argument `u` in for `ar` is ignored (and generates warnings since `D` became the default.
  All it does is avoid updating unchanged objects (shouldn't have any impact on our build)
- Enable `LUA_USE_MKSTEMP` to force the use of `mkstemp()` instead of `tmpname()` (which is dead
  code in redis anyway).
- Remove unused variable `c` in `f_parser()`
- Removed misleadingly indented space in `luaL_loadfile()` and ``addfield()`

Co-authored-by: Oran Agra <oran@redislabs.com>
2020-09-29 17:10:54 +03:00
Oran Agra
ef33252c43
warning: comparison between signed and unsigned integer in 32bit build (#7838) 2020-09-29 17:03:47 +03:00
David CARLIER
f971a5d8ed
Add support for Haiku OS (#7435) 2020-09-29 15:52:13 +03:00
Gavrie Philipson
49383a4af7
Fix typo in module API docs (#7861) 2020-09-29 13:10:08 +03:00
David CARLIER
ce8bfc56ad
getting rss size implementation for netbsd (#7293) 2020-09-29 08:49:35 +03:00
WuYunlong
c2e5546071
Normalize sds test mechanism together with some compile warnings. (#7854) 2020-09-28 11:27:26 +03:00
Oran Agra
8aa083bd28
Fix new obuf-limits tests to work with TLS (#7848)
Also stabilize new shutdown tests on slow machines (valgrind)
2020-09-27 17:13:33 +03:00
caozb
a295770e32
ignore slaveof no one in redis.conf (#7842)
when slaveof config is "no one", reset any pre-existing config and resume.

also solve a memory leak if slaveof appears twice.
and fail loading if port number is out of range or not an integer.

Co-authored-by: caozhengbin <caozb@yidingyun.com>
Co-authored-by: Oran Agra <oran@redislabs.com>
2020-09-27 15:40:07 +03:00
Wang Yuan
c9f00bcce2
Don't support Gopher if enable io threads to read queries (#7851)
There's currently an issue with IO threads and gopher (issuing lookupKey from within the thread).
simply fix is to just not support it for now.
2020-09-27 12:35:16 +03:00
Wang Yuan
f1863a1fe7
Set 'loading' and 'shutdown_asap' to volatile sig_atomic_t type (#7845)
We may access and modify these two variables in signal handler function,
to guarantee them async-signal-safe, so we should set them to volatile
sig_atomic_t type.

It doesn't look like this could have caused any real issue, and it seems that
signals are handled in main thread on most platforms. But we want to follow C
and POSIX standard in signal handler function.
2020-09-25 16:25:47 +03:00
Uri Shachar
c30bd02c9d
Fix config rewrite file handling to make it really atomic (#7824)
Make sure we handle short writes correctly, sync to disk after writing  and use
rename to make sure the replacement is actually atomic.
In any case of failure old configuration will remain in place.

Also, add some additional logging to make it easier to diagnose rewrite problems.
2020-09-25 12:55:45 +03:00
WuYunlong
0d62caab21
Add fsync to readSyncBulkPayload(). (#7839)
We should sync temp DB file before renaming as rdb_fsync_range does not use
flag `SYNC_FILE_RANGE_WAIT_AFTER`.

Refer to `Linux Programmer's Manual`:
SYNC_FILE_RANGE_WAIT_AFTER
    Wait upon write-out of all pages in the range after performing any write.
2020-09-25 08:08:06 +03:00
Wen Hui
323029baa6
rdb.c: handle fclose error case differently to avoid double fclose (#7307)
When fclose would fail, the previous implementation would have attempted to do fclose again
this can in theory lead to segfault.

other changes:
check for non-zero return value as failure rather than a specific error code.
this doesn't fix a real bug, just a minor cleanup.
2020-09-24 18:17:53 +03:00
Wang Yuan
57709c4bc6
Don't write replies if close the client ASAP (#7202)
Before this commit, we would have continued to add replies to the reply buffer even if client
output buffer limit is reached, so the used memory would keep increasing over the configured limit.
What's more, we shouldn’t write any reply to the client if it is set 'CLIENT_CLOSE_ASAP' flag
because that doesn't conform to its definition and we will close all clients flagged with
'CLIENT_CLOSE_ASAP' in ‘beforeSleep’.

Because of code execution order, before this, we may firstly write to part of the replies to
the socket before disconnecting it, but in fact, we may can’t send the full replies to clients
since OS socket buffer is limited. But this unexpected behavior makes some commands work well,
for instance ACL DELUSER, if the client deletes the current user, we need to send reply to client
and close the connection, but before, we close the client firstly and write the reply to reply
buffer. secondly, we shouldn't do this despite the fact it works well in most cases.

We add a flag 'CLIENT_CLOSE_AFTER_COMMAND' to mark clients, this flag means we will close the
client after executing commands and send all entire replies, so that we can write replies to
reply buffer during executing commands, send replies to clients, and close them later.

We also fix some implicit problems. If client output buffer limit is enforced in 'multi/exec',
all commands will be executed completely in redis and clients will not read any reply instead of
partial replies. Even more, if the client executes 'ACL deluser' the using user in 'multi/exec',
it will not read the replies after 'ACL deluser' just like before executing 'client kill' itself
in 'multi/exec'.

We added some tests for output buffer limit breach during multi-exec and using a pipeline of
many small commands rather than one with big response.

Co-authored-by: Oran Agra <oran@redislabs.com>
2020-09-24 16:01:41 +03:00
Guy Korland
b464afb9e2
Fix RedisModule_HashGet examples (#6697) 2020-09-24 12:45:30 +03:00
valentinogeron
795c454db1
Stream: Inconsistency between master and replica some XREADGROUP case (#7526)
XREADGROUP auto-creates the consumer inside the consumer group the
first time it saw it.
When XREADGROUP is being used with NOACK option, the message will not
be added into the client's PEL and XGROUP SETID would be propagated.
When the replica gets the XGROUP SETID it will only update the last delivered
id of the group, but will not create the consumer.

So, in this commit XGROUP CREATECONSUMER is being added.
Command pattern: XGROUP CREATECONSUMER <key> <group> <consumer>.

When NOACK option is being used, createconsumer command would be
propagated as well.

In case of AOFREWRITE, consumer with an empty PEL would be saved with
XGROUP CREATECONSUMER whereas consumer with pending entries would be
saved with XCLAIM
2020-09-24 12:02:40 +03:00
bodong.ybd
b7ce583a5e Refactor multi-key command get keys proc 2020-09-24 08:59:14 +03:00
bodong.ybd
e08bf16637 Add ZINTER/ZUNION command
Syntax: ZINTER/ZUNION numkeys key [key ...] [WEIGHTS weight [weight ...]]
[AGGREGATE SUM|MIN|MAX] [WITHSCORES]

see #7624
2020-09-24 08:59:14 +03:00
alexronke-channeladvisor
66a13ccbdf
Add GT and LT options to ZADD for conditional score updates (#7818)
Co-authored-by: Alex Ronke <w.alex.ronke@gmail.com>
2020-09-23 21:56:16 +03:00
Oran Agra
a735bf5c2a
fix recently broken TLS build error, and add coverage for CI (#7833) 2020-09-23 11:30:24 +03:00
David CARLIER
c3edaa7941
Further NetBSD update and build fixes. (#7831)
mainly backtrace and register dump support.
2020-09-23 10:00:31 +03:00
WuYunlong
63cd4d4e20
Fix redundancy use of semicolon in do-while macros in ziplist.c. (#7832)
this is very dangerous bug, but it looks like it didn't cause any harm.
2020-09-23 09:09:48 +03:00
yixiang
b96c3595af
Fix connGetSocketError usage (#7811) 2020-09-22 12:53:36 +03:00
Oran Agra
26ca04a8a2
RM_GetContextFlags - document missing flags (#7821) 2020-09-22 12:11:19 +03:00
Yossi Gottlieb
1980f639b1
Fix occasional hangs on replication reconnection. (#7830)
This happens only on diskless replicas when attempting to reconnect after 
failing to load an RDB file. It is more likely to occur with larger datasets.

After reconnection is initiated, replicationEmptyDbCallback() may get called 
and try to write to an unconnected socket. This triggered another issue where
the connection is put into an error state and the connect handler never gets
called. The problem is a regression introduced by commit c17e597.
2020-09-22 11:38:52 +03:00
Ariel Shtul
63a05dde46
Fix redis-check-rdb support for modules aux data (#7826)
redis-check-rdb was unable to parse rdb files containing module aux data.

Co-authored-by: Oran Agra <oran@redislabs.com>
2020-09-22 10:18:07 +03:00
Wang Yuan
1bb5794a1f
Kill disk-based fork child when all replicas drop and 'save' is not enabled (#7819)
When all replicas waiting for a bgsave get disconnected (possibly due to output buffer limit),
It may be good to kill the bgsave child. in diskless replication it already happens, but in
disk-based, the child may still serve some purpose (for persistence).

By killing the child, we prevent it from eating COW memory in vain, and we also allow a new child fork sooner for the next full synchronization or bgsave.
We do that only if rdb persistence wasn't enabled in the configuration.

Btw, now, rdbRemoveTempFile in killRDBChild won't block server, so we can killRDBChild safely.
2020-09-22 09:47:58 +03:00
Wen Hui
23b50bcccc
refactor rewriteStreamObject code for adding missing streamIteratorStop call (#7829)
This commit adds streamIteratorStop call in rewriteStreamObject function in some of the return statement. Although currently this will not cause memory leak since stream id is only 16 bytes long.
2020-09-22 09:05:47 +03:00
YoongHM
9216b96b41
Fix compilation warning in jemalloc's malloc_vsnprintf (#7789)
Change `val` to `unsigned char` before being tested.
The fix is identical to the one that's been made in upstream jemalloc.

warning is:
src/malloc_io.c: In function ‘malloc_vsnprintf’:
src/malloc_io.c:369:2: warning: case label value exceeds maximum value for type
  369 |  case '?' | 0x80:      \
      |  ^~~~
src/malloc_io.c:581:5: note: in expansion of macro ‘GET_ARG_NUMERIC’
  581 |     GET_ARG_NUMERIC(val, 'p');
      |     ^~~~~~~~~~~~~~~
2020-09-21 17:04:34 +03:00
YoongHM
d3faed875d
Fix warning from jemalloc configure script (#7790)
jemalloc configure shows this:
    configure: WARNING: unrecognized options: --enable-cc-silence

The changelog of jemalloc 4.0 has:
  - Replace --enable-cc-silence with --disable-cc-silence to suppress spurious
    warnings by default.
2020-09-21 16:57:15 +03:00
WuYunlong
c37acb82aa Make IO threads killable so that they can be canceled at any time.
This commit can be cherry picked to 6.0 only if we also cherry pick f866023.
2020-09-21 12:10:19 +03:00
WuYunlong
647cac5bb4 Make main thread killable so that it can be canceled at any time.
Refine comment of makeThreadKillable().

This commit can be backported to 5.0, only if we also backport 8b70cb0.

Co-authored-by: Oran Agra <oran@redislabs.com>
2020-09-21 12:10:19 +03:00
Oran Agra
f11a0c8f30
Fix C11 detection in the makefile (#7822)
445a4b6 introudced a makefile script that detects if the toolchain
supports c11, and it looked that it was passing on MacOS and fails on
Ubuntu, looks like Ubuntu's Dash was spawning a background process,
deleted foo.c before gcc tried to compile it.
2020-09-21 11:17:48 +03:00
Oran Agra
2458e54814
RM_GetContextFlags provides indication that we're in a fork child (#7783) 2020-09-20 13:43:28 +03:00
Wen Hui
dfe9714c86
Add Swapdb Module Event (#7804) 2020-09-20 13:36:20 +03:00
Daniel Dai
1b3b75208c
fix make warnings in debug.c MacOS (#7805)
Co-authored-by: Oran Agra <oran@redislabs.com>
2020-09-20 12:06:17 +03:00
David CARLIER
eabe3eaec0
debug.c: NetBSD build warning fix. (#7810)
The symbol base address is a const on this system.
2020-09-19 12:24:40 +03:00
Wang Yuan
b002d2b4f1
Remove tmp rdb file in background thread (#7762)
We're already using bg_unlink in several places to delete the rdb file in the background,
and avoid paying the cost of the deletion from our main thread.
This commit uses bg_unlink to remove the temporary rdb file in the background too.

However, in case we delete that rdb file just before exiting, we don't actually wait for the
background thread or the main thread to delete it, and just let the OS clean up after us.
i.e. we open the file, unlink it and exit with the fd still open.

Furthermore, rdbRemoveTempFile can be called from a thread and was using snprintf which is
not async-signal-safe, we now use ll2string instead.
2020-09-17 18:20:10 +03:00
Wang Yuan
445a4b669a
Implement redisAtomic to replace _Atomic C11 builtin (#7707)
Redis 6.0 introduces I/O threads, it is so cool and efficient, we use C11
_Atomic to establish inter-thread synchronization without mutex. But the
compiler that must supports C11 _Atomic can compile redis code, that brings a
lot of inconvenience since some common platforms can't support by default such
as CentOS7, so we want to implement redis atomic type to make it more portable.

We have implemented our atomic variable for redis that only has 'relaxed'
operations in src/atomicvar.h, so we implement some operations with
'sequentially-consistent', just like the default behavior of C11 _Atomic that
can establish inter-thread synchronization. And we replace all uses of C11
_Atomic with redis atomic variable.

Our implementation of redis atomic variable uses C11 _Atomic, __atomic or
__sync macros if available, it supports most common platforms, and we will
detect automatically which feature we use. In Makefile we use a dummy file to
detect if the compiler supports C11 _Atomic. Now for gcc, we can compile redis
code theoretically if your gcc version is not less than 4.1.2(starts to support
__sync_xxx operations). Otherwise, we remove use mutex fallback to implement
redis atomic variable for performance and test. You will get compiling errors
if your compiler doesn't support all features of above.

For cover redis atomic variable tests, we add other CI jobs that build redis on
CentOS6 and CentOS7 and workflow daily jobs that run the tests on them.
For them, we just install gcc by default in order to cover different compiler
versions, gcc is 4.4.7 by default installation on CentOS6 and 4.8.5 on CentOS7.

We restore the feature that we can test redis with Helgrind to find data race
errors. But you need install Valgrind in the default path configuration firstly
before running your tests, since we use macros in helgrind.h to tell Helgrind
inter-thread happens-before relationship explicitly for avoiding false positives.
Please open an issue on github if you find data race errors relate to this commit.

Unrelated:
- Fix redefinition of typedef 'RedisModuleUserChangedFunc'
  For some old version compilers, they will report errors or warnings, if we
  re-define function type.
2020-09-17 16:01:45 +03:00
Oran Agra
092cfca522
Add printf attribute and fix warnings and a minor bug (#7803)
The fix in error handling of rdbGenericLoadStringObject is an actual bugfix
2020-09-16 20:21:04 +03:00
WuYunlong
f866023399 bio: doFastMemoryTest should try to kill io threads as well. 2020-09-16 14:15:02 +03:00
WuYunlong
8b70cb0ef8 bio: fix doFastMemoryTest.
If one thread got SIGSEGV, function sigsegvHandler() would be triggered,
it would call bioKillThreads(). But call pthread_cancel() to cancel itself
would make it block. Also note that if SIGSEGV is caught by bio thread, it
should kill the main thread in order to give a positive report.
2020-09-16 14:15:02 +03:00
Jim Brunner
810e28a397
Incremental eviction processing (#7653)
Rather than blindly evicting until maxmemory limit is achieved, this
update adds a time limit to eviction.  While over the maxmemory limit,
eviction will process before each command AND as a timeProc when no
commands are running.

This will reduce the latency impact on many cases, especially pathological
cases like massive used memory increase during dict rehashing.

There is a risk that some other edge cases (like massive pipelined use
of MGET) could cause Redis memory usage to keep growing despite the
eviction attempts, so a new maxmemory-eviction-tenacity config is
introduced to let users mitigate that.
2020-09-16 09:16:01 +03:00