554 Commits

Author SHA1 Message Date
Oran Agra
fb732f7a94 optimize memory usage of deferred replies
When deffered reply is added the previous reply node cannot be used so
all the extra space we allocated in it is wasted. in case someone uses
deffered replies in a loop, each time adding a small reply, each of
these reply nodes (the small string reply) would have consumed a 16k
block.
now when we add anther diferred reply node, we trim the unused portion
of the previous reply block.

see #7123
2020-04-24 17:20:28 +03:00
antirez
c6ccc2f690 Tracking: NOLOOP further implementation and fixes. 2020-04-21 17:29:18 +02:00
zhenwei pi
5010da6ab0 Threaded IO: set thread name for redis-server
Set thread name for each thread of redis-server, this helps us to
monitor the utilization and optimise the performance.

And suggested-by Salvatore, implement this feature for multi
platforms. Currently support linux and bsd, ignore other OS.

An exmaple on Linux:
 # top -d 5 -p `pidof redis-server ` -H

    PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
3682671 root      20   0  227744   8248   3836 R 99.2  0.0   0:19.53 redis-server
3682677 root      20   0  227744   8248   3836 S 26.4  0.0   0:04.15 io_thd_3
3682675 root      20   0  227744   8248   3836 S 23.6  0.0   0:03.98 io_thd_1
3682676 root      20   0  227744   8248   3836 S 23.6  0.0   0:03.97 io_thd_2
3682672 root      20   0  227744   8248   3836 S  0.2  0.0   0:00.02 bio_close_file
3682673 root      20   0  227744   8248   3836 S  0.2  0.0   0:00.02 bio_aof_fsync
3682674 root      20   0  227744   8248   3836 S  0.0  0.0   0:00.00 bio_lazy_free
3682678 root      20   0  227744   8248   3836 S  0.0  0.0   0:00.00 jemalloc_bg_thd
3682682 root      20   0  227744   8248   3836 S  0.0  0.0   0:00.00 jemalloc_bg_thd
3682683 root      20   0  227744   8248   3836 S  0.0  0.0   0:00.00 jemalloc_bg_thd
3682684 root      20   0  227744   8248   3836 S  0.0  0.0   0:00.00 jemalloc_bg_thd
3682685 root      20   0  227744   8248   3836 S  0.0  0.0   0:00.00 jemalloc_bg_thd
3682687 root      20   0  227744   8248   3836 S  0.0  0.0   0:00.00 jemalloc_bg_thd

Another exmaple on FreeBSD-12.1:
  PID USERNAME    PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
 5212 root        100    0    48M  7280K CPU2     2   0:26  99.52% redis-server{redis-server}
 5212 root         38    0    48M  7280K umtxn    4   0:06  26.94% redis-server{io_thd_3}
 5212 root         36    0    48M  7280K umtxn    6   0:06  26.84% redis-server{io_thd_1}
 5212 root         39    0    48M  7280K umtxn    1   0:06  25.30% redis-server{io_thd_2}
 5212 root         20    0    48M  7280K uwait    3   0:00   0.00% redis-server{redis-server}
 5212 root         21    0    48M  7280K uwait    2   0:00   0.00% redis-server{bio_close_file}
 5212 root         21    0    48M  7280K uwait    3   0:00   0.00% redis-server{bio_aof_fsync}
 5212 root         21    0    48M  7280K uwait    0   0:00   0.00% redis-server{bio_lazy_free}

Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
2020-04-18 11:17:21 +08:00
antirez
c9388ecc73 RESP3: fix HELLO map len in Sentinel mode.
See #6160.
2020-04-16 11:21:52 +02:00
antirez
7f794a122f Fix HELLO reply in Sentinel mode, see #6160. 2020-04-15 15:59:52 +02:00
antirez
f69876280c Speedup INFO by counting client memory incrementally.
Related to #5145.

Design note: clients may change type when they turn into replicas or are
moved into the Pub/Sub category and so forth. Moreover the recomputation
of the bytes used is problematic for obvious reasons: it changes
continuously, so as a conservative way to avoid accumulating errors,
each client remembers the contribution it gave to the sum, and removes
it when it is freed or before updating it with the new memory usage.
2020-04-07 12:07:54 +02:00
Salvatore Sanfilippo
8d6dc136e1
Merge pull request #7041 from hwware/trackingfix
CLIENT TRACKING Command Fix: Add More Checking for OPTIN/OPTOUT mode
2020-04-02 11:22:30 +02:00
antirez
1d13ff0b39 Minor changes to #7037. 2020-03-31 17:10:09 +02:00
Guy Benoish
d6eb3afd13 Modules: Test MULTI/EXEC replication of RM_Replicate
Makse sure call() doesn't wrap replicated commands with
a redundant MULTI/EXEC

Other, unrelated changes:
1. Formatting compiler warning in INFO CLIENTS
2. Use CLIENT_ID_AOF instead of UINT64_MAX
2020-03-31 13:55:51 +03:00
hwware
86c76cad74 add check for not switching between optin optout mode directly 2020-03-29 23:20:54 -04:00
hwware
1ef44f8243 add check for not providing both optin optout flag 2020-03-29 23:06:50 -04:00
Yossi Gottlieb
2dab5015b7 Fix crashes related to failed/rejected accepts. 2020-03-22 14:47:44 +02:00
antirez
f9c56dbb09 ACL: default user off should not allow automatic authentication.
This fixes issue #7011.
2020-03-20 12:45:48 +01:00
Salvatore Sanfilippo
15338ab694
Merge pull request #6991 from soloestoy/io-threads-bugfix
Threaded IO: bugfix client kill may crash redis
2020-03-16 10:53:54 +01:00
zhaozhao.zz
a6a0e05a1a Threaded IO: bugfix client kill may crash redis 2020-03-15 23:30:25 +08:00
antirez
573c4673ee Aesthetic changes in PR #6989. 2020-03-15 16:10:37 +01:00
zhaozhao.zz
606a01df70 Threaded IO: bugfix #6988 process events while blocked 2020-03-15 22:07:36 +08:00
antirez
b6378edcd6 Tracking: optin/out implemented. 2020-02-24 19:09:45 +01:00
Madelyn Olson
c1b5220ee7 Give an error message if you specify redirect twice 2020-02-16 05:41:39 -08:00
Madelyn Olson
d63a435362 Minor CSC fixes and fixed documentation 2020-02-16 05:28:29 -08:00
antirez
47177c9edc Tracking: fix operators precedence error in bcast check. 2020-02-14 14:27:45 +01:00
antirez
6922ccc0b9 Tracking: fix sending messages bug + tracking off bug. 2020-02-13 16:58:07 +01:00
antirez
40194a2a68 Tracking: BCAST: basic feature now works. 2020-02-12 19:22:04 +01:00
antirez
3f7ba86255 Tracking: BCAST: registration in the prefix table. 2020-02-11 17:26:29 +01:00
antirez
dfe126f3e9 Tracking: BCAST: parsing of the options + skeleton. 2020-02-10 17:18:11 +01:00
Salvatore Sanfilippo
7cf53252ee
Merge pull request #6849 from oranagra/free_client_mutex
freeClientAsync don't lock mutex if there's just one thread
2020-02-06 10:29:46 +01:00
Guy Benoish
91c41b6dde Some refactroing using getClientType instead of CLIENT_SLAVE 2020-02-06 14:12:08 +05:30
Guy Benoish
e33fffbde1 Fix small bugs related to replica and monitor ambiguity
1. server.repl_no_slaves_since can be set when a MONITOR client disconnects
2. c->repl_ack_time can be set by a newline from a MONITOR client
3. Improved comments
2020-02-06 14:10:02 +05:30
Oran Agra
86e302f5f3 freeClientAsync don't lock mutex if there's just one thread 2020-02-06 10:07:17 +02:00
antirez
7d5fb5df3f Setting N I/O threads should mean N-1 additional + 1 main thread. 2020-01-13 18:53:36 +01:00
antirez
d28948b143 Port PR #6110 to new connection object code. 2020-01-13 12:54:39 +01:00
antirez
a8c912ead6 A few comments about main thread serving I/O as well.
Related to #6110.
2020-01-13 12:50:26 +01:00
Salvatore Sanfilippo
24896427fc
Merge pull request #6110 from soloestoy/enhance-io-threaded
Enhance IO Threaded: use main thread to handle read/write work
2020-01-13 12:46:40 +01:00
antirez
6e4f70b817 Fix duplicated CLIENT SETNAME reply.
Happened when we set the name to "" to cancel the name.
Was introduced during the RESP3 refactoring.

See #6036.
2019-12-29 15:44:59 +01:00
antirez
5521910de7 Inline protocol: handle empty strings well.
This bug is from the first version of Redis. Probably the problem here
is that before we used an SDS split function that created empty strings
for additional spaces, like in "SET    foo          bar".
AFAIK later we replaced it with the curretn sdssplitarg() API that has
no such a problem. As a result, we introduced a bug, where it is no
longer possible to do something like:

    SET foo ""

Using the inline protocol. Now it is fixed.
2019-12-29 15:40:40 +01:00
Madelyn Olson
034dcf185c Add module APIs for custom authentication 2019-12-17 06:59:59 +00:00
Salvatore Sanfilippo
bbdbfabf9e
Merge pull request #6531 from oranagra/rm_save_long_double
Module API for loading and saving long double
2019-11-04 17:33:35 +01:00
Oran Agra
779aebc91c Module API for loading and saving long double
looks like each platform implements long double differently (different bit count)
so we can't save them as binary, and we also want to avoid creating a new RDB
format version, so we save these are hex strings using "%La".

This commit includes a change in the arguments of ld2string to support this.
as well as tests for coverage and short reads.

coded by @guybe7
2019-11-03 16:42:31 +02:00
Oran Agra
51c3ff8d75 Modules hooks: complete missing hooks for the initial set of hooks
* replication hooks: role change, master link status, replica online/offline
* persistence hooks: saving, loading, loading progress
* misc hooks: cron loop, shutdown, module loaded/unloaded
* change the way hooks test work, and add tests for all of the above

startLoading() now gets flag indicating what is loaded.
stopLoading() now gets an indication of success or failure.
adding startSaving() and stopSaving() with similar args and role.
2019-10-29 17:59:09 +02:00
antirez
8cccc6b468 Modules hooks: don't call the hook for fake clients. 2019-10-23 18:39:53 +02:00
antirez
94c50314f1 Modules hooks: fix sub event in calls from client change. 2019-10-23 18:39:53 +02:00
antirez
019ac37e9c Modules hooks: fix define / linker issues. Implement one test event. 2019-10-23 18:39:53 +02:00
Yossi Gottlieb
0db3b0a0ff Merge remote-tracking branch 'upstream/unstable' into tls 2019-10-16 17:08:07 +03:00
Oran Agra
6b6294807c TLS: Implement support for write barrier. 2019-10-07 21:06:30 +03:00
Oran Agra
5a47794606 diskless replication rdb transfer uses pipe, and writes to sockets form the parent process.
misc:
- handle SSL_has_pending by iterating though these in beforeSleep, and setting timeout of 0 to aeProcessEvents
- fix issue with epoll signaling EPOLLHUP and EPOLLERR only to the write handlers. (needed to detect the rdb pipe was closed)
- add key-load-delay config for testing
- trim connShutdown which is no longer needed
- rioFdsetWrite -> rioFdWrite - simplified since there's no longer need to write to multiple FDs
- don't detect rdb child exited (don't call wait3) until we detect the pipe is closed
- Cleanup bad optimization from rio.c, add another one
2019-10-07 21:06:30 +03:00
Yossi Gottlieb
b087dd1db6 TLS: Connections refactoring and TLS support.
* Introduce a connection abstraction layer for all socket operations and
integrate it across the code base.
* Provide an optional TLS connections implementation based on OpenSSL.
* Pull a newer version of hiredis with TLS support.
* Tests, redis-cli updates for TLS support.
2019-10-07 21:06:13 +03:00
antirez
cb384127d1 RESP3: Use verbatim in CLIENT LIST. 2019-09-18 18:52:13 +02:00
antirez
e57067b677 More strict checks and better comments in flushSlaveOutputBuffers().
Related to #6296.
2019-08-31 14:46:24 +02:00
antirez
2303ba1441 Merge branch 'unstable' of github.com:/antirez/redis into unstable 2019-08-31 14:41:28 +02:00
antirez
ca5ccc58c5 Improve comment in flushSlavesOutputBuffers(). 2019-08-31 14:40:09 +02:00