Commit Graph

5056 Commits

Author SHA1 Message Date
zhaozhao.zz
cc81c6f1d3 Streams: lookupKey[Read->Write]OrReply in xdel and xtrim 2018-06-11 00:14:17 +08:00
michael-grunder
c04082cf13 Abort in XGROUP if the key is not a stream 2018-06-10 08:55:58 -07:00
Itamar Haber
e654b68d1f
Merge branch 'unstable' into module_command_help 2018-06-09 21:10:53 +03:00
Itamar Haber
fefde6e3e4 Capitalizes subcommands & orders lexicographically 2018-06-09 21:03:52 +03:00
Itamar Haber
21ef0376fe
Capitalizes subscommands 2018-06-09 20:54:05 +03:00
shenlongxing
1210dd8a20 fix integer case error 2018-06-09 22:24:33 +08:00
Salvatore Sanfilippo
be899b824e
Merge pull request #4519 from soloestoy/zset-int-problem
Zset int problem
2018-06-08 12:45:11 +02:00
Salvatore Sanfilippo
5ebaadc96b
Merge pull request #4800 from soloestoy/dict-expand
adjust position of _dictNextPower in dictExpand
2018-06-08 12:32:18 +02:00
Salvatore Sanfilippo
9fa2e7020e
Merge pull request #4861 from soloestoy/rdb-dict-expand
RDB: expand dict if needed when rdb load object
2018-06-08 12:12:34 +02:00
antirez
269e80526f Implement DEBUG htstats-key. 2018-06-08 11:17:20 +02:00
antirez
97e1f6812c Merge branch 'unstable' of github.com:/antirez/redis into unstable 2018-06-08 11:02:30 +02:00
antirez
2268d7e5dd redis-cli inline help updated. 2018-06-07 18:53:00 +02:00
Salvatore Sanfilippo
66c6218a3c
Merge pull request #4977 from michael-grunder/xpending-early-return
Return early in XPENDING if sent a nonexistent consumer group.
2018-06-07 18:49:09 +02:00
Itamar Haber
c199280edb Globally applies addReplySubSyntaxError 2018-06-07 18:39:36 +03:00
Itamar Haber
76ad23d012 Adds MODULE HELP and implements addReplySubSyntaxError 2018-06-07 18:34:58 +03:00
antirez
a7dbe37d53 Typo: entires -> entries in several places. 2018-06-07 14:36:56 +02:00
antirez
dfb3bf1b1e Streams: make macro node limits configurable. 2018-06-07 14:36:13 +02:00
antirez
6d34ff3645 Streams: max node limits only checked if non zero. 2018-06-07 14:30:25 +02:00
antirez
19a438e2c0 Streams: use non static macro node limits.
Also add the concept of size/items limit, instead of just having as
limit the number of bytes.
2018-06-07 14:24:49 +02:00
shenlongxing
c85ae56edc Fix write() errno error 2018-06-06 13:06:42 +02:00
antirez
b61416bdf4 Fix streamIteratorRemoveEntry() to update elements count.
Close #4989.
2018-06-06 11:40:22 +02:00
antirez
0a698e499a ZPOP: invert score-ele to match ZRANGE WITHSCORES order. 2018-06-05 17:06:25 +02:00
antirez
9e25f3e1de Remove XINFO <key> special form.
As observed by Michael Grunder this usage while practical is
inconsistent because for instance it does not work against a key called
HELP. Removed.
2018-06-05 16:34:34 +02:00
antirez
2f123ea6a3 Merge branch 'unstable' of github.com:/antirez/redis into unstable 2018-06-04 17:30:11 +02:00
antirez
36b392a0b2 XGROUP SETID implemented + consumer groups core fixes.
Now that we have SETID, the inetrnals of consumer groups should be able
to handle the case of the same message delivered multiple times just
as a side effect of calling XREADGROUP. Normally this should never
happen but if the admin manually "XGROUP SETID mykey mygroup 0",
messages will get re-delivered to clients waiting for the ">" special
ID. The consumer groups internals were not able to handle the case of a
message re-delivered in this circumstances that was already assigned to
another owner.
2018-06-04 17:28:03 +02:00
antirez
05a2996641 Rax library updated. 2018-06-04 17:26:16 +02:00
Salvatore Sanfilippo
b1783d3ab9
Merge pull request #4981 from yossigo/gcc7
Clean gcc 7.x warnings, redis-cli cluster fix.
2018-06-04 17:18:39 +02:00
antirez
7c6f1be5df XGROUP DESTROY implemented. 2018-06-04 12:58:26 +02:00
赵磊
10dedc2586 Fix core dump when using 'command getkeys' with wrong arguments. 2018-06-04 15:14:50 +08:00
Yossi Gottlieb
e1222d8b10 Clean gcc 7.x warnings, redis-cli cluster fix. 2018-06-03 15:54:30 +03:00
michael-grunder
1749fe7a26 Return early in XPENDING if sent a nonexistent consumer group. 2018-06-02 18:22:20 -07:00
WuYunlong
7d269d5e63 Fix DEBUG LOADAOF so that redis-server will not crash unexpectedly
and will not be inconsistent after we call debug loadaof.
Before this commit, there were 2 problems:

1, When appendonly is set to no and there is not a appendonly file,
   redis-server will crash if we call DEBUG LOADAOF.
2, When appendonly is set to no and there is a appendonly file,
   redis-server will hold different data after loading  appendonly
   file.
2018-06-02 14:32:16 +08:00
Salvatore Sanfilippo
86de089a1e
Merge pull request #4907 from youjiali1995/fix-dictScan
Fix dictScan(): It can't scan all buckets when dict is shrinking.
2018-06-01 16:54:05 +02:00
artix
d168b245b3 Cluster Manager: fixed master_id check in clusterManagerNodeLoadInfo 2018-06-01 16:19:47 +02:00
Salvatore Sanfilippo
d49d023828
Merge pull request #4971 from soloestoy/bugfix-bzpop
ZPOP: unblock multiple clients in right way
2018-05-31 18:40:53 +02:00
zhaozhao.zz
b9d19371e4 ZPOP: unblock multiple clients in right way 2018-05-31 23:35:47 +08:00
Salvatore Sanfilippo
2dee9a4118
Merge pull request #4945 from GitHubMota/debug_recover
Fix debug crash-and-recover help info.
2018-05-31 17:32:11 +02:00
Salvatore Sanfilippo
5bf08449f9
Merge pull request #4953 from soloestoy/fix-memory-overhead
MEMORY: fix the missing of monitor clients buffers
2018-05-31 17:31:24 +02:00
Salvatore Sanfilippo
f3b48978f2
Merge pull request #4964 from remicollet/issue-uint64
include stdint.h for unit64_t definition
2018-05-31 17:20:33 +02:00
antirez
2f963080c1 Capitalize OBJECT HELP subcommands. 2018-05-31 17:11:46 +02:00
artix
4bce88b11f Cluster Manager: fixed infinite loop in rebalance (Issue #4941). 2018-05-31 15:59:16 +02:00
Remi Collet
9561fec496 include stdint.h for unit64_t definition 2018-05-30 15:33:06 +02:00
antirez
3d5e2c62e2 Merge branch 'unstable' of github.com:/antirez/redis into unstable 2018-05-29 12:41:07 +02:00
antirez
49147f36e9 Don't expire keys while loading RDB from AOF preamble.
The AOF tail of a combined RDB+AOF is based on the premise of applying
the AOF commands to the exact state that there was in the server while
the RDB was persisted. By expiring keys while loading the RDB file, we
change the state, so applying the AOF tail later may change the state.

Test case:

* Time1: SET a 10
* Time2: EXPIREAT a $time5
* Time3: INCR a
* Time4: PERSIT A. Start bgrewiteaof with RDB preamble. The value of a is 11 without expire time.
* Time5: Restart redis from the RDB+AOF: consistency violation.

Thanks to @soloestoy for providing the patch.
Thanks to @trevor211 for the original issue report and the initial fix.

Check issue #4950 for more info.
2018-05-29 12:37:42 +02:00
WuYunlong
2a887bd53f Fix rdb save by allowing dumping of expire keys, so that when
we add a new slave, and do a failover, eighter by manual or
not, other local slaves will delete the expired keys properly.
2018-05-29 12:35:15 +02:00
zhaozhao.zz
ce2e0c02fc MEMORY: fix the missing of monitor clients buffers 2018-05-27 16:34:58 +08:00
antirez
6536ce27a4 Streams: fix comments referring to old ID format. 2018-05-25 17:25:19 +02:00
zhaozhao.zz
c2d7e6cea7 avoid warning in t_stream.c 2018-05-25 22:49:49 +08:00
antirez
fb9c1c3dbd Simplify XADD ID syntax error code path,. 2018-05-25 16:29:37 +02:00
Salvatore Sanfilippo
a60f553b68
Merge pull request #4832 from 0xtonyxia/fix-xadd-ID-parse
Bugfix: xadd command ID parse
2018-05-25 16:27:21 +02:00
antirez
95fa564add Avoid warning by explicitly initializing var to NULL. 2018-05-25 16:19:14 +02:00
antirez
b2b4e186a2 Explicitly cast void pointers to char pointers for printf style logging. 2018-05-25 16:15:59 +02:00
zhaozhao.zz
2711f053f2 append -lrt to FINAL_LIBS for linux 2018-05-25 20:16:57 +08:00
antirez
e4f5c04930 Don't append libjemalloc.a otherwise it will be after -ldl. 2018-05-25 13:36:51 +02:00
Mota
af3471a5dc Fix debug crash-and-recover help info. 2018-05-25 16:49:23 +08:00
antirez
694145bd74 Merge branch 'unstable' of github.com:/antirez/redis into unstable 2018-05-24 18:21:23 +02:00
antirez
98d5d3f118 Make active defragmentation tests optional.
They failed when active defrag could not be activated because the
Jemalloc version does not include the additional APIs.
2018-05-24 18:04:21 +02:00
Salvatore Sanfilippo
b23a501135
Merge pull request #4099 from JunHe77/aarch64_stack_cflag
Fixed stack trace generation on aarch64
2018-05-24 16:16:12 +02:00
Salvatore Sanfilippo
18b7f71130
Merge pull request #4932 from 0xtonyxia/fix-redis-cli-set-prefer-memory-leak
Fix redis-cli memory leak when sending set :preference command.
2018-05-24 16:15:21 +02:00
Salvatore Sanfilippo
ef7ccad180
Merge pull request #4922 from 0xtonyxia/enhance-cli-security
Enhance security for redis-cli
2018-05-24 16:14:42 +02:00
Salvatore Sanfilippo
8a200b04c3
Merge pull request #4831 from 0xtonyxia/fix-xadd-arity-check
Bugfix: xadd comand <field,value> arity check
2018-05-24 16:07:19 +02:00
antirez
de4b6cb27b Merge branch 'unstable' of github.com:/antirez/redis into unstable 2018-05-23 17:14:59 +02:00
antirez
8631e64779 Sentinel: fix delay in detecting ODOWN.
See issue #2819 for details. The gist is that when we want to send INFO
because we are over the time, we used to send only INFO commands, no
longer sending PING commands. However if a master fails exactly when we
are about to send an INFO command, the PING times will result zero
because the PONG reply was already received, and we'll fail to send more
PINGs, since we try only to send INFO commands: the failure detector
will delay until the connection is closed and re-opened for "long
timeout".

This commit changes the logic so that we can send the three kind of
messages regardless of the fact we sent another one already in the same
code path. It could happen that we go over the message limit for the
link by a few messages, but this is not significant. However now we'll
not introduce delays in sending commands just because there was
something else to send at the same time.
2018-05-23 17:13:44 +02:00
Salvatore Sanfilippo
1ab3c82375
Merge pull request #4908 from soloestoy/aof-rdb-preamble-compatible-checksum-no
AOF & RDB: be compatible with rdbchecksum no
2018-05-23 17:11:00 +02:00
zhaozhao.zz
4e7a160b9b ZPOP: fix the wrong keyc, should be 1 2018-05-22 21:31:22 +08:00
dejun.xdj
46e241ac8e Fix redis-cli memory leak when sending set preference command. 2018-05-21 12:19:37 +08:00
dejun.xdj
95b988b6c6 Check if the repeat value is positive in while loop of cliSendCommand().
In case that the incoming repeat parameter is negative and causes a
deadless loop.
2018-05-21 12:06:48 +08:00
dejun.xdj
cc7ffdfdf2 Change the type of repeat argument to long for function cliSendCommand.
To be in consistent with the original definition.
2018-05-21 12:04:53 +08:00
dejun.xdj
b2762f1ff2 Fix negtive repeat command value issue.
If command like "-1 set a b" is sent with redis-cli, it will cause a deadless loop. So some repeat value checking logic is added to avoid this.
2018-05-19 22:50:40 +08:00
dejun.xdj
c2e2314640 Detect and stop saving history for auth command with repeat option.
Put the repeat option checking code a little forward to avoid repeat logic.
2018-05-18 11:40:05 +08:00
dejun.xdj
ef931ef93e Change the warning message a little bit to avoid trademark issuses. 2018-05-18 11:37:31 +08:00
antirez
3c968ff030 Merge branch 'unstable' of github.com:/antirez/redis into unstable 2018-05-17 16:53:14 +02:00
antirez
3c43e984db Merge remote-tracking branch 'artix/trib2cli' into unstable 2018-05-17 16:52:31 +02:00
Oran Agra
ad133e1023 Active defrag fixes for 32bit builds
problems fixed:
* failing to read fragmentation information from jemalloc
* overflow in jemalloc fragmentation hint to the defragger
* test suite not triggering eviction after population
2018-05-17 09:52:00 +03:00
artix
e47c751c74 Removed TODO in redis-cli 2018-05-16 18:04:13 +02:00
artix
1e4fb1b33a Cluster Manager: fixed unprinted reply error 2018-05-16 17:49:18 +02:00
dejun.xdj
b263c7c465 Stop saving auth command in redis-cli history. 2018-05-16 16:18:00 +08:00
dejun.xdj
c082221aef Add warning message when using password on command line 2018-05-16 16:15:12 +08:00
artix
2f9c032a13 Cluster Manager: print flags as strings. 2018-05-15 18:41:46 +02:00
antirez
25f017e563 ZPOP: fix replication of blocking ZPOP. 2018-05-15 16:03:56 +02:00
artix
3c039996b5 - Fixed mistyped redis command (clusterManagerGetNodeWithMostKeysInSlot)
- Cluster node structure is now updated after ADDSLOTS
2018-05-11 18:29:12 +02:00
antirez
56bbab238a ZPOP: change sync ZPOP to have a count argument instead of N keys.
Usually blocking operations make a lot of sense with multiple keys so
that we can listen to multiple queues (or whatever the app models) with
a single connection. However in the synchronous case it is more useful
to be able to ask for N elements. This is a change that I also wanted to
perform soon or later in the blocking list variant, but here it is more
natural since there is no reply type difference.
2018-05-11 18:00:32 +02:00
antirez
6efb6c1e06 ZPOP: renaming to have explicit MIN/MAX score idea.
This commit also adds a top comment about a subtle behavior of mixing
blocking operations of different types in the same key.
2018-05-11 17:31:53 +02:00
antirez
6b026b70a8 Merge branch 'zpop' of https://github.com/itamarhaber/redis into zpop 2018-05-11 10:40:13 +02:00
antirez
b85aae78df Fix rdb.c dictionary iterator release in 2 more places. 2018-05-09 12:06:37 +02:00
antirez
cd87b3c71f Fix rdb.c dictionary iterator release.
Some times it was not released on error, sometimes it was released two
times because the error path expected the "di" var to be NULL if the
iterator was already released. Thanks to @oranagra for pinging me about
potential problems of this kind inside rdb.c.
2018-05-09 11:03:27 +02:00
Salvatore Sanfilippo
aaef5f00cc
Merge pull request #4594 from smallstool/bugfix-freeMemoryIfneeded-int-overflow
fix int overflow problem in freeMemoryIfNeeded
2018-05-08 17:27:18 +02:00
zhaozhao.zz
edb92db533 AOF & RDB: be compatible with rdbchecksum no 2018-05-08 19:22:13 +08:00
赵磊
8d93f924ea Fix dictScan(): It can't scan all buckets when dict is shrinking. 2018-05-08 15:30:11 +08:00
artix
3312de067c Cluster Manager: --cluster options can now be placed everywhere 2018-05-07 17:31:34 +02:00
Krzysztof Filipek
fd9177dd33 Typo in preprocessor condition 2018-05-06 20:18:48 +02:00
Itamar Haber
49890c8ee9 Adds memory information about the script's cache to INFO
Implementation notes: as INFO is "already broken", I didn't want to break it further. Instead of computing the server.lua_script dict size on every call, I'm keeping a running sum of the body's length and dict overheads.

This implementation is naive as it **does not** take into consideration dict rehashing, but that inaccuracy pays off in speed ;)

Demo time:

```bash
$ redis-cli info memory | grep "script"
used_memory_scripts:96
used_memory_scripts_human:96B
number_of_cached_scripts:0
$ redis-cli eval "" 0 ; redis-cli info memory | grep "script"
(nil)
used_memory_scripts:120
used_memory_scripts_human:120B
number_of_cached_scripts:1
$ redis-cli script flush ; redis-cli info memory | grep "script"
OK
used_memory_scripts:96
used_memory_scripts_human:96B
number_of_cached_scripts:0
$ redis-cli eval "return('Hello, Script Cache :)')" 0 ; redis-cli info memory | grep "script"
"Hello, Script Cache :)"
used_memory_scripts:152
used_memory_scripts_human:152B
number_of_cached_scripts:1
$ redis-cli eval "return redis.sha1hex(\"return('Hello, Script Cache :)')\")" 0 ; redis-cli info memory | grep "script"
"1be72729d43da5114929c1260a749073732dc822"
used_memory_scripts:232
used_memory_scripts_human:232B
number_of_cached_scripts:2
✔ 19:03:54 redis [lua_scripts-in-info-memory L ✚…⚑] $ redis-cli evalsha 1be72729d43da5114929c1260a749073732dc822 0
"Hello, Script Cache :)"
```
2018-04-30 19:33:01 +03:00
Itamar Haber
438125b47c Implements [B]Z[REV]POP and the respective unit tests
An implementation of the
[Ze POP Redis Module](https://github.com/itamarhaber/zpop) as core
Redis commands.

Fixes #1861.
2018-04-30 02:10:42 +03:00
artix
be94e89031 Cluster Manager: fixed expected slots calculation (rebalance)
Cluster Manager: fixed argument parsing after --cluster-weight
2018-04-23 16:28:48 +02:00
artix
2f31545beb Cluster Manager: fixed bug when parsing CLUSTER NODES reply (clusterManagerNodeLoadInfo) 2018-04-23 16:28:48 +02:00
artix
5bc2c98789 Cluster Manager: code improvements and more comments added. 2018-04-23 16:28:48 +02:00
artix
5f358dae33 Cluster Manager: set-timeout command 2018-04-23 16:28:48 +02:00
artix
52f17f6f8e - Cluster Manager: del-node command.
- Cluster Manager: fixed bug in clusterManagerNodeWithLeastReplicas
2018-04-23 16:28:48 +02:00
artix
aeaf6ee1c3 Cluster Manager: add-node command. 2018-04-23 16:28:48 +02:00
artix
efa51f1617 Cluster Manager: added clusterManagerCheckCluster to import command 2018-04-23 16:28:48 +02:00
artix
1f548359cb Cluster Manager: import command 2018-04-23 16:28:48 +02:00
artix
3f8a4adb49 Cluster Manager: fix command. 2018-04-23 16:28:48 +02:00
artix
6d1a7cec23 Cluster Manager: rebalance command 2018-04-23 16:28:48 +02:00
artix
adebee8adc clusterManagerAddSlots: changed the way ADDSLOTS command is built 2018-04-23 16:28:48 +02:00
artix
a4a1c7bb52 ClusterManager: fixed --cluster-from 'all' parsing 2018-04-23 16:28:48 +02:00
Artix
5b3d73b3d7 Cluster Manager: fixed some memory error 2018-04-23 16:28:48 +02:00
artix
220375b424 Fixed memory write error in clusterManagerGetConfigSignature 2018-04-23 16:28:48 +02:00
artix
99da9c9508 Cluster Manager: reshard command, fixed slots
parsing bug and other minor bugs.
2018-04-23 16:28:48 +02:00
artix
7d609ff952 Cluster Manager:
- Almost all Cluster Manager related code moved to
  the same section.
- Many macroes converted to functions
- Added various comments
- Little code restyling
2018-04-23 16:28:48 +02:00
artix
4e0c2f9c3c - Fixed bug in clusterManagerGetAntiAffinityScore
- Code improvements
2018-04-23 16:28:48 +02:00
artix
18910013cd Cluster Manager: colorized output 2018-04-23 16:28:48 +02:00
artix
307d995f75 Cluster Manager: improved cleanup/error handling in various functions 2018-04-23 16:28:48 +02:00
artix
2d677e2bf3 Cluster Manager: 'call' command. 2018-04-23 16:28:48 +02:00
artix
1ad1f00163 Cluster Manager: CLUSTER_MANAGER_NODE_CONNECT macro 2018-04-23 16:28:48 +02:00
artix
2f48d62423 ClusterManager: added replicas count to clusterManagerNode 2018-04-23 16:28:48 +02:00
artix
c002b95d89 Cluster Manager: cluster is considered consistent if only one node has been found 2018-04-23 16:28:48 +02:00
artix
4cc8de1a37 Cluster Manager: reply error catch for MEET command 2018-04-23 16:28:48 +02:00
artix
65d37960e7 Cluster Manager: slots coverage check. 2018-04-23 16:28:48 +02:00
artix
b3e0ca3412 - Cluster Manager: fixed various memory leaks
- Cluster Manager: fixed flags assignment in
  clusterManagerNodeLoadInfo
2018-04-23 16:28:48 +02:00
artix
8c7ad80f9f Added check for open slots (clusterManagerCheckCluster) 2018-04-23 16:28:47 +02:00
artix
486c7af7b8 Cluster Manager: 'create', 'info' and 'check' commands 2018-04-23 16:28:47 +02:00
artix
ad911a338a Cluster Manager mode 2018-04-23 16:28:47 +02:00
zhaozhao.zz
24036b4d32 RDB: expand dict if needed when rdb load object 2018-04-22 22:30:44 +08:00
antirez
e6b0e8d9ec Streams: XTRIM command added. 2018-04-19 16:25:29 +02:00
antirez
19ae809458 Streams: XADD, handle signaling of write in different ways. 2018-04-19 16:11:43 +02:00
antirez
aba76320d5 Streams: XDEL command. 2018-04-18 13:12:09 +02:00
antirez
9c149bf1f1 Streams: higher level "delete by ID" API for streams. 2018-04-17 17:55:56 +02:00
antirez
24ac2b4c74 Streams: iterator entry deletion abilities. 2018-04-17 17:18:00 +02:00
zhaozhao.zz
775adf544c Modules Timer API: fix wrong raxInsert() usage 2018-04-14 01:20:02 +08:00
antirez
e07af6a2b7 Modules API: Add call to get the blocked client handle from the context.
This is useful in the reply and timeout callback, if the module wants to
do some cleanup of the blocked client handle that may be stored around
in the module-private data structures.
2018-04-13 13:48:11 +02:00
antirez
da0e192277 Modules API: moduleGetReplyClient() refactoring. 2018-04-13 13:42:49 +02:00
antirez
404160a271 Modules API: blocked client disconnection callback. 2018-04-12 13:21:48 +02:00
antirez
005c932f22 Modules API: fix timer example. 2018-04-12 13:00:18 +02:00
antirez
4e53624052 Modules API: fix OOM_WARNING flag implementation. 2018-04-11 16:25:54 +02:00
antirez
0afac6939a Modules API: OOM_WARNING flags.
In some modules it may be useful to have an idea about being near to
OOM. Anyway additionally an explicit call to get the fill ratio will be
added in the future.
2018-04-11 16:22:52 +02:00
antirez
de7de53e64 getMaxmemoryState() fixed and improved. 2018-04-11 12:48:26 +02:00
dejun.xdj
b6b00f35ca Return more accurate message when specified ID is invalid 2018-04-10 17:11:10 +08:00
dejun.xdj
3c20b3fc24 Bugfix: xadd command ID parse
strictly check the string to be converted,
strtoull() in libc may not set errno to EINVAL when the string contains
invalid digits.
2018-04-10 17:05:48 +08:00
dejun.xdj
07125b8d22 Bugfix: xadd comand <field,value> arity check
missing parenthesis causes wrong arithmetic priority.
2018-04-10 16:36:05 +08:00
antirez
f97efe0cac Modules: context flags now include OOM flag.
Plus freeMemoryIfNeeded() refactoring to improve legibility.
Please review this commit for sanity.
2018-04-09 17:44:30 +02:00
antirez
6888c1a10d freeMemoryIfNeeded() lacked a top comment. 2018-04-09 17:20:36 +02:00
antirez
9a0dbbb594 Modules: remove trailing empty spaces. 2018-04-09 17:16:55 +02:00
antirez
eaafea4828 Modules API: experimental APIs version.
This way it is possible to use conditional compilation to be compatible
with a larger amount of Redis versions, however note that this breaks
binary compatibiltiy, so the module must be compiled with the
corresponding redismodule.h file depending on the version of Redis
targeted.
2018-04-09 12:17:19 +02:00
antirez
49e098234a Modules API: blocked client free callback modified to get a context.
Note that this was an experimental API that can only be enabled with
REIDSMODULE_EXPERIMENTAL_API, so it is subject to change until its
promoted to stable API. Sorry for the breakage, it is trivial to
resolve btw. This change will not be back ported to Redis 4.0.
2018-04-09 11:54:44 +02:00
charpty
0fd2b25c8d Remove unnecessary return statements
Signed-off-by: charpty <charpty@gmail.com>
2018-04-06 18:46:24 +08:00
antirez
b2868c7b9c Modules API: RM_GetRandomBytes() / GetRandomHexChars(). 2018-04-05 13:24:22 +02:00
antirez
c75582889a Modules Cluster API: GetClusterSize() added. 2018-04-01 16:36:32 +02:00
antirez
27f9c8108c Modules Cluster API: GetMyClusterID() added. 2018-04-01 16:20:57 +02:00
antirez
19c42c901b Modules Timer API: add example of API. 2018-03-31 10:12:34 +02:00
antirez
ee982f4031 Modules Timer API: Wait at least 1 ms per iteration. Convert to ms. 2018-03-31 10:11:30 +02:00
antirez
4c11bc6cf0 Modules Timer API: fix wrong raxRemove() key argument. 2018-03-31 09:58:43 +02:00
antirez
2f7da0fd1a Modules Timer API: fix infinite loop and export API. 2018-03-31 00:44:46 +02:00
antirez
b85a465c25 Modules Timer API: timer handling implemented. 2018-03-30 22:50:21 +02:00
antirez
561039c125 Modules Timer API: initial implementation. 2018-03-30 20:40:35 +02:00
antirez
192361b562 Modules Cluster API: node API exported, example improved. 2018-03-30 17:00:45 +02:00
antirez
16178b692e Modules Cluster API: nodes list and info API. 2018-03-30 16:16:47 +02:00
antirez
83ec35770e Modules Cluster API: node information struct and flags. 2018-03-30 13:16:55 +02:00
antirez
a97df1a6e1 Modules Cluster API: make node IDs pointers constant. 2018-03-30 13:16:07 +02:00
antirez
061f03d730 Modules Cluster API: add a simple example module. 2018-03-30 12:49:45 +02:00
antirez
82004f9dbe Modules Cluster API: fix new API calls exporting. 2018-03-30 12:49:16 +02:00
antirez
b4dc782e4e Modules Cluster API: sending / receiving API first implementation. 2018-03-30 11:06:08 +02:00
antirez
0701cad3de Modules Cluster API: message bus implementation. 2018-03-29 15:13:31 +02:00
zhaozhao.zz
83cf0e3668 adjust position of _dictNextPower in dictExpand 2018-03-29 17:36:15 +08:00
antirez
8ac7af1c5d Fix ae.c when a timer finalizerProc adds an event.
While this feature is not used by Redis, ae.c implements the ability for
a timer to call a finalizer callback when an timer event is deleted.
This feature was bugged since the start, and because it was never used
we never noticed a problem. However Anthony LaTorre was using the same
library in order to implement a different system: he found a bug that he
describes as follows, and which he fixed with the patch in this commit,
sent me by private email:

    --- Anthony email ---

've found one bug in the current implementation of the timed events.
It's possible to lose track of a timed event if an event is added in
the finalizerProc of another event.

For example, suppose you start off with three timed events 1, 2, and
3. Then the linked list looks like:

3 -> 2 -> 1

Then, you run processTimeEvents and events 2 and 3 finish, so now the
list looks like:

-1 -> -1 -> 2

Now, on the next iteration of processTimeEvents it starts by deleting
the first event, and suppose this finalizerProc creates a new event,
so that the list looks like this:

4 -> -1 -> 2

On the next iteration of the while loop, when it gets to the second
event, the variable prev is still set to NULL, so that the head of the
event loop after the next event will be set to 2, i.e. after deleting
the next event the event loop will look like:

2

and the event with id 4 will be lost.

I've attached an example program to illustrate the issue. If you run
it you will see that it prints:

```
foo id = 0
spam!
```

But if you uncomment line 29 and run it again it won't print "spam!".

    --- End of email ---

Test.c source code is as follows:

    #include "ae.h"
    #include <stdio.h>

    aeEventLoop *el;

    int foo(struct aeEventLoop *el, long long id, void *data)
    {
	printf("foo id = %lld\n", id);

	return AE_NOMORE;
    }

    int spam(struct aeEventLoop *el, long long id, void *data)
    {
	printf("spam!\n");

	return AE_NOMORE;
    }

    void bar(struct aeEventLoop *el, void *data)
    {
	aeCreateTimeEvent(el, 0, spam, NULL, NULL);
    }

    int main(int argc, char **argv)
    {
	el = aeCreateEventLoop(100);

	//aeCreateTimeEvent(el, 0, foo, NULL, NULL);
	aeCreateTimeEvent(el, 0, foo, NULL, bar);

	aeMain(el);

	return 0;
    }

Anthony fixed the problem by using a linked list for the list of timers, and
sent me back this patch after he tested the code in production for some time.
The code looks sane to me, so committing it to Redis.
2018-03-28 14:11:04 +02:00
antirez
28d28ef3cf AOF: enable RDB-preamble rewriting by default.
There are too many advantages in doing this, RDB is faster to persist,
more compact, much faster to load back. The main issues here are that
the code is less tested because this was not the old default (so we are
enabling it for the new 5.0 release), and that the AOF is no longer a
trivially parsable format from now on. However the non-preamble mode
will be supported in the future as well, if new data types will be
added.
2018-03-25 11:43:30 +02:00
antirez
8b20112860 Merge branch 'unstable' of github.com:/antirez/redis into unstable 2018-03-23 17:21:46 +01:00
antirez
1392c83fb8 CG: AOF rewriting implemented. 2018-03-23 17:21:31 +01:00
Salvatore Sanfilippo
e08d6bb37e
Merge pull request #4703 from gechunlin/unstable
Update object.c
2018-03-22 16:32:45 +01:00
Salvatore Sanfilippo
bad679137e
Merge pull request #4679 from youjiali1995/fix-lru
Some commands will udpate lfu-counter many times in one call.
2018-03-22 16:26:01 +01:00
Salvatore Sanfilippo
44f2cfa631
Merge pull request #4722 from charsyam/feature/refactoring-call-aeDeleteFileEvent-twice-in-freeClusterLink
Refactoring to call aeDeleteFileEvent twice as once
2018-03-22 16:23:40 +01:00
Salvatore Sanfilippo
b38f52169b
Merge pull request #4777 from soloestoy/avoid-double-close
anet: avoid double close
2018-03-22 16:22:32 +01:00
Salvatore Sanfilippo
3163c9cb63
Merge pull request #4781 from guybe7/block_list_notify
Make blocking list commands send keyspace notifications
2018-03-22 16:21:19 +01:00
antirez
6c4cb1670a Add top comments in two addReply*() functions. 2018-03-22 11:45:04 +01:00
antirez
b86c26b2fd Massivily simplify addReply*() functions in networking.c 2018-03-22 11:42:50 +01:00
Guy Benoish
fa00e20b16 Make blocking list commands send keyspace notifications 2018-03-22 17:22:26 +07:00
Salvatore Sanfilippo
5d8c89163c
Merge pull request #4757 from charsyam/feature/fix-invalid-listpack-type-in-makefile
fix listpack.c to listpack.o in Makefile
2018-03-22 09:20:17 +01:00
Salvatore Sanfilippo
7acf0e0a18
Merge pull request #3826 from charsyam/feature/fix_dlopen_leak
fix missed dlclose leak in module.c
2018-03-22 09:19:26 +01:00
Salvatore Sanfilippo
da621783f0
Merge pull request #4691 from oranagra/active_defrag_v2
Active defrag v2
2018-03-22 09:16:32 +01:00
charsyam
71c7477fd4 fix memory leak for streamPropagateXCLAIM 2018-03-21 17:41:59 +09:00
zhaozhao.zz
46f767aa19 fix missed call on freeaddrinfo 2018-03-21 15:34:13 +08:00
zhaozhao.zz
11baa8487c anet: avoid double close 2018-03-21 15:13:51 +08:00
antirez
233da29f4d CG: propagate XACK by incrementing server.dirty.
This should be more than enough, even if in case of partial IDs that are
not found, we send all the IDs to the slave/AOF, but this is definitely
a corner case without bad effects if not some wasted space.
2018-03-20 17:53:45 +01:00
antirez
0f2c6b6a18 Streams: improve MEMORY USAGE computation, include CGs. 2018-03-20 17:50:40 +01:00
antirez
5577130451 CG: Make XINFO Great Again (and more Redis-ish).
With XINFO out of the blue I invented a new syntax for commands never
used in Redis in the past... Let's fix it and make it

        Great Again!!11one (TM)
2018-03-20 11:52:42 +01:00
antirez
2ba9a57c9a CG: Add JUSTID to XCLAIM propagation to reduce CPU usage. 2018-03-20 11:25:49 +01:00
antirez
ad716e29c0 CG: Fix propagate() arg count in streamPropagateXCLAIM(). 2018-03-20 11:13:41 +01:00
antirez
3cae5f0321 CG: XGROUP CREATE/DELCONUSMER replication. 2018-03-20 11:07:01 +01:00
antirez
0b58ad301e CG: Replication WIP 1: XREADGROUP and XCLAIM propagated as XCLAIM. 2018-03-19 18:02:19 +01:00
antirez
36b78e8dfe Aesthetic changes to PR #4749. 2018-03-16 16:57:53 +01:00
antirez
7e3c583532 Merge branch 'hyperloglog-improvement' of https://github.com/oertl/redis into oertl-hyperloglog-improvement 2018-03-16 16:27:38 +01:00
antirez
38bc8ea531 RDB version 9. 2018-03-16 13:48:44 +01:00
antirez
8b0cfb1e66 RDB: Implement future-proof module AUX data loading. 2018-03-16 13:47:10 +01:00
zhaozhao.zz
54cae05ea7 rdb: incremental fsync when redis saves rdb 2018-03-16 00:44:50 +08:00
antirez
b1aae86fc6 RDB: make RDB check aware of LFU/LRU opcodes. 2018-03-15 16:39:33 +01:00
antirez
8176a2ee76 RDB: LRU/LFU branches missed continue. 2018-03-15 16:33:18 +01:00
antirez
1ce50a7adf RDB: Ability to load LFU/LRU info. 2018-03-15 16:24:53 +01:00
antirez
d7a5c0eb71 RDB: Ability to save LFU/LRU info.
This is a big win for caching use cases, since on reloading Redis will
still have some idea about what is worth to evict and what not.
However this only solves part of the problem because the information is
only partially propagated to slaves (on write operations). Reads will
not affect slaves LFU and LRU counters, so after a failover the eviction
decisions are kinda random until keys start to collect some aging/freq info.

However since new slaves are initially populated via RDB file transfer,
this means that if we spin up a new slave from a master, and perform an
immediate manual failover (for instance in order to upgrade the master),
the slave will have eviction informations to use for some time.

The LFU/LRU info is persisted only if the maxmemory policy is set to one
of the relevant type, even if no actual "maxmemory"  memory limit is
set.
2018-03-15 13:15:55 +01:00
antirez
6614361615 CG: XINFO STREAM. 2018-03-15 12:54:10 +01:00
antirez
d7d8cd0b2f CG: XINFO GROUPS + output format changes.
XINFO is mainly an observability command that will be used more by
humans than computers, and even when used by computers it will be a very
low traffic command. For this reason the format was changed in order to
have field names. They'll consume some bandwidth and CPU cycles, but in
this context this is much better than having to understand what the
numbers in the output array are.
2018-03-15 12:54:10 +01:00
antirez
0cf6b1e3ae CG: XINFO CONSUMERS implemented. 2018-03-15 12:54:10 +01:00
antirez
67eeeb0b10 Streams: fix XREAD missing check for NULL object. 2018-03-15 12:54:10 +01:00
antirez
8d8755c7b5 CG: throw an error if XREADGROUP is used without GROUP. 2018-03-15 12:54:10 +01:00
antirez
d4f81ebdba CG: XGROUP DELCONSUMER implemented. 2018-03-15 12:54:10 +01:00
antirez
9b423ae237 CG: remove stale comment. XREADGROUP is now implemented. 2018-03-15 12:54:10 +01:00
antirez
e6d434c167 CG: implement RETRYCOUNT in XCLAIM. 2018-03-15 12:54:10 +01:00
antirez
f3d9520ccb CG: fix CG RDB loading not found conditional. 2018-03-15 12:54:10 +01:00
antirez
34614a1508 CG: NACK created by XCLAIM FORCE must be set in global PEL. 2018-03-15 12:54:10 +01:00
antirez
596264aee9 CG: implement XCLAIM FORCE option. 2018-03-15 12:54:10 +01:00
antirez
b26f03bd69 CG: XCLAIM now updates the idle time of the message. 2018-03-15 12:54:10 +01:00
antirez
f7d4c3acdf Streams: trap more errors in stream loading + RDB check type name. 2018-03-15 12:54:10 +01:00
antirez
8727b4845b CG: XCLAIM, use minidle and fix array len. 2018-03-15 12:54:10 +01:00
antirez
09e3b3b975 CG: remove unused argument from streamReplyWithRangeFromConsumerPEL(). 2018-03-15 12:54:10 +01:00
antirez
13ff7bc3ef CG: fix RDB saving when there are no consumer groups. 2018-03-15 12:54:10 +01:00
antirez
267f7f2c97 Streams: fix error description for XADD when specified ID is small. 2018-03-15 12:54:10 +01:00
antirez
0a6780e560 CG: XCLAIM initial draft. 2018-03-15 12:54:10 +01:00
antirez
00a29b1a81 Make addReplyError...() family functions able to get error codes.
Now you can use:

    addReplyError("-MYERRORCODE some message");

If the error code is omitted, the behavior is like in the past,
the generic -ERR will be used.
2018-03-15 12:54:10 +01:00
antirez
c9d86c2b16 CG: More specific duplicated group error. 2018-03-15 12:54:10 +01:00
antirez
9f60a6bcee CG: RDB loading, fix inverted conditional. 2018-03-15 12:54:10 +01:00
antirez
f4e1a4de25 CG: RDB loading first implementation. 2018-03-15 12:54:10 +01:00
antirez
db7a5f23b4 CG: RDB saving part 2, consumers. 2018-03-15 12:54:10 +01:00
antirez
8fb6048ed0 CG: RDB saving part 1, metadata and PEL. 2018-03-15 12:54:10 +01:00
antirez
e76fb4ab25 CG: XPENDING should not create consumers and obey to count. 2018-03-15 12:54:10 +01:00
antirez
f3708af7f9 CG: XPENDING with start/stop/count variant implemented. 2018-03-15 12:54:10 +01:00
antirez
1bc31666da CG: XPENDING without start/stop variant implemented. 2018-03-15 12:54:10 +01:00
antirez
b65fe09bb8 CG: Now XREADGROUP + blocking operations work. 2018-03-15 12:54:10 +01:00
antirez
5ad29325fe CG: XACK should return zero when nothing is processed. 2018-03-15 12:54:10 +01:00
antirez
388c69fe4e CG: XACK implementation. 2018-03-15 12:54:10 +01:00
antirez
5bbd117c29 CG: XREADGROUP can fetch data from the consumer PEL. 2018-03-15 12:54:10 +01:00
antirez
aa808394f6 CG: first draft of streamReplyWithRangeFromConsumerPEL(). 2018-03-15 12:54:10 +01:00
antirez
bbec4569a5 CG: Fix order of calls in streamReplyWithRange().
We need to check if we are going to serve the request via the PEL before
inserting a deferred array len in the client output buffer.
2018-03-15 12:54:10 +01:00
antirez
41809fd969 CG: creation of NACK entries in PELs. 2018-03-15 12:54:10 +01:00
antirez
1ffb6723f5 CG: fix XREADGROUP ">" special ID parsing due to missing "continue". 2018-03-15 12:54:10 +01:00
antirez
6c0af37b6e CG: streamCompareID() + group last_id updating. 2018-03-15 12:54:10 +01:00
antirez
86fe8fde20 CG: consumer lookup + initial streamReplyWithRange() work to supprot CG. 2018-03-15 12:54:10 +01:00
antirez
ccdae09046 CG: add & populate group+consumer in the blocking state. 2018-03-15 12:54:10 +01:00
antirez
b8e5232161 CG: fix parsing in XREADGROUP and streamLookupCG() NULL check. 2018-03-15 12:54:10 +01:00
antirez
bd1c11dc35 CG: add XREADGROUP in the command table. 2018-03-15 12:54:10 +01:00
antirez
2bbb2bf427 CG: XGROUPREAD group option parsing and groups lookup. 2018-03-15 12:54:10 +01:00
antirez
1fafe7def1 CG: fix raxFind() retval check in streamCreateCG(). 2018-03-15 12:54:10 +01:00
antirez
58f0c000a5 CG: data structures design + XGROUP CREATE implementation. 2018-03-15 12:54:10 +01:00
charsyam
c76f890209 fix listpack.c to listpack.o in Makefile 2018-03-15 20:32:08 +09:00
Otmar Ertl
15d7e61701 fixed compilation error when using clang as reported by michael-grunder 2018-03-14 21:00:06 +01:00
antirez
432bf4770e Cluster: ability to prevent slaves from failing over their masters.
This commit, in some parts derived from PR #3041 which is no longer
possible to merge (because the user deleted the original branch),
implements the ability of slaves to have a special configuration
preventing that they try to start a failover when the master is failing.

There are multiple reasons for wanting this, and the feautre was
requested in issue #3021 time ago.

The differences between this patch and the original PR are the
following:

1. The flag is saved/loaded on the nodes configuration.
2. The 'myself' node is now flag-aware, the flag is updated as needed
   when the configuration is changed via CONFIG SET.
3. The flag name uses NOFAILOVER instead of NO_FAILOVER to be consistent
   with existing NOADDR.
4. The redis.conf documentation was rewritten.

Thanks to @deep011 for the original patch.
2018-03-14 14:01:38 +01:00
Oran Agra
806736cdf9 Adding real allocator fragmentation to INFO and MEMORY command + active defrag test
other fixes / improvements:
- LUA script memory isn't taken from zmalloc (taken from libc malloc)
  so it can cause high fragmentation ratio to be displayed (which is false)
- there was a problem with "fragmentation" info being calculated from
  RSS and used_memory sampled at different times (now sampling them together)

other details:
- adding a few more allocator info fields to INFO and MEMORY commands
- improve defrag test to measure defrag latency of big keys
- increasing the accuracy of the defrag test (by looking at real grag info)
  this way we can use an even lower threshold and still avoid false positives
- keep the old (total) "fragmentation" field unchanged, but add new ones for spcific things
- add these the MEMORY DOCTOR command
- deduct LUA memory from the rss in case of non jemalloc allocator (one for which we don't "allocator active/used")
- reduce sampling rate of the rss and allocator info
2018-03-12 15:08:52 +02:00
Oran Agra
be1b4aa9aa active defrag v2
- big keys are not defragged in one go from within the dict scan
  instead they are scanned in parts after the main dict hash bucket is done.
- add latency monitor sample for defrag
- change default active-defrag-cycle-min to induce lower latency
- make active defrag start a new scan right away if needed, so it's easier
  (for the test suite) to detect when it's done
- make active defrag quick the current cycle after each db / big key
- defrag  some non key long term global allocations
- some refactoring for smaller functions and more reusable code
- during dict rehashing, one scan iteration of the dict, can end up scanning
  one bucket in the smaller dict and many many buckets in the larger dict.
  so waiting for 16 scan iterations before checking the time, may be much too long.
2018-03-12 15:07:43 +02:00
Otmar Ertl
97bde9f623 use all 64 bits of the hash value instead of 63 2018-03-11 09:18:00 +01:00
Otmar Ertl
44698f45e7 made constant static 2018-03-10 20:44:20 +01:00
Otmar Ertl
633983d479 improved definition of HLL_Q 2018-03-10 20:22:42 +01:00
Otmar Ertl
1e9a774871 improved HyperLogLog cardinality estimation
based on method described in https://arxiv.org/abs/1702.01284
that does not rely on any magic constants
2018-03-10 20:13:21 +01:00
Otmar Ertl
6470b21f59 replaced tab by spaces 2018-03-10 20:09:41 +01:00
pan.liangp
f4eb64cd35 move get clients max buffer calculate into info clients command 2018-03-02 17:16:00 +08:00
antirez
84b281209a Stream: update the listpack pointer in streamTrimByLength(). 2018-03-01 17:26:02 +01:00
antirez
efcbc01fbd Remove warning from lpGet snprintf(). 2018-03-01 15:26:27 +01:00
antirez
d63caaa820 redis-cli: fix missed unit in array. Change define name. 2018-03-01 15:06:41 +01:00
charsyam
da7f5700cf refactoring-call-aeDeleteFileEvent-twice-in-freeClusterLink 2018-03-01 22:30:39 +09:00
charsyam
51a03f6356 fix dlopen leak 2018-03-01 21:22:42 +09:00
Salvatore Sanfilippo
83b5b5a476
Merge pull request #4714 from charsyam/feature/fix-out-of-index-range
[BugFix] Fix out of array index range for findBigKeys in redis-cli
2018-03-01 03:39:15 -08:00
antirez
3a5bf75ede Actually use ae_flags to add AE_BARRIER if needed.
Many thanks to @Plasma that spotted this problem reviewing the code.
2018-02-28 18:03:51 +01:00
Salvatore Sanfilippo
7a73db7512
Merge pull request #4715 from charsyam/feature/refactoring-make-condition-clear-for-rdb
[BugFix] fix calculation length in rdbSaveAuxField
2018-02-27 10:15:27 -08:00
antirez
92696e49d2 expireIfNeeded() needed a top comment documenting the behavior. 2018-02-27 16:44:43 +01:00
antirez
b00c4ffab5 expireIfNeeded() comment: claim -> pretend. 2018-02-27 16:37:37 +01:00
charsyam
76386c48b8 refactoring-make-condition-clear-for-rdb 2018-02-27 21:55:20 +09:00
charsyam
6168d5a1a6 fix-out-of-index-range-for-redis-cli-findbigkey 2018-02-27 21:46:19 +09:00
antirez
956350ef89 ae.c: insetad of not firing, on AE_BARRIER invert the sequence.
AE_BARRIER was implemented like:

    - Fire the readable event.
    - Do not fire the writabel event if the readable fired.

However this may lead to the writable event to never be called if the
readable event is always fired. There is an alterantive, we can just
invert the sequence of the calls in case AE_BARRIER is set. This commit
does that.
2018-02-27 13:06:42 +01:00
antirez
75987431f0 AOF: fix a bug that may prevent proper fsyncing when fsync=always.
In case the write handler is already installed, it could happen that we
serve the reply of a query in the same event loop cycle we received it,
preventing beforeSleep() from guaranteeing that we do the AOF fsync
before sending the reply to the client.

The AE_BARRIER mechanism, introduced in a previous commit, prevents this
problem. This commit makes actual use of this new feature to fix the
bug.
2018-02-27 13:06:42 +01:00
antirez
533d0e0375 Cluster: improve crash-recovery safety after failover auth vote.
Add AE_BARRIER to the writable event loop so that slaves requesting
votes can't be served before we re-enter the event loop in the next
iteration, so clusterBeforeSleep() will fsync to disk in time.
Also add the call to explicitly fsync, given that we modified the last
vote epoch variable.
2018-02-27 13:06:42 +01:00
antirez
548e478e40 ae.c: introduce the concept of read->write barrier.
AOF fsync=always, and certain Redis Cluster bus operations, require to
fsync data on disk before replying with an acknowledge.
In such case, in order to implement Group Commits, we want to be sure
that queries that are read in a given cycle of the event loop, are never
served to clients in the same event loop iteration. This way, by using
the event loop "before sleep" callback, we can fsync the information
just one time before returning into the event loop for the next cycle.
This is much more efficient compared to calling fsync() multiple times.

Unfortunately because of a bug, this was not always guaranteed: the
actual way the events are installed was the sole thing that could
control. Normally this problem is hard to trigger when AOF is enabled
with fsync=always, because we try to flush the output buffers to the
socekt directly in the beforeSleep() function of Redis. However if the
output buffers are full, we actually install a write event, and in such
a case, this bug could happen.

This change to ae.c modifies the event loop implementation to make this
concept explicit. Write events that are registered with:

    AE_WRITABLE|AE_BARRIER

Are guaranteed to never fire after the readable event was fired for the
same file descriptor. In this way we are sure that data is persisted to
disk before the client performing the operation receives an
acknowledged.

However note that this semantics does not provide all the guarantees
that one may believe are automatically provided. Take the example of the
blocking list operations in Redis.

With AOF and fsync=always we could have:

    Client A doing: BLPOP myqueue 0
    Client B doing: RPUSH myqueue a b c

In this scenario, Client A will get the "a" elements immediately after
the Client B RPUSH will be executed, even before the operation is persisted.
However when Client B will get the acknowledge, it can be sure that
"b,c" are already safe on disk inside the list.

What to note here is that it cannot be assumed that Client A receiving
the element is a guaranteed that the operation succeeded from the point
of view of Client B.

This is due to the fact that the barrier exists within the same socket,
and not between different sockets. However in the case above, the
element "a" was not going to be persisted regardless, so it is a pretty
synthetic argument.
2018-02-27 13:06:42 +01:00
Salvatore Sanfilippo
d8830200b4
Merge pull request #3828 from oranagra/sdsnewlen_pr
add SDS_NOINIT option to sdsnewlen to avoid unnecessary memsets.
2018-02-27 04:04:32 -08:00
antirez
813960dbdd Fix ziplist prevlen encoding description. See #4705. 2018-02-23 12:19:35 +01:00
gechunlin
d4e6d1086f
Update object.c 2018-02-22 20:57:54 -06:00
antirez
ffde73c57d Track number of logically expired keys still in memory.
This commit adds two new fields in the INFO output, stats section:

expired_stale_perc:0.34
expired_time_cap_reached_count:58

The first field is an estimate of the number of keys that are yet in
memory but are already logically expired. They reason why those keys are
yet not reclaimed is because the active expire cycle can't spend more
time on the process of reclaiming the keys, and at the same time nobody
is accessing such keys. However as the active expire cycle runs, while
it will eventually have to return to the caller, because of time limit
or because there are less than 25% of keys logically expired in each
given database, it collects the stats in order to populate this INFO
field.

Note that expired_stale_perc is a running average, where the current
sample accounts for 5% and the history for 95%, so you'll see it
changing smoothly over time.

The other field, expired_time_cap_reached_count, counts the number
of times the expire cycle had to stop, even if still it was finding a
sizeable number of keys yet to expire, because of the time limit.
This allows people handling operations to understand if the Redis
server, during mass-expiration events, is able to collect keys fast
enough usually. It is normal for this field to increment during mass
expires, but normally it should very rarely increment. When instead it
constantly increments, it means that the current workloads is using
a very important percentage of CPU time to expire keys.

This feature was created thanks to the hints of Rashmi Ramesh and
Bart Robinson from Twitter. In private email exchanges, they noted how
it was important to improve the observability of this parameter in the
Redis server. Actually in big deployments, the amount of keys that are
yet to expire in each server, even if they are logically expired, may
account for a very big amount of wasted memory.
2018-02-19 11:12:49 +01:00
antirez
aa57481d8c Remove non semantical spaces from module.c. 2018-02-15 21:41:03 +01:00
Salvatore Sanfilippo
7830f8492f
Merge pull request #4479 from dvirsky/notify
Keyspace notifications API for modules
2018-02-15 21:36:32 +01:00
antirez
f4dc736cca Fix typo in notifyKeyspaceEvent() comment. 2018-02-15 21:33:06 +01:00
Dvir Volk
0a36196ce4 Add doc comment about notification flags 2018-02-14 21:54:00 +02:00
Dvir Volk
10efdf307b Add REDISMODULE_NOTIFY_STREAM flag to support stream notifications 2018-02-14 21:50:42 +02:00
Dvir Volk
613831f820 Fix indentation and comment style in testmodule 2018-02-14 21:43:06 +02:00
Dvir Volk
f27a64232e Use one static client for all keyspace notification callbacks 2018-02-14 21:40:10 +02:00
Dvir Volk
3aab12414f Remove the NOTIFY_MODULE flag and simplify the module notification flow if there aren't subscribers 2018-02-14 21:40:10 +02:00
Dvir Volk
a8e2e99a88 Document flags for notifications 2018-02-14 21:38:58 +02:00
Dvir Volk
d4d753dae4 removed some trailing whitespaces 2018-02-14 21:38:58 +02:00
Dvir Volk
5b7b12e38f removed hellonotify.c 2018-02-14 21:38:58 +02:00
Dvir Volk
896db12b41 fixed test 2018-02-14 21:38:58 +02:00
Dvir Volk
2136035e47 finished implementation of notifications. Tests unfinished 2018-02-14 21:38:58 +02:00
charsyam
9d41436115 getting rid of duplicated code 2018-02-14 00:12:13 +09:00
antirez
ae29bcd8e2 More verbose logging when slave sends errors to master.
See #3832.
2018-02-13 16:01:31 +01:00
Salvatore Sanfilippo
756df19134
Merge pull request #3832 from oranagra/slave_reply_to_master_pr
when a slave responds with an error on commands that come from master, log it
2018-02-13 15:55:26 +01:00
Salvatore Sanfilippo
f9e6c2046f
Merge pull request #3745 from guybe7/unstable
enlarged buffer given to ld2string
2018-02-13 15:50:21 +01:00
antirez
c14ba46e3a Make it explicit with a comment why we kill the old AOF rewrite.
See #3858.
2018-02-13 15:43:34 +01:00
Guy Benoish
f782006782 rewriteAppendOnlyFileBackground() failure fix
It is possible to do BGREWRITEAOF even if appendonly=no. This is by design.
stopAppendonly() didn't turn off aof_rewrite_scheduled (it can be turned on
again by BGREWRITEAOF even while appendonly is off anyway).
After configuring `appendonly yes` it will see that the state is AOF_OFF,
there's no RDB fork, so it will do rewriteAppendOnlyFileBackground() which
will fail since the aof_child_pid is set (was scheduled and started by cron).

Solution:
stopAppendonly() will turn off the schedule flag (regardless of who asked for it).
startAppendonly() will terminate any existing fork and start a new one (so it is the most recent).
2018-02-13 15:41:06 +01:00
Oran Agra
8e8d957ff8 fix to latency monitor reporting wrong max latency
in some cases LATENCY HISTORY reported latency that was
higher than the max latency reported by LATENCY LATEST / DOCTOR
2018-02-13 15:58:40 +02:00
赵磊
aacecbc997 Remove updateLFU() in dbOverwrite(). 2018-02-11 21:02:07 +08:00
antirez
32ac4c64ba Rax updated to latest antirez/rax commit. 2018-02-02 11:10:18 +01:00
zhaozhao.zz
968cb26693 config: handle special configuration "" for auth 2018-01-26 22:49:39 +08:00
Salvatore Sanfilippo
4aa2ecd98b
Merge pull request #4269 from jianqingdu/unstable
fix not call va_end() when syncWrite() failed
2018-01-24 10:55:25 +01:00
Mark Nunberg
062bd733da
redismodule.h: Check ModuleNameBusy before calling it
Older versions might not have this function.
2018-01-23 10:49:18 -05:00
antirez
727dd43614 Fix migrateCommand() access of not initialized byte. 2018-01-18 12:41:05 +01:00
Guy Benoish
fd8efb7c36 Replication buffer fills up on high rate traffic.
When feeding the master with a high rate traffic the the slave's feed is much slower.
This causes the replication buffer to grow (indefinitely) which leads to slave disconnection.
The problem is that writeToClient() decides to stop writing after NET_MAX_WRITES_PER_EVENT
writes (In order to be fair to clients).
We should ignore this when the client is a slave.
It's better if clients wait longer, the alternative is that the slave has no chance to stay in
sync in this situation.
2018-01-18 12:10:48 +01:00
antirez
1673a3f32c Cluster: improve anti-affinity algo in redis-trib.rb.
See #3462 and related PRs.

We use a simple algorithm to calculate the level of affinity violation,
and then an optimizer that performs random swaps until things improve.
2018-01-18 11:44:19 +01:00
antirez
e1e0bbe04d Remove useless comment from serverCron().
The behavior is well specified by the code itself.
2018-01-17 11:23:41 +01:00
Salvatore Sanfilippo
a18e4c964e
Merge pull request #4546 from hqin6/unstable
fixbug for #4545 dead loop aof rewrite
2018-01-17 11:21:55 +01:00
heqin
3d3faa0a19 fixbug for #4545 dead loop aof rewrite 2018-01-17 18:08:30 +08:00
Salvatore Sanfilippo
81401878de
Merge pull request #4609 from Qinch/unstable
fix assert problem in ZIP_DECODE_PREVLENSIZE macro
2018-01-17 10:45:11 +01:00
antirez
b23927b240 Hopefully more clear comment to explain the change in #4607. 2018-01-16 15:52:13 +01:00
qinchao
1e0e168570 fix assert problem in ZIP_DECODE_PREVLENSIZE
, see issue: https://github.com/antirez/redis/issues/4587
2018-01-16 22:43:06 +08:00
Oran Agra
689b64c3ad PSYNC2 fix - promoted slave should hold on to it's backlog
after a slave is promoted (assuming it has no slaves
and it booted over an hour ago), it will lose it's replication
backlog at the next replication cron, rather than waiting for slaves
to connect to it.
so on a simple master/slave faiover, if the new slave doesn't connect
immediately, it may be too later and PSYNC2 will fail.
2018-01-16 10:10:42 +02:00
zhaozhao.zz
1b8eec3e53 aof: format code and comment 2018-01-15 13:01:03 +01:00
antirez
c45366be0a Put more details in the comment introduced by #4601. 2018-01-15 12:50:08 +01:00
Salvatore Sanfilippo
1ed5ac7ce5
Merge pull request #4601 from soloestoy/fix-memoryleak-for-lazy-server-del
lazyfree: fix memory leak for lazyfree-lazy-server-del
2018-01-15 12:43:55 +01:00
zhaozhao.zz
0517ab8397 lazyfree: fix memory leak for lazyfree-lazy-server-del 2018-01-15 00:45:37 +08:00
Salvatore Sanfilippo
aeeb747796
Merge pull request #4575 from soloestoy/bugfix-benchmark
redis-benchmark: bugfix - handle zero liveclients in right way
2018-01-12 17:43:01 +01:00
Salvatore Sanfilippo
72187fa8a5
Merge pull request #4581 from dvirsky/module_unlink
Added RM_UnlinkKey - a low level analog to UNLINK command
2018-01-12 17:41:09 +01:00
antirez
a5b6bc2bd7 Merge branch 'unstable' of github.com:/antirez/redis into unstable 2018-01-12 17:16:38 +01:00
Salvatore Sanfilippo
71914387ba
Merge pull request #4586 from gnuhpc/fix-crashlog-typo
Fix a typo(maybe instruction?) in crash log
2018-01-12 17:16:12 +01:00
antirez
2f8476df91 Fix getKeysUsingCommandTable() in the case of nagative arity.
This fixes a crash with Redis Cluster when OBJECT is mis-used, because
getKeysUsingCommandTable() will call serverPanic() detecting we are
accessing an invalid argument in the case "OBJECT foo" is called.

This bug was introduced when OBJECT HELP was introduced, because the key
argument is set fixed at index 2 in the command table, however now
OBJECT may be called with an insufficient number of arguments to extract
the key.

The "Right Thing" would be to have a specific function to extract keys
from the OBJECT command, however this is kinda of an overkill, so I
preferred to make getKeysUsingCommandTable() more robust and just return
no keys when it's not possible to honor the command table, because new
commands are often added and also there are a number with an HELP
subcommand violating the normal form, and crashing for this trivial
reason or having many command-specific key extraction functions is not
great.
2018-01-12 11:26:29 +01:00
antirez
13650446ac proto-max-querybuf-len -> client-query-buffer-limit.
We already had client buffer limits exported as configuration options.
Stick with the naming scheme already used.

See #4568.
2018-01-11 11:36:26 +01:00
antirez
8075572207 New config options about protocol prefixed with "proto".
Related to #4568.
2018-01-11 11:27:41 +01:00
Salvatore Sanfilippo
c2fa4b8a60
Merge pull request #4568 from oranagra/restore_size_limit
fix RESTORE command size limits
2018-01-11 11:20:02 +01:00
huijing.whj
f0e09bf803 fix int overflow problem in freeMemoryIfNeeded 2018-01-10 23:47:02 +08:00
Salvatore Sanfilippo
e954742fbd
Merge pull request #3356 from yusaku/fix-module-firstkey
Fix the firstkey, lastkey, and keystep of moduleCommand
2018-01-09 18:58:31 +01:00
antirez
3ce1c28d47 Rewrite MIGRATE AUTH option.
See PR #2507. This is a reimplementation of the fix that contained
different problems.
2018-01-09 18:48:26 +01:00
gnuhpc
36e71b88b5 Fix a typo(maybe instruction?) in crash log 2018-01-09 16:13:02 +08:00
Dvir Volk
e76dfc90a7 Added RM_UnlinkKey - a low level analog to UNLINK command 2018-01-07 16:41:43 +02:00
gnuhpc
5113a53df6 Fix memory usage list bug 2018-01-05 12:16:24 +08:00
zhaozhao.zz
c0f611f12a redis-benchmark: bugfix - handle zero liveclients in right way 2018-01-03 12:47:25 +08:00
Oran Agra
b509a14c3e Add config options for max-bulk-len and max-querybuf-len mainly to support RESTORE of large keys 2017-12-29 12:43:48 +02:00
Oran Agra
60a4f12f8b fix processing of large bulks (above 2GB)
- protocol parsing (processMultibulkBuffer) was limitted to 32big positions in the buffer
  readQueryFromClient potential overflow
- rioWriteBulkCount used int, although rioWriteBulkString gave it size_t
- several places in sds.c that used int for string length or index.
- bugfix in RM_SaveAuxField (return was 1 or -1 and not length)
- RM_SaveStringBuffer was limitted to 32bit length
2017-12-29 12:24:19 +02:00
antirez
0b561883b4 Hyperloglog: refresh hdr variable correctly.
This is a fix for the #3819 improvements. The o->ptr may change because
of hllSparseSet() calls, so 'hdr' must be correctly re-fetched.
2017-12-22 11:26:31 +01:00
antirez
27b81f3fbc Hyperloglog: Support for PFMERGE sparse encoding as target.
This is a fix for #3819.
2017-12-22 11:01:27 +01:00
antirez
cbe9d725a7 Hyperloglog: refactoring of sparse/dense add function.
The commit splits the add functions into a set() and add() set of
functions, so that it's possible to set registers in an independent way
just having the index and count.

Related to #3819, otherwise a fix is not possible.
2017-12-22 11:00:22 +01:00
heqin
365e6c45ea fixbug for #4545 dead loop aof rewrite 2017-12-18 17:59:03 +08:00
Salvatore Sanfilippo
238c9bd086
Merge pull request #4540 from hqin6/unstable
fixbug for #4538 Error opening /setting AOF rewrite IPC pipes: No suc…
2017-12-15 12:41:02 +01:00
heqin
ee724cac49 fixbug for #4538 Error opening /setting AOF rewrite IPC pipes: No such file or directory 2017-12-15 11:06:58 +08:00
Salvatore Sanfilippo
d745e8dfb3
Merge pull request #4536 from tporadowski/antirez_unstable-cli-history
Always enable command history in redis-cli run on a TTY
2017-12-14 22:14:45 +01:00
Salvatore Sanfilippo
398b2084af
Merge pull request #4528 from nashe/patch-oob-read-4527
Prevent off-by-one read in stringmatchlen()
2017-12-14 12:21:32 +01:00
antirez
12e65a424c safe_write -> aofWrite. Function commented.
Related to #4498.
2017-12-14 12:19:36 +01:00
Salvatore Sanfilippo
5182dcd645
Merge pull request #4498 from soloestoy/aof-safe-write
Aof safe write -- fix the short write
2017-12-14 12:15:46 +01:00
Tomasz Poradowski
77653ebe8a always enable command history in redis-cli
- when redis-cli is running in a TTY - always enable command history
buffering, regardless if history file path can be successfully
determined
2017-12-13 23:49:41 +01:00
antirez
de276b6a43 Cluster: allow read-only EVAL/EVALSHA in slaves.
Fix #3665.
2017-12-13 13:36:01 +01:00
nashe
f43eb5adcf Prevent off-by-one read in stringmatchlen() (fixes #4527) 2017-12-12 01:25:03 +01:00
zhaozhao.zz
109ee497be zset: change the span of zskiplistNode to unsigned long 2017-12-08 16:09:27 +08:00
zhaozhao.zz
e8901b2fe4 zset: fix the int problem 2017-12-08 15:37:08 +08:00
antirez
522760fac7 Change indentation and other minor details of PR #4489.
The main change introduced by this commit is pretending that help
arrays are more text than code, thus indenting them at level 0. This
improves readability, and is an old practice when defining arrays of
C strings describing text.

Additionally a few useless return statements are removed, and the HELP
subcommand capitalized when printed to the user.
2017-12-06 12:05:14 +01:00
Itamar Haber
482d678e95 C style 2017-12-05 19:09:19 +02:00
Itamar Haber
b23c8babed Uses an offset in addReplyHelp 2017-12-05 18:17:14 +02:00
Itamar Haber
8b51121998 Merge remote-tracking branch 'upstream/unstable' into help_subcommands 2017-12-05 18:14:59 +02:00
antirez
62a4b817c6 add linkClient(): adds the client and caches the list node.
We have this operation in two places: when caching the master and
when linking a new client after the client creation. By having an API
for this we avoid incurring in errors when modifying one of the two
places forgetting the other. The function is also a good place where to
document why we cache the linked list node.

Related to #4497 and #4210.
2017-12-05 16:02:03 +01:00
Salvatore Sanfilippo
03cfc8bf3a
Merge pull request #4497 from soloestoy/optimize-unlink-client
networking: optimize unlinkClient() in freeClient()
2017-12-05 15:51:15 +01:00
Salvatore Sanfilippo
c56fbb246c
Merge pull request #4508 from trevor211/fixNotes
fix some notes
2017-12-05 15:47:05 +01:00
antirez
479ea57797 Merge branch 'unstable' of github.com:/antirez/redis into unstable 2017-12-05 15:38:21 +01:00
zhaozhao.zz
7c6ddbc37d dict: fix the int problem for defrag 2017-12-05 15:38:03 +01:00
zhaozhao.zz
d1176b582c dict: fix the int problem 2017-12-05 15:37:59 +01:00
Salvatore Sanfilippo
6830fa25e3
Merge pull request #4488 from itamarhaber/debug_arity
Standardizes arity handling of DEBUG
2017-12-05 15:29:42 +01:00
zhaozhao.zz
42387d6c1a set: fix the int problem for qsort 2017-12-05 17:42:19 +08:00
zhaozhao.zz
de809666f8 set: fix the int problem for SPOP & SRANDMEMBER 2017-12-05 17:19:19 +08:00
WuYunlong
3f232ebfb8 fix some notes 2017-12-05 14:41:16 +08:00
zhaozhao.zz
b9491b65d9 quicklist: change the len of quicklist to unsigned long 2017-12-04 19:49:40 +08:00
zhaozhao.zz
664bbfe760 quicklist: fix the return value of quicklistCount 2017-12-04 19:49:23 +08:00
antirez
2869284e44 Streams: fix a few type mismatches in t_stream.c. 2017-12-04 11:59:15 +01:00
antirez
60d26acfc8 Refactoring: improve luaCreateFunction() API.
The function in its initial form, and after the fixes for the PSYNC2
bugs, required code duplication in multiple spots. This commit modifies
it in order to always compute the script name independently, and to
return the SDS of the SHA of the body: this way it can be used in all
the places, including for SCRIPT LOAD, without duplicating the code to
create the Lua function name. Note that this requires to re-compute the
body SHA1 in the case of EVAL seeing a script for the first time, but
this should not change scripting performance in any way because new
scripts definition is a rare event happening the first time a script is
seen, and the SHA1 computation is anyway not a very slow process against
the typical Redis script and compared to the actua Lua byte compiling of
the body.

Note that the function used to assert() if a duplicated script was
loaded, however actually now two times over three, we want the function
to handle duplicated scripts just fine: this happens in SCRIPT LOAD and
in RDB AUX "lua" loading. Moreover the assert was not defending against
some obvious failure mode, so now the function always tests against
already defined functions at start.
2017-12-04 11:25:20 +01:00
antirez
c6eca690ee Remove useless variable check from luaCreateFunction().
The block is already inside if (allow_dup).
2017-12-04 10:55:54 +01:00
antirez
68681f2bcf Fix issue #4505, Lua RDB AUX field loading of existing scripts.
Unfortunately, as outlined by @soloestoy in #4505, "lua" AUX RDB field
loading in case of duplicated script was still broken. This commit fixes
this problem and also a memory leak introduced by the past commit.

Note that now we have a regression test able to duplicate the issue, so
this commit was actually tested against the regression. The original PR
also had a valid fix, but I prefer to hide the details of scripting.c
outside scripting.c, and later "SCRIPT LOAD" should also be able to use
the function luaCreateFunction() instead of redoing the work.
2017-12-04 10:33:04 +01:00
antirez
6a1bf07a46 DEBUG change-repl-id implemented.
With PSYNC2 to force a full SYNC in tests is hard. With this new DEBUG
subcommand we just need to call it and then CLIENT KILL TYPE master in
the slave.
2017-12-04 10:24:52 +01:00
Itamar Haber
51eb6cb395 Adds help to CONFIG 2017-12-03 19:34:31 +02:00
Itamar Haber
bd5af03dbd Adds help to CLUSTER command 2017-12-03 19:05:10 +02:00
Itamar Haber
ee3884e63c Improve slowlog help 2017-12-03 17:39:52 +02:00
Itamar Haber
d884ba4bc9 Helps CLIENT 2017-12-03 16:49:29 +02:00
antirez
65a9740fa8 Fix loading of RDB files lua AUX fields when the script is defined.
In the case of slaves loading the RDB from master, or in other similar
cases, the script is already defined, and the function registering the
script should not fail in the assert() call.
2017-12-01 16:01:10 +01:00
antirez
8ac76be5f2 Streams: DEBUG DIGEST support. 2017-12-01 15:04:05 +01:00
antirez
f42df6f43a Streams: add code to compute the stream memory usage.
It's a bit of black magic without actually tracking it inside rax.c,
however Redis usage of the radix tree for the stream data structure is
quite consistent, so a few magic constants apparently are producing
results that make sense.
2017-12-01 12:50:27 +01:00
antirez
115d076d65 Streams: fix lp-count field for non-same-fields entries. 2017-12-01 10:24:25 +01:00
antirez
9bb18e5438 Streams: XRANGE REV option -> XREVRANGE command. 2017-12-01 10:24:25 +01:00
antirez
9dc79c039a Streams: fix reverse iterator discarding of items out of range. 2017-12-01 10:24:25 +01:00
antirez
6919280cc5 Streams: fix reverse iteration next node jumping. 2017-12-01 10:24:25 +01:00
antirez
ee3490ec48 Streams: state machine for reverse iteration WIP 1. 2017-12-01 10:24:25 +01:00
antirez
3c5d773f82 Streams: augment stream entries to allow backward scanning. 2017-12-01 10:24:25 +01:00
antirez
0381931b4c Streams: Update listpack to fix 32bit strings encoding error.
Note that streams produced by XADD in previous broken versions having
elements with 4096 bytes or more will be permanently broken and must be
created again from scratch.

Fix #4428
Fix #4349
2017-12-01 10:24:24 +01:00
antirez
020fe26bd6 Streams: fix COUNT parsing, issue #4433. 2017-12-01 10:24:24 +01:00
antirez
abab0b7817 Streams: fix redis-cli to understand the stream type. 2017-12-01 10:24:24 +01:00
antirez
671b1f6a9d Streams: fix TYPE for stream type. 2017-12-01 10:24:24 +01:00
antirez
5082ec6419 Streams: move ID ms/seq separator from '.' to '-'
After checking with the community via Twitter (here:
https://twitter.com/antirez/status/915130876861788161) the verdict was to
use ":". However I later realized, after users lamented the fact that
it's hard to copy IDs just with double click, that this was the reason
why I moved to "." in the first instance. Fortunately "-", that was the
other option with most votes, also gets selected with double click on
most terminal applications on Linux and MacOS.

So my reasoning was:

1) We can't retain "." because it's actually confusing to newcomers, it
looks like a floating number, people may be tricked into thinking they
can order IDs numerically as floats.

2) Moving to a double-click-to-select format is much better. People will
work with such IDs for long time when coding / debugging. Why making now
a choice that will impact this for the next years?

The only other viable option was "-", and that's what I did. Thanks.
2017-12-01 10:24:24 +01:00
antirez
50595a5889 Streams: fix XADD + MAXLEN propagation due to var shadowing.
Clang should be more prone to return warnings by default when there is
same-var-name shadowing. GCC does this and can avoid bugs like that.
2017-12-01 10:24:24 +01:00
antirez
a4e6aae6b8 Streams: fix memory leak in streamTrimByLength(). 2017-12-01 10:24:24 +01:00
antirez
0248a6b125 Streams: fix streamTrimByLength() standalone items skipping. 2017-12-01 10:24:24 +01:00
antirez
0540803288 Streams: XADD MAXLEN implementation.
The core of this change is the implementation of stream trimming, and
the resulting MAXLEN option of XADD as a trivial result of having
trimming functionalities. MAXLEN already works but in order to be more
efficient listpack GC should be implemented, currently marked as a TODO
item inside the comments.
2017-12-01 10:24:24 +01:00
antirez
0c00fd7834 Streams: reduce listpack max size to 2k to speedup range queries.
Listpack max size is a tradeoff between space and time. A 2k max entry
puts the memory usage approximately at a similar order of magnitude (5
million entries went from 96 to 120 MB), but the range queries speed
doubled (because there are half entries to scan in the average case).

Lower values could be considered, or maybe this parameter should be
made tunable.
2017-12-01 10:24:24 +01:00
antirez
f24d3a7de0 Streams: delta encode IDs based on key. Add count + deleted fields.
We used to have the master ID stored at the start of the listpack,
however using the key directly makes more sense in order to create a
space efficient representation: anyway the key at the radix tree is very
unlikely to change because of how the stream is implemented. Moreover on
nodes merging, to rewrite the merged listpacks is anyway the most
sensible operation, and we can use the iterator and the append-to-stream
function in order to avoid re-implementing the code needed for merging.

This commit also adds two items at the start of the listpack: the
number of valid items inside the listpack, and the number of items
marked as deleted. This means that there is no need to scan a listpack
in order to understand if it's a good candidate for garbage collection,
if the ration between valid/deleted items triggers the GC.
2017-12-01 10:24:24 +01:00
antirez
cea421a021 Streams: specify better how the master enty works. 2017-12-01 10:24:24 +01:00
antirez
3f2d7e277e Streams: items compression implemented.
The approach used is to set a fixed header at the start of every
listpack blob (that contains many entries). The header contains a
"master" ID and fields, that are initially just obtained from the first
entry inserted in the listpack, so that the first enty is always well
compressed. Later every new entry is checked against these fields, and
if it matches, the SAMEFIELD flag is set in the entry so that we know to
just use the master entry flags. The IDs are always delta-encoded
against the first entry. This approach avoids cascading effects in which
entries are encoded depending on the previous entries, in order to avoid
complexity and rewritings of the data when data is removed in the middle
(which is a planned feature).
2017-12-01 10:24:24 +01:00
antirez
8f00cf85a7 Streams: fixed memory leaks when blocking again for same stream.
blockForKeys() was not freeing the allocation holding the ID when the
key was already found busy. Fortunately the unit test checked explicitly
for blocking multiple times for the same key (copying a regression in
the blocking lists tests), so the bug was detected by the Redis test leak
checker.
2017-12-01 10:24:24 +01:00
antirez
26d4f8e3ec Streams: AOF rewriting + minor iterator improvements. 2017-12-01 10:24:24 +01:00
antirez
01ea018c40 Streams: export iteration API. 2017-12-01 10:24:24 +01:00
antirez
9ed40f0fc3 Streams: implement streamReplyWithRange() in terms of the iterator. 2017-12-01 10:24:24 +01:00
antirez
a58733cacf Streams: stream iteration refactoring, WIP 2. 2017-12-01 10:24:24 +01:00
antirez
b1ec333633 Streams: stream iteration refactoring, WIP 1. 2017-12-01 10:24:24 +01:00
antirez
1a603e1a87 Streams: fix bug in XREAD last received ID processing. 2017-12-01 10:24:24 +01:00
antirez
94af55c5ea Streams: fix memory leak in freeStream(). 2017-12-01 10:24:24 +01:00
antirez
3a0b78bc52 Streams: rewrite XADD ID argument for AOF/slaves. 2017-12-01 10:24:24 +01:00
antirez
19b06935d5 Streams: fix XADD API and keyspace notifications.
XADD was suboptimal in the first incarnation of the command, not being
able to accept an ID (very useufl for replication), nor options for
having capped streams.

The keyspace notification for streams was not implemented.
2017-12-01 10:24:24 +01:00
antirez
db89f7474d Streams: When XREAD blocks without COUNT, set a default one.
A client may lose a lot of time between invocations of blocking XREAD,
for example because it is processing the messages or for any other
cause. When it returns back, it may provide a low enough message ID that
the server will block to send an unreasonable number of messages in a
single call. For this reason we set a COUNT when the client is blocked
with XREAD calls, even if no COUNT is given. This is arbitrarily set to
1000 because it's enough to avoid slowing down the reception of many
messages, but low enough to avoid to block.
2017-12-01 10:24:24 +01:00
antirez
c128190026 Streams: fix handleClientsBlockedOnKeys() access to invalid ID. 2017-12-01 10:24:24 +01:00
antirez
6468cb2e82 Streams: fix XREAD ready-key signaling.
With lists we need to signal only on key creation, but streams can
provide data to clients listening at every new item added.
To make this slightly more efficient we now track different classes of
blocked clients to avoid signaling keys when there is nobody listening.
A typical case is when the stream is used as a time series DB and
accessed only by range with XRANGE.
2017-12-01 10:24:24 +01:00
antirez
b5be5093fe Streams: fix XREAD timeout handling, zero is valid. 2017-12-01 10:24:24 +01:00
antirez
2cacdcd6f8 Streams: XREAD related code to serve blocked clients. 2017-12-01 10:24:24 +01:00
antirez
0adb43b68f Streams: XREAD ability to block fixed. 2017-12-01 10:24:24 +01:00
antirez
6a1c92d52d Streams: synchronous xread fixes and improvements. 2017-12-01 10:24:24 +01:00
antirez
a7d898334a Streams: XREAD get-key method fixed. 2017-12-01 10:24:24 +01:00
antirez
110041825c Streams: XREAD get-keys method. 2017-12-01 10:24:24 +01:00
antirez
fa61720d30 Streams: XREAD, first draft. Handling of blocked clients still missing. 2017-12-01 10:24:24 +01:00
antirez
e65b4825f0 Streams: XREAD arguments parsing. 2017-12-01 10:24:24 +01:00
antirez
4086dff477 Streams: augment client.bpop with XREAD specific fields. 2017-12-01 10:24:24 +01:00
antirez
f80dfbf464 Streams: more internal preparation for blocking XREAD. 2017-12-01 10:24:24 +01:00
antirez
4a377cecd8 Streams: initial work to use blocking lists logic for streams XREAD. 2017-12-01 10:24:24 +01:00
antirez
439120c620 Streams: implement stream object release. 2017-12-01 10:24:24 +01:00
antirez
ec9bbe96bf Streams: XLEN command. 2017-12-01 10:24:24 +01:00
antirez
98d184db12 Streams: Save stream->length in RDB. 2017-12-01 10:24:24 +01:00
antirez
cd18f06e9c Streams: change listpack allocator to zmalloc. 2017-12-01 10:24:24 +01:00
antirez
edd70c1993 Streams: RDB loading. RDB saving modified.
After a few attempts it looked quite saner to just add the last item ID
at the end of the serialized listpacks, instead of scanning the last
listpack loaded from head to tail just to fetch it. It's a disk space VS
CPU-and-simplicity tradeoff basically.
2017-12-01 10:24:24 +01:00
antirez
485014cc74 Streams: RDB saving. 2017-12-01 10:24:24 +01:00
antirez
100d43c1ac Streams: assign value of 6 to OBJ_STREAM + some refactoring. 2017-12-01 10:24:24 +01:00
antirez
79866a6361 Streams: 12 commits squashed into the initial Streams implementation. 2017-12-01 10:24:24 +01:00
antirez
045d65c3af PSYNC2: Fix off by one buffer size in luaCreateFunction(). 2017-11-30 18:38:29 +01:00
antirez
452ad2e928 PSYNC2: just store script bodies into RDB.
Related to #4483. As suggested by @soloestoy, we can retrieve the SHA1
from the body. Given that in the new implementation using AUX fields we
ended copying around a lot to create new objects and strings, extremize
such concept and trade CPU for space inside the RDB file.
2017-11-30 18:38:26 +01:00
antirez
28dfdca733 PSYNC2: luaCreateFunction() should handle NULL client parameter.
See #4483. This is needed because luaCreateFunction() is now called
from RDB loading code outside a client context.
2017-11-30 18:37:52 +01:00
antirez
f11a7585a8 PSYNC2: Save Lua scripts state into RDB file.
This is currently needed in order to fix #4483, but this can be
useful in other contexts, so maybe later we may want to remove the
conditionals and always save/load scripts.

Note that we are using the "lua" AUX field here, in order to guarantee
backward compatibility of the RDB file. The unknown AUX fields must be
discarded by past versions of Redis.
2017-11-30 18:37:52 +01:00
antirez
3b9be93fda Prevent corruption of server.executable after DEBUG RESTART.
Doing the following ended with a broken server.executable:

1. Start Redis with src/redis-server
2. Send CONFIG SET DIR /tmp/
3. Send DEBUG RESTART

At this point we called execve with an argv[0] that is no longer related
to the new path. So after the restart the absolute path of the
executable is recomputed in the wrong way. With this fix we pass the
absolute path already computed as argv[0].
2017-11-30 18:30:06 +01:00
antirez
d8f8701032 Be more verbose when DEBUG RESTART fails. 2017-11-30 18:08:21 +01:00
zhaozhao.zz
43be967690 networking: optimize unlinkClient() in freeClient() 2017-11-30 18:11:05 +08:00
zhaozhao.zz
1b5f56d042 aof: cast sdslen to ssize_t 2017-11-30 10:27:12 +08:00
zhaozhao.zz
2d73cf2367 aof: fix the short write 2017-11-30 10:22:12 +08:00
Itamar Haber
0752a834f9 Check arity in SLOWLOG before accessing arg 2017-11-30 00:30:30 +02:00
antirez
2785d6caa0 Merge branch 'lfu-fixes' into unstable 2017-11-29 17:16:13 +01:00
Itamar Haber
59d52f7fab Standardizes the 'help' subcommand
This adds a new `addReplyHelp` helper that's used by commands
when returning a help text. The following commands have been
touched: DEBUG, OBJECT, COMMAND, PUBSUB, SCRIPT and SLOWLOG.

WIP

Fix entry command table entry for OBJECT for HELP option.

After #4472 the command may have just 2 arguments.

Improve OBJECT HELP descriptions.

See #4472.

WIP 2

WIP 3
2017-11-28 21:15:45 +02:00
Salvatore Sanfilippo
565e139a56
Merge pull request #4200 from jeesyn/fix_typo
fix a typo
2017-11-28 18:44:11 +01:00
Salvatore Sanfilippo
923502a70b
Merge pull request #4166 from charpty/wip-redisclic-typo
redis-cli.c typo: helpe -> helper.
2017-11-28 18:41:51 +01:00
Salvatore Sanfilippo
26826329f5
Merge pull request #4167 from charpty/wip-redisclic-typo2
redis-cli.c typo: Requets -> Requests.
2017-11-28 18:41:28 +01:00
Salvatore Sanfilippo
3508b9c440
Merge pull request #4170 from TehWebby/patch-2
Fix typo
2017-11-28 18:40:43 +01:00
antirez
851e9fc48b t_hash.c: clarify calling two times the same function. 2017-11-28 18:39:00 +01:00
antirez
c44732ac58 adlist: fix listJoin() in the case the second list is empty.
See #4192, the original PR removed lines of code that are actually
needed, so thanks to @chunqiulfq for reporting the problem, but merging
solution from @jeesyn after checking, together with @artix75, that the
logic covers all the cases.
2017-11-28 18:25:14 +01:00
Salvatore Sanfilippo
a13106e001
Merge pull request #4374 from rouzier/patch-1
Fix file descriptor leak and error handling
2017-11-28 17:33:23 +01:00
Salvatore Sanfilippo
bf71b120f1
Merge pull request #4451 from devnexen/minor_build_fixes
Fix undefined behavior constant defined.
2017-11-28 17:23:48 +01:00
Itamar Haber
8c7f90e91e Standardizes arity handling of DEBUG 2017-11-28 18:18:45 +02:00
antirez
06ca9d6839 LFU: Fix LFUDecrAndReturn() to just decrement.
Splitting the popularity in half actually just needs decrementing the
counter because the counter is logarithmic.
2017-11-28 12:18:30 +01:00
zhaozhao.zz
9f131c9a89 LFU: add hotkeys option to redis-cli 2017-11-27 18:39:29 +01:00
zhaozhao.zz
583c314725 LFU: do some changes about LFU to find hotkeys
Firstly, use access time to replace the decreas time of LFU.
For function LFUDecrAndReturn,
it should only try to get decremented counter,
not update LFU fields, we will update it in an explicit way.
And we will times halve the counter according to the times of
elapsed time than server.lfu_decay_time.
Everytime a key is accessed, we should update the LFU
including update access time, and increment the counter after
call function LFUDecrAndReturn.
If a key is overwritten, the LFU should be also updated.
Then we can use `OBJECT freq` command to get a key's frequence,
and LFUDecrAndReturn should be called in `OBJECT freq` command
in case of the key has not been accessed for a long time,
because we update the access time only when the key is read or
overwritten.
2017-11-27 18:39:22 +01:00
zhaozhao.zz
53cea97204 LFU: change lfu* parameters to int 2017-11-27 18:38:55 +01:00
zhaozhao.zz
dfc42ec447 LFU: fix the missing of config get and rewrite 2017-11-27 18:38:33 +01:00
antirez
75fa7879e6 Improve OBJECT HELP descriptions.
See #4472.
2017-11-27 18:09:08 +01:00
antirez
b412c544fd Fix entry command table entry for OBJECT for HELP option.
After #4472 the command may have just 2 arguments.
2017-11-27 13:16:07 +01:00
Salvatore Sanfilippo
29252391c4
Merge pull request #4472 from itamarhaber/object_patch
A minor fix and `help` subcommand for `OBJECT`
2017-11-27 12:41:02 +01:00