Commit Graph

535 Commits

Author SHA1 Message Date
antirez
ce03d68332 Rename var to fixed_time_expire now that is more general. 2019-11-19 11:28:04 +01:00
antirez
84b01f63db Expire cycle: introduce configurable effort. 2019-11-18 11:30:05 +01:00
antirez
2f6fe5ce3a Expire cycle: introduce the new state needed for the new algo. 2019-11-14 18:28:01 +01:00
Oran Agra
0bc3dab095 Adjustments for active defrag defaults and tuning
Reduce default minimum effort, so that when fragmentation is just detected,
the impact on the latency will be minor.

Reduce the default maximum effort, mainly to prevent a case were a sudden
massive deletions, won't trigger an aggressive defrag that will cause latency.

When activedefrag is disabled mid-run, reset the 'running' info field, and
clear the scan cursor, so that when it'll be re-enabled, a new fresh scan will
start.

Clearing the 'running' variable is important since lowering the defragger
tunables mid-scan won't help, the defragger only considers new threshold when
a new scan starts, and during a scan it can only become more aggressive,
(when more severe fragmentation is detected), it'll never go less aggressive.
So by temporarily disabling activedefrag, one can lower th the tunables.

Removing the experimantal warning.
2019-11-12 15:05:29 +02:00
David Carlier
960c25ac0a Add missing header 2019-11-11 13:06:08 +00:00
Oran Agra
28c20b4ef9 rename RN_SetLRUOrLFU -> RM_SetLRU and RN_SetLFU
- the API name was odd, separated to two apis one for LRU and one for LFU
- the LRU idle time was in 1 second resolution, which might be ok for RDB
  and RESTORE, but i think modules may need higher resolution
- adding tests for LFU and for handling maxmemory policy mismatch
2019-11-10 09:27:01 +02:00
antirez
0f026af185 Merge branch 'unstable' of github.com:/antirez/redis into unstable 2019-11-06 17:44:42 +01:00
antirez
8b2c0f9044 Update PR #6537: use a fresh time outside call().
One problem with the solution proposed so far in #6537 is that key
lookups outside a command execution via call(), still used a cached
time. The cached time needed to be refreshed in multiple places,
especially because of modules callbacks from timers, cluster bus, and
thread safe contexts, that may use RM_Open().

In order to avoid this problem, this commit introduces the ability to
detect if we are inside call(): this way we can use the reference fixed
time only when we are in the context of a command execution or Lua
script, but for the asynchronous lookups, we can still use mstime() to
get a fresh time reference.
2019-11-06 09:57:29 +01:00
antirez
824f5f0b7a Update PR #6537 patch to for generality.
After the thread in #6537 and thanks to the suggestions received, this
commit updates the original patch in order to:

1. Solve the problem of updating the time in multiple places by updating
it in call().
2. Avoid introducing a new field but use our cached time.

This required some minor refactoring to the function updating the time,
and the introduction of a new cached time in microseconds in order to
use less gettimeofday() calls.
2019-11-05 10:14:34 +01:00
zhaozhao.zz
e542132b07 expires: refactoring judgment about whether a key is expired
Calling lookupKey*() many times to search a key in one command
may get different result.

That's because lookupKey*() calls expireIfNeeded(), and delete
the key when reach the expire time. So we can get an robj before
the expire time, but a NULL after the expire time.

The worst is that may lead to Redis crash, for example
`RPOPLPUSH foo foo` the first time we get a list form `foo` and
hold the pointer, but when we get `foo` again it's expired and
deleted. Now we hold a freed memory, when execute rpoplpushHandlePush()
redis crash.

To fix it, we can refactor the judgment about whether a key is expired,
using the same basetime `server.cmd_start_mstime` instead of calling
mstime() everytime.
2019-11-05 09:56:39 +01:00
Salvatore Sanfilippo
18b62a51e2
Merge branch 'unstable' into module_lru_lfu 2019-11-04 11:04:49 +01:00
Salvatore Sanfilippo
327cb45ed5
Merge pull request #6532 from oranagra/rm_misc_commands
Module API for PUBLISH, FLUSHALL, RANDOMKEY, DBSIZE
2019-11-04 11:02:37 +01:00
Salvatore Sanfilippo
da44c54897
Merge branch 'unstable' into module_hooks 2019-11-04 10:37:06 +01:00
Oran Agra
87332ce524 Module API for PUBLISH, FLUSHALL, RANDOMKEY, DBSIZE 2019-11-04 07:40:52 +02:00
Oran Agra
4d580438b0 Add module api for looking into INFO fields
- Add RM_GetServerInfo and friends
- Add auto memory for new opaque struct
- Add tests for new APIs

other minor fixes:
- add const in various char pointers
- requested_section in modulesCollectInfo was actually not sds but char*
- extract new string2d out of getDoubleFromObject for code reuse

Add module API for
2019-11-03 15:02:25 +02:00
antirez
91f4bdc9f9 Modules: block on keys: use a better interface.
Using the is_key_ready() callback plus the reply callback later, creates
different issues AFAIK:

1. More complex API.
2. We need to call the reply callback() ASAP if the is_key_ready()
interface returned success, however the internals do not work in that
way, so when the reply callback is called the setup could be different.
To fix that, there is to break the current design that handles the
unblocked clients asyncrhonously, and run the list ASAP.
2019-10-31 11:35:07 +01:00
antirez
215b72c0ba Modules: block on keys: implement the internals. 2019-10-30 10:57:44 +01:00
Oran Agra
51c3ff8d75 Modules hooks: complete missing hooks for the initial set of hooks
* replication hooks: role change, master link status, replica online/offline
* persistence hooks: saving, loading, loading progress
* misc hooks: cron loop, shutdown, module loaded/unloaded
* change the way hooks test work, and add tests for all of the above

startLoading() now gets flag indicating what is loaded.
stopLoading() now gets an indication of success or failure.
adding startSaving() and stopSaving() with similar args and role.
2019-10-29 17:59:09 +02:00
Oran Agra
e978bdf9ef Module API for controlling LRU and LFU, and OpenKey without TOUCH
Some commands would want to open a key without touching it's LRU/LFU
similarly to the OBJECT or DEBUG command do.

Other commands may want to implement logic similar to what RESTORE
does (and in the future MIGRATE) and get/set the LRU or LFU.
2019-10-29 15:03:05 +02:00
antirez
019ac37e9c Modules hooks: fix define / linker issues. Implement one test event. 2019-10-23 18:39:53 +02:00
antirez
d54652682d Modules hooks: handle module unloading + API export. 2019-10-23 18:39:53 +02:00
antirez
97f6e314c1 Modules: allow to check for AOF loading client. 2019-10-17 18:10:50 +02:00
Yossi Gottlieb
0db3b0a0ff Merge remote-tracking branch 'upstream/unstable' into tls 2019-10-16 17:08:07 +03:00
Yossi Gottlieb
d7f2681a0c TLS: Improve CA certifiate configuration options.
This adds support for explicit configuration of a CA certs directory (in
addition to the previously supported bundle file).  For redis-cli, if no
explicit CA configuration is supplied the system-wide default
configuration will be adopted.
2019-10-08 17:58:50 +03:00
Yossi Gottlieb
61733ded14 TLS: Configuration options.
Add configuration options for TLS protocol versions, ciphers/cipher
suites selection, etc.
2019-10-07 21:07:27 +03:00
Oran Agra
6b6294807c TLS: Implement support for write barrier. 2019-10-07 21:06:30 +03:00
Oran Agra
5a47794606 diskless replication rdb transfer uses pipe, and writes to sockets form the parent process.
misc:
- handle SSL_has_pending by iterating though these in beforeSleep, and setting timeout of 0 to aeProcessEvents
- fix issue with epoll signaling EPOLLHUP and EPOLLERR only to the write handlers. (needed to detect the rdb pipe was closed)
- add key-load-delay config for testing
- trim connShutdown which is no longer needed
- rioFdsetWrite -> rioFdWrite - simplified since there's no longer need to write to multiple FDs
- don't detect rdb child exited (don't call wait3) until we detect the pipe is closed
- Cleanup bad optimization from rio.c, add another one
2019-10-07 21:06:30 +03:00
Yossi Gottlieb
b087dd1db6 TLS: Connections refactoring and TLS support.
* Introduce a connection abstraction layer for all socket operations and
integrate it across the code base.
* Provide an optional TLS connections implementation based on OpenSSL.
* Pull a newer version of hiredis with TLS support.
* Tests, redis-cli updates for TLS support.
2019-10-07 21:06:13 +03:00
Oran Agra
bf759cc9c3 Merge remote-tracking branch 'antirez/unstable' into jemalloc_purge_bg 2019-10-04 13:53:40 +03:00
antirez
ee1cef189f Minor aesthetic changes to #6419. 2019-10-04 12:00:41 +02:00
Salvatore Sanfilippo
264778af0a
Merge pull request #6419 from oranagra/lua_timeout_sha
On LUA script timeout, print the script SHA to the log
2019-10-04 11:58:05 +02:00
antirez
e938bbc543 Modules: implement RM_Replicate() from async callbacks. 2019-10-03 10:56:37 +02:00
antirez
758b39be99 Speedup INFO server section. 2019-10-02 11:30:20 +02:00
Oran Agra
98426e9886 On LUA script timeout, print the script SHA to the log
since the slowlog and other means that can help you detect the bad script
are only exposed after the script is done. it might be a good idea to at least
print the script name (sha) to the log when it timeouts.
2019-10-02 08:40:35 +03:00
Oran Agra
d5c14c70b7 Merge remote-tracking branch 'antirez/unstable' into modules_info 2019-09-30 20:47:35 +03:00
antirez
0a07f8ffee TerminateModuleForkChild(): fix function prototype. 2019-09-27 12:23:07 +02:00
antirez
721d3c9e0c TerminateModuleForkChild(): move safety checks there.
We don't want that the API could be used directly in an unsafe way,
without checking if there is an active child. Now the safety checks are
moved directly in the function performing the operations.
2019-09-27 12:17:47 +02:00
antirez
de1f82aa33 Function renamed hasForkChild() -> hasActiveChildProcess(). 2019-09-27 12:03:09 +02:00
antirez
ae3ef964c1 Modules fork: improve SIGUSR1 handling, fix include.
We can't expect SIGUSR1 to have any specific value range, so let's
define an exit code that we can handle in a special way.
This also fixes an #include <wait.h> that is not standard.
2019-09-27 11:39:45 +02:00
Salvatore Sanfilippo
6129758558
Merge branch 'unstable' into modules_fork 2019-09-27 11:24:06 +02:00
Salvatore Sanfilippo
c1ea6175c5
Merge pull request #6024 from itamarhaber/info_modules
Adds a "Modules" section to `INFO`
2019-09-26 11:58:52 +02:00
Salvatore Sanfilippo
959fb5cf68
Merge pull request #6235 from oranagra/module_rdb_load_errors
Allow modules to handle RDB loading errors.
2019-09-26 11:52:42 +02:00
antirez
06d490342f RESP3: fix cases of NULL reported instead of empty aggregate. 2019-09-02 12:50:47 +02:00
Oran Agra
78bbb9b58d Modlue fork is killed when the parent exists 2019-08-25 10:11:48 +03:00
antirez
0e9b5adbd3 Replication: clarify why repl_put_online_on_ack exists at all. 2019-08-05 17:38:15 +02:00
Salvatore Sanfilippo
fc5c2052b5
Merge pull request #6266 from madolson/dev-unstable-hide-auth-and-hello
Hide HELLO and AUTH from slowlog and monitor
2019-07-31 11:12:46 +02:00
Oran Agra
e5187ad2ae Merge remote-tracking branch 'oss/unstable' into module_rdb_load_errors 2019-07-30 15:14:08 +03:00
Oran Agra
4339706e07 Avoid diskelss-load if modules did not declare they handle read errors 2019-07-30 15:11:57 +03:00
Salvatore Sanfilippo
f42846e8c7
Merge pull request #6264 from oranagra/modules_api_aux_rdb
Implement module api for aux data in rdb
2019-07-30 11:50:44 +02:00
Oran Agra
e91d9a6fff Extend modules API to allow modules report to redis INFO
this implements #6012
2019-07-24 12:58:15 +03:00
antirez
32efd9adf8 Client side caching: config option for table fill rate. 2019-07-24 11:35:01 +02:00
antirez
c98e7717bb Client side caching: show tracking slots usage in INFO. 2019-07-23 11:02:14 +02:00
antirez
9268493e8d Client side caching: implement full slot limit function. 2019-07-23 10:57:22 +02:00
Madelyn Olson
7d21754710 Hide HELLO and AUTH from slowlog and monitor 2019-07-22 22:53:15 -07:00
Oran Agra
3b6aeea44c Implement module api for aux data in rdb
Other changes:
* fix memory leak in error handling of rdb loading of type OBJ_MODULE
2019-07-22 21:15:33 +03:00
antirez
c41f94d2a3 Client side caching: split invalidation into key / slot. 2019-07-22 18:59:53 +02:00
Oran Agra
d7d028a7a7 Allow modules to handle RDB loading errors.
This is especially needed in diskless loading, were a short read could have
caused redis to exit. now the module can handle the error and return to the
caller gracefully.

this fixes #5326
2019-07-21 18:19:32 +03:00
Oran Agra
56258c6b7d Module API for Forking
* create module API for forking child processes.
* refactor duplicate code around creating and tracking forks by AOF and RDB.
* child processes listen to SIGUSR1 and dies exitFromChild in order to
  eliminate a valgrind warning of unhandled signal.
* note that BGSAVE error reply has changed.

valgrind error is:
  Process terminating with default action of signal 10 (SIGUSR1)
2019-07-17 16:40:24 +03:00
zhaozhao.zz
6191ea90a1 Client side caching: implement trackingInvalidateKeysOnFlush() 2019-07-17 20:33:52 +08:00
Oran Agra
2de544cfcc diskless replication on slave side (don't store rdb to file), plus some other related fixes
The implementation of the diskless replication was currently diskless only on the master side.
The slave side was still storing the received rdb file to the disk before loading it back in and parsing it.

This commit adds two modes to load rdb directly from socket:
1) when-empty
2) using "swapdb"
the third mode of using diskless slave by flushdb is risky and currently not included.

other changes:
--------------
distinguish between aof configuration and state so that we can re-enable aof only when sync eventually
succeeds (and not when exiting from readSyncBulkPayload after a failed attempt)
also a CONFIG GET and INFO during rdb loading would have lied

When loading rdb from the network, don't kill the server on short read (that can be a network error)

Fix rdb check when performed on preamble AOF

tests:
run replication tests for diskless slave too
make replication test a bit more aggressive
Add test for diskless load swapdb
2019-07-08 15:37:48 +03:00
Salvatore Sanfilippo
722446510f
Merge pull request #6116 from AngusP/scan-types
SCAN: New Feature `SCAN cursor [TYPE type]` modifier suggested in issue #6107
2019-07-08 12:53:34 +02:00
Angus Pearson
6eb52e200c Change typeNameCanonicalize -> getObjectTypeName, and other style changes 2019-07-08 11:04:37 +01:00
antirez
6b29f2d83d Client side caching: RESP2 support. 2019-07-05 12:24:28 +02:00
antirez
46edb55de9 Client side caching: implement trackingInvalidateKey(). 2019-07-03 19:16:20 +02:00
antirez
506764b3f8 Client side caching: hook inside call() for tracking. 2019-07-03 12:42:16 +02:00
antirez
c29f3bcf2e Client side caching: enable tracking mode. 2019-06-30 06:19:08 -04:00
antirez
45d64f229e Client side caching: fields and flags for tracking mode. 2019-06-29 20:08:41 -04:00
Angus Pearson
38cd5fd9f6 Spelling cannonical -> canonical 2019-06-13 17:49:33 +01:00
Angus Pearson
e2adea2188 Add char* typeNameCanonicalize(robj*) to remove duplicate code between SCAN and TYPE commands,
and to keep OBJ_* enum to string canonicalization in one place.
2019-06-10 17:41:44 +01:00
Oran Agra
09f99c2a92 make redis purge jemalloc after flush, and enable background purging thread
jemalloc 5 doesn't immediately release memory back to the OS, instead there's a decaying
mechanism, which doesn't work when there's no traffic (no allocations).
this is most evident if there's no traffic after flushdb, the RSS will remain high.

1) enable jemalloc background purging
2) explicitly purge in flushdb
2019-06-02 15:33:14 +03:00
Salvatore Sanfilippo
e633254ccf
Merge pull request #6053 from soloestoy/enhance-aof-fsync-everysec
aof: enhance AOF_FSYNC_EVERYSEC, more details in #5985
2019-05-10 18:06:40 +02:00
antirez
1c0c436757 Threaded IO: ability to disable reads from threaded path. 2019-05-06 18:02:51 +02:00
antirez
6ab6a97fe6 Threaded IO: parsing WIP 2: refactoring to parse from thread. 2019-05-06 18:02:51 +02:00
antirez
63a0ffd36a Threaded IO: read side WIP 3. 2019-05-06 18:02:51 +02:00
antirez
dd5b105c73 Threaded IO: read side WIP. 2019-05-06 18:02:51 +02:00
antirez
9814b2a5f3 Threaded IO: make num of I/O threads configurable. 2019-05-06 18:02:51 +02:00
Ubuntu
9bf7f302a7 Threaded IO: stop threads when no longer needed + C11 in Makefile.
Now threads are stopped even when the connections drop immediately to
zero, not allowing the networking code to detect the condition and stop
the threads. serverCron() will handle that.
2019-05-06 18:02:51 +02:00
antirez
f468e653b5 Threaded IO: implement handleClientsWithPendingWritesUsingThreads().
This is just an experiment for now, there are a couple of race
conditions, mostly harmless for the performance gain experiment that
this commit represents so far.

The general idea here is to take Redis single threaded and instead
fan-out on expansive kernel calls: write(2) in this case, but the same
concept could be easily implemented for read(2) and protcol parsing.

However just threading writes like in this commit, is enough to evaluate
if the approach is sounding.
2019-05-06 18:02:51 +02:00
zhaozhao.zz
bcac165fab aof: enhance AOF_FSYNC_EVERYSEC, more details in #5985 2019-04-29 14:38:28 +08:00
Itamar Haber
52686f4866 Adds a "Modules" section to INFO
Fixes #6012.

As long as "INFO is broken", this should be adequate IMO. Once we rework
`INFO`, perhaps into RESP3, this implementation should be revisited.
2019-04-16 22:16:12 +03:00
Oran Agra
acba2fc9b4 slave corrupts replication stream when module blocked client uses large reply (or POSTPONED_ARRAY)
when redis appends the blocked client reply list to the real client, it didn't
bother to check if it is in fact the master client. so a slave executing that
module command will send replies to the master, causing the master to send the
slave error responses, which will mess up the replication offset
(slave will advance it's replication offset, and the master does not)
2019-03-24 14:17:37 +02:00
Salvatore Sanfilippo
5e8caca036
Merge pull request #5944 from yossigo/command-filtering
Command Filtering API
2019-03-22 17:43:49 +01:00
Salvatore Sanfilippo
122f42844a
Merge pull request #5945 from dvirsky/miss_notification
Added keyspace miss notifications support
2019-03-22 17:41:00 +01:00
Dvir Volk
bc269c85e1 remove extra linebreak 2019-03-21 12:48:37 +02:00
Dvir Volk
99c2fe0bcf added special flag for keyspace miss notifications 2019-03-21 11:47:14 +02:00
Yossi Gottlieb
c3e187190b Initial command filter experiment. 2019-03-18 13:50:34 +02:00
Yossi Gottlieb
a88264d934 Add RedisModule_GetKeyNameFromIO(). 2019-03-15 10:23:27 +02:00
Salvatore Sanfilippo
0cce98f2f9
Merge pull request #5834 from guybe7/trim_module_sds
Trim SDS free space of retained module strings
2019-03-14 12:41:31 +01:00
antirez
6fd4af1f12 Merge branch 'sharedapi' into unstable 2019-03-14 11:24:48 +01:00
Yuan Zhou
feb4ebff45 server.h: remove dead code
hashTypeTryObjectEncoding() is not used now
2019-03-07 18:38:27 +08:00
Salvatore Sanfilippo
88b720672c
Merge pull request #5877 from vattezhang/unstable_sentinel_cmd
fix: fix sentinel command table and new flags format
2019-02-27 15:45:03 +01:00
vattezhang
9d632230b6 fix: fix sentinel command table and new flags format 2019-02-27 21:35:58 +08:00
antirez
a7780f716e Merge branch 'gopher' into unstable 2019-02-25 18:16:58 +01:00
antirez
3b420034bb RESP3: allow HELLO to be used with version = 2. 2019-02-25 16:41:00 +01:00
antirez
e00b22e090 Gopher: initial request handling. 2019-02-21 23:13:08 +01:00
antirez
3de9ccf190 Gopher: config setting to turn support on/off. 2019-02-21 17:28:53 +01:00
zhaozhao.zz
14507457a0 ACL: show categories in COMMAND reply
Adding another new filed categories at the end of
command reply, it's easy to read and distinguish
flags and categories, also compatible with old format.
2019-02-14 00:13:01 +08:00
Guy Benoish
bdd9a8002a Trim SDS free space of retained module strings
In some cases processMultibulkBuffer uses sdsMakeRoomFor to
expand the querybuf, but later in some cases it uses that query
buffer as is for an argv element (see "Optimization"), which means
that the sds in argv may have a lot of wasted space, and then in case
modules keep that argv RedisString inside their data structure, this
space waste will remain for long (until restarted from rdb).
2019-02-12 14:21:21 +01:00
zhaozhao.zz
ea9d3aefec ACL: add masteruser configuration for replication
In mostly production environment, normal user's behavior should be
limited.

Now in redis ACL mechanism we can do it like that:

    user default on +@all ~* -@dangerous nopass
    user admin on +@all ~* >someSeriousPassword

Then the default normal user can not execute dangerous commands like
FLUSHALL/KEYS.

But some admin commands are in dangerous category too like PSYNC,
and the configurations above will forbid replica from sync with master.

Finally I think we could add a new configuration for replication,
it is masteruser option, like this:

    masteruser admin
    masterauth someSeriousPassword

Then replica will try AUTH admin someSeriousPassword and get privilege
to execute PSYNC. If masteruser is NULL, replica would AUTH with only
masterauth like before.
2019-02-12 17:12:37 +08:00
antirez
80f987726d ACL: load ACL file at startup. Prevent silly configurations. 2019-02-07 17:20:03 +01:00
antirez
775bf6193d ACL: implement rewriting of users in redis.conf. 2019-02-05 10:48:17 +01:00
antirez
500b3e128f ACL: implement ACLLoadConfiguredUsers(). 2019-02-04 16:35:15 +01:00
antirez
68fd4a97fa ACL: better error reporting in users configuration errors. 2019-02-04 13:04:35 +01:00
antirez
b166c41edd ACL: make ACLAppendUserForLoading() able to report bad argument. 2019-02-04 13:00:58 +01:00
antirez
21e84cdae2 ACL: initial appending of users in user loading list. 2019-02-04 12:55:48 +01:00
antirez
c7cd10dfe9 ACL: flags refactoring, function to describe user. 2019-01-31 16:49:22 +01:00
antirez
f99e0f59ef ACL: populate category flags from command table. 2019-01-23 16:59:09 +01:00
antirez
91ec53ed13 ACL: define category names and flags. 2019-01-23 16:47:29 +01:00
antirez
70e541b7bc ACL: better define name, and the idea of reserved ID. 2019-01-23 08:10:57 +01:00
antirez
711e514ea4 ACL: update comments in command flags. 2019-01-22 19:02:50 +01:00
antirez
4dc69497f5 Refactoring: always kill AOF/RDB child via helper functions. 2019-01-21 11:28:44 +01:00
antirez
c8391388c2 ACL: remove server.requirepass + some refactoring. 2019-01-18 11:49:30 +01:00
antirez
7b65605ab2 ACL: reimplement requirepass option in term of ACLs. 2019-01-17 18:05:43 +01:00
antirez
4a3419acfc ACL: fix and improve ACL key checking. 2019-01-16 18:31:05 +01:00
antirez
cca64672f4 ACL: AUTH uses users. ACL WHOAMI implemented. 2019-01-15 18:16:20 +01:00
antirez
b39409bcf8 ACL: nopass user setting.
This is needed in order to model the current behavior of authenticating
the connection directly when no password is set. Now with ACLs this will
be obtained by setting the default user as "nopass" user. Moreover this
flag can be used in order to create other users that do not require any
password but will work with "AUTH username <any-password>".
2019-01-15 13:16:31 +01:00
antirez
7aea02fa87 ACL: initial implementation of the ACL command. 2019-01-15 09:36:12 +01:00
antirez
a2e376ba52 ACL: ACLCheckCommandPerm() implementation WIP. 2019-01-14 18:35:21 +01:00
antirez
a0a4fb85ff ACL: Fix compilation by adding prototype and c->cmd fix. 2019-01-14 13:22:56 +01:00
antirez
2da2e452ab ACL: ACLLCOMMAND flags. 2019-01-14 13:21:21 +01:00
antirez
aced0328e3 ACL: avoid a radix tree lookup for the default user. 2019-01-11 11:32:41 +01:00
antirez
6bb6a6d3a8 ACL: implement ACLCreateUser(). 2019-01-10 17:01:12 +01:00
antirez
29c88a9ce5 ACL: initialization function. 2019-01-10 16:39:32 +01:00
antirez
4278104acc ACL: add a reference to the user in each client. 2019-01-10 16:34:13 +01:00
antirez
4729f71495 ACL: improved version of the user structure. 2019-01-10 12:47:52 +01:00
antirez
7fc882c578 ACL: use a fixed table for command IDs. 2019-01-09 21:31:29 +01:00
antirez
91f1d8026b ACL: introduce the concept of command ID. 2019-01-09 17:20:47 +01:00
antirez
b43d70df56 ACL: refactoring of the original authentication code. 2019-01-09 17:00:30 +01:00
antirez
709a6612eb RESP3: addReplyString() -> addReplyProto().
The function naming was totally nuts. Let's fix it as we break PRs
anyway with RESP3 refactoring and changes.
2019-01-09 17:00:30 +01:00
antirez
e291170385 RESP3: verbatim reply API + DEBUG PROTOCOL support. 2019-01-09 17:00:30 +01:00
antirez
809e3a44a7 RESP3: addReplyBool() implemented. 2019-01-09 17:00:29 +01:00
antirez
4f0860cbfd RESP3: initial implementation of the HELLO command. 2019-01-09 17:00:29 +01:00
antirez
3fd78f41e8 RESP3: restore the concept of null array for RESP2 compat. 2019-01-09 17:00:29 +01:00
antirez
2ad6e875ba RESP3: add shared.nullarray for better RESP2 compat. 2019-01-09 17:00:29 +01:00
antirez
1a17cdfadf RESP3: addReplyNullArray() added for better RESP2 compat. 2019-01-09 17:00:29 +01:00
antirez
317f8b9d38 RESP3: most null replies converted. 2019-01-09 17:00:29 +01:00
antirez
1b7298e66a RESP3: addReplyNull() added. 2019-01-09 17:00:29 +01:00
antirez
fc9a3de97d RESP3: remove other pointless shared object. 2019-01-09 17:00:29 +01:00
antirez
b7e8b734c9 RESP3: remove certain constants to spot places to fix. 2019-01-09 17:00:29 +01:00
antirez
13966522ea RESP3: bring RESP2 compatibility to previous changes. 2019-01-09 17:00:29 +01:00
antirez
e14aabf936 RESP3: addReply*Len() support for RESP2 backward comp. 2019-01-09 17:00:29 +01:00
antirez
1ac6926647 RESP3: put RESP version in the client structure. 2019-01-09 17:00:29 +01:00
antirez
073293693e RESP3: Use new deferred len API in server.c. 2019-01-09 17:00:29 +01:00
antirez
57c5a766a2 RESP3: Aggregate deferred lengths functions. 2019-01-09 17:00:29 +01:00
antirez
27f6e9bb9b Modules shared API: initial core functions.
Based on ideas and code in PR #5560 by @MeirShpilraien.
2018-12-20 17:57:35 +01:00
antirez
850b64c116 Revert shared APIs to modify the design. 2018-12-20 17:56:38 +01:00
MeirShpilraien
ab37289fa6 added module ability to register api to be used by other modules 2018-12-20 17:55:18 +01:00
antirez
129f2d2746 freeMemoryIfNeeded() small refactoring.
Related to issue #5686 and PR #5689.
2018-12-12 11:37:15 +01:00
antirez
274531396c Reject EXEC containing write commands against RO replica.
Thanks to @soloestoy for discovering this issue in #5667.
This is an alternative fix in order to avoid both cycling the clients
and also disconnecting clients just having valid read-only transactions
pending.
2018-12-11 11:39:21 +01:00
Oran Agra
b587c54c24 fix #5580, display fragmentation and rss overhead bytes as signed
these metrics become negative when RSS is smaller than the used_memory.
This can easily happen when the program allocated a lot of memory and haven't
written to it yet, in which case the kernel doesn't allocate any pages to the process
2018-12-02 15:29:20 +02:00
antirez
e3446fea9e Streams: XSTREAM SETID -> XSETID.
Keep vanilla stream commands at toplevel, see #5426.
2018-10-16 13:17:14 +02:00
Salvatore Sanfilippo
af09df08d7
Merge pull request #5426 from soloestoy/feature-xstream
Bugfix data inconsistency after aof rewrite, and add XSTREAM command.
2018-10-16 13:10:36 +02:00
antirez
c9d9ae7baa Fix propagation of consumer groups last ID.
Issue #5433.
2018-10-10 12:51:02 +02:00
antirez
69c30965eb Introduce protectClient() + some refactoring.
The idea is to have an API for the cases like -BUSY state and DEBUG
RELOAD where we have to manually deinstall the read handler.
See #4804.
2018-10-09 13:15:41 +02:00
zhaozhao.zz
ec511fa709 Streams: add a new command XTREAM
XSTREAM CREATE <key> <id or *> -- Create a new empty stream.
XSTREAM SETID <key> <id or $>  -- Set the current stream ID.
2018-10-09 13:11:04 +08:00
antirez
744fe7f348 Module cluster flags: initial vars / defines added. 2018-09-19 11:20:52 +02:00
antirez
43385c4375 LOLWUT: wrap it into a proper command. 2018-09-12 11:34:10 +02:00
antirez
ef2c7a5bbb Slave removal: SLAVEOF -> REPLICAOF. SLAVEOF is now an alias. 2018-09-11 15:32:28 +02:00
antirez
6c001bfc0d Unblocked clients API refactoring. See #4418. 2018-09-03 18:39:18 +02:00
antirez
3e7349fdaf Make pending buffer processing safe for CLIENT_MASTER client.
Related to #5305.
2018-09-03 18:17:31 +02:00
antirez
067647a783 Introduce repl_slave_ignore_maxmemory flag internally.
Note: this breaks backward compatibility with Redis 4, since now slaves
by default are exact copies of masters and do not try to evict keys
independently.
2018-08-27 12:20:27 +02:00
Salvatore Sanfilippo
19880ab851
Merge pull request #5248 from soloestoy/rewrite-brpoplpush
rewrite BRPOPLPUSH as RPOPLPUSH to propagate
2018-08-26 16:31:24 +02:00
zhaozhao.zz
8a1219d93b block: rewrite BRPOPLPUSH as RPOPLPUSH to propagate 2018-08-14 20:58:58 +08:00
zhaozhao.zz
14c4ddb5a6 pipeline: do not sdsrange querybuf unless all commands processed
This is an optimization for processing pipeline, we discussed a
problem in issue #5229: clients may be paused if we apply `CLIENT
PAUSE` command, and then querybuf may grow too large, the cost of
memmove in sdsrange after parsing a completed command will be
horrible. The optimization is that parsing all commands in queyrbuf
, after that we can just call sdsrange only once.
2018-08-14 00:43:42 +08:00
Salvatore Sanfilippo
92b39a0abf
Merge pull request #5189 from soloestoy/refactor-dbOverwrite
refactor dbOverwrite to make lazyfree work
2018-07-31 16:40:35 +02:00
antirez
5401fe7fb9 Introduce writeCommandsDeniedByDiskError(). 2018-07-31 13:09:38 +02:00
zhaozhao.zz
fddeeae724 refactor dbOverwrite to make lazyfree work 2018-07-31 12:07:57 +08:00
antirez
c426d85c4c Control dynamic HZ via server configuration. 2018-07-30 13:37:30 +02:00
antirez
4e9c30a6ca Merge branch 'dynamic-hz' into unstable 2018-07-30 13:31:23 +02:00
Salvatore Sanfilippo
445a2a2b1b
Merge pull request #4883 from itamarhaber/lua_scripts-in-info-memory
Adds memory information about the scripts' cache to INFO
2018-07-23 18:43:05 +02:00
antirez
b65ddfb16a Dynamic HZ: adapt cron frequency to number of clients. 2018-07-23 14:21:04 +02:00
antirez
e6ea603ad3 Dynamic HZ: separate hz from the configured hz.
This way we can remember what the user configured HZ is, but change the
actual HZ dynamically if needed in the dynamic HZ feature
implementation.
2018-07-23 14:13:58 +02:00
Itamar Haber
993716c351 Adds Lua overheads to MEMORY STATS, smartens the MEMORY DOCTOR 2018-07-22 21:16:00 +03:00
antirez
f9c84d6d39 Hopefully improve commenting of #5126.
Reading the PR gave me the opportunity to better specify what the code
was doing in places where I was not immediately sure about what was
going on. Moreover I documented the structure in server.h so that people
reading the header file will immediately understand what the structure
is useful for.
2018-07-16 17:56:54 +02:00
Oran Agra
bf680b6f8c slave buffers were wasteful and incorrectly counted causing eviction
A) slave buffers didn't count internal fragmentation and sds unused space,
   this caused them to induce eviction although we didn't mean for it.

B) slave buffers were consuming about twice the memory of what they actually needed.
- this was mainly due to sdsMakeRoomFor growing to twice as much as needed each time
  but networking.c not storing more than 16k (partially fixed recently in 237a38737).
- besides it wasn't able to store half of the new string into one buffer and the
  other half into the next (so the above mentioned fix helped mainly for small items).
- lastly, the sds buffers had up to 30% internal fragmentation that was wasted,
  consumed but not used.

C) inefficient performance due to starting from a small string and reallocing many times.

what i changed:
- creating dedicated buffers for reply list, counting their size with zmalloc_size
- when creating a new reply node from, preallocate it to at least 16k.
- when appending a new reply to the buffer, first fill all the unused space of the
  previous node before starting a new one.

other changes:
- expose mem_not_counted_for_evict info field for the benefit of the test suite
- add a test to make sure slave buffers are counted correctly and that they don't cause eviction
2018-07-16 16:43:42 +03:00
dejun.xdj
61f12973f7 Bugfix: PEL is incorrect when consumer is blocked using xreadgroup with NOACK option.
Save NOACK option into client.blockingState structure.
2018-07-09 13:40:29 +02:00
antirez
81778d91bf Cache timezone and daylight active flag for safer logging.
With such information will be able to use a private localtime()
implementation serverLog(), which does not use any locking and is both
thread and fork() safe.
2018-07-04 16:45:00 +02:00
Jack Drogon
93238575f7 Fix typo 2018-07-03 18:19:46 +02:00
antirez
94b3ee6142 Clarify the pending_querybuf field of clients. 2018-07-03 13:25:41 +02:00
chendianqiang
cbb2ac0799
Merge branch 'unstable' into pending-querybuf 2018-07-03 10:07:26 +08:00
antirez
2edcafb35d addReplySubSyntaxError() renamed to addReplySubcommandSyntaxError(). 2018-07-02 18:49:34 +02:00
Salvatore Sanfilippo
bc6a004588
Merge pull request #4998 from itamarhaber/module_command_help
Module command help
2018-07-02 18:46:56 +02:00
chendianqiang
7de1ada070 limit the size of pending-querybuf in masterclient 2018-07-01 14:43:53 +08:00
zhaozhao.zz
b9cbd04b57 clients: add type option for client list 2018-06-28 17:43:05 +08:00
antirez
fb39bfd7af Take clients in a ID -> Client handle dictionary. 2018-06-27 14:08:42 +02:00
Guy Benoish
b5197f1fc9 Enhance RESTORE with RDBv9 new features
RESTORE now supports:
1. Setting LRU/LFU
2. Absolute-time TTL

Other related changes:
1. RDB loading will not override LRU bits when RDB file
   does not contain the LRU opcode.
2. RDB loading will not set LRU/LFU bits if the server's
   maxmemory-policy does not match.
2018-06-20 15:11:08 +07:00
Oran Agra
482785ac62 add malloc_usable_size for libc malloc
this reduces the extra 8 bytes we save before each pointer.
but more importantly maybe, it makes the valgrind runs to be more similiar
to our normal runs.

note: the change in malloc_stats struct in server.h is to eliminate an name conflict.
structs that are not typedefed are resolved from a separate name space.
2018-06-19 18:18:23 +03:00
antirez
bd92389c2d Refactor createObjectFromLongLong() to be suitable for value objects. 2018-06-18 16:55:16 +02:00
Salvatore Sanfilippo
94658303e9
Merge pull request #4758 from soloestoy/rdb-save-incremental-fsync
Rdb save incremental fsync
2018-06-16 10:59:37 +02:00
Itamar Haber
e654b68d1f
Merge branch 'unstable' into module_command_help 2018-06-09 21:10:53 +03:00
Salvatore Sanfilippo
be899b824e
Merge pull request #4519 from soloestoy/zset-int-problem
Zset int problem
2018-06-08 12:45:11 +02:00
Itamar Haber
76ad23d012 Adds MODULE HELP and implements addReplySubSyntaxError 2018-06-07 18:34:58 +03:00
antirez
19a438e2c0 Streams: use non static macro node limits.
Also add the concept of size/items limit, instead of just having as
limit the number of bytes.
2018-06-07 14:24:49 +02:00
antirez
56bbab238a ZPOP: change sync ZPOP to have a count argument instead of N keys.
Usually blocking operations make a lot of sense with multiple keys so
that we can listen to multiple queues (or whatever the app models) with
a single connection. However in the synchronous case it is more useful
to be able to ask for N elements. This is a change that I also wanted to
perform soon or later in the blocking list variant, but here it is more
natural since there is no reply type difference.
2018-05-11 18:00:32 +02:00
antirez
6efb6c1e06 ZPOP: renaming to have explicit MIN/MAX score idea.
This commit also adds a top comment about a subtle behavior of mixing
blocking operations of different types in the same key.
2018-05-11 17:31:53 +02:00
Itamar Haber
49890c8ee9 Adds memory information about the script's cache to INFO
Implementation notes: as INFO is "already broken", I didn't want to break it further. Instead of computing the server.lua_script dict size on every call, I'm keeping a running sum of the body's length and dict overheads.

This implementation is naive as it **does not** take into consideration dict rehashing, but that inaccuracy pays off in speed ;)

Demo time:

```bash
$ redis-cli info memory | grep "script"
used_memory_scripts:96
used_memory_scripts_human:96B
number_of_cached_scripts:0
$ redis-cli eval "" 0 ; redis-cli info memory | grep "script"
(nil)
used_memory_scripts:120
used_memory_scripts_human:120B
number_of_cached_scripts:1
$ redis-cli script flush ; redis-cli info memory | grep "script"
OK
used_memory_scripts:96
used_memory_scripts_human:96B
number_of_cached_scripts:0
$ redis-cli eval "return('Hello, Script Cache :)')" 0 ; redis-cli info memory | grep "script"
"Hello, Script Cache :)"
used_memory_scripts:152
used_memory_scripts_human:152B
number_of_cached_scripts:1
$ redis-cli eval "return redis.sha1hex(\"return('Hello, Script Cache :)')\")" 0 ; redis-cli info memory | grep "script"
"1be72729d43da5114929c1260a749073732dc822"
used_memory_scripts:232
used_memory_scripts_human:232B
number_of_cached_scripts:2
✔ 19:03:54 redis [lua_scripts-in-info-memory L ✚…⚑] $ redis-cli evalsha 1be72729d43da5114929c1260a749073732dc822 0
"Hello, Script Cache :)"
```
2018-04-30 19:33:01 +03:00
Itamar Haber
438125b47c Implements [B]Z[REV]POP and the respective unit tests
An implementation of the
[Ze POP Redis Module](https://github.com/itamarhaber/zpop) as core
Redis commands.

Fixes #1861.
2018-04-30 02:10:42 +03:00
antirez
e6b0e8d9ec Streams: XTRIM command added. 2018-04-19 16:25:29 +02:00
antirez
aba76320d5 Streams: XDEL command. 2018-04-18 13:12:09 +02:00
antirez
de7de53e64 getMaxmemoryState() fixed and improved. 2018-04-11 12:48:26 +02:00