Commit Graph

500 Commits

Author SHA1 Message Date
Yossi Gottlieb
64c360c515
Module API: fix missing RM_CLIENTINFO_FLAG_SSL. (#7666)
The `REDISMODULE_CLIENTINFO_FLAG_SSL` flag was already a part of the `RedisModuleClientInfo` structure but was not implemented.
2020-08-17 17:46:54 +03:00
杨博东
229327ad8b
Avoid redundant calls to signalKeyAsReady (#7625)
signalKeyAsReady has some overhead (namely dictFind) so we should
only call it when there are clients blocked on the relevant type (BLOCKED_*)
2020-08-11 08:18:09 +03:00
Meir Shpilraien (Spielrein)
3f494cc49d
see #7544, added RedisModule_HoldString api. (#7577)
Added RedisModule_HoldString that either returns a
shallow copy of the given String (by increasing
the String ref count) or a new deep copy of String
in case its not possible to get a shallow copy.

Co-authored-by: Itamar Haber <itamar@redislabs.com>
2020-08-09 06:11:47 +03:00
Wang Yuan
1ef014ee6b
Fix applying zero offset to null pointer when creating moduleFreeContextReusedClient (#7323)
Before this fix we where attempting to select a db before creating db the DB, see: #7323

This issue doesn't seem to have any implications, since the selected DB index is 0,
the db pointer remains NULL, and will later be correctly set before using this dummy
client for the first time.

As we know, we call 'moduleInitModulesSystem()' before 'initServer()'. We will allocate
memory for server.db in 'initServer', but we call 'createClient()' that will call 'selectDb()'
in 'moduleInitModulesSystem()', before the databases where created. Instead, we should call
'createClient()' for moduleFreeContextReusedClient after 'initServer()'.
2020-08-08 14:36:41 +03:00
Yossi Gottlieb
7af05f07ff
Clarify RM_BlockClient() error condition. (#6093) 2020-07-29 17:03:38 +03:00
Meir Shpilraien (Spielrein)
8d82639319
This PR introduces a new loaded keyspace event (#7536)
Co-authored-by: Oran Agra <oran@redislabs.com>
Co-authored-by: Itamar Haber <itamar@redislabs.com>
2020-07-23 12:38:51 +03:00
yoav-steinberg
d484b8a04e
Support passing stack allocated module strings to moduleCreateArgvFromUserFormat (#7528)
Specifically, the key passed to the module aof_rewrite callback is a stack allocated robj. When passing it to RedisModule_EmitAOF (with appropriate "s" fmt string) redis used to panic when trying to inc the ref count of the stack allocated robj. Now support such robjs by coying them to a new heap robj. This doesn't affect performance because using the alternative "c" or "b" format strings also copies the input to a new heap robj.
2020-07-16 20:59:38 +03:00
WuYunlong
dc690161d5
Refactor RM_KeyType() by using macro. (#7486) 2020-07-15 12:37:44 +03:00
meir@redislabs.com
a89bf734a9 Fix RM_ScanKey module api not to return int encoded strings
The scan key module API provides the scan callback with the current
field name and value (if it exists). Those arguments are RedisModuleString*
which means it supposes to point to robj which is encoded as a string.
Using createStringObjectFromLongLong function might return robj that
points to an integer and so break a module that tries for example to
use RedisModule_StringPtrLen on the given field/value.

The PR introduces a fix that uses the createObject function and sdsfromlonglong function.
Using those function promise that the field and value pass to the to the
scan callback will be Strings.

The PR also changes the Scan test module to use RedisModule_StringPtrLen
to catch the issue. without this, the issue is hidden because
RedisModule_ReplyWithString knows to handle integer encoding of the
given robj (RedisModuleString).

The PR also introduces a new test to verify the issue is solved.
2020-06-14 11:20:15 +03:00
antirez
94f2e7f9f9 Tracking: NOLOOP internals implementation. 2020-04-21 10:51:46 +02:00
Salvatore Sanfilippo
eb9ca33414
Merge pull request #6694 from oranagra/signal_modified_key
modules don't signalModifiedKey in setKey() since that's done (optionally) in RM_CloseKey
2020-04-02 19:00:20 +02:00
Salvatore Sanfilippo
0062194d74
Merge pull request #6654 from oranagra/fix_module_info_warning
fix possible warning on incomplete struct init
2020-04-02 16:32:20 +02:00
antirez
80ec0431e8 Simplify comment in moduleTryServeClientBlockedOnKey(). 2020-04-02 11:20:09 +02:00
Guy Benoish
c4dc5b80b2 Fix memory corruption in moduleHandleBlockedClients
By using a "circular BRPOPLPUSH"-like scenario it was
possible the get the same client on db->blocking_keys
twice (See comment in moduleTryServeClientBlockedOnKey)

The fix was actually already implememnted in
moduleTryServeClientBlockedOnKey but it had a bug:
the funxction should return 0 or 1 (not OK or ERR)

Other changes:
1. Added two commands to blockonkeys.c test module (To
   reproduce the case described above)
2. Simplify blockonkeys.c in order to make testing easier
3. cast raxSize() to avoid warning with format spec
2020-04-01 12:53:26 +03:00
antirez
9dcf878f1b Fix module commands propagation double MULTI bug.
37a10cef introduced automatic wrapping of MULTI/EXEC for the
alsoPropagate API. However this collides with the built-in mechanism
already present in module.c. To avoid complex changes near Redis 6 GA
this commit introduces the ability to exclude call() MUTLI/EXEC wrapping
for also propagate in order to continue to use the old code paths in
module.c.
2020-03-31 11:00:45 +02:00
antirez
7698316fda Fix RM_Call() stale comment due to cut&paste. 2020-03-31 10:27:47 +02:00
antirez
19f5be231d Modules: updated function doc after #7003. 2020-03-23 11:17:50 +01:00
Salvatore Sanfilippo
7c5dc07016
Merge pull request #7003 from guybe7/rm_context_flags_handle_null
Allow RM_GetContextFlags to work with ctx==NULL
2020-03-23 11:15:42 +01:00
Guy Benoish
f16eaadd4f Allow RM_GetContextFlags to work with ctx==NULL 2020-03-18 18:34:27 +05:30
Salvatore Sanfilippo
ebf1acd33c
Merge pull request #6918 from chendq8/fixModuleUnregisterUsedApi
Fix module unregister used api
2020-03-13 12:46:27 +01:00
antirez
c8ae90fef6 Modules: reformat RM_Scan() top comment a bit. 2020-02-28 18:09:46 +01:00
antirez
8a3e9a6d44 Modules: more details in RM_Scan API top comment. 2020-02-28 18:06:30 +01:00
Salvatore Sanfilippo
86cbddc475
Merge pull request #6838 from oranagra/rm_scan_dict_rehash
RM_Scan disable dict rehashing
2020-02-28 17:47:52 +01:00
Salvatore Sanfilippo
4ccb56d0e6
Merge pull request #6828 from guybe7/module_string_from_double
Add RM_CreateStringFromDouble
2020-02-28 11:06:02 +01:00
Salvatore Sanfilippo
f0d9a4e1a4
Merge pull request #6893 from oranagra/api_doc_aux_save
module api docs for aux_save and aux_load
2020-02-28 10:22:53 +01:00
Salvatore Sanfilippo
ca825701dc
Merge pull request #6876 from MeirShpilraien/changed_log_level_of_module_fork_api
Changed log level for module fork api from 'notice' to 'verbos'.
2020-02-27 10:26:55 +01:00
Salvatore Sanfilippo
37612118c8
Merge pull request #6919 from ashtul/Fix-ThreadSafeContext-functions-names
fix ThreadSafeContext lock/unlock function names
2020-02-27 10:01:48 +01:00
Guy Benoish
376a806bfb Modules: Do not auto-unblock clients if not blocked on keys 2020-02-23 19:36:46 +05:30
Ariel
28bde89172 fix ThreadSafeContext lock/unlock function names 2020-02-22 23:49:23 +02:00
chendianqiang
9947956d7b use correct list for moduleUnregisterUsedAPI 2020-02-22 15:03:01 +08:00
Oran Agra
6fff2cf9b6 module api docs for aux_save and aux_load 2020-02-16 15:43:19 +02:00
Salvatore Sanfilippo
c21c23bbba
Merge pull request #6863 from oranagra/module_commands_no_slowlog
add no-slowlog option to RM_CreateCommand
2020-02-13 17:02:19 +01:00
meir@redislabs.com
67dbc5f365 Changed log level for module fork api from 'notice' to 'verbos'. 2020-02-10 12:10:32 +02:00
Oran Agra
46216b0e83 add no-slowlog option to RM_CreateCommand 2020-02-06 15:06:33 +02:00
Oran Agra
28ef18a894 RM_Scan disable dict rehashing
The callback approach we took is very efficient, the module can do any
filtering of keys without building any list and cloning strings, it can
also read data from the key's value. but if the user tries to re-open
the key, or any other key, this can cause dict re-hashing (dictFind does
that), and that's very bad to do from inside dictScan.

this commit protects the dict from doing any rehashing during scan, but
also warns the user not to attempt any writes or command calls from
within the callback, for fear of unexpected side effects and crashes.
2020-02-06 11:48:12 +02:00
Salvatore Sanfilippo
5bfba06a3b
Merge pull request #6821 from guybe7/key_miss_notify
Exclude "keymiss" notification from NOTIFY_ALL
2020-02-06 10:43:05 +01:00
Salvatore Sanfilippo
2e1dd00c80
Merge pull request #6837 from oranagra/signal_modified_key_doc
update RM_SignalModifiedKey doc comment
2020-02-06 10:38:36 +01:00
Salvatore Sanfilippo
13741fb99d
Merge pull request #6846 from oranagra/module_client_flags
Fix client flags to be int64 in module.c
2020-02-06 10:32:35 +01:00
Oran Agra
85cc696f50 moduleRDBLoadError, add key name, and use panic rather than exit
using panic rather than exit means you get s stack trace of the code
path that experianced the error, and possibly other info.
2020-02-06 09:37:04 +02:00
Oran Agra
d454d5a4f5 Fix client flags to be int64 in module.c
currently there's no bug since the flags these functions handle are
always lower than 32bit, but still better fix the type to prevent future
bugs.
2020-02-06 09:33:20 +02:00
Oran Agra
3795aaf42a update RM_SignalModifiedKey doc comment 2020-02-05 18:15:38 +02:00
Guy Benoish
1c7a2269d6 Add RM_CreateStringFromDouble 2020-02-04 19:28:09 +05:30
Oran Agra
138ed120dd fix uninitialized info_cb var in module.c 2020-02-03 16:35:39 +02:00
Guy Benoish
2fda5f5c98 Exclude "keymiss" notification from NOTIFY_ALL
Because "keymiss" is "special" compared to the rest of
the notifications (Trying not to break existing apps
using the 'A' format for notifications)

Also updated redis.conf and module.c docs
2020-02-03 15:43:44 +05:30
Salvatore Sanfilippo
53ac8c7df2
Merge pull request #6812 from guybe7/str_convert_fix
ld2string should fail if string contains \0 in the middle
2020-01-30 17:51:04 +01:00
Guy Benoish
2deb55512f ld2string should fail if string contains \0 in the middle
This bug affected RM_StringToLongDouble and HINCRBYFLOAT.
I added tests for both cases.

Main changes:
1. Fixed string2ld to fail if string contains \0 in the middle
2. Use string2ld in getLongDoubleFromObject - No point of
   having duplicated code here

The two changes above broke RM_SaveLongDouble/RM_LoadLongDouble
because the long double string was saved with length+1 (An innocent
mistake, but it's actually a bug - The length passed to
RM_SaveLongDouble should not include the last \0).
2020-01-30 18:15:17 +05:30
antirez
17ff3173d6 Add more info in the unblockClientFromModule() function. 2020-01-29 12:47:50 +01:00
Salvatore Sanfilippo
bf53f9280a
Merge pull request #6699 from guybe7/module_blocked_on_key_timeout_memleak
Modules: Fix blocked-client-related memory leak
2020-01-29 12:06:14 +01:00
Guy Benoish
d7d13721d3 Modules: Fix blocked-client-related memory leak
If a blocked module client times-out (or disconnects, unblocked
by CLIENT command, etc.) we need to call moduleUnblockClient
in order to free memory allocated by the module sub-system
and blocked-client private data

Other changes:
Made blockedonkeys.tcl tests a bit more aggressive in order
to smoke-out potential memory leaks
2019-12-30 10:10:59 +05:30
Yossi Gottlieb
0c898bdb07 Add REDISMODULE_CTX_FLAGS_MULTI_DIRTY. 2019-12-23 19:43:51 +02:00
Oran Agra
660645c742 modules don't signalModifiedKey in setKey() since that's done (optionally) in RM_CloseKey 2019-12-23 10:15:52 +02:00
antirez
324e22accf Fix ip and missing mode in RM_GetClusterNodeInfo(). 2019-12-20 12:29:02 +01:00
Salvatore Sanfilippo
d3a9dff6b9
Merge pull request #6615 from soloestoy/wrap-also-propagate-as-multi
Wrap also propagate as multi
2019-12-19 09:24:52 +01:00
antirez
c5bc1c14c0 Modules: rewrite top function doc of AvoidReplicaTraffic(). 2019-12-18 17:11:03 +01:00
Salvatore Sanfilippo
ef0b45ece8
Merge pull request #6497 from oranagra/avoid_replica_traffic
Add config and module API for AvoidReplicaTraffic
2019-12-18 17:06:05 +01:00
antirez
5b9d3ac6c6 Avoid changing setKey() API after #6679 fix. 2019-12-18 11:58:02 +01:00
zhaozhao.zz
24044f3356 add a new SET option KEEPTTL that doesn't remove expire time 2019-12-18 15:20:36 +08:00
Salvatore Sanfilippo
b7c78b7651
Merge pull request #5916 from madolson/dev-unstable-acl-module-pr
Add module APIs for custom authentication
2019-12-17 09:58:26 +01:00
Madelyn Olson
576a08908b Split error message so dependandent callers give a useful result 2019-12-16 23:34:37 -08:00
Madelyn Olson
44aa22c635 Added better exception handling around scripting and module 2019-12-16 23:33:53 -08:00
Madelyn Olson
f891b64a83 Tweaking the documentation 2019-12-17 07:28:55 +00:00
Madelyn Olson
034dcf185c Add module APIs for custom authentication 2019-12-17 06:59:59 +00:00
antirez
aa3f13761f Merge branch 'unstable' of github.com:/antirez/redis into unstable 2019-12-16 11:18:20 +01:00
antirez
096592506e Modules: allow to execute RM_Call() from the module init function. 2019-12-13 17:34:37 +01:00
Yossi Gottlieb
0283db5883 Improve RM_ModuleTypeReplaceValue() API.
With the previous API, a NULL return value was ambiguous and could
represent either an old value of NULL or an error condition. The new API
returns a status code and allows the old value to be returned
by-reference.

This commit also includes test coverage based on
tests/modules/datatype.c which did not exist at the time of the original
commit.
2019-12-12 18:50:11 +02:00
antirez
476333d192 Remove useless space from moduleLoad(). 2019-12-12 09:29:10 +01:00
Salvatore Sanfilippo
2968d8e3cb
Merge branch 'unstable' into ModuleSecurity 2019-12-12 09:27:14 +01:00
Oran Agra
b5f3247ca5 Add module API for AvoidReplicaTraffic
This is useful to tell redis and modules to try to avoid doing things that may
increment the replication offset, and should be used when draining a master
and waiting for replicas to be in perfect sync before a failover.
2019-12-12 08:38:09 +02:00
Salvatore Sanfilippo
c6fb9d0963
Merge pull request #6656 from oranagra/leak_rm_load_from_str
fix leak in RM_LoadDataTypeFromString, and save
2019-12-11 11:39:27 +01:00
Oran Agra
a37cca3644 fix leak in RM_LoadDataTypeFromString, and save 2019-12-11 12:35:00 +02:00
Oran Agra
173cca5413 fix possible warning on incomplete struct init 2019-12-10 11:16:13 +02:00
antirez
e4b6070c0f Modules: more clarification about disconnection callback. 2019-12-09 12:15:38 +01:00
antirez
2271cf03f5 Modules: clarify when the disconnection callback is called. 2019-12-05 10:38:18 +01:00
antirez
7b5f4b175b Modules: create timers in contexts without a client. 2019-12-03 16:21:23 +01:00
zhaozhao.zz
2c970532dc Propagation: flag module client as CLIENT_MULTI if needed
in case of nested MULTI/EXEC
2019-11-22 16:20:30 +08:00
zhaozhao.zz
37a10cef02 Propagation: wrap commands in also_propagate array with MULIT/EXEC
Random command like SPOP with count is replicated as
some SREM operations, and store them in also_propagate
array to propagate after the call, but this would break
atomicity.

To keep the command's atomicity, wrap also_propagate
array with MULTI/EXEC.
2019-11-22 15:42:49 +08:00
Salvatore Sanfilippo
64c2508ee3
Merge branch 'unstable' into rm_get_server_info 2019-11-21 10:06:15 +01:00
Salvatore Sanfilippo
f1f259de5b
Merge pull request #6547 from guybe7/module_api_streams
Support streams in general module API functions
2019-11-21 10:03:50 +01:00
antirez
da47d52c79 Recomment PR #6346. 2019-11-21 10:01:49 +01:00
Salvatore Sanfilippo
70469b76d6
Merge branch 'unstable' into expose_zmalloc_capabilities 2019-11-21 09:57:19 +01:00
Salvatore Sanfilippo
e7144fbed8
Merge branch 'unstable' into module-long-double 2019-11-19 12:15:45 +01:00
Salvatore Sanfilippo
e916058f0b
Merge pull request #6557 from oranagra/rm_lru_lfu_revized
rename RN_SetLRUOrLFU -> RM_SetLRU and RN_SetLFU
2019-11-19 11:58:07 +01:00
Salvatore Sanfilippo
06a24b9cfa
Merge pull request #6597 from yossigo/rm-call-extended-errors
Improve RM_Call() errno classification.
2019-11-19 11:41:32 +01:00
Yossi Gottlieb
2d1e893b3e Improve RM_Call() errno classification.
RM_Call() will now use EBADF and ENONET in addition to EINVAL in order
to provide more information about errors (i.e. when return value is
NULL).
2019-11-19 12:10:48 +02:00
Salvatore Sanfilippo
656e40eed2
Merge branch 'unstable' into scan_module_impl 2019-11-19 11:08:02 +01:00
Salvatore Sanfilippo
1530a3cfc6
Merge pull request #6577 from oranagra/module_doc
module docs, missing LOADING flag
2019-11-19 11:02:58 +01:00
Salvatore Sanfilippo
32a8301073
Merge pull request #4076 from yossigo/add_mt_replacevalue
Add RM_ModuleTypeReplaceValue.
2019-11-19 11:00:01 +01:00
Salvatore Sanfilippo
3d89210477
Merge pull request #3383 from yossigo/datatype_load_save
Redis Module API calls to allow re-use of data type RDB save/load.
2019-11-19 10:55:42 +01:00
Oran Agra
c426bbf3a5 Slightly more efficient RM_ReplyWithEmptyString
trimming talk about RESP protocol from API docs (should be independent to that anyway)
2019-11-14 09:46:46 +02:00
Oran Agra
2d30afc45f module docs, missing LOADING flag 2019-11-14 09:09:10 +02:00
Yossi Gottlieb
9c76875f41 Add RM_ModuleTypeReplaceValue.
This is a light-weight replace function, useful for use cases such as
realloc()ing an existing value, etc.  Using RM_ModuleTypeSetValue() in
such cases is wasteful and complex as it attempts to delete the old
value, call its destructor, etc.
2019-11-14 08:50:11 +02:00
Oran Agra
0f8692b464 Add RM_ScanKey to scan hash, set, zset, changes to RM_Scan API
- Adding RM_ScanKey
- Adding tests for RM_ScanKey
- Refactoring RM_Scan API

Changes in RM_Scan
- cleanup in docs and coding convention
- Moving out of experimantal Api
- Adding ctx to scan callback
- Dont use cursor of -1 as an indication of done (can be a valid cursor)
- Set errno when returning 0 for various reasons
- Rename Cursor to ScanCursor
- Test filters key that are not strings, and opens a key if NULL
2019-11-11 16:05:55 +02:00
meir@redislabs.com
11c6ce812a Added scan implementation to module api.
The implementation expose the following new functions:
1. RedisModule_CursorCreate - allow to create a new cursor object for
keys scanning
2. RedisModule_CursorRestart - restart an existing cursor to restart the
scan
3. RedisModule_CursorDestroy - destroy an existing cursor
4. RedisModule_Scan - scan keys

The RedisModule_Scan function gets a cursor object, a callback and void*
(used as user private data).
The callback will be called for each key in the database proving the key
name and the value as RedisModuleKey.
2019-11-11 16:01:41 +02:00
Oran Agra
28c20b4ef9 rename RN_SetLRUOrLFU -> RM_SetLRU and RN_SetLFU
- the API name was odd, separated to two apis one for LRU and one for LFU
- the LRU idle time was in 1 second resolution, which might be ok for RDB
  and RESTORE, but i think modules may need higher resolution
- adding tests for LFU and for handling maxmemory policy mismatch
2019-11-10 09:27:01 +02:00
meir@redislabs.com
e45239e350 fix documentation 2019-11-06 12:26:03 +02:00
Guy Benoish
1833d008b3 Support streams in general module API functions
Fixes GitHub issue #6492
Added stream support in RM_KeyType and RM_ValueLength.
Also moduleDelKeyIfEmpty was updated, even though it has
no effect now (It will be relevant when stream type direct
API will be coded - i.e. RM_StreamAdd)
2019-11-06 15:52:19 +05:30
meir@redislabs.com
c032dc45d1 changed GetUsedMemoryPresentage -> GetUsedMemoryRatio and return value between 0 and 1. 2019-11-06 12:17:09 +02:00
meir@redislabs.com
aded138a59 return value between 0 to 100 instead of 0 to 1. 2019-11-06 11:28:52 +02:00
meir@redislabs.com
089ee5979d expose used memory via redismodule api
The exposed functions:
1. RedisModule_GetUsedMemoryPercentage - return the used memory
2. RedisModue_MallocSize - return for a given pointer, the amount of memory allocated for this pointer
2019-11-06 11:28:52 +02:00
Yossi Gottlieb
5350e7669e Add ModuleDataType to/from string serialization.
Add two new functions that leverage the RedisModuleDataType mechanism
for RDB serialization/deserialization and make it possible to use it
to/from arbitrary strings:

* RM_SaveDataTypeToString()
* RM_LoadDataTypeFromString()
2019-11-04 21:53:59 +02:00