Commit Graph

402 Commits

Author SHA1 Message Date
meir@redislabs.com
089ee5979d expose used memory via redismodule api
The exposed functions:
1. RedisModule_GetUsedMemoryPercentage - return the used memory
2. RedisModue_MallocSize - return for a given pointer, the amount of memory allocated for this pointer
2019-11-06 11:28:52 +02:00
Yossi Gottlieb
5350e7669e Add ModuleDataType to/from string serialization.
Add two new functions that leverage the RedisModuleDataType mechanism
for RDB serialization/deserialization and make it possible to use it
to/from arbitrary strings:

* RM_SaveDataTypeToString()
* RM_LoadDataTypeFromString()
2019-11-04 21:53:59 +02:00
artix
a68c19df6c RM_CreateStringFromLongDouble: use new ld2string 'mode' type 2019-11-04 18:49:56 +01:00
artix
60ec2b78b3 Start support for long double in modules
New API:

- RedisModule_StringToLongDouble
- RedisModule_CreateStringFromLongDouble
- RedisModule_ReplyWithLongDouble
2019-11-04 18:49:42 +01:00
Salvatore Sanfilippo
bbdbfabf9e
Merge pull request #6531 from oranagra/rm_save_long_double
Module API for loading and saving long double
2019-11-04 17:33:35 +01:00
Salvatore Sanfilippo
18b62a51e2
Merge branch 'unstable' into module_lru_lfu 2019-11-04 11:04:49 +01:00
Salvatore Sanfilippo
327cb45ed5
Merge pull request #6532 from oranagra/rm_misc_commands
Module API for PUBLISH, FLUSHALL, RANDOMKEY, DBSIZE
2019-11-04 11:02:37 +01:00
Salvatore Sanfilippo
da44c54897
Merge branch 'unstable' into module_hooks 2019-11-04 10:37:06 +01:00
Oran Agra
0423309768 Add RM_ServerInfoGetFieldUnsigned
rename RM_ServerInfoGetFieldNumerical RM_ServerInfoGetFieldSigned
move string2ull to util.c
fix leak in RM_GetServerInfo when duplicate info fields exist
2019-11-04 08:50:29 +02:00
Oran Agra
deebed23e1 Adding RM_ServerInfoGetFieldC 2019-11-04 07:57:52 +02:00
Oran Agra
87332ce524 Module API for PUBLISH, FLUSHALL, RANDOMKEY, DBSIZE 2019-11-04 07:40:52 +02:00
Oran Agra
779aebc91c Module API for loading and saving long double
looks like each platform implements long double differently (different bit count)
so we can't save them as binary, and we also want to avoid creating a new RDB
format version, so we save these are hex strings using "%La".

This commit includes a change in the arguments of ld2string to support this.
as well as tests for coverage and short reads.

coded by @guybe7
2019-11-03 16:42:31 +02:00
Oran Agra
4d580438b0 Add module api for looking into INFO fields
- Add RM_GetServerInfo and friends
- Add auto memory for new opaque struct
- Add tests for new APIs

other minor fixes:
- add const in various char pointers
- requested_section in modulesCollectInfo was actually not sds but char*
- extract new string2d out of getDoubleFromObject for code reuse

Add module API for
2019-11-03 15:02:25 +02:00
antirez
fdaea2a7a7 Modules: fix thread safe context creation crash.
See #6525, this likely creates a NULL deference if the client was
terminated by Redis between the creation of the blocked client and the
creation of the thread safe context.
2019-10-31 18:07:33 +01:00
antirez
1ee195d255 Merge branch 'unstable' of github.com:/antirez/redis into unstable 2019-10-31 17:55:18 +01:00
antirez
825adcf3f5 Modules: block on keys: finish implementing RM_UnblockClient(). 2019-10-31 17:39:05 +01:00
antirez
66f55bc5c1 Modules: block on keys: fix bugs in processing order. 2019-10-31 12:23:55 +01:00
antirez
629081f839 Modules: block on keys: fix the top comments. 2019-10-31 11:43:45 +01:00
antirez
91f4bdc9f9 Modules: block on keys: use a better interface.
Using the is_key_ready() callback plus the reply callback later, creates
different issues AFAIK:

1. More complex API.
2. We need to call the reply callback() ASAP if the is_key_ready()
interface returned success, however the internals do not work in that
way, so when the reply callback is called the setup could be different.
To fix that, there is to break the current design that handles the
unblocked clients asyncrhonously, and run the list ASAP.
2019-10-31 11:35:07 +01:00
antirez
4534960b29 Modules: remove spurious call from moduleHandleBlockedClients().
Now we handle propagation when we free the context.
2019-10-31 10:32:59 +01:00
antirez
215b72c0ba Modules: block on keys: implement the internals. 2019-10-30 10:57:44 +01:00
antirez
fb6110ac20 Modules: block on keys: export APIs. 2019-10-30 10:20:28 +01:00
antirez
3649568ff2 Modules: block on keys functions layout and mechanism. 2019-10-30 10:11:58 +01:00
Oran Agra
51c3ff8d75 Modules hooks: complete missing hooks for the initial set of hooks
* replication hooks: role change, master link status, replica online/offline
* persistence hooks: saving, loading, loading progress
* misc hooks: cron loop, shutdown, module loaded/unloaded
* change the way hooks test work, and add tests for all of the above

startLoading() now gets flag indicating what is loaded.
stopLoading() now gets an indication of success or failure.
adding startSaving() and stopSaving() with similar args and role.
2019-10-29 17:59:09 +02:00
Oran Agra
e978bdf9ef Module API for controlling LRU and LFU, and OpenKey without TOUCH
Some commands would want to open a key without touching it's LRU/LFU
similarly to the OBJECT or DEBUG command do.

Other commands may want to implement logic similar to what RESTORE
does (and in the future MIGRATE) and get/set the LRU or LFU.
2019-10-29 15:03:05 +02:00
Salvatore Sanfilippo
764b420f5f
Merge pull request #6487 from oranagra/module_modified_key
Module API for explicit SignalModifiedKey instead of implicit one.
2019-10-29 10:40:22 +01:00
Salvatore Sanfilippo
90169cdf16
Merge pull request #6495 from oranagra/rm_call_argv
Module api RM_CallArgv and test
2019-10-29 10:07:05 +01:00
Oran Agra
0399b5a27e Module api tests for RM_Call
Adding a test for coverage for RM_Call in a new "misc" unit
to be used for various short simple tests

also solves compilation warnings in redismodule.h and fork.c
2019-10-28 12:39:57 +02:00
Guy Benoish
cee6dead3c Modules: Allow notifying custom keyspace events
Also, add an API for getting server.notify_keyspace_events

Other (unrelated) changes:
Add RM_GetKeynameFromModuleKey
2019-10-28 15:53:35 +05:30
Oran Agra
a12f07792f Module API for explicit SignalModifiedKey instead of implicit one.
This commit also fixes an uninitialized module struct member (that luckily never got released)
2019-10-28 12:09:25 +02:00
Salvatore Sanfilippo
5aab012676
Merge pull request #6498 from oranagra/rm_latency_add_sample
Module API for LatencyAddSample
2019-10-28 09:59:26 +01:00
Salvatore Sanfilippo
d070221707
Merge pull request #6505 from swilly22/replaywith-module-API-additions
Introduce ReplyWithVerbatimString, ReplyWithEmptyArray, ReplyWithNull…
2019-10-28 09:56:56 +01:00
swilly22
56a7c45521 Introduce ReplyWithVerbatimString, ReplyWithEmptyArray, ReplyWithNullArray and ReplyWithEmptyString to redis module API 2019-10-28 08:50:25 +02:00
zhaozhao.zz
c74398e130 Modules: make unloading module more safe
As we know if a module exports module-side data types,
unload it is not allowed. This rule is the same with
blocked clients in module, because we use background
threads to implement module blocked clients, and it's
not safe to unload a module if there are background
threads running. So it's necessary to check if any
blocked clients running in this module when unload it.

Moreover, after that we can ensure that if no modules,
then no module blocked clients even module unloaded.
So, we can call moduleHandleBlockedClients only when
we have installed modules.
2019-10-28 11:31:58 +08:00
Oran Agra
e3b1d6d3ad Module API for LatencyAddSample 2019-10-24 14:24:55 +03:00
antirez
6e98214f74 Modules hooks: test flush event. 2019-10-24 10:51:03 +02:00
antirez
7cefbeaad5 Modules hooks: select DB zero by default. 2019-10-23 18:51:44 +02:00
antirez
61ac2aef2b Modules hooks: for nested calls, create new fake clients.
We can't use the same client at the same time when re-entering the hook.
2019-10-23 18:39:53 +02:00
antirez
83de18805c Modules hooks: do not re-enter in hooks.
Calling a module hook callback may result in callback operations in turn
triggering other events the module is subscribed too. We don't want to
trigger those, it's unsafe and quite confusing, and to do it correcly we
would need to maintain an event list: quite a more complex
implementation.
2019-10-23 18:39:53 +02:00
antirez
89af7e4c85 Modules hooks: select the right DB for FLUSHDB events. 2019-10-23 18:39:53 +02:00
antirez
bc1ef48e56 Modules hooks: FLUSHDB event example. 2019-10-23 18:39:53 +02:00
antirez
ed833c9f99 Modules hooks: implement the FLUSHDB event. 2019-10-23 18:39:53 +02:00
antirez
8ec2fc3969 Modules hooks: unify structures definitions. 2019-10-23 18:39:53 +02:00
antirez
db8c9a8520 Modules hooks: document what yet to implement WIP 2. 2019-10-23 18:39:53 +02:00
antirez
18d1607f5e Modules hooks: document what yet to implement WIP 1. 2019-10-23 18:39:53 +02:00
antirez
cf52e0ad44 Module hooks: fixes and an example module. 2019-10-23 18:39:53 +02:00
antirez
019ac37e9c Modules hooks: fix define / linker issues. Implement one test event. 2019-10-23 18:39:53 +02:00
antirez
d54652682d Modules hooks: handle module unloading + API export. 2019-10-23 18:39:53 +02:00
antirez
1e78681df8 Modules hooks: fix a leak and a few more issues. 2019-10-23 18:39:53 +02:00
antirez
b9af7e24e4 Modules hooks: event firing logic. 2019-10-23 18:39:53 +02:00
antirez
6e56f513b4 Modules: refactoring of RM_GetClientInfoById(). 2019-10-23 18:39:53 +02:00
antirez
21d8544128 Modules: fix RedisModule_GetClientInfoById() example. 2019-10-23 18:39:53 +02:00
antirez
e9b5484ddd Modules: export GetClientInfoById(). 2019-10-23 18:39:53 +02:00
antirez
6ea8641ae0 Modules: add the client ID to the client info structure. 2019-10-23 18:39:53 +02:00
antirez
d2d74ca0f3 Modules: RM_GetClientInfoById() draft. 2019-10-23 18:39:53 +02:00
antirez
bebb2e1994 Modules hooks: a first version of events and some API. 2019-10-23 18:39:53 +02:00
antirez
3170f63330 Modules hooks: initial design of data structures. 2019-10-23 18:39:53 +02:00
antirez
ed2ba31e4c Modules: fix moduleCreateArgvFromUserFormat() casting bug.
In 32 bit systems casting to "long" will cut the result to 32 bit.
2019-10-23 18:39:14 +02:00
antirez
673c9d7029 module: fix propagation API bug. 2019-10-17 18:28:34 +02:00
antirez
97f6e314c1 Modules: allow to check for AOF loading client. 2019-10-17 18:10:50 +02:00
antirez
8651e6a10d Modules: add new flags to context, replica state + more. 2019-10-17 17:49:33 +02:00
Yossi Gottlieb
0db3b0a0ff Merge remote-tracking branch 'upstream/unstable' into tls 2019-10-16 17:08:07 +03:00
Salvatore Sanfilippo
b8e02f2b40
Merge pull request #5926 from JimB123/unstable
Addition of RedisModule_OnUnload hook
2019-10-16 11:13:12 +02:00
Yossi Gottlieb
b087dd1db6 TLS: Connections refactoring and TLS support.
* Introduce a connection abstraction layer for all socket operations and
integrate it across the code base.
* Provide an optional TLS connections implementation based on OpenSSL.
* Pull a newer version of hiredis with TLS support.
* Tests, redis-cli updates for TLS support.
2019-10-07 21:06:13 +03:00
antirez
3eaff2941e Modules: RM_Call(): give pointer to documentation. 2019-10-04 11:46:53 +02:00
antirez
2a81e49dde Modules: RM_Call/Replicate() ability to exclude AOF/replicas. 2019-10-04 11:44:53 +02:00
antirez
1bca62c4b7 Modules: RM_Replicate() in thread safe contexts. 2019-10-03 11:03:46 +02:00
antirez
e938bbc543 Modules: implement RM_Replicate() from async callbacks. 2019-10-03 10:56:37 +02:00
antirez
c549513acd Modules: handle propagation when ctx is freed. Flag modules commands ctx. 2019-10-02 18:33:40 +02:00
Oran Agra
1b4f888109 Use sdscatfmt instead of sdscatprintf in module info
sdscatfmt is faster
2019-09-30 21:13:13 +03:00
Oran Agra
d5c14c70b7 Merge remote-tracking branch 'antirez/unstable' into modules_info 2019-09-30 20:47:35 +03:00
antirez
23f5cb4233 Change a bit the style of #6385. 2019-09-30 10:58:15 +02:00
Salvatore Sanfilippo
ad45d7e407
Merge pull request #6385 from filipecosta90/perf-reply-ss-error
Improve performance of RM_ReplyWithSimpleString and RM_ReplyWi…
2019-09-30 10:57:05 +02:00
antirez
916c4e5d86 moduleRDBLoadError(): io->ctx may be NULL.
The correct way to access the module about a given IO context is to
deference io->type->module, since io->ctx is only populated if the user
requests an explicit context from an IO object.
2019-09-27 18:42:38 +02:00
antirez
8b5848a4f5 Fix memory leak in moduleLoadFromQueue(). 2019-09-27 18:33:21 +02:00
antirez
b394817754 Fix memory leak in RM_UnregisterCommandFilter(). 2019-09-27 18:33:05 +02:00
antirez
0a07f8ffee TerminateModuleForkChild(): fix function prototype. 2019-09-27 12:23:07 +02:00
antirez
721d3c9e0c TerminateModuleForkChild(): move safety checks there.
We don't want that the API could be used directly in an unsafe way,
without checking if there is an active child. Now the safety checks are
moved directly in the function performing the operations.
2019-09-27 12:17:47 +02:00
antirez
de1f82aa33 Function renamed hasForkChild() -> hasActiveChildProcess(). 2019-09-27 12:03:09 +02:00
antirez
82845f8d04 TerminateModuleForkChild(): use wait4 for safety.
In theory currently there is only one active child, but the API may
change or for bugs in the implementation we may have several (it was
like that for years because of a bug). Better to wait for a specific
pid and avoid consuing other pending children information.
2019-09-27 11:59:58 +02:00
antirez
ae3ef964c1 Modules fork: improve SIGUSR1 handling, fix include.
We can't expect SIGUSR1 to have any specific value range, so let's
define an exit code that we can handle in a special way.
This also fixes an #include <wait.h> that is not standard.
2019-09-27 11:39:45 +02:00
Salvatore Sanfilippo
6129758558
Merge branch 'unstable' into modules_fork 2019-09-27 11:24:06 +02:00
antirez
b7b23bdfb8 INFO: more info about loaded modules.
Related to #6024.
2019-09-26 12:18:55 +02:00
Salvatore Sanfilippo
c1ea6175c5
Merge pull request #6024 from itamarhaber/info_modules
Adds a "Modules" section to `INFO`
2019-09-26 11:58:52 +02:00
Salvatore Sanfilippo
959fb5cf68
Merge pull request #6235 from oranagra/module_rdb_load_errors
Allow modules to handle RDB loading errors.
2019-09-26 11:52:42 +02:00
Salvatore Sanfilippo
b0a90d8fa8
Merge pull request #4980 from yossigo/rm-assert
Add RedisModule_Assert() API call.
2019-09-26 11:39:29 +02:00
filipecosta90
af15b285fa [fix] fixed the un-refactor bug. 2019-09-25 17:28:42 +01:00
filipecosta90
733280d9cb [fix] un-refactor the code. [perf] replyWithStatus now makes usage of addReplyProto 2019-09-23 23:45:31 +01:00
filipecosta90
4a30a26f8f [add] improved performance of RM_ReplyWithSimpleString and RM_ReplyWithError by making usage addReplyProto instead of addReplySds 2019-09-15 21:16:30 +01:00
Doug Nelson
cc8f06ece9 Typo fixes in API documentation 2019-09-05 16:25:06 +01:00
Oran Agra
78bbb9b58d Modlue fork is killed when the parent exists 2019-08-25 10:11:48 +03:00
Oran Agra
61853ad8de Module INFO, support default section for simple modules 2019-08-18 10:01:57 +03:00
Oran Agra
1d6e5dc4dc Module INFO, add support for dict fields, rename API to have common prefix 2019-08-18 09:41:45 +03:00
Oran Agra
0a97149dec RM_ReplyWithCString was missing registration 2019-08-07 13:06:11 +03:00
Oran Agra
e5187ad2ae Merge remote-tracking branch 'oss/unstable' into module_rdb_load_errors 2019-07-30 15:14:08 +03:00
Oran Agra
4339706e07 Avoid diskelss-load if modules did not declare they handle read errors 2019-07-30 15:11:57 +03:00
Salvatore Sanfilippo
f42846e8c7
Merge pull request #6264 from oranagra/modules_api_aux_rdb
Implement module api for aux data in rdb
2019-07-30 11:50:44 +02:00
Oran Agra
e91d9a6fff Extend modules API to allow modules report to redis INFO
this implements #6012
2019-07-24 12:58:15 +03:00
Oran Agra
3b6aeea44c Implement module api for aux data in rdb
Other changes:
* fix memory leak in error handling of rdb loading of type OBJ_MODULE
2019-07-22 21:15:33 +03:00
Oran Agra
d7d028a7a7 Allow modules to handle RDB loading errors.
This is especially needed in diskless loading, were a short read could have
caused redis to exit. now the module can handle the error and return to the
caller gracefully.

this fixes #5326
2019-07-21 18:19:32 +03:00