redict/tests/unit/moduleapi
Meir Shpilraien (Spielrein) 2237131e15
Unified Lua and modules reply parsing and added RESP3 support to RM_Call (#9202)
## Current state
1. Lua has its own parser that handles parsing `reds.call` replies and translates them
  to Lua objects that can be used by the user Lua code. The parser partially handles
  resp3 (missing big number, verbatim, attribute, ...)
2. Modules have their own parser that handles parsing `RM_Call` replies and translates
  them to RedisModuleCallReply objects. The parser does not support resp3.

In addition, in the future, we want to add Redis Function (#8693) that will probably
support more languages. At some point maintaining so many parsers will stop
scaling (bug fixes and protocol changes will need to be applied on all of them).
We will probably end up with different parsers that support different parts of the
resp protocol (like we already have today with Lua and modules)

## PR Changes
This PR attempt to unified the reply parsing of Lua and modules (and in the future
Redis Function) by introducing a new parser unit (`resp_parser.c`). The new parser
handles parsing the reply and calls different callbacks to allow the users (another
unit that uses the parser, i.e, Lua, modules, or Redis Function) to analyze the reply.

### Lua API Additions
The code that handles reply parsing on `scripting.c` was removed. Instead, it uses
the resp_parser to parse and create a Lua object out of the reply. As mentioned
above the Lua parser did not handle parsing big numbers, verbatim, and attribute.
The new parser can handle those and so Lua also gets it for free.
Those are translated to Lua objects in the following way:
1. Big Number - Lua table `{'big_number':'<str representation for big number>'}`
2. Verbatim - Lua table `{'verbatim_string':{'format':'<verbatim format>', 'string':'<verbatim string value>'}}`
3. Attribute - currently ignored and not expose to the Lua parser, another issue will be open to decide how to expose it.

Tests were added to check resp3 reply parsing on Lua

### Modules API Additions
The reply parsing code on `module.c` was also removed and the new resp_parser is used instead.
In addition, the RedisModuleCallReply was also extracted to a separate unit located on `call_reply.c`
(in the future, this unit will also be used by Redis Function). A nice side effect of unified parsing is
that modules now also support resp3. Resp3 can be enabled by giving `3` as a parameter to the
fmt argument of `RM_Call`. It is also possible to give `0`, which will indicate an auto mode. i.e, Redis
will automatically chose the reply protocol base on the current client set on the RedisModuleCtx
(this mode will mostly be used when the module want to pass the reply to the client as is).
In addition, the following RedisModuleAPI were added to allow analyzing resp3 replies:

* New RedisModuleCallReply types:
   * `REDISMODULE_REPLY_MAP`
   * `REDISMODULE_REPLY_SET`
   * `REDISMODULE_REPLY_BOOL`
   * `REDISMODULE_REPLY_DOUBLE`
   * `REDISMODULE_REPLY_BIG_NUMBER`
   * `REDISMODULE_REPLY_VERBATIM_STRING`
   * `REDISMODULE_REPLY_ATTRIBUTE`

* New RedisModuleAPI:
   * `RedisModule_CallReplyDouble` - getting double value from resp3 double reply
   * `RedisModule_CallReplyBool` - getting boolean value from resp3 boolean reply
   * `RedisModule_CallReplyBigNumber` - getting big number value from resp3 big number reply
   * `RedisModule_CallReplyVerbatim` - getting format and value from resp3 verbatim reply
   * `RedisModule_CallReplySetElement` - getting element from resp3 set reply
   * `RedisModule_CallReplyMapElement` - getting key and value from resp3 map reply
   * `RedisModule_CallReplyAttribute` - getting a reply attribute
   * `RedisModule_CallReplyAttributeElement` - getting key and value from resp3 attribute reply
   
* New context flags:
   * `REDISMODULE_CTX_FLAGS_RESP3` - indicate that the client is using resp3

Tests were added to check the new RedisModuleAPI

### Modules API Changes
* RM_ReplyWithCallReply might return REDISMODULE_ERR if the given CallReply is in resp3
  but the client expects resp2. This is not a breaking change because in order to get a resp3
  CallReply one needs to specifically specify `3` as a parameter to the fmt argument of
  `RM_Call` (as mentioned above).

Tests were added to check this change

### More small Additions
* Added `debug set-disable-deny-scripts` that allows to turn on and off the commands no-script
flag protection. This is used by the Lua resp3 tests so it will be possible to run `debug protocol`
and check the resp3 parsing code.

Co-authored-by: Oran Agra <oran@redislabs.com>
Co-authored-by: Yossi Gottlieb <yossigo@gmail.com>
2021-08-04 16:28:07 +03:00
..
auth.tcl Fixed some typos, add a spell check ci and others minor fix (#8890) 2021-06-10 15:39:33 +03:00
basics.tcl Unified Lua and modules reply parsing and added RESP3 support to RM_Call (#9202) 2021-08-04 16:28:07 +03:00
blockedclient.tcl Handle output buffer limits for Module blocked clients (#8141) 2020-12-08 16:41:20 +02:00
blockonbackground.tcl Fix CLIENT UNBLOCK crashing modules. (#9167) 2021-07-01 17:11:27 +03:00
blockonkeys.tcl Test that module can wake up module blocked on non-empty list key (#8382) 2021-01-22 16:19:37 +02:00
commandfilter.tcl Remove const from CommandFilterArgGet result (#9247) 2021-08-01 11:29:32 +03:00
datatype2.tcl Enhance mem_usage/free_effort/unlink/copy callbacks and add GetDbFromIO api. (#8999) 2021-06-16 09:45:49 +03:00
datatype.tcl Add module data-type support for COPY. (#8112) 2020-12-09 20:22:45 +02:00
defrag.tcl Tests: fix new defrag test to be skipped when not supported (#8185) 2020-12-14 11:13:46 +02:00
fork.tcl tests: fix exec fails when grep exists with status other than 0 (#9066) 2021-07-25 09:58:21 +03:00
getkeys.tcl Modules: add RM_GetCommandKeys(). 2020-10-11 16:04:14 +03:00
hash.tcl Modules: In RM_HashSet, add COUNT_ALL flag and set errno (#8446) 2021-02-15 11:40:05 +02:00
hooks.tcl Add Swapdb Module Event (#7804) 2020-09-20 13:36:20 +03:00
infotest.tcl Escape unsafe field name characters in INFO. (#8492) 2021-02-15 17:08:53 +02:00
infra.tcl add test for modules load/unload and config rewrite 2021-06-01 13:43:48 +03:00
keyspace_events.tcl Modules: adding a module type for key space notification (#8759) 2021-04-19 21:33:26 +03:00
misc.tcl Fixed some typos, add a spell check ci and others minor fix (#8890) 2021-06-10 15:39:33 +03:00
propagate.tcl Modules: Replicate lazy-expire even if replication is not allowed (#8816) 2021-04-19 17:16:02 +03:00
reply.tcl Module api support for RESP3 (#8521) 2021-08-03 11:37:19 +03:00
scan.tcl Fix RM_ScanKey module api not to return int encoded strings 2020-06-14 11:20:15 +03:00
stream.tcl Add modules API for streams (#8288) 2021-01-28 16:19:43 +02:00
test_lazyfree.tcl Modules callbacks for lazy free effort, and unlink (#7912) 2020-11-16 10:34:04 +02:00
testrdb.tcl test infra - reduce disk space usage 2020-09-06 09:59:19 +03:00
timer.tcl Add timer module API tests. (#8041) 2020-11-11 22:57:33 +02:00
zset.tcl RM_ZsetRem: Delete key if empty (#8453) 2021-02-05 19:54:01 +02:00