mirror of
https://codeberg.org/redict/redict.git
synced 2025-01-22 16:18:28 -05:00
abc345ad28
### Summary of API additions * `RedisModule_AddPostNotificationJob` - new API to call inside a key space notification (and on more locations in the future) and allow to add a post job as describe above. * New module option, `REDISMODULE_OPTIONS_ALLOW_NESTED_KEYSPACE_NOTIFICATIONS`, allows to disable Redis protection of nested key-space notifications. * `RedisModule_GetModuleOptionsAll` - gets the mask of all supported module options so a module will be able to check if a given option is supported by the current running Redis instance. ### Background The following PR is a proposal of handling write operations inside module key space notifications. After a lot of discussions we came to a conclusion that module should not perform any write operations on key space notification. Some examples of issues that such write operation can cause are describe on the following links: * Bad replication oreder - https://github.com/redis/redis/pull/10969 * Used after free - https://github.com/redis/redis/pull/10969#issuecomment-1223771006 * Used after free - https://github.com/redis/redis/pull/9406#issuecomment-1221684054 There are probably more issues that are yet to be discovered. The underline problem with writing inside key space notification is that the notification runs synchronously, this means that the notification code will be executed in the middle on Redis logic (commands logic, eviction, expire). Redis **do not assume** that the data might change while running the logic and such changes can crash Redis or cause unexpected behaviour. The solution is to state that modules **should not** perform any write command inside key space notification (we can chose whether or not we want to force it). To still cover the use-case where module wants to perform a write operation as a reaction to key space notifications, we introduce a new API , `RedisModule_AddPostNotificationJob`, that allows to register a callback that will be called by Redis when the following conditions hold: * It is safe to perform any write operation. * The job will be called atomically along side the operation that triggers it (in our case, key space notification). Module can use this new API to safely perform any write operation and still achieve atomicity between the notification and the write. Although currently the API is supported on key space notifications, the API is written in a generic way so that in the future we will be able to use it on other places (server events for example). ### Technical Details Whenever a module uses `RedisModule_AddPostNotificationJob` the callback is added to a list of callbacks (called `modulePostExecUnitJobs`) that need to be invoke after the current execution unit ends (whether its a command, eviction, or active expire). In order to trigger those callback atomically with the notification effect, we call those callbacks on `postExecutionUnitOperations` (which was `propagatePendingCommands` before this PR). The new function fires the post jobs and then calls `propagatePendingCommands`. If the callback perform more operations that triggers more key space notifications. Those keys space notifications might register more callbacks. Those callbacks will be added to the end of `modulePostExecUnitJobs` list and will be invoke atomically after the current callback ends. This raises a concerns of entering an infinite loops, we consider infinite loops as a logical bug that need to be fixed in the module, an attempt to protect against infinite loops by halting the execution could result in violation of the feature correctness and so **Redis will make no attempt to protect the module from infinite loops** In addition, currently key space notifications are not nested. Some modules might want to allow nesting key-space notifications. To allow that and keep backward compatibility, we introduce a new module option called `REDISMODULE_OPTIONS_ALLOW_NESTED_KEYSPACE_NOTIFICATIONS`. Setting this option will disable the Redis key-space notifications nesting protection and will pass this responsibility to the module. ### Redis infrastructure This PR promotes the existing `propagatePendingCommands` to an "Execution Unit" concept, which is called after each atomic unit of execution, Co-authored-by: Oran Agra <oran@redislabs.com> Co-authored-by: Yossi Gottlieb <yossigo@gmail.com> Co-authored-by: Madelyn Olson <34459052+madolson@users.noreply.github.com>
205 lines
6.0 KiB
Tcl
205 lines
6.0 KiB
Tcl
set testmodule [file normalize tests/modules/postnotifications.so]
|
|
|
|
tags "modules" {
|
|
start_server [list overrides [list loadmodule "$testmodule"]] {
|
|
|
|
test {Test write on post notification callback} {
|
|
set repl [attach_to_replication_stream]
|
|
|
|
r set string_x 1
|
|
assert_equal {1} [r get string_changed{string_x}]
|
|
assert_equal {1} [r get string_total]
|
|
|
|
r set string_x 2
|
|
assert_equal {2} [r get string_changed{string_x}]
|
|
assert_equal {2} [r get string_total]
|
|
|
|
assert_replication_stream $repl {
|
|
{multi}
|
|
{select *}
|
|
{set string_x 1}
|
|
{incr string_changed{string_x}}
|
|
{incr string_total}
|
|
{exec}
|
|
{multi}
|
|
{set string_x 2}
|
|
{incr string_changed{string_x}}
|
|
{incr string_total}
|
|
{exec}
|
|
}
|
|
close_replication_stream $repl
|
|
}
|
|
|
|
test {Test write on post notification callback from module thread} {
|
|
r flushall
|
|
set repl [attach_to_replication_stream]
|
|
|
|
assert_equal {OK} [r postnotification.async_set]
|
|
assert_equal {1} [r get string_changed{string_x}]
|
|
assert_equal {1} [r get string_total]
|
|
|
|
assert_replication_stream $repl {
|
|
{multi}
|
|
{select *}
|
|
{set string_x 1}
|
|
{incr string_changed{string_x}}
|
|
{incr string_total}
|
|
{exec}
|
|
}
|
|
close_replication_stream $repl
|
|
}
|
|
|
|
test {Test active expire} {
|
|
r flushall
|
|
set repl [attach_to_replication_stream]
|
|
|
|
r set x 1
|
|
r pexpire x 10
|
|
|
|
wait_for_condition 100 50 {
|
|
[r keys expired] == {expired}
|
|
} else {
|
|
puts [r keys *]
|
|
fail "Failed waiting for x to expired"
|
|
}
|
|
|
|
assert_replication_stream $repl {
|
|
{select *}
|
|
{set x 1}
|
|
{pexpireat x *}
|
|
{multi}
|
|
{del x}
|
|
{incr expired}
|
|
{exec}
|
|
}
|
|
close_replication_stream $repl
|
|
}
|
|
|
|
test {Test lazy expire} {
|
|
r flushall
|
|
r DEBUG SET-ACTIVE-EXPIRE 0
|
|
set repl [attach_to_replication_stream]
|
|
|
|
r set x 1
|
|
r pexpire x 1
|
|
after 10
|
|
assert_equal {} [r get x]
|
|
|
|
assert_replication_stream $repl {
|
|
{select *}
|
|
{set x 1}
|
|
{pexpireat x *}
|
|
{multi}
|
|
{del x}
|
|
{incr expired}
|
|
{exec}
|
|
}
|
|
close_replication_stream $repl
|
|
r DEBUG SET-ACTIVE-EXPIRE 1
|
|
} {OK} {needs:debug}
|
|
|
|
test {Test lazy expire inside post job notification} {
|
|
r flushall
|
|
r DEBUG SET-ACTIVE-EXPIRE 0
|
|
set repl [attach_to_replication_stream]
|
|
|
|
r set x 1
|
|
r pexpire x 1
|
|
after 10
|
|
assert_equal {OK} [r set read_x 1]
|
|
|
|
assert_replication_stream $repl {
|
|
{select *}
|
|
{set x 1}
|
|
{pexpireat x *}
|
|
{multi}
|
|
{set read_x 1}
|
|
{del x}
|
|
{incr expired}
|
|
{exec}
|
|
}
|
|
close_replication_stream $repl
|
|
r DEBUG SET-ACTIVE-EXPIRE 1
|
|
} {OK} {needs:debug}
|
|
|
|
test {Test nested keyspace notification} {
|
|
r flushall
|
|
set repl [attach_to_replication_stream]
|
|
|
|
assert_equal {OK} [r set write_sync_write_sync_x 1]
|
|
|
|
assert_replication_stream $repl {
|
|
{multi}
|
|
{select *}
|
|
{set x 1}
|
|
{set write_sync_x 1}
|
|
{set write_sync_write_sync_x 1}
|
|
{exec}
|
|
}
|
|
close_replication_stream $repl
|
|
}
|
|
|
|
test {Test eviction} {
|
|
r flushall
|
|
set repl [attach_to_replication_stream]
|
|
r set x 1
|
|
r config set maxmemory-policy allkeys-random
|
|
r config set maxmemory 1
|
|
|
|
assert_error {OOM *} {r set y 1}
|
|
|
|
assert_replication_stream $repl {
|
|
{select *}
|
|
{set x 1}
|
|
{multi}
|
|
{del x}
|
|
{incr evicted}
|
|
{exec}
|
|
}
|
|
close_replication_stream $repl
|
|
} {} {needs:config-maxmemory}
|
|
}
|
|
}
|
|
|
|
set testmodule2 [file normalize tests/modules/keyspace_events.so]
|
|
|
|
tags "modules" {
|
|
start_server [list overrides [list loadmodule "$testmodule"]] {
|
|
r module load $testmodule2
|
|
test {Test write on post notification callback} {
|
|
set repl [attach_to_replication_stream]
|
|
|
|
r set string_x 1
|
|
assert_equal {1} [r get string_changed{string_x}]
|
|
assert_equal {1} [r get string_total]
|
|
|
|
r set string_x 2
|
|
assert_equal {2} [r get string_changed{string_x}]
|
|
assert_equal {2} [r get string_total]
|
|
|
|
r set string1_x 1
|
|
assert_equal {1} [r get string_changed{string1_x}]
|
|
assert_equal {3} [r get string_total]
|
|
|
|
assert_replication_stream $repl {
|
|
{multi}
|
|
{select *}
|
|
{set string_x 1}
|
|
{incr string_changed{string_x}}
|
|
{incr string_total}
|
|
{exec}
|
|
{multi}
|
|
{set string_x 2}
|
|
{incr string_changed{string_x}}
|
|
{incr string_total}
|
|
{exec}
|
|
{multi}
|
|
{set string1_x 1}
|
|
{incr string_changed{string1_x}}
|
|
{incr string_total}
|
|
{exec}
|
|
}
|
|
close_replication_stream $repl
|
|
}
|
|
}
|
|
} |