redict/tests/unit/moduleapi/propagate.tcl

244 lines
8.9 KiB
Tcl
Raw Normal View History

set testmodule [file normalize tests/modules/propagate.so]
tags "modules" {
test {Modules can propagate in async and threaded contexts} {
start_server {} {
set replica [srv 0 client]
set replica_host [srv 0 host]
set replica_port [srv 0 port]
start_server [list overrides [list loadmodule "$testmodule"]] {
set master [srv 0 client]
set master_host [srv 0 host]
set master_port [srv 0 port]
# Start the replication process...
$replica replicaof $master_host $master_port
wait_for_sync $replica
after 1000
Remove read-only flag from non-keyspace cmds, different approach for EXEC to propagate MULTI (#8216) In the distant history there was only the read flag for commands, and whatever command that didn't have the read flag was a write one. Then we added the write flag, but some portions of the code still used !read Also some commands that don't work on the keyspace at all, still have the read flag. Changes in this commit: 1. remove the read-only flag from TIME, ECHO, ROLE and LASTSAVE 2. EXEC command used to decides if it should propagate a MULTI by looking at the command flags (!read & !admin). When i was about to change it to look at the write flag instead, i realized that this would cause it not to propagate a MULTI for PUBLISH, EVAL, and SCRIPT, all 3 are not marked as either a read command or a write one (as they should), but all 3 are calling forceCommandPropagation. So instead of introducing a new flag to denote a command that "writes" but not into the keyspace, and still needs propagation, i decided to rely on the forceCommandPropagation, and just fix the code to propagate MULTI when needed rather than depending on the command flags at all. The implication of my change then is that now it won't decide to propagate MULTI when it sees one of these: SELECT, PING, INFO, COMMAND, TIME and other commands which are neither read nor write. 3. Changing getNodeByQuery and clusterRedirectBlockedClientIfNeeded in cluster.c to look at !write rather than read flag. This should have no implications, since these code paths are only reachable for commands which access keys, and these are always marked as either read or write. This commit improve MULTI propagation tests, for modules and a bunch of other special cases, all of which used to pass already before that commit. the only one that test change that uncovered a change of behavior is the one that DELs a non-existing key, it used to propagate an empty multi-exec block, and no longer does.
2020-12-22 05:03:49 -05:00
test {module propagates from timer} {
set repl [attach_to_replication_stream]
$master propagate-test.timer
wait_for_condition 5000 10 {
[$replica get timer] eq "3"
} else {
fail "The two counters don't match the expected value."
}
assert_replication_stream $repl {
{select *}
{multi}
{incr timer}
{exec}
{multi}
{incr timer}
{exec}
{multi}
{incr timer}
{exec}
}
close_replication_stream $repl
}
Fix some issues with modules and MULTI/EXEC (#8617) Bug 1: When a module ctx is freed moduleHandlePropagationAfterCommandCallback is called and handles propagation. We want to prevent it from propagating commands that were not replicated by the same context. Example: 1. module1.foo does: RM_Replicate(cmd1); RM_Call(cmd2); RM_Replicate(cmd3) 2. RM_Replicate(cmd1) propagates MULTI and adds cmd1 to also_propagagte 3. RM_Call(cmd2) create a new ctx, calls call() and destroys the ctx. 4. moduleHandlePropagationAfterCommandCallback is called, calling alsoPropagates EXEC (Note: EXEC is still not written to socket), setting server.in_trnsaction = 0 5. RM_Replicate(cmd3) is called, propagagting yet another MULTI (now we have nested MULTI calls, which is no good) and then cmd3 We must prevent RM_Call(cmd2) from resetting server.in_transaction. REDISMODULE_CTX_MULTI_EMITTED was revived for that purpose. Bug 2: Fix issues with nested RM_Call where some have '!' and some don't. Example: 1. module1.foo does RM_Call of module2.bar without replication (i.e. no '!') 2. module2.bar internally calls RM_Call of INCR with '!' 3. at the end of module1.foo we call RM_ReplicateVerbatim We want the replica/AOF to see only module1.foo and not the INCR from module2.bar Introduced a global replication_allowed flag inside RM_Call to determine whether we need to replicate or not (even if '!' was specified) Other changes: Split beforePropagateMultiOrExec to beforePropagateMulti afterPropagateExec just for better readability
2021-03-10 11:02:17 -05:00
test {module propagates nested ctx case1} {
set repl [attach_to_replication_stream]
$master del timer-nested-start
$master del timer-nested-end
$master propagate-test.timer-nested
wait_for_condition 5000 10 {
[$replica get timer-nested-end] eq "1"
} else {
fail "The two counters don't match the expected value."
}
assert_replication_stream $repl {
{select *}
{multi}
{incrby timer-nested-start 1}
{incrby timer-nested-end 1}
{exec}
}
close_replication_stream $repl
}
test {module propagates nested ctx case2} {
set repl [attach_to_replication_stream]
$master del timer-nested-start
$master del timer-nested-end
$master propagate-test.timer-nested-repl
wait_for_condition 5000 10 {
[$replica get timer-nested-end] eq "1"
} else {
fail "The two counters don't match the expected value."
}
# Note the 'after-call' and 'timer-nested-start' propagation below is out of order (known limitation)
assert_replication_stream $repl {
{select *}
{multi}
{incr using-call}
{incr counter-1}
{incr counter-2}
{incr after-call}
{incr counter-3}
{incr counter-4}
{incrby timer-nested-start 1}
{incrby timer-nested-end 1}
{exec}
}
close_replication_stream $repl
}
Remove read-only flag from non-keyspace cmds, different approach for EXEC to propagate MULTI (#8216) In the distant history there was only the read flag for commands, and whatever command that didn't have the read flag was a write one. Then we added the write flag, but some portions of the code still used !read Also some commands that don't work on the keyspace at all, still have the read flag. Changes in this commit: 1. remove the read-only flag from TIME, ECHO, ROLE and LASTSAVE 2. EXEC command used to decides if it should propagate a MULTI by looking at the command flags (!read & !admin). When i was about to change it to look at the write flag instead, i realized that this would cause it not to propagate a MULTI for PUBLISH, EVAL, and SCRIPT, all 3 are not marked as either a read command or a write one (as they should), but all 3 are calling forceCommandPropagation. So instead of introducing a new flag to denote a command that "writes" but not into the keyspace, and still needs propagation, i decided to rely on the forceCommandPropagation, and just fix the code to propagate MULTI when needed rather than depending on the command flags at all. The implication of my change then is that now it won't decide to propagate MULTI when it sees one of these: SELECT, PING, INFO, COMMAND, TIME and other commands which are neither read nor write. 3. Changing getNodeByQuery and clusterRedirectBlockedClientIfNeeded in cluster.c to look at !write rather than read flag. This should have no implications, since these code paths are only reachable for commands which access keys, and these are always marked as either read or write. This commit improve MULTI propagation tests, for modules and a bunch of other special cases, all of which used to pass already before that commit. the only one that test change that uncovered a change of behavior is the one that DELs a non-existing key, it used to propagate an empty multi-exec block, and no longer does.
2020-12-22 05:03:49 -05:00
test {module propagates from thread} {
set repl [attach_to_replication_stream]
$master propagate-test.thread
wait_for_condition 5000 10 {
[$replica get a-from-thread] eq "3"
} else {
fail "The two counters don't match the expected value."
}
assert_replication_stream $repl {
{select *}
{incr a-from-thread}
{incr b-from-thread}
{incr a-from-thread}
{incr b-from-thread}
{incr a-from-thread}
{incr b-from-thread}
}
close_replication_stream $repl
}
Remove read-only flag from non-keyspace cmds, different approach for EXEC to propagate MULTI (#8216) In the distant history there was only the read flag for commands, and whatever command that didn't have the read flag was a write one. Then we added the write flag, but some portions of the code still used !read Also some commands that don't work on the keyspace at all, still have the read flag. Changes in this commit: 1. remove the read-only flag from TIME, ECHO, ROLE and LASTSAVE 2. EXEC command used to decides if it should propagate a MULTI by looking at the command flags (!read & !admin). When i was about to change it to look at the write flag instead, i realized that this would cause it not to propagate a MULTI for PUBLISH, EVAL, and SCRIPT, all 3 are not marked as either a read command or a write one (as they should), but all 3 are calling forceCommandPropagation. So instead of introducing a new flag to denote a command that "writes" but not into the keyspace, and still needs propagation, i decided to rely on the forceCommandPropagation, and just fix the code to propagate MULTI when needed rather than depending on the command flags at all. The implication of my change then is that now it won't decide to propagate MULTI when it sees one of these: SELECT, PING, INFO, COMMAND, TIME and other commands which are neither read nor write. 3. Changing getNodeByQuery and clusterRedirectBlockedClientIfNeeded in cluster.c to look at !write rather than read flag. This should have no implications, since these code paths are only reachable for commands which access keys, and these are always marked as either read or write. This commit improve MULTI propagation tests, for modules and a bunch of other special cases, all of which used to pass already before that commit. the only one that test change that uncovered a change of behavior is the one that DELs a non-existing key, it used to propagate an empty multi-exec block, and no longer does.
2020-12-22 05:03:49 -05:00
test {module propagates from from command} {
set repl [attach_to_replication_stream]
$master propagate-test.simple
$master propagate-test.mixed
# Note the 'after-call' propagation below is out of order (known limitation)
assert_replication_stream $repl {
{select *}
{multi}
{incr counter-1}
{incr counter-2}
{exec}
{multi}
{incr using-call}
{incr after-call}
{incr counter-1}
{incr counter-2}
{exec}
}
close_replication_stream $repl
}
test {module propagates from from command after good EVAL} {
set repl [attach_to_replication_stream]
assert_equal [ $master eval { return "hello" } 0 ] {hello}
$master propagate-test.simple
$master propagate-test.mixed
# Note the 'after-call' propagation below is out of order (known limitation)
assert_replication_stream $repl {
{select *}
{multi}
{incr counter-1}
{incr counter-2}
{exec}
{multi}
{incr using-call}
{incr after-call}
{incr counter-1}
{incr counter-2}
{exec}
}
close_replication_stream $repl
}
test {module propagates from from command after bad EVAL} {
set repl [attach_to_replication_stream]
catch { $master eval { return "hello" } -12 } e
assert_equal $e {ERR Number of keys can't be negative}
$master propagate-test.simple
$master propagate-test.mixed
# Note the 'after-call' propagation below is out of order (known limitation)
assert_replication_stream $repl {
{select *}
{multi}
{incr counter-1}
{incr counter-2}
Remove read-only flag from non-keyspace cmds, different approach for EXEC to propagate MULTI (#8216) In the distant history there was only the read flag for commands, and whatever command that didn't have the read flag was a write one. Then we added the write flag, but some portions of the code still used !read Also some commands that don't work on the keyspace at all, still have the read flag. Changes in this commit: 1. remove the read-only flag from TIME, ECHO, ROLE and LASTSAVE 2. EXEC command used to decides if it should propagate a MULTI by looking at the command flags (!read & !admin). When i was about to change it to look at the write flag instead, i realized that this would cause it not to propagate a MULTI for PUBLISH, EVAL, and SCRIPT, all 3 are not marked as either a read command or a write one (as they should), but all 3 are calling forceCommandPropagation. So instead of introducing a new flag to denote a command that "writes" but not into the keyspace, and still needs propagation, i decided to rely on the forceCommandPropagation, and just fix the code to propagate MULTI when needed rather than depending on the command flags at all. The implication of my change then is that now it won't decide to propagate MULTI when it sees one of these: SELECT, PING, INFO, COMMAND, TIME and other commands which are neither read nor write. 3. Changing getNodeByQuery and clusterRedirectBlockedClientIfNeeded in cluster.c to look at !write rather than read flag. This should have no implications, since these code paths are only reachable for commands which access keys, and these are always marked as either read or write. This commit improve MULTI propagation tests, for modules and a bunch of other special cases, all of which used to pass already before that commit. the only one that test change that uncovered a change of behavior is the one that DELs a non-existing key, it used to propagate an empty multi-exec block, and no longer does.
2020-12-22 05:03:49 -05:00
{exec}
{multi}
{incr using-call}
{incr after-call}
{incr counter-1}
{incr counter-2}
{exec}
}
close_replication_stream $repl
}
test {module propagates from from multi-exec} {
set repl [attach_to_replication_stream]
$master multi
$master propagate-test.simple
$master propagate-test.mixed
$master exec
wait_for_ofs_sync $master $replica
# Note the 'after-call' propagation below is out of order (known limitation)
assert_replication_stream $repl {
{select *}
{multi}
{incr counter-1}
{incr counter-2}
{incr using-call}
{incr after-call}
{incr counter-1}
{incr counter-2}
{exec}
}
close_replication_stream $repl
}
assert_equal [s -1 unexpected_error_replies] 0
}
}
}
}
tags "modules aof" {
test {Modules RM_Replicate replicates MULTI/EXEC correctly} {
start_server [list overrides [list loadmodule "$testmodule"]] {
# Enable the AOF
r config set appendonly yes
r config set auto-aof-rewrite-percentage 0 ; # Disable auto-rewrite.
waitForBgrewriteaof r
Remove read-only flag from non-keyspace cmds, different approach for EXEC to propagate MULTI (#8216) In the distant history there was only the read flag for commands, and whatever command that didn't have the read flag was a write one. Then we added the write flag, but some portions of the code still used !read Also some commands that don't work on the keyspace at all, still have the read flag. Changes in this commit: 1. remove the read-only flag from TIME, ECHO, ROLE and LASTSAVE 2. EXEC command used to decides if it should propagate a MULTI by looking at the command flags (!read & !admin). When i was about to change it to look at the write flag instead, i realized that this would cause it not to propagate a MULTI for PUBLISH, EVAL, and SCRIPT, all 3 are not marked as either a read command or a write one (as they should), but all 3 are calling forceCommandPropagation. So instead of introducing a new flag to denote a command that "writes" but not into the keyspace, and still needs propagation, i decided to rely on the forceCommandPropagation, and just fix the code to propagate MULTI when needed rather than depending on the command flags at all. The implication of my change then is that now it won't decide to propagate MULTI when it sees one of these: SELECT, PING, INFO, COMMAND, TIME and other commands which are neither read nor write. 3. Changing getNodeByQuery and clusterRedirectBlockedClientIfNeeded in cluster.c to look at !write rather than read flag. This should have no implications, since these code paths are only reachable for commands which access keys, and these are always marked as either read or write. This commit improve MULTI propagation tests, for modules and a bunch of other special cases, all of which used to pass already before that commit. the only one that test change that uncovered a change of behavior is the one that DELs a non-existing key, it used to propagate an empty multi-exec block, and no longer does.
2020-12-22 05:03:49 -05:00
r propagate-test.simple
r propagate-test.mixed
r multi
Remove read-only flag from non-keyspace cmds, different approach for EXEC to propagate MULTI (#8216) In the distant history there was only the read flag for commands, and whatever command that didn't have the read flag was a write one. Then we added the write flag, but some portions of the code still used !read Also some commands that don't work on the keyspace at all, still have the read flag. Changes in this commit: 1. remove the read-only flag from TIME, ECHO, ROLE and LASTSAVE 2. EXEC command used to decides if it should propagate a MULTI by looking at the command flags (!read & !admin). When i was about to change it to look at the write flag instead, i realized that this would cause it not to propagate a MULTI for PUBLISH, EVAL, and SCRIPT, all 3 are not marked as either a read command or a write one (as they should), but all 3 are calling forceCommandPropagation. So instead of introducing a new flag to denote a command that "writes" but not into the keyspace, and still needs propagation, i decided to rely on the forceCommandPropagation, and just fix the code to propagate MULTI when needed rather than depending on the command flags at all. The implication of my change then is that now it won't decide to propagate MULTI when it sees one of these: SELECT, PING, INFO, COMMAND, TIME and other commands which are neither read nor write. 3. Changing getNodeByQuery and clusterRedirectBlockedClientIfNeeded in cluster.c to look at !write rather than read flag. This should have no implications, since these code paths are only reachable for commands which access keys, and these are always marked as either read or write. This commit improve MULTI propagation tests, for modules and a bunch of other special cases, all of which used to pass already before that commit. the only one that test change that uncovered a change of behavior is the one that DELs a non-existing key, it used to propagate an empty multi-exec block, and no longer does.
2020-12-22 05:03:49 -05:00
r propagate-test.simple
r propagate-test.mixed
r exec
# Load the AOF
r debug loadaof
assert_equal [s 0 unexpected_error_replies] 0
}
}
}