Avoid saving module aux on RDB if no aux data was saved by the module. (#11374)
### Background
The issue is that when saving an RDB with module AUX data, the module AUX metadata
(moduleid, when, ...) is saved to the RDB even though the module did not saved any actual data.
This prevent loading the RDB in the absence of the module (although there is no actual data in
the RDB that requires the module to be loaded).
### Solution
The solution suggested in this PR is that module AUX will be saved on the RDB only if the module
actually saved something during `aux_save` function.
To support backward compatibility, we introduce `aux_save2` callback that acts the same as
`aux_save` with the tiny change of avoid saving the aux field if no data was actually saved by
the module. Modules can use the new API to make sure that if they have no data to save,
then it will be possible to load the created RDB even without the module.
### Concerns
A module may register for the aux load and save hooks just in order to be notified when
saving or loading starts or completed (there are better ways to do that, but it still possible
that someone used it).
However, if a module didn't save a single field in the save callback, it means it's not allowed
to read in the read callback, since it has no way to distinguish between empty and non-empty
payloads. furthermore, it means that if the module did that, it must never change it, since it'll
break compatibility with it's old RDB files, so this is really not a valid use case.
Since some modules (ones who currently save one field indicating an empty payload), need
to know if saving an empty payload is valid, and if Redis is gonna ignore an empty payload
or store it, we opted to add a new API (rather than change behavior of an existing API and
expect modules to check the redis version)
### Technical Details
To avoid saving AUX data on RDB, we change the code to first save the AUX metadata
(moduleid, when, ...) into a temporary buffer. The buffer is then flushed to the rio at the first
time the module makes a write operation inside the `aux_save` function. If the module saves
nothing (and `aux_save2` was used), the entire temporary buffer is simply dropped and no
data about this AUX field is saved to the RDB. This make it possible to load the RDB even in
the absence of the module.
Test was added to verify the fix.
2022-10-18 12:45:46 -04:00
|
|
|
# This module can be configure with multiple options given as flags on module load time
|
|
|
|
|
# 0 - not aux fields will be declared (this is the default)
|
|
|
|
|
# 1 << 0 - use aux_save2 api
|
|
|
|
|
# 1 << 1 - call aux callback before key space
|
|
|
|
|
# 1 << 2 - call aux callback after key space
|
|
|
|
|
# 1 << 3 - do not save data on aux callback
|
2019-07-21 10:41:03 -04:00
|
|
|
set testmodule [file normalize tests/modules/testrdb.so]
|
|
|
|
|
|
|
|
|
|
tags "modules" {
|
2019-10-24 02:45:25 -04:00
|
|
|
test {modules are able to persist types} {
|
|
|
|
|
start_server [list overrides [list loadmodule "$testmodule"]] {
|
2019-07-21 10:41:03 -04:00
|
|
|
r testrdb.set.key key1 value1
|
|
|
|
|
assert_equal "value1" [r testrdb.get.key key1]
|
|
|
|
|
r debug reload
|
|
|
|
|
assert_equal "value1" [r testrdb.get.key key1]
|
|
|
|
|
}
|
2019-10-24 02:45:25 -04:00
|
|
|
}
|
2019-07-21 10:41:03 -04:00
|
|
|
|
2019-10-24 02:45:25 -04:00
|
|
|
test {modules global are lost without aux} {
|
|
|
|
|
set server_path [tmpdir "server.module-testrdb"]
|
2020-08-31 03:44:43 -04:00
|
|
|
start_server [list overrides [list loadmodule "$testmodule" "dir" $server_path] keep_persistence true] {
|
2019-07-21 10:41:03 -04:00
|
|
|
r testrdb.set.before global1
|
|
|
|
|
assert_equal "global1" [r testrdb.get.before]
|
2019-10-24 02:45:25 -04:00
|
|
|
}
|
|
|
|
|
start_server [list overrides [list loadmodule "$testmodule" "dir" $server_path]] {
|
2019-07-21 10:41:03 -04:00
|
|
|
assert_equal "" [r testrdb.get.before]
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
Avoid saving module aux on RDB if no aux data was saved by the module. (#11374)
### Background
The issue is that when saving an RDB with module AUX data, the module AUX metadata
(moduleid, when, ...) is saved to the RDB even though the module did not saved any actual data.
This prevent loading the RDB in the absence of the module (although there is no actual data in
the RDB that requires the module to be loaded).
### Solution
The solution suggested in this PR is that module AUX will be saved on the RDB only if the module
actually saved something during `aux_save` function.
To support backward compatibility, we introduce `aux_save2` callback that acts the same as
`aux_save` with the tiny change of avoid saving the aux field if no data was actually saved by
the module. Modules can use the new API to make sure that if they have no data to save,
then it will be possible to load the created RDB even without the module.
### Concerns
A module may register for the aux load and save hooks just in order to be notified when
saving or loading starts or completed (there are better ways to do that, but it still possible
that someone used it).
However, if a module didn't save a single field in the save callback, it means it's not allowed
to read in the read callback, since it has no way to distinguish between empty and non-empty
payloads. furthermore, it means that if the module did that, it must never change it, since it'll
break compatibility with it's old RDB files, so this is really not a valid use case.
Since some modules (ones who currently save one field indicating an empty payload), need
to know if saving an empty payload is valid, and if Redis is gonna ignore an empty payload
or store it, we opted to add a new API (rather than change behavior of an existing API and
expect modules to check the redis version)
### Technical Details
To avoid saving AUX data on RDB, we change the code to first save the AUX metadata
(moduleid, when, ...) into a temporary buffer. The buffer is then flushed to the rio at the first
time the module makes a write operation inside the `aux_save` function. If the module saves
nothing (and `aux_save2` was used), the entire temporary buffer is simply dropped and no
data about this AUX field is saved to the RDB. This make it possible to load the RDB even in
the absence of the module.
Test was added to verify the fix.
2022-10-18 12:45:46 -04:00
|
|
|
test {aux that saves no data are not saved to the rdb when aux_save2 is used} {
|
|
|
|
|
set server_path [tmpdir "server.module-testrdb"]
|
|
|
|
|
puts $server_path
|
|
|
|
|
# 15 == 1111 - use aux_save2 before and after key space without data
|
|
|
|
|
start_server [list overrides [list loadmodule "$testmodule 15" "dir" $server_path] keep_persistence true] {
|
|
|
|
|
r set x 1
|
|
|
|
|
r save
|
|
|
|
|
}
|
|
|
|
|
start_server [list overrides [list "dir" $server_path] keep_persistence true] {
|
|
|
|
|
# make sure server started successfully without the module.
|
|
|
|
|
assert_equal {1} [r get x]
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
test {aux that saves no data are saved to the rdb when aux_save is used} {
|
|
|
|
|
set server_path [tmpdir "server.module-testrdb"]
|
|
|
|
|
puts $server_path
|
|
|
|
|
# 14 == 1110 - use aux_save before and after key space without data
|
|
|
|
|
start_server [list overrides [list loadmodule "$testmodule 14" "dir" $server_path] keep_persistence true] {
|
|
|
|
|
r set x 1
|
|
|
|
|
r save
|
|
|
|
|
}
|
|
|
|
|
start_server [list overrides [list loadmodule "$testmodule 14" "dir" $server_path] keep_persistence true] {
|
|
|
|
|
# make sure server started successfully and aux_save was called twice.
|
|
|
|
|
assert_equal {1} [r get x]
|
|
|
|
|
assert_equal {2} [r testrdb.get.n_aux_load_called]
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
foreach test_case {6 7} {
|
|
|
|
|
# 6 == 0110 - use aux_save before and after key space with data
|
|
|
|
|
# 7 == 0111 - use aux_save2 before and after key space with data
|
2019-10-24 02:45:25 -04:00
|
|
|
test {modules are able to persist globals before and after} {
|
|
|
|
|
set server_path [tmpdir "server.module-testrdb"]
|
2023-04-18 09:14:26 -04:00
|
|
|
start_server [list overrides [list loadmodule "$testmodule $test_case" "dir" $server_path "save" "900 1"] keep_persistence true] {
|
2019-07-21 10:41:03 -04:00
|
|
|
r testrdb.set.before global1
|
|
|
|
|
r testrdb.set.after global2
|
|
|
|
|
assert_equal "global1" [r testrdb.get.before]
|
|
|
|
|
assert_equal "global2" [r testrdb.get.after]
|
2019-10-24 02:45:25 -04:00
|
|
|
}
|
2023-04-18 09:14:26 -04:00
|
|
|
start_server [list overrides [list loadmodule "$testmodule $test_case" "dir" $server_path "save" "900 1"]] {
|
2019-07-21 10:41:03 -04:00
|
|
|
assert_equal "global1" [r testrdb.get.before]
|
|
|
|
|
assert_equal "global2" [r testrdb.get.after]
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
}
|
Avoid saving module aux on RDB if no aux data was saved by the module. (#11374)
### Background
The issue is that when saving an RDB with module AUX data, the module AUX metadata
(moduleid, when, ...) is saved to the RDB even though the module did not saved any actual data.
This prevent loading the RDB in the absence of the module (although there is no actual data in
the RDB that requires the module to be loaded).
### Solution
The solution suggested in this PR is that module AUX will be saved on the RDB only if the module
actually saved something during `aux_save` function.
To support backward compatibility, we introduce `aux_save2` callback that acts the same as
`aux_save` with the tiny change of avoid saving the aux field if no data was actually saved by
the module. Modules can use the new API to make sure that if they have no data to save,
then it will be possible to load the created RDB even without the module.
### Concerns
A module may register for the aux load and save hooks just in order to be notified when
saving or loading starts or completed (there are better ways to do that, but it still possible
that someone used it).
However, if a module didn't save a single field in the save callback, it means it's not allowed
to read in the read callback, since it has no way to distinguish between empty and non-empty
payloads. furthermore, it means that if the module did that, it must never change it, since it'll
break compatibility with it's old RDB files, so this is really not a valid use case.
Since some modules (ones who currently save one field indicating an empty payload), need
to know if saving an empty payload is valid, and if Redis is gonna ignore an empty payload
or store it, we opted to add a new API (rather than change behavior of an existing API and
expect modules to check the redis version)
### Technical Details
To avoid saving AUX data on RDB, we change the code to first save the AUX metadata
(moduleid, when, ...) into a temporary buffer. The buffer is then flushed to the rio at the first
time the module makes a write operation inside the `aux_save` function. If the module saves
nothing (and `aux_save2` was used), the entire temporary buffer is simply dropped and no
data about this AUX field is saved to the RDB. This make it possible to load the RDB even in
the absence of the module.
Test was added to verify the fix.
2022-10-18 12:45:46 -04:00
|
|
|
}
|
2019-07-21 10:41:03 -04:00
|
|
|
|
Avoid saving module aux on RDB if no aux data was saved by the module. (#11374)
### Background
The issue is that when saving an RDB with module AUX data, the module AUX metadata
(moduleid, when, ...) is saved to the RDB even though the module did not saved any actual data.
This prevent loading the RDB in the absence of the module (although there is no actual data in
the RDB that requires the module to be loaded).
### Solution
The solution suggested in this PR is that module AUX will be saved on the RDB only if the module
actually saved something during `aux_save` function.
To support backward compatibility, we introduce `aux_save2` callback that acts the same as
`aux_save` with the tiny change of avoid saving the aux field if no data was actually saved by
the module. Modules can use the new API to make sure that if they have no data to save,
then it will be possible to load the created RDB even without the module.
### Concerns
A module may register for the aux load and save hooks just in order to be notified when
saving or loading starts or completed (there are better ways to do that, but it still possible
that someone used it).
However, if a module didn't save a single field in the save callback, it means it's not allowed
to read in the read callback, since it has no way to distinguish between empty and non-empty
payloads. furthermore, it means that if the module did that, it must never change it, since it'll
break compatibility with it's old RDB files, so this is really not a valid use case.
Since some modules (ones who currently save one field indicating an empty payload), need
to know if saving an empty payload is valid, and if Redis is gonna ignore an empty payload
or store it, we opted to add a new API (rather than change behavior of an existing API and
expect modules to check the redis version)
### Technical Details
To avoid saving AUX data on RDB, we change the code to first save the AUX metadata
(moduleid, when, ...) into a temporary buffer. The buffer is then flushed to the rio at the first
time the module makes a write operation inside the `aux_save` function. If the module saves
nothing (and `aux_save2` was used), the entire temporary buffer is simply dropped and no
data about this AUX field is saved to the RDB. This make it possible to load the RDB even in
the absence of the module.
Test was added to verify the fix.
2022-10-18 12:45:46 -04:00
|
|
|
foreach test_case {4 5} {
|
|
|
|
|
# 4 == 0100 - use aux_save after key space with data
|
|
|
|
|
# 5 == 0101 - use aux_save2 after key space with data
|
2019-10-24 02:45:25 -04:00
|
|
|
test {modules are able to persist globals just after} {
|
|
|
|
|
set server_path [tmpdir "server.module-testrdb"]
|
2023-04-18 09:14:26 -04:00
|
|
|
start_server [list overrides [list loadmodule "$testmodule $test_case" "dir" $server_path "save" "900 1"] keep_persistence true] {
|
2019-07-21 10:41:03 -04:00
|
|
|
r testrdb.set.after global2
|
|
|
|
|
assert_equal "global2" [r testrdb.get.after]
|
2019-10-24 02:45:25 -04:00
|
|
|
}
|
2023-04-18 09:14:26 -04:00
|
|
|
start_server [list overrides [list loadmodule "$testmodule $test_case" "dir" $server_path "save" "900 1"]] {
|
2019-07-21 10:41:03 -04:00
|
|
|
assert_equal "global2" [r testrdb.get.after]
|
|
|
|
|
}
|
|
|
|
|
}
|
Avoid saving module aux on RDB if no aux data was saved by the module. (#11374)
### Background
The issue is that when saving an RDB with module AUX data, the module AUX metadata
(moduleid, when, ...) is saved to the RDB even though the module did not saved any actual data.
This prevent loading the RDB in the absence of the module (although there is no actual data in
the RDB that requires the module to be loaded).
### Solution
The solution suggested in this PR is that module AUX will be saved on the RDB only if the module
actually saved something during `aux_save` function.
To support backward compatibility, we introduce `aux_save2` callback that acts the same as
`aux_save` with the tiny change of avoid saving the aux field if no data was actually saved by
the module. Modules can use the new API to make sure that if they have no data to save,
then it will be possible to load the created RDB even without the module.
### Concerns
A module may register for the aux load and save hooks just in order to be notified when
saving or loading starts or completed (there are better ways to do that, but it still possible
that someone used it).
However, if a module didn't save a single field in the save callback, it means it's not allowed
to read in the read callback, since it has no way to distinguish between empty and non-empty
payloads. furthermore, it means that if the module did that, it must never change it, since it'll
break compatibility with it's old RDB files, so this is really not a valid use case.
Since some modules (ones who currently save one field indicating an empty payload), need
to know if saving an empty payload is valid, and if Redis is gonna ignore an empty payload
or store it, we opted to add a new API (rather than change behavior of an existing API and
expect modules to check the redis version)
### Technical Details
To avoid saving AUX data on RDB, we change the code to first save the AUX metadata
(moduleid, when, ...) into a temporary buffer. The buffer is then flushed to the rio at the first
time the module makes a write operation inside the `aux_save` function. If the module saves
nothing (and `aux_save2` was used), the entire temporary buffer is simply dropped and no
data about this AUX field is saved to the RDB. This make it possible to load the RDB even in
the absence of the module.
Test was added to verify the fix.
2022-10-18 12:45:46 -04:00
|
|
|
}
|
2019-07-21 10:41:03 -04:00
|
|
|
|
2022-02-11 13:15:52 -05:00
|
|
|
test {Verify module options info} {
|
|
|
|
|
start_server [list overrides [list loadmodule "$testmodule"]] {
|
|
|
|
|
assert_match "*\[handle-io-errors|handle-repl-async-load\]*" [r info modules]
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
2019-07-21 11:18:11 -04:00
|
|
|
tags {repl} {
|
|
|
|
|
test {diskless loading short read with module} {
|
|
|
|
|
start_server [list overrides [list loadmodule "$testmodule"]] {
|
|
|
|
|
set replica [srv 0 client]
|
|
|
|
|
set replica_host [srv 0 host]
|
|
|
|
|
set replica_port [srv 0 port]
|
|
|
|
|
start_server [list overrides [list loadmodule "$testmodule"]] {
|
|
|
|
|
set master [srv 0 client]
|
|
|
|
|
set master_host [srv 0 host]
|
|
|
|
|
set master_port [srv 0 port]
|
2019-07-21 10:41:03 -04:00
|
|
|
|
2019-07-21 11:18:11 -04:00
|
|
|
# Set master and replica to use diskless replication
|
|
|
|
|
$master config set repl-diskless-sync yes
|
|
|
|
|
$master config set rdbcompression no
|
|
|
|
|
$replica config set repl-diskless-load swapdb
|
Accelerate diskless master connections, and general re-connections (#6271)
Diskless master has some inherent latencies.
1) fork starts with delay from cron rather than immediately
2) replica is put online only after an ACK. but the ACK
was sent only once a second.
3) but even if it would arrive immediately, it will not
register in case cron didn't yet detect that the fork is done.
Besides that, when a replica disconnects, it doesn't immediately
attempts to re-connect, it waits for replication cron (one per second).
in case it was already online, it may be important to try to re-connect
as soon as possible, so that the backlog at the master doesn't vanish.
In case it disconnected during rdb transfer, one can argue that it's
not very important to re-connect immediately, but this is needed for the
"diskless loading short read" test to be able to run 100 iterations in 5
seconds, rather than 3 (waiting for replication cron re-connection)
changes in this commit:
1) sync command starts a fork immediately if no sync_delay is configured
2) replica sends REPLCONF ACK when done reading the rdb (rather than on 1s cron)
3) when a replica unexpectedly disconnets, it immediately tries to
re-connect rather than waiting 1s
4) when when a child exits, if there is another replica waiting, we spawn a new
one right away, instead of waiting for 1s replicationCron.
5) added a call to connectWithMaster from replicationSetMaster. which is called
from the REPLICAOF command but also in 3 places in cluster.c, in all of
these the connection attempt will now be immediate instead of delayed by 1
second.
side note:
we can add a call to rdbPipeReadHandler in replconfCommand when getting
a REPLCONF ACK from the replica to solve a race where the replica got
the entire rdb and EOF marker before we detected that the pipe was
closed.
in the test i did see this race happens in one about of some 300 runs,
but i concluded that this race is unlikely in real life (where the
replica is on another host and we're more likely to first detect the
pipe was closed.
the test runs 100 iterations in 3 seconds, so in some cases it'll take 4
seconds instead (waiting for another REPLCONF ACK).
Removing unneeded startBgsaveForReplication from updateSlavesWaitingForBgsave
Now that CheckChildrenDone is calling the new replicationStartPendingFork
(extracted from serverCron) there's actually no need to call
startBgsaveForReplication from updateSlavesWaitingForBgsave anymore,
since as soon as updateSlavesWaitingForBgsave returns, CheckChildrenDone is
calling replicationStartPendingFork that handles that anyway.
The code in updateSlavesWaitingForBgsave had a bug in which it ignored
repl-diskless-sync-delay, but removing that code shows that this bug was
hiding another bug, which is that the max_idle should have used >= and
not >, this one second delay has a big impact on my new test.
2020-08-06 09:53:06 -04:00
|
|
|
$master config set hz 500
|
|
|
|
|
$replica config set hz 500
|
|
|
|
|
$master config set dynamic-hz no
|
|
|
|
|
$replica config set dynamic-hz no
|
|
|
|
|
set start [clock clicks -milliseconds]
|
2019-07-21 11:18:11 -04:00
|
|
|
for {set k 0} {$k < 30} {incr k} {
|
|
|
|
|
r testrdb.set.key key$k [string repeat A [expr {int(rand()*1000000)}]]
|
|
|
|
|
}
|
2019-07-21 10:41:03 -04:00
|
|
|
|
Accelerate diskless master connections, and general re-connections (#6271)
Diskless master has some inherent latencies.
1) fork starts with delay from cron rather than immediately
2) replica is put online only after an ACK. but the ACK
was sent only once a second.
3) but even if it would arrive immediately, it will not
register in case cron didn't yet detect that the fork is done.
Besides that, when a replica disconnects, it doesn't immediately
attempts to re-connect, it waits for replication cron (one per second).
in case it was already online, it may be important to try to re-connect
as soon as possible, so that the backlog at the master doesn't vanish.
In case it disconnected during rdb transfer, one can argue that it's
not very important to re-connect immediately, but this is needed for the
"diskless loading short read" test to be able to run 100 iterations in 5
seconds, rather than 3 (waiting for replication cron re-connection)
changes in this commit:
1) sync command starts a fork immediately if no sync_delay is configured
2) replica sends REPLCONF ACK when done reading the rdb (rather than on 1s cron)
3) when a replica unexpectedly disconnets, it immediately tries to
re-connect rather than waiting 1s
4) when when a child exits, if there is another replica waiting, we spawn a new
one right away, instead of waiting for 1s replicationCron.
5) added a call to connectWithMaster from replicationSetMaster. which is called
from the REPLICAOF command but also in 3 places in cluster.c, in all of
these the connection attempt will now be immediate instead of delayed by 1
second.
side note:
we can add a call to rdbPipeReadHandler in replconfCommand when getting
a REPLCONF ACK from the replica to solve a race where the replica got
the entire rdb and EOF marker before we detected that the pipe was
closed.
in the test i did see this race happens in one about of some 300 runs,
but i concluded that this race is unlikely in real life (where the
replica is on another host and we're more likely to first detect the
pipe was closed.
the test runs 100 iterations in 3 seconds, so in some cases it'll take 4
seconds instead (waiting for another REPLCONF ACK).
Removing unneeded startBgsaveForReplication from updateSlavesWaitingForBgsave
Now that CheckChildrenDone is calling the new replicationStartPendingFork
(extracted from serverCron) there's actually no need to call
startBgsaveForReplication from updateSlavesWaitingForBgsave anymore,
since as soon as updateSlavesWaitingForBgsave returns, CheckChildrenDone is
calling replicationStartPendingFork that handles that anyway.
The code in updateSlavesWaitingForBgsave had a bug in which it ignored
repl-diskless-sync-delay, but removing that code shows that this bug was
hiding another bug, which is that the max_idle should have used >= and
not >, this one second delay has a big impact on my new test.
2020-08-06 09:53:06 -04:00
|
|
|
if {$::verbose} {
|
|
|
|
|
set end [clock clicks -milliseconds]
|
|
|
|
|
set duration [expr $end - $start]
|
|
|
|
|
puts "filling took $duration ms (TODO: use pipeline)"
|
|
|
|
|
set start [clock clicks -milliseconds]
|
|
|
|
|
}
|
|
|
|
|
|
2019-07-21 11:18:11 -04:00
|
|
|
# Start the replication process...
|
2020-07-10 01:28:22 -04:00
|
|
|
set loglines [count_log_lines -1]
|
2019-07-21 11:18:11 -04:00
|
|
|
$master config set repl-diskless-sync-delay 0
|
|
|
|
|
$replica replicaof $master_host $master_port
|
|
|
|
|
|
|
|
|
|
# kill the replication at various points
|
Accelerate diskless master connections, and general re-connections (#6271)
Diskless master has some inherent latencies.
1) fork starts with delay from cron rather than immediately
2) replica is put online only after an ACK. but the ACK
was sent only once a second.
3) but even if it would arrive immediately, it will not
register in case cron didn't yet detect that the fork is done.
Besides that, when a replica disconnects, it doesn't immediately
attempts to re-connect, it waits for replication cron (one per second).
in case it was already online, it may be important to try to re-connect
as soon as possible, so that the backlog at the master doesn't vanish.
In case it disconnected during rdb transfer, one can argue that it's
not very important to re-connect immediately, but this is needed for the
"diskless loading short read" test to be able to run 100 iterations in 5
seconds, rather than 3 (waiting for replication cron re-connection)
changes in this commit:
1) sync command starts a fork immediately if no sync_delay is configured
2) replica sends REPLCONF ACK when done reading the rdb (rather than on 1s cron)
3) when a replica unexpectedly disconnets, it immediately tries to
re-connect rather than waiting 1s
4) when when a child exits, if there is another replica waiting, we spawn a new
one right away, instead of waiting for 1s replicationCron.
5) added a call to connectWithMaster from replicationSetMaster. which is called
from the REPLICAOF command but also in 3 places in cluster.c, in all of
these the connection attempt will now be immediate instead of delayed by 1
second.
side note:
we can add a call to rdbPipeReadHandler in replconfCommand when getting
a REPLCONF ACK from the replica to solve a race where the replica got
the entire rdb and EOF marker before we detected that the pipe was
closed.
in the test i did see this race happens in one about of some 300 runs,
but i concluded that this race is unlikely in real life (where the
replica is on another host and we're more likely to first detect the
pipe was closed.
the test runs 100 iterations in 3 seconds, so in some cases it'll take 4
seconds instead (waiting for another REPLCONF ACK).
Removing unneeded startBgsaveForReplication from updateSlavesWaitingForBgsave
Now that CheckChildrenDone is calling the new replicationStartPendingFork
(extracted from serverCron) there's actually no need to call
startBgsaveForReplication from updateSlavesWaitingForBgsave anymore,
since as soon as updateSlavesWaitingForBgsave returns, CheckChildrenDone is
calling replicationStartPendingFork that handles that anyway.
The code in updateSlavesWaitingForBgsave had a bug in which it ignored
repl-diskless-sync-delay, but removing that code shows that this bug was
hiding another bug, which is that the max_idle should have used >= and
not >, this one second delay has a big impact on my new test.
2020-08-06 09:53:06 -04:00
|
|
|
set attempts 100
|
|
|
|
|
if {$::accurate} { set attempts 500 }
|
2019-07-21 11:18:11 -04:00
|
|
|
for {set i 0} {$i < $attempts} {incr i} {
|
|
|
|
|
# wait for the replica to start reading the rdb
|
|
|
|
|
# using the log file since the replica only responds to INFO once in 2mb
|
2020-07-28 04:15:29 -04:00
|
|
|
set res [wait_for_log_messages -1 {"*Loading DB in memory*"} $loglines 2000 1]
|
|
|
|
|
set loglines [lindex $res 1]
|
2019-07-21 11:18:11 -04:00
|
|
|
|
|
|
|
|
# add some additional random sleep so that we kill the master on a different place each time
|
2020-07-28 04:15:29 -04:00
|
|
|
after [expr {int(rand()*50)}]
|
2019-07-21 11:18:11 -04:00
|
|
|
|
|
|
|
|
# kill the replica connection on the master
|
|
|
|
|
set killed [$master client kill type replica]
|
|
|
|
|
|
2021-11-09 15:37:18 -05:00
|
|
|
set res [wait_for_log_messages -1 {"*Internal error in RDB*" "*Finished with success*" "*Successful partial resynchronization*"} $loglines 500 10]
|
2020-07-28 04:15:29 -04:00
|
|
|
if {$::verbose} { puts $res }
|
|
|
|
|
set log_text [lindex $res 0]
|
|
|
|
|
set loglines [lindex $res 1]
|
|
|
|
|
if {![string match "*Internal error in RDB*" $log_text]} {
|
2019-07-21 11:18:11 -04:00
|
|
|
# force the replica to try another full sync
|
2020-07-28 04:15:29 -04:00
|
|
|
$master multi
|
2019-07-21 11:18:11 -04:00
|
|
|
$master client kill type replica
|
|
|
|
|
$master set asdf asdf
|
Replication backlog and replicas use one global shared replication buffer (#9166)
## Background
For redis master, one replica uses one copy of replication buffer, that is a big waste of memory,
more replicas more waste, and allocate/free memory for every reply list also cost much.
If we set client-output-buffer-limit small and write traffic is heavy, master may disconnect with
replicas and can't finish synchronization with replica. If we set client-output-buffer-limit big,
master may be OOM when there are many replicas that separately keep much memory.
Because replication buffers of different replica client are the same, one simple idea is that
all replicas only use one replication buffer, that will effectively save memory.
Since replication backlog content is the same as replicas' output buffer, now we
can discard replication backlog memory and use global shared replication buffer
to implement replication backlog mechanism.
## Implementation
I create one global "replication buffer" which contains content of replication stream.
The structure of "replication buffer" is similar to the reply list that exists in every client.
But the node of list is `replBufBlock`, which has `id, repl_offset, refcount` fields.
```c
/* Replication buffer blocks is the list of replBufBlock.
*
* +--------------+ +--------------+ +--------------+
* | refcount = 1 | ... | refcount = 0 | ... | refcount = 2 |
* +--------------+ +--------------+ +--------------+
* | / \
* | / \
* | / \
* Repl Backlog Replia_A Replia_B
*
* Each replica or replication backlog increments only the refcount of the
* 'ref_repl_buf_node' which it points to. So when replica walks to the next
* node, it should first increase the next node's refcount, and when we trim
* the replication buffer nodes, we remove node always from the head node which
* refcount is 0. If the refcount of the head node is not 0, we must stop
* trimming and never iterate the next node. */
/* Similar with 'clientReplyBlock', it is used for shared buffers between
* all replica clients and replication backlog. */
typedef struct replBufBlock {
int refcount; /* Number of replicas or repl backlog using. */
long long id; /* The unique incremental number. */
long long repl_offset; /* Start replication offset of the block. */
size_t size, used;
char buf[];
} replBufBlock;
```
So now when we feed replication stream into replication backlog and all replicas, we only need
to feed stream into replication buffer `feedReplicationBuffer`. In this function, we set some fields of
replication backlog and replicas to references of the global replication buffer blocks. And we also
need to check replicas' output buffer limit to free if exceeding `client-output-buffer-limit`, and trim
replication backlog if exceeding `repl-backlog-size`.
When sending reply to replicas, we also need to iterate replication buffer blocks and send its
content, when totally sending one block for replica, we decrease current node count and
increase the next current node count, and then free the block which reference is 0 from the
head of replication buffer blocks.
Since now we use linked list to manage replication backlog, it may cost much time for iterating
all linked list nodes to find corresponding replication buffer node. So we create a rax tree to
store some nodes for index, but to avoid rax tree occupying too much memory, i record
one per 64 nodes for index.
Currently, to make partial resynchronization as possible as much, we always let replication
backlog as the last reference of replication buffer blocks, backlog size may exceeds our setting
if slow replicas that reference vast replication buffer blocks, and this method doesn't increase
memory usage since they share replication buffer. To avoid freezing server for freeing unreferenced
replication buffer blocks when we need to trim backlog for exceeding backlog size setting,
we trim backlog incrementally (free 64 blocks per call now), and make it faster in
`beforeSleep` (free 640 blocks).
### Other changes
- `mem_total_replication_buffers`: we add this field in INFO command, it means the total
memory of replication buffers used.
- `mem_clients_slaves`: now even replica is slow to replicate, and its output buffer memory
is not 0, but it still may be 0, since replication backlog and replicas share one global replication
buffer, only if replication buffer memory is more than the repl backlog setting size, we consider
the excess as replicas' memory. Otherwise, we think replication buffer memory is the consumption
of repl backlog.
- Key eviction
Since all replicas and replication backlog share global replication buffer, we think only the
part of exceeding backlog size the extra separate consumption of replicas.
Because we trim backlog incrementally in the background, backlog size may exceeds our
setting if slow replicas that reference vast replication buffer blocks disconnect.
To avoid massive eviction loop, we don't count the delayed freed replication backlog into
used memory even if there are no replicas, i.e. we also regard this memory as replicas's memory.
- `client-output-buffer-limit` check for replica clients
It doesn't make sense to set the replica clients output buffer limit lower than the repl-backlog-size
config (partial sync will succeed and then replica will get disconnected). Such a configuration is
ignored (the size of repl-backlog-size will be used). This doesn't have memory consumption
implications since the replica client will share the backlog buffers memory.
- Drop replication backlog after loading data if needed
We always create replication backlog if server is a master, we need it because we put DELs in
it when loading expired keys in RDB, but if RDB doesn't have replication info or there is no rdb,
it is not possible to support partial resynchronization, to avoid extra memory of replication backlog,
we drop it.
- Multi IO threads
Since all replicas and replication backlog use global replication buffer, if I/O threads are enabled,
to guarantee data accessing thread safe, we must let main thread handle sending the output buffer
to all replicas. But before, other IO threads could handle sending output buffer of all replicas.
## Other optimizations
This solution resolve some other problem:
- When replicas disconnect with master since of out of output buffer limit, releasing the output
buffer of replicas may freeze server if we set big `client-output-buffer-limit` for replicas, but now,
it doesn't cause freezing.
- This implementation may mitigate reply list copy cost time(also freezes server) when one replication
has huge reply buffer and another replica can copy buffer for full synchronization. now, we just copy
reference info, it is very light.
- If we set replication backlog size big, it also may cost much time to copy replication backlog into
replica's output buffer. But this commit eliminates this problem.
- Resizing replication backlog size doesn't empty current replication backlog content.
2021-10-25 02:24:31 -04:00
|
|
|
# fill replication backlog with new content
|
|
|
|
|
$master config set repl-backlog-size 16384
|
|
|
|
|
for {set keyid 0} {$keyid < 10} {incr keyid} {
|
|
|
|
|
$master set "$keyid string_$keyid" [string repeat A 16384]
|
|
|
|
|
}
|
2020-07-28 04:15:29 -04:00
|
|
|
$master exec
|
2019-07-21 11:18:11 -04:00
|
|
|
}
|
2021-11-24 05:46:43 -05:00
|
|
|
|
2019-07-21 11:18:11 -04:00
|
|
|
# wait for loading to stop (fail)
|
2021-11-24 05:46:43 -05:00
|
|
|
# After a loading successfully, next loop will enter `async_loading`
|
2020-07-28 04:15:29 -04:00
|
|
|
wait_for_condition 1000 1 {
|
2021-11-24 05:46:43 -05:00
|
|
|
[s -1 async_loading] eq 0 &&
|
2019-07-21 11:18:11 -04:00
|
|
|
[s -1 loading] eq 0
|
|
|
|
|
} else {
|
|
|
|
|
fail "Replica didn't disconnect"
|
|
|
|
|
}
|
|
|
|
|
}
|
Accelerate diskless master connections, and general re-connections (#6271)
Diskless master has some inherent latencies.
1) fork starts with delay from cron rather than immediately
2) replica is put online only after an ACK. but the ACK
was sent only once a second.
3) but even if it would arrive immediately, it will not
register in case cron didn't yet detect that the fork is done.
Besides that, when a replica disconnects, it doesn't immediately
attempts to re-connect, it waits for replication cron (one per second).
in case it was already online, it may be important to try to re-connect
as soon as possible, so that the backlog at the master doesn't vanish.
In case it disconnected during rdb transfer, one can argue that it's
not very important to re-connect immediately, but this is needed for the
"diskless loading short read" test to be able to run 100 iterations in 5
seconds, rather than 3 (waiting for replication cron re-connection)
changes in this commit:
1) sync command starts a fork immediately if no sync_delay is configured
2) replica sends REPLCONF ACK when done reading the rdb (rather than on 1s cron)
3) when a replica unexpectedly disconnets, it immediately tries to
re-connect rather than waiting 1s
4) when when a child exits, if there is another replica waiting, we spawn a new
one right away, instead of waiting for 1s replicationCron.
5) added a call to connectWithMaster from replicationSetMaster. which is called
from the REPLICAOF command but also in 3 places in cluster.c, in all of
these the connection attempt will now be immediate instead of delayed by 1
second.
side note:
we can add a call to rdbPipeReadHandler in replconfCommand when getting
a REPLCONF ACK from the replica to solve a race where the replica got
the entire rdb and EOF marker before we detected that the pipe was
closed.
in the test i did see this race happens in one about of some 300 runs,
but i concluded that this race is unlikely in real life (where the
replica is on another host and we're more likely to first detect the
pipe was closed.
the test runs 100 iterations in 3 seconds, so in some cases it'll take 4
seconds instead (waiting for another REPLCONF ACK).
Removing unneeded startBgsaveForReplication from updateSlavesWaitingForBgsave
Now that CheckChildrenDone is calling the new replicationStartPendingFork
(extracted from serverCron) there's actually no need to call
startBgsaveForReplication from updateSlavesWaitingForBgsave anymore,
since as soon as updateSlavesWaitingForBgsave returns, CheckChildrenDone is
calling replicationStartPendingFork that handles that anyway.
The code in updateSlavesWaitingForBgsave had a bug in which it ignored
repl-diskless-sync-delay, but removing that code shows that this bug was
hiding another bug, which is that the max_idle should have used >= and
not >, this one second delay has a big impact on my new test.
2020-08-06 09:53:06 -04:00
|
|
|
if {$::verbose} {
|
|
|
|
|
set end [clock clicks -milliseconds]
|
|
|
|
|
set duration [expr $end - $start]
|
|
|
|
|
puts "test took $duration ms"
|
|
|
|
|
}
|
2019-07-21 11:18:11 -04:00
|
|
|
# enable fast shutdown
|
|
|
|
|
$master config set rdb-key-save-delay 0
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
}
|
Replica keep serving data during repl-diskless-load=swapdb for better availability (#9323)
For diskless replication in swapdb mode, considering we already spend replica memory
having a backup of current db to restore in case of failure, we can have the following benefits
by instead swapping database only in case we succeeded in transferring db from master:
- Avoid `LOADING` response during failed and successful synchronization for cases where the
replica is already up and running with data.
- Faster total time of diskless replication, because now we're moving from Transfer + Flush + Load
time to Transfer + Load only. Flushing the tempDb is done asynchronously after swapping.
- This could be implemented also for disk replication with similar benefits if consumers are willing
to spend the extra memory usage.
General notes:
- The concept of `backupDb` becomes `tempDb` for clarity.
- Async loading mode will only kick in if the replica is syncing from a master that has the same
repl-id the one it had before. i.e. the data it's getting belongs to a different time of the same timeline.
- New property in INFO: `async_loading` to differentiate from the blocking loading
- Slot to Key mapping is now a field of `redisDb` as it's more natural to access it from both server.db
and the tempDb that is passed around.
- Because this is affecting replicas only, we assume that if they are not readonly and write commands
during replication, they are lost after SYNC same way as before, but we're still denying CONFIG SET
here anyways to avoid complications.
Considerations for review:
- We have many cases where server.loading flag is used and even though I tried my best, there may
be cases where async_loading should be checked as well and cases where it shouldn't (would require
very good understanding of whole code)
- Several places that had different behavior depending on the loading flag where actually meant to just
handle commands coming from the AOF client differently than ones coming from real clients, changed
to check CLIENT_ID_AOF instead.
**Additional for Release Notes**
- Bugfix - server.dirty was not incremented for any kind of diskless replication, as effect it wouldn't
contribute on triggering next database SAVE
- New flag for RM_GetContextFlags module API: REDISMODULE_CTX_FLAGS_ASYNC_LOADING
- Deprecated RedisModuleEvent_ReplBackup. Starting from Redis 7.0, we don't fire this event.
Instead, we have the new RedisModuleEvent_ReplAsyncLoad holding 3 sub-events: STARTED,
ABORTED and COMPLETED.
- New module flag REDISMODULE_OPTIONS_HANDLE_REPL_ASYNC_LOAD for RedisModule_SetModuleOptions
to allow modules to declare they support the diskless replication with async loading (when absent, we fall
back to disk-based loading).
Co-authored-by: Eduardo Semprebon <edus@saxobank.com>
Co-authored-by: Oran Agra <oran@redislabs.com>
2021-11-04 04:46:50 -04:00
|
|
|
|
|
|
|
|
# Module events for diskless load swapdb when async_loading (matching master replid)
|
Avoid saving module aux on RDB if no aux data was saved by the module. (#11374)
### Background
The issue is that when saving an RDB with module AUX data, the module AUX metadata
(moduleid, when, ...) is saved to the RDB even though the module did not saved any actual data.
This prevent loading the RDB in the absence of the module (although there is no actual data in
the RDB that requires the module to be loaded).
### Solution
The solution suggested in this PR is that module AUX will be saved on the RDB only if the module
actually saved something during `aux_save` function.
To support backward compatibility, we introduce `aux_save2` callback that acts the same as
`aux_save` with the tiny change of avoid saving the aux field if no data was actually saved by
the module. Modules can use the new API to make sure that if they have no data to save,
then it will be possible to load the created RDB even without the module.
### Concerns
A module may register for the aux load and save hooks just in order to be notified when
saving or loading starts or completed (there are better ways to do that, but it still possible
that someone used it).
However, if a module didn't save a single field in the save callback, it means it's not allowed
to read in the read callback, since it has no way to distinguish between empty and non-empty
payloads. furthermore, it means that if the module did that, it must never change it, since it'll
break compatibility with it's old RDB files, so this is really not a valid use case.
Since some modules (ones who currently save one field indicating an empty payload), need
to know if saving an empty payload is valid, and if Redis is gonna ignore an empty payload
or store it, we opted to add a new API (rather than change behavior of an existing API and
expect modules to check the redis version)
### Technical Details
To avoid saving AUX data on RDB, we change the code to first save the AUX metadata
(moduleid, when, ...) into a temporary buffer. The buffer is then flushed to the rio at the first
time the module makes a write operation inside the `aux_save` function. If the module saves
nothing (and `aux_save2` was used), the entire temporary buffer is simply dropped and no
data about this AUX field is saved to the RDB. This make it possible to load the RDB even in
the absence of the module.
Test was added to verify the fix.
2022-10-18 12:45:46 -04:00
|
|
|
foreach test_case {6 7} {
|
|
|
|
|
# 6 == 0110 - use aux_save before and after key space with data
|
|
|
|
|
# 7 == 0111 - use aux_save2 before and after key space with data
|
Replica keep serving data during repl-diskless-load=swapdb for better availability (#9323)
For diskless replication in swapdb mode, considering we already spend replica memory
having a backup of current db to restore in case of failure, we can have the following benefits
by instead swapping database only in case we succeeded in transferring db from master:
- Avoid `LOADING` response during failed and successful synchronization for cases where the
replica is already up and running with data.
- Faster total time of diskless replication, because now we're moving from Transfer + Flush + Load
time to Transfer + Load only. Flushing the tempDb is done asynchronously after swapping.
- This could be implemented also for disk replication with similar benefits if consumers are willing
to spend the extra memory usage.
General notes:
- The concept of `backupDb` becomes `tempDb` for clarity.
- Async loading mode will only kick in if the replica is syncing from a master that has the same
repl-id the one it had before. i.e. the data it's getting belongs to a different time of the same timeline.
- New property in INFO: `async_loading` to differentiate from the blocking loading
- Slot to Key mapping is now a field of `redisDb` as it's more natural to access it from both server.db
and the tempDb that is passed around.
- Because this is affecting replicas only, we assume that if they are not readonly and write commands
during replication, they are lost after SYNC same way as before, but we're still denying CONFIG SET
here anyways to avoid complications.
Considerations for review:
- We have many cases where server.loading flag is used and even though I tried my best, there may
be cases where async_loading should be checked as well and cases where it shouldn't (would require
very good understanding of whole code)
- Several places that had different behavior depending on the loading flag where actually meant to just
handle commands coming from the AOF client differently than ones coming from real clients, changed
to check CLIENT_ID_AOF instead.
**Additional for Release Notes**
- Bugfix - server.dirty was not incremented for any kind of diskless replication, as effect it wouldn't
contribute on triggering next database SAVE
- New flag for RM_GetContextFlags module API: REDISMODULE_CTX_FLAGS_ASYNC_LOADING
- Deprecated RedisModuleEvent_ReplBackup. Starting from Redis 7.0, we don't fire this event.
Instead, we have the new RedisModuleEvent_ReplAsyncLoad holding 3 sub-events: STARTED,
ABORTED and COMPLETED.
- New module flag REDISMODULE_OPTIONS_HANDLE_REPL_ASYNC_LOAD for RedisModule_SetModuleOptions
to allow modules to declare they support the diskless replication with async loading (when absent, we fall
back to disk-based loading).
Co-authored-by: Eduardo Semprebon <edus@saxobank.com>
Co-authored-by: Oran Agra <oran@redislabs.com>
2021-11-04 04:46:50 -04:00
|
|
|
foreach testType {Successful Aborted} {
|
Avoid saving module aux on RDB if no aux data was saved by the module. (#11374)
### Background
The issue is that when saving an RDB with module AUX data, the module AUX metadata
(moduleid, when, ...) is saved to the RDB even though the module did not saved any actual data.
This prevent loading the RDB in the absence of the module (although there is no actual data in
the RDB that requires the module to be loaded).
### Solution
The solution suggested in this PR is that module AUX will be saved on the RDB only if the module
actually saved something during `aux_save` function.
To support backward compatibility, we introduce `aux_save2` callback that acts the same as
`aux_save` with the tiny change of avoid saving the aux field if no data was actually saved by
the module. Modules can use the new API to make sure that if they have no data to save,
then it will be possible to load the created RDB even without the module.
### Concerns
A module may register for the aux load and save hooks just in order to be notified when
saving or loading starts or completed (there are better ways to do that, but it still possible
that someone used it).
However, if a module didn't save a single field in the save callback, it means it's not allowed
to read in the read callback, since it has no way to distinguish between empty and non-empty
payloads. furthermore, it means that if the module did that, it must never change it, since it'll
break compatibility with it's old RDB files, so this is really not a valid use case.
Since some modules (ones who currently save one field indicating an empty payload), need
to know if saving an empty payload is valid, and if Redis is gonna ignore an empty payload
or store it, we opted to add a new API (rather than change behavior of an existing API and
expect modules to check the redis version)
### Technical Details
To avoid saving AUX data on RDB, we change the code to first save the AUX metadata
(moduleid, when, ...) into a temporary buffer. The buffer is then flushed to the rio at the first
time the module makes a write operation inside the `aux_save` function. If the module saves
nothing (and `aux_save2` was used), the entire temporary buffer is simply dropped and no
data about this AUX field is saved to the RDB. This make it possible to load the RDB even in
the absence of the module.
Test was added to verify the fix.
2022-10-18 12:45:46 -04:00
|
|
|
start_server [list overrides [list loadmodule "$testmodule $test_case"] tags [list external:skip]] {
|
Replica keep serving data during repl-diskless-load=swapdb for better availability (#9323)
For diskless replication in swapdb mode, considering we already spend replica memory
having a backup of current db to restore in case of failure, we can have the following benefits
by instead swapping database only in case we succeeded in transferring db from master:
- Avoid `LOADING` response during failed and successful synchronization for cases where the
replica is already up and running with data.
- Faster total time of diskless replication, because now we're moving from Transfer + Flush + Load
time to Transfer + Load only. Flushing the tempDb is done asynchronously after swapping.
- This could be implemented also for disk replication with similar benefits if consumers are willing
to spend the extra memory usage.
General notes:
- The concept of `backupDb` becomes `tempDb` for clarity.
- Async loading mode will only kick in if the replica is syncing from a master that has the same
repl-id the one it had before. i.e. the data it's getting belongs to a different time of the same timeline.
- New property in INFO: `async_loading` to differentiate from the blocking loading
- Slot to Key mapping is now a field of `redisDb` as it's more natural to access it from both server.db
and the tempDb that is passed around.
- Because this is affecting replicas only, we assume that if they are not readonly and write commands
during replication, they are lost after SYNC same way as before, but we're still denying CONFIG SET
here anyways to avoid complications.
Considerations for review:
- We have many cases where server.loading flag is used and even though I tried my best, there may
be cases where async_loading should be checked as well and cases where it shouldn't (would require
very good understanding of whole code)
- Several places that had different behavior depending on the loading flag where actually meant to just
handle commands coming from the AOF client differently than ones coming from real clients, changed
to check CLIENT_ID_AOF instead.
**Additional for Release Notes**
- Bugfix - server.dirty was not incremented for any kind of diskless replication, as effect it wouldn't
contribute on triggering next database SAVE
- New flag for RM_GetContextFlags module API: REDISMODULE_CTX_FLAGS_ASYNC_LOADING
- Deprecated RedisModuleEvent_ReplBackup. Starting from Redis 7.0, we don't fire this event.
Instead, we have the new RedisModuleEvent_ReplAsyncLoad holding 3 sub-events: STARTED,
ABORTED and COMPLETED.
- New module flag REDISMODULE_OPTIONS_HANDLE_REPL_ASYNC_LOAD for RedisModule_SetModuleOptions
to allow modules to declare they support the diskless replication with async loading (when absent, we fall
back to disk-based loading).
Co-authored-by: Eduardo Semprebon <edus@saxobank.com>
Co-authored-by: Oran Agra <oran@redislabs.com>
2021-11-04 04:46:50 -04:00
|
|
|
set replica [srv 0 client]
|
|
|
|
|
set replica_host [srv 0 host]
|
|
|
|
|
set replica_port [srv 0 port]
|
|
|
|
|
set replica_log [srv 0 stdout]
|
Avoid saving module aux on RDB if no aux data was saved by the module. (#11374)
### Background
The issue is that when saving an RDB with module AUX data, the module AUX metadata
(moduleid, when, ...) is saved to the RDB even though the module did not saved any actual data.
This prevent loading the RDB in the absence of the module (although there is no actual data in
the RDB that requires the module to be loaded).
### Solution
The solution suggested in this PR is that module AUX will be saved on the RDB only if the module
actually saved something during `aux_save` function.
To support backward compatibility, we introduce `aux_save2` callback that acts the same as
`aux_save` with the tiny change of avoid saving the aux field if no data was actually saved by
the module. Modules can use the new API to make sure that if they have no data to save,
then it will be possible to load the created RDB even without the module.
### Concerns
A module may register for the aux load and save hooks just in order to be notified when
saving or loading starts or completed (there are better ways to do that, but it still possible
that someone used it).
However, if a module didn't save a single field in the save callback, it means it's not allowed
to read in the read callback, since it has no way to distinguish between empty and non-empty
payloads. furthermore, it means that if the module did that, it must never change it, since it'll
break compatibility with it's old RDB files, so this is really not a valid use case.
Since some modules (ones who currently save one field indicating an empty payload), need
to know if saving an empty payload is valid, and if Redis is gonna ignore an empty payload
or store it, we opted to add a new API (rather than change behavior of an existing API and
expect modules to check the redis version)
### Technical Details
To avoid saving AUX data on RDB, we change the code to first save the AUX metadata
(moduleid, when, ...) into a temporary buffer. The buffer is then flushed to the rio at the first
time the module makes a write operation inside the `aux_save` function. If the module saves
nothing (and `aux_save2` was used), the entire temporary buffer is simply dropped and no
data about this AUX field is saved to the RDB. This make it possible to load the RDB even in
the absence of the module.
Test was added to verify the fix.
2022-10-18 12:45:46 -04:00
|
|
|
start_server [list overrides [list loadmodule "$testmodule $test_case"]] {
|
Replica keep serving data during repl-diskless-load=swapdb for better availability (#9323)
For diskless replication in swapdb mode, considering we already spend replica memory
having a backup of current db to restore in case of failure, we can have the following benefits
by instead swapping database only in case we succeeded in transferring db from master:
- Avoid `LOADING` response during failed and successful synchronization for cases where the
replica is already up and running with data.
- Faster total time of diskless replication, because now we're moving from Transfer + Flush + Load
time to Transfer + Load only. Flushing the tempDb is done asynchronously after swapping.
- This could be implemented also for disk replication with similar benefits if consumers are willing
to spend the extra memory usage.
General notes:
- The concept of `backupDb` becomes `tempDb` for clarity.
- Async loading mode will only kick in if the replica is syncing from a master that has the same
repl-id the one it had before. i.e. the data it's getting belongs to a different time of the same timeline.
- New property in INFO: `async_loading` to differentiate from the blocking loading
- Slot to Key mapping is now a field of `redisDb` as it's more natural to access it from both server.db
and the tempDb that is passed around.
- Because this is affecting replicas only, we assume that if they are not readonly and write commands
during replication, they are lost after SYNC same way as before, but we're still denying CONFIG SET
here anyways to avoid complications.
Considerations for review:
- We have many cases where server.loading flag is used and even though I tried my best, there may
be cases where async_loading should be checked as well and cases where it shouldn't (would require
very good understanding of whole code)
- Several places that had different behavior depending on the loading flag where actually meant to just
handle commands coming from the AOF client differently than ones coming from real clients, changed
to check CLIENT_ID_AOF instead.
**Additional for Release Notes**
- Bugfix - server.dirty was not incremented for any kind of diskless replication, as effect it wouldn't
contribute on triggering next database SAVE
- New flag for RM_GetContextFlags module API: REDISMODULE_CTX_FLAGS_ASYNC_LOADING
- Deprecated RedisModuleEvent_ReplBackup. Starting from Redis 7.0, we don't fire this event.
Instead, we have the new RedisModuleEvent_ReplAsyncLoad holding 3 sub-events: STARTED,
ABORTED and COMPLETED.
- New module flag REDISMODULE_OPTIONS_HANDLE_REPL_ASYNC_LOAD for RedisModule_SetModuleOptions
to allow modules to declare they support the diskless replication with async loading (when absent, we fall
back to disk-based loading).
Co-authored-by: Eduardo Semprebon <edus@saxobank.com>
Co-authored-by: Oran Agra <oran@redislabs.com>
2021-11-04 04:46:50 -04:00
|
|
|
set master [srv 0 client]
|
|
|
|
|
set master_host [srv 0 host]
|
|
|
|
|
set master_port [srv 0 port]
|
|
|
|
|
|
|
|
|
|
set start [clock clicks -milliseconds]
|
|
|
|
|
|
|
|
|
|
# Set master and replica to use diskless replication on swapdb mode
|
|
|
|
|
$master config set repl-diskless-sync yes
|
|
|
|
|
$master config set repl-diskless-sync-delay 0
|
|
|
|
|
$master config set save ""
|
|
|
|
|
$replica config set repl-diskless-load swapdb
|
|
|
|
|
$replica config set save ""
|
|
|
|
|
|
|
|
|
|
# Initial sync to have matching replids between master and replica
|
|
|
|
|
$replica replicaof $master_host $master_port
|
|
|
|
|
|
|
|
|
|
# Let replica finish initial sync with master
|
|
|
|
|
wait_for_condition 100 100 {
|
|
|
|
|
[s -1 master_link_status] eq "up"
|
|
|
|
|
} else {
|
|
|
|
|
fail "Master <-> Replica didn't finish sync"
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
# Set global values on module so we can check if module event callbacks will pick it up correctly
|
|
|
|
|
$master testrdb.set.before value1_master
|
|
|
|
|
$replica testrdb.set.before value1_replica
|
|
|
|
|
|
|
|
|
|
# Put different data sets on the master and replica
|
|
|
|
|
# We need to put large keys on the master since the replica replies to info only once in 2mb
|
|
|
|
|
$replica debug populate 200 slave 10
|
|
|
|
|
$master debug populate 1000 master 100000
|
|
|
|
|
$master config set rdbcompression no
|
|
|
|
|
|
|
|
|
|
# Force the replica to try another full sync (this time it will have matching master replid)
|
|
|
|
|
$master multi
|
|
|
|
|
$master client kill type replica
|
|
|
|
|
# Fill replication backlog with new content
|
|
|
|
|
$master config set repl-backlog-size 16384
|
|
|
|
|
for {set keyid 0} {$keyid < 10} {incr keyid} {
|
|
|
|
|
$master set "$keyid string_$keyid" [string repeat A 16384]
|
|
|
|
|
}
|
|
|
|
|
$master exec
|
|
|
|
|
|
|
|
|
|
switch $testType {
|
|
|
|
|
"Aborted" {
|
|
|
|
|
# Set master with a slow rdb generation, so that we can easily intercept loading
|
|
|
|
|
# 10ms per key, with 1000 keys is 10 seconds
|
|
|
|
|
$master config set rdb-key-save-delay 10000
|
|
|
|
|
|
|
|
|
|
test {Diskless load swapdb RedisModuleEvent_ReplAsyncLoad handling: during loading, can keep module variable same as before} {
|
|
|
|
|
# Wait for the replica to start reading the rdb and module for acknowledgement
|
|
|
|
|
# We wanna abort only after the temp db was populated by REDISMODULE_AUX_BEFORE_RDB
|
|
|
|
|
wait_for_condition 100 100 {
|
|
|
|
|
[s -1 async_loading] eq 1 && [$replica testrdb.async_loading.get.before] eq "value1_master"
|
|
|
|
|
} else {
|
|
|
|
|
fail "Module didn't receive or react to REDISMODULE_SUBEVENT_REPL_ASYNC_LOAD_STARTED"
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
assert_equal [$replica dbsize] 200
|
|
|
|
|
assert_equal value1_replica [$replica testrdb.get.before]
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
# Make sure that next sync will not start immediately so that we can catch the replica in between syncs
|
|
|
|
|
$master config set repl-diskless-sync-delay 5
|
|
|
|
|
|
|
|
|
|
# Kill the replica connection on the master
|
|
|
|
|
set killed [$master client kill type replica]
|
|
|
|
|
|
|
|
|
|
test {Diskless load swapdb RedisModuleEvent_ReplAsyncLoad handling: when loading aborted, can keep module variable same as before} {
|
|
|
|
|
# Wait for loading to stop (fail) and module for acknowledgement
|
|
|
|
|
wait_for_condition 100 100 {
|
|
|
|
|
[s -1 async_loading] eq 0 && [$replica testrdb.async_loading.get.before] eq ""
|
|
|
|
|
} else {
|
|
|
|
|
fail "Module didn't receive or react to REDISMODULE_SUBEVENT_REPL_ASYNC_LOAD_ABORTED"
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
assert_equal [$replica dbsize] 200
|
|
|
|
|
assert_equal value1_replica [$replica testrdb.get.before]
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
# Speed up shutdown
|
|
|
|
|
$master config set rdb-key-save-delay 0
|
|
|
|
|
}
|
|
|
|
|
"Successful" {
|
|
|
|
|
# Let replica finish sync with master
|
|
|
|
|
wait_for_condition 100 100 {
|
|
|
|
|
[s -1 master_link_status] eq "up"
|
|
|
|
|
} else {
|
|
|
|
|
fail "Master <-> Replica didn't finish sync"
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
test {Diskless load swapdb RedisModuleEvent_ReplAsyncLoad handling: after db loaded, can set module variable with new value} {
|
|
|
|
|
assert_equal [$replica dbsize] 1010
|
|
|
|
|
assert_equal value1_master [$replica testrdb.get.before]
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
if {$::verbose} {
|
|
|
|
|
set end [clock clicks -milliseconds]
|
|
|
|
|
set duration [expr $end - $start]
|
|
|
|
|
puts "test took $duration ms"
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
}
|
Avoid saving module aux on RDB if no aux data was saved by the module. (#11374)
### Background
The issue is that when saving an RDB with module AUX data, the module AUX metadata
(moduleid, when, ...) is saved to the RDB even though the module did not saved any actual data.
This prevent loading the RDB in the absence of the module (although there is no actual data in
the RDB that requires the module to be loaded).
### Solution
The solution suggested in this PR is that module AUX will be saved on the RDB only if the module
actually saved something during `aux_save` function.
To support backward compatibility, we introduce `aux_save2` callback that acts the same as
`aux_save` with the tiny change of avoid saving the aux field if no data was actually saved by
the module. Modules can use the new API to make sure that if they have no data to save,
then it will be possible to load the created RDB even without the module.
### Concerns
A module may register for the aux load and save hooks just in order to be notified when
saving or loading starts or completed (there are better ways to do that, but it still possible
that someone used it).
However, if a module didn't save a single field in the save callback, it means it's not allowed
to read in the read callback, since it has no way to distinguish between empty and non-empty
payloads. furthermore, it means that if the module did that, it must never change it, since it'll
break compatibility with it's old RDB files, so this is really not a valid use case.
Since some modules (ones who currently save one field indicating an empty payload), need
to know if saving an empty payload is valid, and if Redis is gonna ignore an empty payload
or store it, we opted to add a new API (rather than change behavior of an existing API and
expect modules to check the redis version)
### Technical Details
To avoid saving AUX data on RDB, we change the code to first save the AUX metadata
(moduleid, when, ...) into a temporary buffer. The buffer is then flushed to the rio at the first
time the module makes a write operation inside the `aux_save` function. If the module saves
nothing (and `aux_save2` was used), the entire temporary buffer is simply dropped and no
data about this AUX field is saved to the RDB. This make it possible to load the RDB even in
the absence of the module.
Test was added to verify the fix.
2022-10-18 12:45:46 -04:00
|
|
|
}
|
2019-07-21 11:18:11 -04:00
|
|
|
}
|
2019-07-21 10:41:03 -04:00
|
|
|
}
|
