Sanitize dump payload: ziplist, listpack, zipmap, intset, stream
When loading an encoded payload we will at least do a shallow validation to
check that the size that's encoded in the payload matches the size of the
allocation.
This let's us later use this encoded size to make sure the various offsets
inside encoded payload don't reach outside the allocation, if they do, we'll
assert/panic, but at least we won't segfault or smear memory.
We can also do 'deep' validation which runs on all the records of the encoded
payload and validates that they don't contain invalid offsets. This lets us
detect corruptions early and reject a RESTORE command rather than accepting
it and asserting (crashing) later when accessing that payload via some command.
configuration:
- adding ACL flag skip-sanitize-payload
- adding config sanitize-dump-payload [yes/no/clients]
For now, we don't have a good way to ensure MIGRATE in cluster resharding isn't
being slowed down by these sanitation, so i'm setting the default value to `no`,
but later on it should be set to `clients` by default.
changes:
- changing rdbReportError not to `exit` in RESTORE command
- adding a new stat to be able to later check if cluster MIGRATE isn't being
slowed down by sanitation.
2020-08-13 09:41:05 -04:00
# tests of corrupt ziplist payload with valid CRC
2020-08-14 09:05:34 -04:00
# * setting crash-memcheck-enabled to no to avoid issues with valgrind
# * setting use-exit-on-panic to yes so that valgrind can search for leaks
2021-06-10 08:39:33 -04:00
# * setting debug set-skip-checksum-validation to 1 on some tests for which we
2020-08-14 09:05:34 -04:00
# didn't bother to fake a valid checksum
# * some tests set sanitize-dump-payload to no and some to yet, depending on
# what we want to test
Sanitize dump payload: ziplist, listpack, zipmap, intset, stream
When loading an encoded payload we will at least do a shallow validation to
check that the size that's encoded in the payload matches the size of the
allocation.
This let's us later use this encoded size to make sure the various offsets
inside encoded payload don't reach outside the allocation, if they do, we'll
assert/panic, but at least we won't segfault or smear memory.
We can also do 'deep' validation which runs on all the records of the encoded
payload and validates that they don't contain invalid offsets. This lets us
detect corruptions early and reject a RESTORE command rather than accepting
it and asserting (crashing) later when accessing that payload via some command.
configuration:
- adding ACL flag skip-sanitize-payload
- adding config sanitize-dump-payload [yes/no/clients]
For now, we don't have a good way to ensure MIGRATE in cluster resharding isn't
being slowed down by these sanitation, so i'm setting the default value to `no`,
but later on it should be set to `clients` by default.
changes:
- changing rdbReportError not to `exit` in RESTORE command
- adding a new stat to be able to later check if cluster MIGRATE isn't being
slowed down by sanitation.
2020-08-13 09:41:05 -04:00
2021-06-09 08:13:24 -04:00
tags { " d u m p " " c o r r u p t i o n " " e x t e r n a l : s k i p " } {
Sanitize dump payload: ziplist, listpack, zipmap, intset, stream
When loading an encoded payload we will at least do a shallow validation to
check that the size that's encoded in the payload matches the size of the
allocation.
This let's us later use this encoded size to make sure the various offsets
inside encoded payload don't reach outside the allocation, if they do, we'll
assert/panic, but at least we won't segfault or smear memory.
We can also do 'deep' validation which runs on all the records of the encoded
payload and validates that they don't contain invalid offsets. This lets us
detect corruptions early and reject a RESTORE command rather than accepting
it and asserting (crashing) later when accessing that payload via some command.
configuration:
- adding ACL flag skip-sanitize-payload
- adding config sanitize-dump-payload [yes/no/clients]
For now, we don't have a good way to ensure MIGRATE in cluster resharding isn't
being slowed down by these sanitation, so i'm setting the default value to `no`,
but later on it should be set to `clients` by default.
changes:
- changing rdbReportError not to `exit` in RESTORE command
- adding a new stat to be able to later check if cluster MIGRATE isn't being
slowed down by sanitation.
2020-08-13 09:41:05 -04:00
2021-08-18 09:00:22 -04:00
# We only run OOM related tests on x86_64 and aarch64, as jemalloc on other
# platforms (notably s390x) may actually succeed very large allocations. As
# a result the test may hang for a very long time at the cleanup phase,
# iterating as many as 2^61 hash table slots.
set arch_name [ exec uname - m]
set run_oom_tests [ expr { $arch_name == " x 8 6 _ 6 4 " || $arch_name == " a a r c h 6 4 " } ]
Sanitize dump payload: ziplist, listpack, zipmap, intset, stream
When loading an encoded payload we will at least do a shallow validation to
check that the size that's encoded in the payload matches the size of the
allocation.
This let's us later use this encoded size to make sure the various offsets
inside encoded payload don't reach outside the allocation, if they do, we'll
assert/panic, but at least we won't segfault or smear memory.
We can also do 'deep' validation which runs on all the records of the encoded
payload and validates that they don't contain invalid offsets. This lets us
detect corruptions early and reject a RESTORE command rather than accepting
it and asserting (crashing) later when accessing that payload via some command.
configuration:
- adding ACL flag skip-sanitize-payload
- adding config sanitize-dump-payload [yes/no/clients]
For now, we don't have a good way to ensure MIGRATE in cluster resharding isn't
being slowed down by these sanitation, so i'm setting the default value to `no`,
but later on it should be set to `clients` by default.
changes:
- changing rdbReportError not to `exit` in RESTORE command
- adding a new stat to be able to later check if cluster MIGRATE isn't being
slowed down by sanitation.
2020-08-13 09:41:05 -04:00
set corrupt_payload_7445 " \x 0 E \x 0 1 \x 1 D \x 1 D \x 0 0 \x 0 0 \x 0 0 \x 1 6 \x 0 0 \x 0 0 \x 0 0 \x 0 3 \x 0 0 \x 0 0 \x 0 4 \x 4 3 \x 4 3 \x 4 3 \x 4 3 \x 0 6 \x 0 4 \x 4 2 \x 4 2 \x 4 2 \x 4 2 \x 0 6 \x 3 F \x 4 1 \x 4 1 \x 4 1 \x 4 1 \x F F \x 0 9 \x 0 0 \x 8 8 \x A 5 \x C A \x A 8 \x C 5 \x 4 1 \x F 4 \x 3 5 "
test { corrupt payload: # 7445 - with sanitize} {
start_server [ list overrides [ list loglevel verbose use-exit-on-panic yes crash-memcheck-enabled no] ] {
r config set sanitize-dump-payload yes
catch {
r restore key 0 $corrupt_payload_7445
} err
assert_match " * B a d d a t a f o r m a t * " $err
2020-11-02 02:35:37 -05:00
verify_log_message 0 " * i n t e g r i t y c h e c k f a i l e d * " 0
Sanitize dump payload: ziplist, listpack, zipmap, intset, stream
When loading an encoded payload we will at least do a shallow validation to
check that the size that's encoded in the payload matches the size of the
allocation.
This let's us later use this encoded size to make sure the various offsets
inside encoded payload don't reach outside the allocation, if they do, we'll
assert/panic, but at least we won't segfault or smear memory.
We can also do 'deep' validation which runs on all the records of the encoded
payload and validates that they don't contain invalid offsets. This lets us
detect corruptions early and reject a RESTORE command rather than accepting
it and asserting (crashing) later when accessing that payload via some command.
configuration:
- adding ACL flag skip-sanitize-payload
- adding config sanitize-dump-payload [yes/no/clients]
For now, we don't have a good way to ensure MIGRATE in cluster resharding isn't
being slowed down by these sanitation, so i'm setting the default value to `no`,
but later on it should be set to `clients` by default.
changes:
- changing rdbReportError not to `exit` in RESTORE command
- adding a new stat to be able to later check if cluster MIGRATE isn't being
slowed down by sanitation.
2020-08-13 09:41:05 -04:00
}
}
test { corrupt payload: hash with valid zip list header, invalid entry len} {
start_server [ list overrides [ list loglevel verbose use-exit-on-panic yes crash-memcheck-enabled no] ] {
2021-08-10 02:18:49 -04:00
catch {
r restore key 0 " \x 0 D \x 1 B \x 1 B \x 0 0 \x 0 0 \x 0 0 \x 1 6 \x 0 0 \x 0 0 \x 0 0 \x 0 4 \x 0 0 \x 0 0 \x 0 2 \x 6 1 \x 0 0 \x 0 4 \x 0 2 \x 6 2 \x 0 0 \x 0 4 \x 1 4 \x 6 3 \x 0 0 \x 0 4 \x 0 2 \x 6 4 \x 0 0 \x F F \x 0 9 \x 0 0 \x D 9 \x 1 0 \x 5 4 \x 9 2 \x 1 5 \x F 5 \x 5 F \x 5 2 "
} err
assert_match " * B a d d a t a f o r m a t * " $err
verify_log_message 0 " * i n t e g r i t y c h e c k f a i l e d * " 0
Sanitize dump payload: ziplist, listpack, zipmap, intset, stream
When loading an encoded payload we will at least do a shallow validation to
check that the size that's encoded in the payload matches the size of the
allocation.
This let's us later use this encoded size to make sure the various offsets
inside encoded payload don't reach outside the allocation, if they do, we'll
assert/panic, but at least we won't segfault or smear memory.
We can also do 'deep' validation which runs on all the records of the encoded
payload and validates that they don't contain invalid offsets. This lets us
detect corruptions early and reject a RESTORE command rather than accepting
it and asserting (crashing) later when accessing that payload via some command.
configuration:
- adding ACL flag skip-sanitize-payload
- adding config sanitize-dump-payload [yes/no/clients]
For now, we don't have a good way to ensure MIGRATE in cluster resharding isn't
being slowed down by these sanitation, so i'm setting the default value to `no`,
but later on it should be set to `clients` by default.
changes:
- changing rdbReportError not to `exit` in RESTORE command
- adding a new stat to be able to later check if cluster MIGRATE isn't being
slowed down by sanitation.
2020-08-13 09:41:05 -04:00
}
}
test { corrupt payload: invalid zlbytes header} {
start_server [ list overrides [ list loglevel verbose use-exit-on-panic yes crash-memcheck-enabled no] ] {
catch {
r restore key 0 " \x 0 D \x 1 B \x 2 5 \x 0 0 \x 0 0 \x 0 0 \x 1 6 \x 0 0 \x 0 0 \x 0 0 \x 0 4 \x 0 0 \x 0 0 \x 0 2 \x 6 1 \x 0 0 \x 0 4 \x 0 2 \x 6 2 \x 0 0 \x 0 4 \x 0 2 \x 6 3 \x 0 0 \x 0 4 \x 0 2 \x 6 4 \x 0 0 \x F F \x 0 9 \x 0 0 \x B 7 \x F 7 \x 6 E \x 9 F \x 4 3 \x 4 3 \x 1 4 \x C 6 "
} err
assert_match " * B a d d a t a f o r m a t * " $err
2021-08-10 02:18:49 -04:00
verify_log_message 0 " * i n t e g r i t y c h e c k f a i l e d * " 0
Sanitize dump payload: ziplist, listpack, zipmap, intset, stream
When loading an encoded payload we will at least do a shallow validation to
check that the size that's encoded in the payload matches the size of the
allocation.
This let's us later use this encoded size to make sure the various offsets
inside encoded payload don't reach outside the allocation, if they do, we'll
assert/panic, but at least we won't segfault or smear memory.
We can also do 'deep' validation which runs on all the records of the encoded
payload and validates that they don't contain invalid offsets. This lets us
detect corruptions early and reject a RESTORE command rather than accepting
it and asserting (crashing) later when accessing that payload via some command.
configuration:
- adding ACL flag skip-sanitize-payload
- adding config sanitize-dump-payload [yes/no/clients]
For now, we don't have a good way to ensure MIGRATE in cluster resharding isn't
being slowed down by these sanitation, so i'm setting the default value to `no`,
but later on it should be set to `clients` by default.
changes:
- changing rdbReportError not to `exit` in RESTORE command
- adding a new stat to be able to later check if cluster MIGRATE isn't being
slowed down by sanitation.
2020-08-13 09:41:05 -04:00
}
}
test { corrupt payload: valid zipped hash header, dup records} {
start_server [ list overrides [ list loglevel verbose use-exit-on-panic yes crash-memcheck-enabled no] ] {
2021-08-10 02:18:49 -04:00
catch {
r restore key 0 " \x 0 D \x 1 B \x 1 B \x 0 0 \x 0 0 \x 0 0 \x 1 6 \x 0 0 \x 0 0 \x 0 0 \x 0 4 \x 0 0 \x 0 0 \x 0 2 \x 6 1 \x 0 0 \x 0 4 \x 0 2 \x 6 2 \x 0 0 \x 0 4 \x 0 2 \x 6 1 \x 0 0 \x 0 4 \x 0 2 \x 6 4 \x 0 0 \x F F \x 0 9 \x 0 0 \x A 1 \x 9 8 \x 3 6 \x 7 8 \x C C \x 8 E \x 9 3 \x 2 E "
} err
assert_match " * B a d d a t a f o r m a t * " $err
verify_log_message 0 " * i n t e g r i t y c h e c k f a i l e d * " 0
Sanitize dump payload: ziplist, listpack, zipmap, intset, stream
When loading an encoded payload we will at least do a shallow validation to
check that the size that's encoded in the payload matches the size of the
allocation.
This let's us later use this encoded size to make sure the various offsets
inside encoded payload don't reach outside the allocation, if they do, we'll
assert/panic, but at least we won't segfault or smear memory.
We can also do 'deep' validation which runs on all the records of the encoded
payload and validates that they don't contain invalid offsets. This lets us
detect corruptions early and reject a RESTORE command rather than accepting
it and asserting (crashing) later when accessing that payload via some command.
configuration:
- adding ACL flag skip-sanitize-payload
- adding config sanitize-dump-payload [yes/no/clients]
For now, we don't have a good way to ensure MIGRATE in cluster resharding isn't
being slowed down by these sanitation, so i'm setting the default value to `no`,
but later on it should be set to `clients` by default.
changes:
- changing rdbReportError not to `exit` in RESTORE command
- adding a new stat to be able to later check if cluster MIGRATE isn't being
slowed down by sanitation.
2020-08-13 09:41:05 -04:00
}
}
test { corrupt payload: quicklist big ziplist prev len} {
start_server [ list overrides [ list loglevel verbose use-exit-on-panic yes crash-memcheck-enabled no] ] {
2020-08-14 09:05:34 -04:00
r config set sanitize-dump-payload no
2021-11-24 06:34:13 -05:00
catch { r restore key 0 " \x 0 E \x 0 1 \x 1 3 \x 1 3 \x 0 0 \x 0 0 \x 0 0 \x 0 E \x 0 0 \x 0 0 \x 0 0 \x 0 2 \x 0 0 \x 0 0 \x 0 2 \x 6 1 \x 0 0 \x 0 E \x 0 2 \x 6 2 \x 0 0 \x F F \x 0 9 \x 0 0 \x 4 9 \x 9 7 \x 3 0 \x B 2 \x 0 D \x A 1 \x E D \x A A " } err
assert_match " * B a d d a t a f o r m a t * " $err
verify_log_message 0 " * i n t e g r i t y c h e c k f a i l e d * " 0
Sanitize dump payload: ziplist, listpack, zipmap, intset, stream
When loading an encoded payload we will at least do a shallow validation to
check that the size that's encoded in the payload matches the size of the
allocation.
This let's us later use this encoded size to make sure the various offsets
inside encoded payload don't reach outside the allocation, if they do, we'll
assert/panic, but at least we won't segfault or smear memory.
We can also do 'deep' validation which runs on all the records of the encoded
payload and validates that they don't contain invalid offsets. This lets us
detect corruptions early and reject a RESTORE command rather than accepting
it and asserting (crashing) later when accessing that payload via some command.
configuration:
- adding ACL flag skip-sanitize-payload
- adding config sanitize-dump-payload [yes/no/clients]
For now, we don't have a good way to ensure MIGRATE in cluster resharding isn't
being slowed down by these sanitation, so i'm setting the default value to `no`,
but later on it should be set to `clients` by default.
changes:
- changing rdbReportError not to `exit` in RESTORE command
- adding a new stat to be able to later check if cluster MIGRATE isn't being
slowed down by sanitation.
2020-08-13 09:41:05 -04:00
}
}
test { corrupt payload: quicklist small ziplist prev len} {
start_server [ list overrides [ list loglevel verbose use-exit-on-panic yes crash-memcheck-enabled no] ] {
r config set sanitize-dump-payload yes
catch {
r restore key 0 " \x 0 E \x 0 1 \x 1 3 \x 1 3 \x 0 0 \x 0 0 \x 0 0 \x 0 E \x 0 0 \x 0 0 \x 0 0 \x 0 2 \x 0 0 \x 0 0 \x 0 2 \x 6 1 \x 0 0 \x 0 2 \x 0 2 \x 6 2 \x 0 0 \x F F \x 0 9 \x 0 0 \x C 7 \x 7 1 \x 0 3 \x 9 7 \x 0 7 \x 7 5 \x B 0 \x 6 3 "
} err
assert_match " * B a d d a t a f o r m a t * " $err
2020-11-02 02:35:37 -05:00
verify_log_message 0 " * i n t e g r i t y c h e c k f a i l e d * " 0
Sanitize dump payload: ziplist, listpack, zipmap, intset, stream
When loading an encoded payload we will at least do a shallow validation to
check that the size that's encoded in the payload matches the size of the
allocation.
This let's us later use this encoded size to make sure the various offsets
inside encoded payload don't reach outside the allocation, if they do, we'll
assert/panic, but at least we won't segfault or smear memory.
We can also do 'deep' validation which runs on all the records of the encoded
payload and validates that they don't contain invalid offsets. This lets us
detect corruptions early and reject a RESTORE command rather than accepting
it and asserting (crashing) later when accessing that payload via some command.
configuration:
- adding ACL flag skip-sanitize-payload
- adding config sanitize-dump-payload [yes/no/clients]
For now, we don't have a good way to ensure MIGRATE in cluster resharding isn't
being slowed down by these sanitation, so i'm setting the default value to `no`,
but later on it should be set to `clients` by default.
changes:
- changing rdbReportError not to `exit` in RESTORE command
- adding a new stat to be able to later check if cluster MIGRATE isn't being
slowed down by sanitation.
2020-08-13 09:41:05 -04:00
}
}
test { corrupt payload: quicklist ziplist wrong count} {
start_server [ list overrides [ list loglevel verbose use-exit-on-panic yes crash-memcheck-enabled no] ] {
2020-08-14 09:05:34 -04:00
r config set sanitize-dump-payload no
2021-11-24 06:34:13 -05:00
catch { r restore key 0 " \x 0 E \x 0 1 \x 1 3 \x 1 3 \x 0 0 \x 0 0 \x 0 0 \x 0 E \x 0 0 \x 0 0 \x 0 0 \x 0 3 \x 0 0 \x 0 0 \x 0 2 \x 6 1 \x 0 0 \x 0 4 \x 0 2 \x 6 2 \x 0 0 \x F F \x 0 9 \x 0 0 \x 4 D \x E 2 \x 0 A \x 2 F \x 0 8 \x 2 5 \x D F \x 9 1 " } err
assert_match " * B a d d a t a f o r m a t * " $err
verify_log_message 0 " * i n t e g r i t y c h e c k f a i l e d * " 0
Sanitize dump payload: ziplist, listpack, zipmap, intset, stream
When loading an encoded payload we will at least do a shallow validation to
check that the size that's encoded in the payload matches the size of the
allocation.
This let's us later use this encoded size to make sure the various offsets
inside encoded payload don't reach outside the allocation, if they do, we'll
assert/panic, but at least we won't segfault or smear memory.
We can also do 'deep' validation which runs on all the records of the encoded
payload and validates that they don't contain invalid offsets. This lets us
detect corruptions early and reject a RESTORE command rather than accepting
it and asserting (crashing) later when accessing that payload via some command.
configuration:
- adding ACL flag skip-sanitize-payload
- adding config sanitize-dump-payload [yes/no/clients]
For now, we don't have a good way to ensure MIGRATE in cluster resharding isn't
being slowed down by these sanitation, so i'm setting the default value to `no`,
but later on it should be set to `clients` by default.
changes:
- changing rdbReportError not to `exit` in RESTORE command
- adding a new stat to be able to later check if cluster MIGRATE isn't being
slowed down by sanitation.
2020-08-13 09:41:05 -04:00
}
}
test { corrupt payload: # 3080 - quicklist} {
start_server [ list overrides [ list loglevel verbose use-exit-on-panic yes crash-memcheck-enabled no] ] {
2020-08-14 09:05:34 -04:00
r config set sanitize-dump-payload no
Sanitize dump payload: ziplist, listpack, zipmap, intset, stream
When loading an encoded payload we will at least do a shallow validation to
check that the size that's encoded in the payload matches the size of the
allocation.
This let's us later use this encoded size to make sure the various offsets
inside encoded payload don't reach outside the allocation, if they do, we'll
assert/panic, but at least we won't segfault or smear memory.
We can also do 'deep' validation which runs on all the records of the encoded
payload and validates that they don't contain invalid offsets. This lets us
detect corruptions early and reject a RESTORE command rather than accepting
it and asserting (crashing) later when accessing that payload via some command.
configuration:
- adding ACL flag skip-sanitize-payload
- adding config sanitize-dump-payload [yes/no/clients]
For now, we don't have a good way to ensure MIGRATE in cluster resharding isn't
being slowed down by these sanitation, so i'm setting the default value to `no`,
but later on it should be set to `clients` by default.
changes:
- changing rdbReportError not to `exit` in RESTORE command
- adding a new stat to be able to later check if cluster MIGRATE isn't being
slowed down by sanitation.
2020-08-13 09:41:05 -04:00
catch {
r RESTORE key 0 " \x 0 E \x 0 1 \x 8 0 \x 0 0 \x 0 0 \x 0 0 \x 1 0 \x 4 1 \x 4 1 \x 4 1 \x 4 1 \x 4 1 \x 4 1 \x 4 1 \x 4 1 \x 0 2 \x 0 0 \x 0 0 \x 8 0 \x 4 1 \x 4 1 \x 4 1 \x 4 1 \x 0 7 \x 0 0 \x 0 3 \x C 7 \x 1 D \x E F \x 5 4 \x 6 8 \x C C \x F 3 "
2020-08-14 09:05:34 -04:00
r DUMP key ; # DUMP was used in the original issue, but now even with shallow sanitization restore safely fails, so this is dead code
} err
Sanitize dump payload: ziplist, listpack, zipmap, intset, stream
When loading an encoded payload we will at least do a shallow validation to
check that the size that's encoded in the payload matches the size of the
allocation.
This let's us later use this encoded size to make sure the various offsets
inside encoded payload don't reach outside the allocation, if they do, we'll
assert/panic, but at least we won't segfault or smear memory.
We can also do 'deep' validation which runs on all the records of the encoded
payload and validates that they don't contain invalid offsets. This lets us
detect corruptions early and reject a RESTORE command rather than accepting
it and asserting (crashing) later when accessing that payload via some command.
configuration:
- adding ACL flag skip-sanitize-payload
- adding config sanitize-dump-payload [yes/no/clients]
For now, we don't have a good way to ensure MIGRATE in cluster resharding isn't
being slowed down by these sanitation, so i'm setting the default value to `no`,
but later on it should be set to `clients` by default.
changes:
- changing rdbReportError not to `exit` in RESTORE command
- adding a new stat to be able to later check if cluster MIGRATE isn't being
slowed down by sanitation.
2020-08-13 09:41:05 -04:00
assert_match " * B a d d a t a f o r m a t * " $err
2020-11-02 02:35:37 -05:00
verify_log_message 0 " * i n t e g r i t y c h e c k f a i l e d * " 0
Sanitize dump payload: ziplist, listpack, zipmap, intset, stream
When loading an encoded payload we will at least do a shallow validation to
check that the size that's encoded in the payload matches the size of the
allocation.
This let's us later use this encoded size to make sure the various offsets
inside encoded payload don't reach outside the allocation, if they do, we'll
assert/panic, but at least we won't segfault or smear memory.
We can also do 'deep' validation which runs on all the records of the encoded
payload and validates that they don't contain invalid offsets. This lets us
detect corruptions early and reject a RESTORE command rather than accepting
it and asserting (crashing) later when accessing that payload via some command.
configuration:
- adding ACL flag skip-sanitize-payload
- adding config sanitize-dump-payload [yes/no/clients]
For now, we don't have a good way to ensure MIGRATE in cluster resharding isn't
being slowed down by these sanitation, so i'm setting the default value to `no`,
but later on it should be set to `clients` by default.
changes:
- changing rdbReportError not to `exit` in RESTORE command
- adding a new stat to be able to later check if cluster MIGRATE isn't being
slowed down by sanitation.
2020-08-13 09:41:05 -04:00
}
}
2021-08-09 10:13:46 -04:00
test { corrupt payload: quicklist with empty ziplist} {
start_server [ list overrides [ list loglevel verbose use-exit-on-panic yes crash-memcheck-enabled no] ] {
r config set sanitize-dump-payload no
r debug set-skip-checksum-validation 1
catch { r restore key 0 " \x 0 E \x 0 1 \x 0 B \x 0 B \x 0 0 \x 0 0 \x 0 0 \x 0 A \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x F F \x 0 9 \x 0 0 \x C 2 \x 6 9 \x 3 7 \x 8 3 \x 3 C \x 7 F \x F E \x 6 F " replace} err
assert_match " * B a d d a t a f o r m a t * " $err
r ping
}
}
2021-11-03 14:47:18 -04:00
test { corrupt payload: quicklist encoded_len is 0 } {
start_server [ list overrides [ list loglevel verbose use-exit-on-panic yes crash-memcheck-enabled no] ] {
2021-11-22 05:30:06 -05:00
catch { r restore _list 0 " \x 1 2 \x 0 1 \x 0 1 \x 0 0 \x 0 a \x 0 0 \x 8 f \x c 6 \x c 0 \x 5 7 \x 1 c \x 0 a \x b 3 \x 3 c " replace } err
2021-11-03 14:47:18 -04:00
assert_match " * B a d d a t a f o r m a t * " $err
r ping
}
}
2021-12-04 09:43:08 -05:00
test { corrupt payload: quicklist listpack entry start with EOF} {
start_server [ list overrides [ list loglevel verbose use-exit-on-panic yes crash-memcheck-enabled no] ] {
r config set sanitize-dump-payload yes
catch { r restore _list 0 " \x 1 2 \x 0 1 \x 0 2 \x 0 b \x 0 b \x 0 0 \x 0 0 \x 0 0 \x 0 1 \x 0 0 \x 8 1 \x 6 1 \x 0 2 \x f f \x f f \x 0 a \x 0 0 \x 7 e \x d 8 \x d e \x 5 b \x 0 d \x d 7 \x 7 0 \x b 8 " replace } err
assert_match " * B a d d a t a f o r m a t * " $err
r ping
}
}
Sanitize dump payload: ziplist, listpack, zipmap, intset, stream
When loading an encoded payload we will at least do a shallow validation to
check that the size that's encoded in the payload matches the size of the
allocation.
This let's us later use this encoded size to make sure the various offsets
inside encoded payload don't reach outside the allocation, if they do, we'll
assert/panic, but at least we won't segfault or smear memory.
We can also do 'deep' validation which runs on all the records of the encoded
payload and validates that they don't contain invalid offsets. This lets us
detect corruptions early and reject a RESTORE command rather than accepting
it and asserting (crashing) later when accessing that payload via some command.
configuration:
- adding ACL flag skip-sanitize-payload
- adding config sanitize-dump-payload [yes/no/clients]
For now, we don't have a good way to ensure MIGRATE in cluster resharding isn't
being slowed down by these sanitation, so i'm setting the default value to `no`,
but later on it should be set to `clients` by default.
changes:
- changing rdbReportError not to `exit` in RESTORE command
- adding a new stat to be able to later check if cluster MIGRATE isn't being
slowed down by sanitation.
2020-08-13 09:41:05 -04:00
test { corrupt payload: # 3080 - ziplist} {
start_server [ list overrides [ list loglevel verbose use-exit-on-panic yes crash-memcheck-enabled no] ] {
2020-08-14 09:05:34 -04:00
# shallow sanitization is enough for restore to safely reject the payload with wrong size
r config set sanitize-dump-payload no
Sanitize dump payload: ziplist, listpack, zipmap, intset, stream
When loading an encoded payload we will at least do a shallow validation to
check that the size that's encoded in the payload matches the size of the
allocation.
This let's us later use this encoded size to make sure the various offsets
inside encoded payload don't reach outside the allocation, if they do, we'll
assert/panic, but at least we won't segfault or smear memory.
We can also do 'deep' validation which runs on all the records of the encoded
payload and validates that they don't contain invalid offsets. This lets us
detect corruptions early and reject a RESTORE command rather than accepting
it and asserting (crashing) later when accessing that payload via some command.
configuration:
- adding ACL flag skip-sanitize-payload
- adding config sanitize-dump-payload [yes/no/clients]
For now, we don't have a good way to ensure MIGRATE in cluster resharding isn't
being slowed down by these sanitation, so i'm setting the default value to `no`,
but later on it should be set to `clients` by default.
changes:
- changing rdbReportError not to `exit` in RESTORE command
- adding a new stat to be able to later check if cluster MIGRATE isn't being
slowed down by sanitation.
2020-08-13 09:41:05 -04:00
catch {
r RESTORE key 0 " \x 0 A \x 8 0 \x 0 0 \x 0 0 \x 0 0 \x 1 0 \x 4 1 \x 4 1 \x 4 1 \x 4 1 \x 4 1 \x 4 1 \x 4 1 \x 4 1 \x 0 2 \x 0 0 \x 0 0 \x 8 0 \x 4 1 \x 4 1 \x 4 1 \x 4 1 \x 0 7 \x 0 0 \x 3 9 \x 5 B \x 4 9 \x E 0 \x C 1 \x C 6 \x D D \x 7 6 "
2020-08-14 09:05:34 -04:00
} err
Sanitize dump payload: ziplist, listpack, zipmap, intset, stream
When loading an encoded payload we will at least do a shallow validation to
check that the size that's encoded in the payload matches the size of the
allocation.
This let's us later use this encoded size to make sure the various offsets
inside encoded payload don't reach outside the allocation, if they do, we'll
assert/panic, but at least we won't segfault or smear memory.
We can also do 'deep' validation which runs on all the records of the encoded
payload and validates that they don't contain invalid offsets. This lets us
detect corruptions early and reject a RESTORE command rather than accepting
it and asserting (crashing) later when accessing that payload via some command.
configuration:
- adding ACL flag skip-sanitize-payload
- adding config sanitize-dump-payload [yes/no/clients]
For now, we don't have a good way to ensure MIGRATE in cluster resharding isn't
being slowed down by these sanitation, so i'm setting the default value to `no`,
but later on it should be set to `clients` by default.
changes:
- changing rdbReportError not to `exit` in RESTORE command
- adding a new stat to be able to later check if cluster MIGRATE isn't being
slowed down by sanitation.
2020-08-13 09:41:05 -04:00
assert_match " * B a d d a t a f o r m a t * " $err
2020-11-02 02:35:37 -05:00
verify_log_message 0 " * i n t e g r i t y c h e c k f a i l e d * " 0
Sanitize dump payload: ziplist, listpack, zipmap, intset, stream
When loading an encoded payload we will at least do a shallow validation to
check that the size that's encoded in the payload matches the size of the
allocation.
This let's us later use this encoded size to make sure the various offsets
inside encoded payload don't reach outside the allocation, if they do, we'll
assert/panic, but at least we won't segfault or smear memory.
We can also do 'deep' validation which runs on all the records of the encoded
payload and validates that they don't contain invalid offsets. This lets us
detect corruptions early and reject a RESTORE command rather than accepting
it and asserting (crashing) later when accessing that payload via some command.
configuration:
- adding ACL flag skip-sanitize-payload
- adding config sanitize-dump-payload [yes/no/clients]
For now, we don't have a good way to ensure MIGRATE in cluster resharding isn't
being slowed down by these sanitation, so i'm setting the default value to `no`,
but later on it should be set to `clients` by default.
changes:
- changing rdbReportError not to `exit` in RESTORE command
- adding a new stat to be able to later check if cluster MIGRATE isn't being
slowed down by sanitation.
2020-08-13 09:41:05 -04:00
}
}
test { corrupt payload: load corrupted rdb with no CRC - # 3505 } {
set server_path [ tmpdir " s e r v e r . r d b - c o r r u p t i o n - t e s t " ]
exec cp tests/ assets/ corrupt_ziplist.rdb $server_path
2020-08-14 09:05:34 -04:00
set srv [ start_server [ list overrides [ list " d i r " $server_path " d b f i l e n a m e " " c o r r u p t _ z i p l i s t . r d b " loglevel verbose use-exit-on-panic yes crash-memcheck-enabled no sanitize-dump-payload no] ] ]
Sanitize dump payload: ziplist, listpack, zipmap, intset, stream
When loading an encoded payload we will at least do a shallow validation to
check that the size that's encoded in the payload matches the size of the
allocation.
This let's us later use this encoded size to make sure the various offsets
inside encoded payload don't reach outside the allocation, if they do, we'll
assert/panic, but at least we won't segfault or smear memory.
We can also do 'deep' validation which runs on all the records of the encoded
payload and validates that they don't contain invalid offsets. This lets us
detect corruptions early and reject a RESTORE command rather than accepting
it and asserting (crashing) later when accessing that payload via some command.
configuration:
- adding ACL flag skip-sanitize-payload
- adding config sanitize-dump-payload [yes/no/clients]
For now, we don't have a good way to ensure MIGRATE in cluster resharding isn't
being slowed down by these sanitation, so i'm setting the default value to `no`,
but later on it should be set to `clients` by default.
changes:
- changing rdbReportError not to `exit` in RESTORE command
- adding a new stat to be able to later check if cluster MIGRATE isn't being
slowed down by sanitation.
2020-08-13 09:41:05 -04:00
# wait for termination
wait_for_condition 100 50 {
2023-10-03 04:09:25 -04:00
! [ is_alive [ dict get $srv pid] ]
Sanitize dump payload: ziplist, listpack, zipmap, intset, stream
When loading an encoded payload we will at least do a shallow validation to
check that the size that's encoded in the payload matches the size of the
allocation.
This let's us later use this encoded size to make sure the various offsets
inside encoded payload don't reach outside the allocation, if they do, we'll
assert/panic, but at least we won't segfault or smear memory.
We can also do 'deep' validation which runs on all the records of the encoded
payload and validates that they don't contain invalid offsets. This lets us
detect corruptions early and reject a RESTORE command rather than accepting
it and asserting (crashing) later when accessing that payload via some command.
configuration:
- adding ACL flag skip-sanitize-payload
- adding config sanitize-dump-payload [yes/no/clients]
For now, we don't have a good way to ensure MIGRATE in cluster resharding isn't
being slowed down by these sanitation, so i'm setting the default value to `no`,
but later on it should be set to `clients` by default.
changes:
- changing rdbReportError not to `exit` in RESTORE command
- adding a new stat to be able to later check if cluster MIGRATE isn't being
slowed down by sanitation.
2020-08-13 09:41:05 -04:00
} else {
fail " r d b l o a d i n g d i d n ' t f a i l "
}
set stdout [ dict get $srv stdout]
assert_equal [ count_message_lines $stdout " T e r m i n a t i n g s e r v e r a f t e r r d b f i l e r e a d i n g f a i l u r e . " ] 1
2020-11-02 02:35:37 -05:00
assert_lessthan 1 [ count_message_lines $stdout " i n t e g r i t y c h e c k f a i l e d " ]
Sanitize dump payload: ziplist, listpack, zipmap, intset, stream
When loading an encoded payload we will at least do a shallow validation to
check that the size that's encoded in the payload matches the size of the
allocation.
This let's us later use this encoded size to make sure the various offsets
inside encoded payload don't reach outside the allocation, if they do, we'll
assert/panic, but at least we won't segfault or smear memory.
We can also do 'deep' validation which runs on all the records of the encoded
payload and validates that they don't contain invalid offsets. This lets us
detect corruptions early and reject a RESTORE command rather than accepting
it and asserting (crashing) later when accessing that payload via some command.
configuration:
- adding ACL flag skip-sanitize-payload
- adding config sanitize-dump-payload [yes/no/clients]
For now, we don't have a good way to ensure MIGRATE in cluster resharding isn't
being slowed down by these sanitation, so i'm setting the default value to `no`,
but later on it should be set to `clients` by default.
changes:
- changing rdbReportError not to `exit` in RESTORE command
- adding a new stat to be able to later check if cluster MIGRATE isn't being
slowed down by sanitation.
2020-08-13 09:41:05 -04:00
kill_server $srv ; # let valgrind look for issues
}
2021-08-09 10:13:46 -04:00
foreach sanitize_dump { no yes} {
test { corrupt payload: load corrupted rdb with empty keys} {
set server_path [ tmpdir " s e r v e r . r d b - c o r r u p t i o n - e m p t y - k e y s - t e s t " ]
exec cp tests/ assets/ corrupt_empty_keys.rdb $server_path
start_server [ list overrides [ list " d i r " $server_path " d b f i l e n a m e " " c o r r u p t _ e m p t y _ k e y s . r d b " " s a n i t i z e - d u m p - p a y l o a d " $sanitize_dump ] ] {
r select 0
assert_equal [ r dbsize] 0
2021-08-05 15:42:20 -04:00
2021-08-09 10:13:46 -04:00
verify_log_message 0 " * s k i p p i n g e m p t y k e y : s e t * " 0
verify_log_message 0 " * s k i p p i n g e m p t y k e y : l i s t _ q u i c k l i s t * " 0
verify_log_message 0 " * s k i p p i n g e m p t y k e y : l i s t _ q u i c k l i s t _ e m p t y _ z i p l i s t * " 0
verify_log_message 0 " * s k i p p i n g e m p t y k e y : l i s t _ z i p l i s t * " 0
verify_log_message 0 " * s k i p p i n g e m p t y k e y : h a s h * " 0
verify_log_message 0 " * s k i p p i n g e m p t y k e y : h a s h _ z i p l i s t * " 0
verify_log_message 0 " * s k i p p i n g e m p t y k e y : z s e t * " 0
verify_log_message 0 " * s k i p p i n g e m p t y k e y : z s e t _ z i p l i s t * " 0
2021-09-09 11:18:53 -04:00
verify_log_message 0 " * s k i p p i n g e m p t y k e y : z s e t _ l i s t p a c k * " 0
verify_log_message 0 " * e m p t y k e y s s k i p p e d : 9 * " 0
2021-08-09 10:13:46 -04:00
}
2021-08-05 15:42:20 -04:00
}
}
Sanitize dump payload: ziplist, listpack, zipmap, intset, stream
When loading an encoded payload we will at least do a shallow validation to
check that the size that's encoded in the payload matches the size of the
allocation.
This let's us later use this encoded size to make sure the various offsets
inside encoded payload don't reach outside the allocation, if they do, we'll
assert/panic, but at least we won't segfault or smear memory.
We can also do 'deep' validation which runs on all the records of the encoded
payload and validates that they don't contain invalid offsets. This lets us
detect corruptions early and reject a RESTORE command rather than accepting
it and asserting (crashing) later when accessing that payload via some command.
configuration:
- adding ACL flag skip-sanitize-payload
- adding config sanitize-dump-payload [yes/no/clients]
For now, we don't have a good way to ensure MIGRATE in cluster resharding isn't
being slowed down by these sanitation, so i'm setting the default value to `no`,
but later on it should be set to `clients` by default.
changes:
- changing rdbReportError not to `exit` in RESTORE command
- adding a new stat to be able to later check if cluster MIGRATE isn't being
slowed down by sanitation.
2020-08-13 09:41:05 -04:00
test { corrupt payload: listpack invalid size header} {
start_server [ list overrides [ list loglevel verbose use-exit-on-panic yes crash-memcheck-enabled no] ] {
2020-08-14 09:05:34 -04:00
r config set sanitize-dump-payload no
Sanitize dump payload: ziplist, listpack, zipmap, intset, stream
When loading an encoded payload we will at least do a shallow validation to
check that the size that's encoded in the payload matches the size of the
allocation.
This let's us later use this encoded size to make sure the various offsets
inside encoded payload don't reach outside the allocation, if they do, we'll
assert/panic, but at least we won't segfault or smear memory.
We can also do 'deep' validation which runs on all the records of the encoded
payload and validates that they don't contain invalid offsets. This lets us
detect corruptions early and reject a RESTORE command rather than accepting
it and asserting (crashing) later when accessing that payload via some command.
configuration:
- adding ACL flag skip-sanitize-payload
- adding config sanitize-dump-payload [yes/no/clients]
For now, we don't have a good way to ensure MIGRATE in cluster resharding isn't
being slowed down by these sanitation, so i'm setting the default value to `no`,
but later on it should be set to `clients` by default.
changes:
- changing rdbReportError not to `exit` in RESTORE command
- adding a new stat to be able to later check if cluster MIGRATE isn't being
slowed down by sanitation.
2020-08-13 09:41:05 -04:00
catch {
r restore key 0 " \x 0 F \x 0 1 \x 1 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 2 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 2 \x 4 0 \x 5 5 \x 5 F \x 0 0 \x 0 0 \x 0 0 \x 0 F \x 0 0 \x 0 1 \x 0 1 \x 0 0 \x 0 1 \x 0 2 \x 0 1 \x 8 8 \x 3 1 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 9 \x 8 8 \x 3 2 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 9 \x 0 0 \x 0 1 \x 0 0 \x 0 1 \x 0 0 \x 0 1 \x 0 0 \x 0 1 \x 0 2 \x 0 2 \x 8 8 \x 3 1 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 9 \x 8 8 \x 6 1 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 9 \x 8 8 \x 3 2 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 9 \x 8 8 \x 6 2 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 9 \x 0 8 \x 0 1 \x F F \x 0 A \x 0 1 \x 0 0 \x 0 0 \x 0 9 \x 0 0 \x 4 5 \x 9 1 \x 0 A \x 8 7 \x 2 F \x A 5 \x F 9 \x 2 E "
} err
assert_match " * B a d d a t a f o r m a t * " $err
verify_log_message 0 " * S t r e a m l i s t p a c k i n t e g r i t y c h e c k f a i l e d * " 0
}
}
test { corrupt payload: listpack too long entry len} {
start_server [ list overrides [ list loglevel verbose use-exit-on-panic yes crash-memcheck-enabled no] ] {
2020-08-14 09:05:34 -04:00
r config set sanitize-dump-payload no
Sanitize dump payload: ziplist, listpack, zipmap, intset, stream
When loading an encoded payload we will at least do a shallow validation to
check that the size that's encoded in the payload matches the size of the
allocation.
This let's us later use this encoded size to make sure the various offsets
inside encoded payload don't reach outside the allocation, if they do, we'll
assert/panic, but at least we won't segfault or smear memory.
We can also do 'deep' validation which runs on all the records of the encoded
payload and validates that they don't contain invalid offsets. This lets us
detect corruptions early and reject a RESTORE command rather than accepting
it and asserting (crashing) later when accessing that payload via some command.
configuration:
- adding ACL flag skip-sanitize-payload
- adding config sanitize-dump-payload [yes/no/clients]
For now, we don't have a good way to ensure MIGRATE in cluster resharding isn't
being slowed down by these sanitation, so i'm setting the default value to `no`,
but later on it should be set to `clients` by default.
changes:
- changing rdbReportError not to `exit` in RESTORE command
- adding a new stat to be able to later check if cluster MIGRATE isn't being
slowed down by sanitation.
2020-08-13 09:41:05 -04:00
catch {
Add stream consumer group lag tracking and reporting (#9127)
Adds the ability to track the lag of a consumer group (CG), that is, the number
of entries yet-to-be-delivered from the stream.
The proposed constant-time solution is in the spirit of "best-effort."
Partially addresses #8737.
## Description of approach
We add a new "entries_added" property to the stream. This starts at 0 for a new
stream and is incremented by 1 with every `XADD`. It is essentially an all-time
counter of the entries added to the stream.
Given the stream's length and this counter value, we can trivially find the logical
"entries_added" counter of the first ID if and only if the stream is contiguous.
A fragmented stream contains one or more tombstones generated by `XDEL`s.
The new "xdel_max_id" stream property tracks the latest tombstone.
The CG also tracks its last delivered ID's as an "entries_read" counter and
increments it independently when delivering new messages, unless the this
read counter is invalid (-1 means invalid offset). When the CG's counter is
available, the reported lag is the difference between added and read counters.
Lastly, this also adds a "first_id" field to the stream structure in order to make
looking it up cheaper in most cases.
## Limitations
There are two cases in which the mechanism isn't able to track the lag.
In these cases, `XINFO` replies with `null` in the "lag" field.
The first case is when a CG is created with an arbitrary last delivered ID,
that isn't "0-0", nor the first or the last entries of the stream. In this case,
it is impossible to obtain a valid read counter (short of an O(N) operation).
The second case is when there are one or more tombstones fragmenting
the stream's entries range.
In both cases, given enough time and assuming that the consumers are
active (reading and lacking) and advancing, the CG should be able to
catch up with the tip of the stream and report zero lag.
Once that's achieved, lag tracking would resume as normal (until the
next tombstone is set).
## API changes
* `XGROUP CREATE` added with the optional named argument `[ENTRIESREAD entries-read]`
for explicitly specifying the new CG's counter.
* `XGROUP SETID` added with an optional positional argument `[ENTRIESREAD entries-read]`
for specifying the CG's counter.
* `XINFO` reports the maximal tombstone ID, the recorded first entry ID, and total
number of entries added to the stream.
* `XINFO` reports the current lag and logical read counter of CGs.
* `XSETID` is an internal command that's used in replication/aof. It has been added with
the optional positional arguments `[ENTRIESADDED entries-added] [MAXDELETEDID max-deleted-entry-id]`
for propagating the CG's offset and maximal tombstone ID of the stream.
## The generic unsolved problem
The current stream implementation doesn't provide an efficient way to obtain the
approximate/exact size of a range of entries. While it could've been nice to have
that ability (#5813) in general, let alone specifically in the context of CGs, the risk
and complexities involved in such implementation are in all likelihood prohibitive.
## A refactoring note
The `streamGetEdgeID` has been refactored to accommodate both the existing seek
of any entry as well as seeking non-deleted entries (the addition of the `skip_tombstones`
argument). Furthermore, this refactoring also migrated the seek logic to use the
`streamIterator` (rather than `raxIterator`) that was, in turn, extended with the
`skip_tombstones` Boolean struct field to control the emission of these.
Co-authored-by: Guy Benoish <guy.benoish@redislabs.com>
Co-authored-by: Oran Agra <oran@redislabs.com>
2022-02-23 15:34:58 -05:00
r restore key 0 " \x 0 F \x 0 1 \x 1 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 2 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 2 \x 4 0 \x 5 5 \x 5 5 \x 0 0 \x 0 0 \x 0 0 \x 0 F \x 0 0 \x 0 1 \x 0 1 \x 0 0 \x 0 1 \x 0 2 \x 0 1 \x 8 8 \x 3 1 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 9 \x 8 8 \x 3 2 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 9 \x 0 0 \x 0 1 \x 0 0 \x 0 1 \x 0 0 \x 0 1 \x 0 0 \x 0 1 \x 0 2 \x 0 2 \x 8 9 \x 3 1 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 9 \x 8 8 \x 6 1 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 9 \x 8 8 \x 3 2 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 9 \x 8 8 \x 6 2 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 9 \x 0 8 \x 0 1 \x F F \x 0 A \x 0 1 \x 0 0 \x 0 0 \x 0 9 \x 0 0 \x 4 0 \x 6 3 \x C 9 \x 3 7 \x 0 3 \x A 2 \x E 5 \x 6 8 "
Sanitize dump payload: ziplist, listpack, zipmap, intset, stream
When loading an encoded payload we will at least do a shallow validation to
check that the size that's encoded in the payload matches the size of the
allocation.
This let's us later use this encoded size to make sure the various offsets
inside encoded payload don't reach outside the allocation, if they do, we'll
assert/panic, but at least we won't segfault or smear memory.
We can also do 'deep' validation which runs on all the records of the encoded
payload and validates that they don't contain invalid offsets. This lets us
detect corruptions early and reject a RESTORE command rather than accepting
it and asserting (crashing) later when accessing that payload via some command.
configuration:
- adding ACL flag skip-sanitize-payload
- adding config sanitize-dump-payload [yes/no/clients]
For now, we don't have a good way to ensure MIGRATE in cluster resharding isn't
being slowed down by these sanitation, so i'm setting the default value to `no`,
but later on it should be set to `clients` by default.
changes:
- changing rdbReportError not to `exit` in RESTORE command
- adding a new stat to be able to later check if cluster MIGRATE isn't being
slowed down by sanitation.
2020-08-13 09:41:05 -04:00
} err
2020-08-14 09:05:34 -04:00
assert_equal [ count_log_message 0 " c r a s h e d b y s i g n a l " ] 0
assert_equal [ count_log_message 0 " A S S E R T I O N F A I L E D " ] 1
Sanitize dump payload: ziplist, listpack, zipmap, intset, stream
When loading an encoded payload we will at least do a shallow validation to
check that the size that's encoded in the payload matches the size of the
allocation.
This let's us later use this encoded size to make sure the various offsets
inside encoded payload don't reach outside the allocation, if they do, we'll
assert/panic, but at least we won't segfault or smear memory.
We can also do 'deep' validation which runs on all the records of the encoded
payload and validates that they don't contain invalid offsets. This lets us
detect corruptions early and reject a RESTORE command rather than accepting
it and asserting (crashing) later when accessing that payload via some command.
configuration:
- adding ACL flag skip-sanitize-payload
- adding config sanitize-dump-payload [yes/no/clients]
For now, we don't have a good way to ensure MIGRATE in cluster resharding isn't
being slowed down by these sanitation, so i'm setting the default value to `no`,
but later on it should be set to `clients` by default.
changes:
- changing rdbReportError not to `exit` in RESTORE command
- adding a new stat to be able to later check if cluster MIGRATE isn't being
slowed down by sanitation.
2020-08-13 09:41:05 -04:00
}
}
test { corrupt payload: listpack very long entry len} {
start_server [ list overrides [ list loglevel verbose use-exit-on-panic yes crash-memcheck-enabled no] ] {
2020-08-14 09:05:34 -04:00
r config set sanitize-dump-payload no
Sanitize dump payload: ziplist, listpack, zipmap, intset, stream
When loading an encoded payload we will at least do a shallow validation to
check that the size that's encoded in the payload matches the size of the
allocation.
This let's us later use this encoded size to make sure the various offsets
inside encoded payload don't reach outside the allocation, if they do, we'll
assert/panic, but at least we won't segfault or smear memory.
We can also do 'deep' validation which runs on all the records of the encoded
payload and validates that they don't contain invalid offsets. This lets us
detect corruptions early and reject a RESTORE command rather than accepting
it and asserting (crashing) later when accessing that payload via some command.
configuration:
- adding ACL flag skip-sanitize-payload
- adding config sanitize-dump-payload [yes/no/clients]
For now, we don't have a good way to ensure MIGRATE in cluster resharding isn't
being slowed down by these sanitation, so i'm setting the default value to `no`,
but later on it should be set to `clients` by default.
changes:
- changing rdbReportError not to `exit` in RESTORE command
- adding a new stat to be able to later check if cluster MIGRATE isn't being
slowed down by sanitation.
2020-08-13 09:41:05 -04:00
catch {
Add stream consumer group lag tracking and reporting (#9127)
Adds the ability to track the lag of a consumer group (CG), that is, the number
of entries yet-to-be-delivered from the stream.
The proposed constant-time solution is in the spirit of "best-effort."
Partially addresses #8737.
## Description of approach
We add a new "entries_added" property to the stream. This starts at 0 for a new
stream and is incremented by 1 with every `XADD`. It is essentially an all-time
counter of the entries added to the stream.
Given the stream's length and this counter value, we can trivially find the logical
"entries_added" counter of the first ID if and only if the stream is contiguous.
A fragmented stream contains one or more tombstones generated by `XDEL`s.
The new "xdel_max_id" stream property tracks the latest tombstone.
The CG also tracks its last delivered ID's as an "entries_read" counter and
increments it independently when delivering new messages, unless the this
read counter is invalid (-1 means invalid offset). When the CG's counter is
available, the reported lag is the difference between added and read counters.
Lastly, this also adds a "first_id" field to the stream structure in order to make
looking it up cheaper in most cases.
## Limitations
There are two cases in which the mechanism isn't able to track the lag.
In these cases, `XINFO` replies with `null` in the "lag" field.
The first case is when a CG is created with an arbitrary last delivered ID,
that isn't "0-0", nor the first or the last entries of the stream. In this case,
it is impossible to obtain a valid read counter (short of an O(N) operation).
The second case is when there are one or more tombstones fragmenting
the stream's entries range.
In both cases, given enough time and assuming that the consumers are
active (reading and lacking) and advancing, the CG should be able to
catch up with the tip of the stream and report zero lag.
Once that's achieved, lag tracking would resume as normal (until the
next tombstone is set).
## API changes
* `XGROUP CREATE` added with the optional named argument `[ENTRIESREAD entries-read]`
for explicitly specifying the new CG's counter.
* `XGROUP SETID` added with an optional positional argument `[ENTRIESREAD entries-read]`
for specifying the CG's counter.
* `XINFO` reports the maximal tombstone ID, the recorded first entry ID, and total
number of entries added to the stream.
* `XINFO` reports the current lag and logical read counter of CGs.
* `XSETID` is an internal command that's used in replication/aof. It has been added with
the optional positional arguments `[ENTRIESADDED entries-added] [MAXDELETEDID max-deleted-entry-id]`
for propagating the CG's offset and maximal tombstone ID of the stream.
## The generic unsolved problem
The current stream implementation doesn't provide an efficient way to obtain the
approximate/exact size of a range of entries. While it could've been nice to have
that ability (#5813) in general, let alone specifically in the context of CGs, the risk
and complexities involved in such implementation are in all likelihood prohibitive.
## A refactoring note
The `streamGetEdgeID` has been refactored to accommodate both the existing seek
of any entry as well as seeking non-deleted entries (the addition of the `skip_tombstones`
argument). Furthermore, this refactoring also migrated the seek logic to use the
`streamIterator` (rather than `raxIterator`) that was, in turn, extended with the
`skip_tombstones` Boolean struct field to control the emission of these.
Co-authored-by: Guy Benoish <guy.benoish@redislabs.com>
Co-authored-by: Oran Agra <oran@redislabs.com>
2022-02-23 15:34:58 -05:00
# This will catch migrated payloads from v6.2.x
r restore key 0 " \x 0 F \x 0 1 \x 1 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 2 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 2 \x 4 0 \x 5 5 \x 5 5 \x 0 0 \x 0 0 \x 0 0 \x 0 F \x 0 0 \x 0 1 \x 0 1 \x 0 0 \x 0 1 \x 0 2 \x 0 1 \x 8 8 \x 3 1 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 9 \x 8 8 \x 3 2 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 9 \x 0 0 \x 0 1 \x 0 0 \x 0 1 \x 0 0 \x 0 1 \x 0 0 \x 0 1 \x 0 2 \x 0 2 \x 8 8 \x 3 1 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 9 \x 8 8 \x 6 1 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 9 \x 8 8 \x 3 2 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 9 \x 9 C \x 6 2 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 9 \x 0 8 \x 0 1 \x F F \x 0 A \x 0 1 \x 0 0 \x 0 0 \x 0 9 \x 0 0 \x 6 3 \x 6 F \x 4 2 \x 8 E \x 7 C \x B 5 \x A 2 \x 9 D "
Sanitize dump payload: ziplist, listpack, zipmap, intset, stream
When loading an encoded payload we will at least do a shallow validation to
check that the size that's encoded in the payload matches the size of the
allocation.
This let's us later use this encoded size to make sure the various offsets
inside encoded payload don't reach outside the allocation, if they do, we'll
assert/panic, but at least we won't segfault or smear memory.
We can also do 'deep' validation which runs on all the records of the encoded
payload and validates that they don't contain invalid offsets. This lets us
detect corruptions early and reject a RESTORE command rather than accepting
it and asserting (crashing) later when accessing that payload via some command.
configuration:
- adding ACL flag skip-sanitize-payload
- adding config sanitize-dump-payload [yes/no/clients]
For now, we don't have a good way to ensure MIGRATE in cluster resharding isn't
being slowed down by these sanitation, so i'm setting the default value to `no`,
but later on it should be set to `clients` by default.
changes:
- changing rdbReportError not to `exit` in RESTORE command
- adding a new stat to be able to later check if cluster MIGRATE isn't being
slowed down by sanitation.
2020-08-13 09:41:05 -04:00
} err
2020-08-14 09:05:34 -04:00
assert_equal [ count_log_message 0 " c r a s h e d b y s i g n a l " ] 0
assert_equal [ count_log_message 0 " A S S E R T I O N F A I L E D " ] 1
Sanitize dump payload: ziplist, listpack, zipmap, intset, stream
When loading an encoded payload we will at least do a shallow validation to
check that the size that's encoded in the payload matches the size of the
allocation.
This let's us later use this encoded size to make sure the various offsets
inside encoded payload don't reach outside the allocation, if they do, we'll
assert/panic, but at least we won't segfault or smear memory.
We can also do 'deep' validation which runs on all the records of the encoded
payload and validates that they don't contain invalid offsets. This lets us
detect corruptions early and reject a RESTORE command rather than accepting
it and asserting (crashing) later when accessing that payload via some command.
configuration:
- adding ACL flag skip-sanitize-payload
- adding config sanitize-dump-payload [yes/no/clients]
For now, we don't have a good way to ensure MIGRATE in cluster resharding isn't
being slowed down by these sanitation, so i'm setting the default value to `no`,
but later on it should be set to `clients` by default.
changes:
- changing rdbReportError not to `exit` in RESTORE command
- adding a new stat to be able to later check if cluster MIGRATE isn't being
slowed down by sanitation.
2020-08-13 09:41:05 -04:00
}
}
test { corrupt payload: listpack too long entry prev len} {
start_server [ list overrides [ list loglevel verbose use-exit-on-panic yes crash-memcheck-enabled no] ] {
r config set sanitize-dump-payload yes
catch {
r restore key 0 " \x 0 F \x 0 1 \x 1 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 2 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 2 \x 4 0 \x 5 5 \x 5 5 \x 0 0 \x 0 0 \x 0 0 \x 0 F \x 0 0 \x 0 1 \x 0 1 \x 0 0 \x 1 5 \x 0 2 \x 0 1 \x 8 8 \x 3 1 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 9 \x 8 8 \x 3 2 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 9 \x 0 0 \x 0 1 \x 0 0 \x 0 1 \x 0 0 \x 0 1 \x 0 0 \x 0 1 \x 0 2 \x 0 2 \x 8 8 \x 3 1 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 9 \x 8 8 \x 6 1 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 9 \x 8 8 \x 3 2 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 9 \x 8 8 \x 6 2 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 9 \x 0 8 \x 0 1 \x F F \x 0 A \x 0 1 \x 0 0 \x 0 0 \x 0 9 \x 0 0 \x 0 6 \x F B \x 4 4 \x 2 4 \x 0 A \x 8 E \x 7 5 \x E A "
} err
assert_match " * B a d d a t a f o r m a t * " $err
verify_log_message 0 " * S t r e a m l i s t p a c k i n t e g r i t y c h e c k f a i l e d * " 0
}
}
2021-12-08 11:11:57 -05:00
test { corrupt payload: stream with duplicate consumers} {
start_server [ list overrides [ list loglevel verbose use-exit-on-panic yes crash-memcheck-enabled no] ] {
catch {
r restore key 0 " \x 0 F \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 1 \x 0 7 \x 6 D \x 7 9 \x 6 7 \x 7 2 \x 6 F \x 7 5 \x 7 0 \x 0 0 \x 0 0 \x 0 0 \x 0 2 \x 0 4 \x 6 E \x 6 1 \x 6 D \x 6 5 \x 2 A \x 4 C \x A A \x 9 A \x 7 D \x 0 1 \x 0 0 \x 0 0 \x 0 0 \x 0 4 \x 6 E \x 6 1 \x 6 D \x 6 5 \x 2 B \x 4 C \x A A \x 9 A \x 7 D \x 0 1 \x 0 0 \x 0 0 \x 0 0 \x 0 A \x 0 0 \x C C \x E D \x 8 C \x A 7 \x 6 2 \x E E \x C 7 \x C 8 "
} err
assert_match " * B a d d a t a f o r m a t * " $err
verify_log_message 0 " * D u p l i c a t e s t r e a m c o n s u m e r d e t e c t e d * " 0
r ping
}
}
2020-11-02 02:35:37 -05:00
test { corrupt payload: hash ziplist with duplicate records} {
# when we do perform full sanitization, we expect duplicate records to fail the restore
start_server [ list overrides [ list loglevel verbose use-exit-on-panic yes crash-memcheck-enabled no] ] {
r config set sanitize-dump-payload yes
r debug set-skip-checksum-validation 1
catch { r RESTORE _hash 0 " \x 0 D \x 3 D \x 3 D \x 0 0 \x 0 0 \x 0 0 \x 3 A \x 0 0 \x 0 0 \x 0 0 \x 1 4 \x 1 3 \x 0 0 \x F 5 \x 0 2 \x F 5 \x 0 2 \x F 2 \x 0 2 \x 5 3 \x 5 F \x 3 1 \x 0 4 \x F 3 \x 0 2 \x F 3 \x 0 2 \x F 7 \x 0 2 \x F 7 \x 0 2 \x F 8 \x 0 2 \x 0 2 \x 5 F \x 3 7 \x 0 4 \x F 1 \x 0 2 \x F 1 \x 0 2 \x F 6 \x 0 2 \x 0 2 \x 5 F \x 3 5 \x 0 4 \x F 4 \x 0 2 \x 0 2 \x 5 F \x 3 3 \x 0 4 \x F A \x 0 2 \x 0 2 \x 5 F \x 3 9 \x 0 4 \x F 9 \x 0 2 \x F 9 \x F F \x 0 9 \x 0 0 \x B 5 \x 4 8 \x D E \x 6 2 \x 3 1 \x D 0 \x E 5 \x 6 3 " } err
assert_match " * B a d d a t a f o r m a t * " $err
}
}
2021-08-10 02:18:49 -04:00
test { corrupt payload: hash listpack with duplicate records} {
# when we do perform full sanitization, we expect duplicate records to fail the restore
start_server [ list overrides [ list loglevel verbose use-exit-on-panic yes crash-memcheck-enabled no] ] {
r config set sanitize-dump-payload yes
r debug set-skip-checksum-validation 1
2021-11-22 05:30:06 -05:00
catch { r RESTORE _hash 0 " \x 1 0 \x 1 7 \x 1 7 \x 0 0 \x 0 0 \x 0 0 \x 0 4 \x 0 0 \x 8 2 \x 6 1 \x 0 0 \x 0 3 \x 8 2 \x 6 2 \x 0 0 \x 0 3 \x 8 2 \x 6 1 \x 0 0 \x 0 3 \x 8 2 \x 6 4 \x 0 0 \x 0 3 \x f f \x 0 a \x 0 0 \x c 0 \x c f \x a 6 \x 8 7 \x e 5 \x a 7 \x c 5 \x b e " } err
2021-08-10 02:18:49 -04:00
assert_match " * B a d d a t a f o r m a t * " $err
}
}
2021-11-21 11:47:10 -05:00
test { corrupt payload: hash listpack with duplicate records - convert} {
# when we do NOT perform full sanitization, but we convert to hash, we expect duplicate records panic
start_server [ list overrides [ list loglevel verbose use-exit-on-panic yes crash-memcheck-enabled no] ] {
r config set sanitize-dump-payload no
r config set hash-max-listpack-entries 1
r debug set-skip-checksum-validation 1
2021-11-22 05:30:06 -05:00
catch { r RESTORE _hash 0 " \x 1 0 \x 1 7 \x 1 7 \x 0 0 \x 0 0 \x 0 0 \x 0 4 \x 0 0 \x 8 2 \x 6 1 \x 0 0 \x 0 3 \x 8 2 \x 6 2 \x 0 0 \x 0 3 \x 8 2 \x 6 1 \x 0 0 \x 0 3 \x 8 2 \x 6 4 \x 0 0 \x 0 3 \x f f \x 0 a \x 0 0 \x c 0 \x c f \x a 6 \x 8 7 \x e 5 \x a 7 \x c 5 \x b e " } err
2021-11-21 11:47:10 -05:00
assert_equal [ count_log_message 0 " c r a s h e d b y s i g n a l " ] 0
assert_equal [ count_log_message 0 " l i s t p a c k w i t h d u p e l e m e n t s " ] 1
}
}
2020-11-02 02:35:37 -05:00
test { corrupt payload: hash ziplist uneven record count} {
2021-11-21 11:47:10 -05:00
# when we do NOT perform full sanitization, but shallow sanitization can detect uneven count
2020-11-02 02:35:37 -05:00
start_server [ list overrides [ list loglevel verbose use-exit-on-panic yes crash-memcheck-enabled no] ] {
2021-11-21 11:47:10 -05:00
r config set sanitize-dump-payload no
2020-11-02 02:35:37 -05:00
r debug set-skip-checksum-validation 1
catch { r RESTORE _hash 0 " \r \x 1 b \x 1 b \x 0 0 \x 0 0 \x 0 0 \x 1 6 \x 0 0 \x 0 0 \x 0 0 \x 0 4 \x 0 0 \x 0 0 \x 0 2 a \x 0 0 \x 0 4 \x 0 2 b \x 0 0 \x 0 4 \x 0 2 a \x 0 0 \x 0 4 \x 0 2 d \x 0 0 \x f f \t \x 0 0 \x a 1 \x 9 8 \x 3 6 x \x c c \x 8 e \x 9 3 \x 2 e " } err
assert_match " * B a d d a t a f o r m a t * " $err
}
}
2021-06-10 08:39:33 -04:00
test { corrupt payload: hash duplicate records} {
2020-11-02 02:35:37 -05:00
# when we do perform full sanitization, we expect duplicate records to fail the restore
start_server [ list overrides [ list loglevel verbose use-exit-on-panic yes crash-memcheck-enabled no] ] {
r config set sanitize-dump-payload yes
r debug set-skip-checksum-validation 1
catch { r RESTORE _hash 0 " \x 0 4 \x 0 2 \x 0 1 a \x 0 1 b \x 0 1 a \x 0 1 d \t \x 0 0 \x c 6 \x 9 c \x a b \x b c \b k \x 0 c \x 0 6 " } err
assert_match " * B a d d a t a f o r m a t * " $err
}
}
2021-08-09 10:13:46 -04:00
test { corrupt payload: hash empty zipmap} {
start_server [ list overrides [ list loglevel verbose use-exit-on-panic yes crash-memcheck-enabled no] ] {
r config set sanitize-dump-payload no
r debug set-skip-checksum-validation 1
catch { r RESTORE _hash 0 " \x 0 9 \x 0 2 \x 0 0 \x F F \x 0 9 \x 0 0 \x C 0 \x F 1 \x B 8 \x 6 7 \x 4 C \x 1 6 \x A C \x E 3 " } err
assert_match " * B a d d a t a f o r m a t * " $err
verify_log_message 0 " * Z i p m a p i n t e g r i t y c h e c k f a i l e d * " 0
}
}
2020-08-14 09:05:34 -04:00
test { corrupt payload: fuzzer findings - NPD in streamIteratorGetID} {
start_server [ list overrides [ list loglevel verbose use-exit-on-panic yes crash-memcheck-enabled no] ] {
r config set sanitize-dump-payload no
r debug set-skip-checksum-validation 1
catch {
r RESTORE key 0 " \x 0 F \x 0 1 \x 1 0 \x 0 0 \x 0 0 \x 0 1 \x 7 3 \x B D \x 6 8 \x 4 8 \x 7 1 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 4 0 \x 4 2 \x 4 2 \x 0 0 \x 0 0 \x 0 0 \x 1 8 \x 0 0 \x 0 3 \x 0 1 \x 0 0 \x 0 1 \x 0 2 \x 0 1 \x 8 4 \x 6 9 \x 7 4 \x 6 5 \x 6 D \x 0 5 \x 8 5 \x 7 6 \x 6 1 \x 6 C \x 7 5 \x 6 5 \x 0 6 \x 0 0 \x 0 1 \x 0 2 \x 0 1 \x 0 0 \x 0 1 \x 0 0 \x 0 1 \x 0 1 \x 0 1 \x 0 0 \x 0 1 \x 0 5 \x 0 1 \x 0 2 \x 0 1 \x 0 0 \x 0 1 \x 0 1 \x 0 1 \x 0 1 \x 0 1 \x 8 2 \x 5 F \x 3 1 \x 0 3 \x 0 5 \x 0 1 \x 0 2 \x 0 1 \x 0 0 \x 0 1 \x 0 2 \x 0 1 \x 0 1 \x 0 1 \x 0 2 \x 0 1 \x 4 8 \x 0 1 \x F F \x 0 3 \x 8 1 \x 0 0 \x 0 0 \x 0 1 \x 7 3 \x B D \x 6 8 \x 4 8 \x 7 1 \x 0 2 \x 0 1 \x 0 7 \x 6 D \x 7 9 \x 6 7 \x 7 2 \x 6 F \x 7 5 \x 7 0 \x 8 1 \x 0 0 \x 0 0 \x 0 1 \x 7 3 \x B D \x 6 8 \x 4 8 \x 7 1 \x 0 0 \x 0 1 \x 0 0 \x 0 0 \x 0 1 \x 7 3 \x B D \x 6 8 \x 4 8 \x 7 1 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 7 2 \x 4 8 \x 6 8 \x B D \x 7 3 \x 0 1 \x 0 0 \x 0 0 \x 0 1 \x 0 1 \x 0 5 \x 4 1 \x 6 C \x 6 9 \x 6 3 \x 6 5 \x 7 2 \x 4 8 \x 6 8 \x B D \x 7 3 \x 0 1 \x 0 0 \x 0 0 \x 0 1 \x 0 0 \x 0 0 \x 0 1 \x 7 3 \x B D \x 6 8 \x 4 8 \x 7 1 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 9 \x 0 0 \x 8 0 \x C D \x B 0 \x D 5 \x 1 A \x C E \x F F \x 1 0 "
r XREVRANGE key 725 233
}
assert_equal [ count_log_message 0 " c r a s h e d b y s i g n a l " ] 0
assert_equal [ count_log_message 0 " A S S E R T I O N F A I L E D " ] 1
}
}
test { corrupt payload: fuzzer findings - listpack NPD on invalid stream} {
start_server [ list overrides [ list loglevel verbose use-exit-on-panic yes crash-memcheck-enabled no] ] {
r config set sanitize-dump-payload no
r debug set-skip-checksum-validation 1
catch {
r RESTORE _stream 0 " \x 0 F \x 0 1 \x 1 0 \x 0 0 \x 0 0 \x 0 1 \x 7 3 \x D C \x B 6 \x 6 B \x F 1 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 4 0 \x 4 2 \x 4 2 \x 0 0 \x 0 0 \x 0 0 \x 1 8 \x 0 0 \x 0 3 \x 0 1 \x 0 0 \x 0 1 \x 0 2 \x 0 1 \x 8 4 \x 6 9 \x 7 4 \x 6 5 \x 6 D \x 0 5 \x 8 5 \x 7 6 \x 6 1 \x 6 C \x 7 5 \x 6 5 \x 0 6 \x 0 0 \x 0 1 \x 0 2 \x 0 1 \x 0 0 \x 0 1 \x 0 0 \x 0 1 \x 0 1 \x 0 1 \x 0 0 \x 0 1 \x 0 5 \x 0 1 \x 0 2 \x 0 1 \x 1 F \x 0 1 \x 0 0 \x 0 1 \x 0 1 \x 0 1 \x 6 D \x 5 F \x 3 1 \x 0 3 \x 0 5 \x 0 1 \x 0 2 \x 0 1 \x 2 9 \x 0 1 \x 0 0 \x 0 1 \x 0 1 \x 0 1 \x 0 2 \x 0 1 \x 0 5 \x 0 1 \x F F \x 0 3 \x 8 1 \x 0 0 \x 0 0 \x 0 1 \x 7 3 \x D C \x B 6 \x 6 C \x 1 A \x 0 0 \x 0 1 \x 0 7 \x 6 D \x 7 9 \x 6 7 \x 7 2 \x 6 F \x 7 5 \x 7 0 \x 8 1 \x 0 0 \x 0 0 \x 0 1 \x 7 3 \x D C \x B 6 \x 6 B \x F 1 \x 0 0 \x 0 1 \x 0 0 \x 0 0 \x 0 1 \x 7 3 \x D C \x B 6 \x 6 B \x F 1 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 4 B \x 6 C \x B 6 \x D C \x 7 3 \x 0 1 \x 0 0 \x 0 0 \x 0 1 \x 0 1 \x 0 5 \x 4 1 \x 6 C \x 6 9 \x 6 3 \x 6 5 \x 3 D \x 6 C \x B 6 \x D C \x 7 3 \x 0 1 \x 0 0 \x 0 0 \x 0 1 \x 0 0 \x 0 0 \x 0 1 \x 7 3 \x D C \x B 6 \x 6 B \x F 1 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 9 \x 0 0 \x C 7 \x 7 D \x 1 C \x D 7 \x 0 4 \x F F \x E 6 \x 9 D "
r XREAD STREAMS _stream 519389898758
}
assert_equal [ count_log_message 0 " c r a s h e d b y s i g n a l " ] 0
assert_equal [ count_log_message 0 " A S S E R T I O N F A I L E D " ] 1
}
}
test { corrupt payload: fuzzer findings - NPD in quicklistIndex} {
start_server [ list overrides [ list loglevel verbose use-exit-on-panic yes crash-memcheck-enabled no] ] {
r config set sanitize-dump-payload no
r debug set-skip-checksum-validation 1
catch {
r RESTORE key 0 " \x 0 E \x 0 1 \x 1 3 \x 1 3 \x 0 0 \x 0 0 \x 0 0 \x 1 0 \x 0 0 \x 0 0 \x 0 0 \x 0 3 \x 1 2 \x 0 0 \x F 3 \x 0 2 \x 0 2 \x 5 F \x 3 1 \x 0 4 \x F 1 \x F F \x 0 9 \x 0 0 \x C 9 \x 4 B \x 3 1 \x F E \x 6 1 \x C 0 \x 9 6 \x F E "
2021-11-24 06:34:13 -05:00
} err
assert_match " * B a d d a t a f o r m a t * " $err
verify_log_message 0 " * i n t e g r i t y c h e c k f a i l e d * " 0
2020-08-14 09:05:34 -04:00
}
}
2021-08-10 02:18:49 -04:00
test { corrupt payload: fuzzer findings - encoded entry header reach outside the allocation} {
2020-08-14 09:05:34 -04:00
start_server [ list overrides [ list loglevel verbose use-exit-on-panic yes crash-memcheck-enabled no] ] {
r debug set-skip-checksum-validation 1
catch {
r RESTORE key 0 " \x 0 D \x 1 9 \x 1 9 \x 0 0 \x 0 0 \x 0 0 \x 1 6 \x 0 0 \x 0 0 \x 0 0 \x 0 6 \x 0 0 \x 0 0 \x F 1 \x 0 2 \x F 1 \x 0 2 \x F 2 \x 0 2 \x 0 2 \x 5 F \x 3 1 \x 0 4 \x 9 9 \x 0 2 \x F 3 \x F F \x 0 9 \x 0 0 \x C 5 \x B 8 \x 1 0 \x C 0 \x 8 A \x F 9 \x 1 6 \x D F "
2021-08-10 02:18:49 -04:00
} err
assert_match " * B a d d a t a f o r m a t * " $err
verify_log_message 0 " * i n t e g r i t y c h e c k f a i l e d * " 0
2020-08-14 09:05:34 -04:00
}
}
test { corrupt payload: fuzzer findings - invalid ziplist encoding} {
start_server [ list overrides [ list loglevel verbose use-exit-on-panic yes crash-memcheck-enabled no] ] {
r config set sanitize-dump-payload yes
r debug set-skip-checksum-validation 1
catch {
r RESTORE _listbig 0 " \x 0 E \x 0 2 \x 1 B \x 1 B \x 0 0 \x 0 0 \x 0 0 \x 1 6 \x 0 0 \x 0 0 \x 0 0 \x 0 5 \x 0 0 \x 0 0 \x 0 2 \x 5 F \x 3 9 \x 0 4 \x F 9 \x 0 2 \x 8 6 \x 5 F \x 3 7 \x 0 4 \x F 7 \x 0 2 \x 0 2 \x 5 F \x 3 5 \x F F \x 1 9 \x 1 9 \x 0 0 \x 0 0 \x 0 0 \x 1 6 \x 0 0 \x 0 0 \x 0 0 \x 0 5 \x 0 0 \x 0 0 \x F 5 \x 0 2 \x 0 2 \x 5 F \x 3 3 \x 0 4 \x F 3 \x 0 2 \x 0 2 \x 5 F \x 3 1 \x 0 4 \x F 1 \x F F \x 0 9 \x 0 0 \x 0 C \x F C \x 9 9 \x 2 C \x 2 3 \x 4 5 \x 1 5 \x 6 0 "
} err
assert_match " * B a d d a t a f o r m a t * " $err
2020-11-02 02:35:37 -05:00
verify_log_message 0 " * i n t e g r i t y c h e c k f a i l e d * " 0
2020-08-14 09:05:34 -04:00
}
}
test { corrupt payload: fuzzer findings - hash crash} {
start_server [ list overrides [ list loglevel verbose use-exit-on-panic yes crash-memcheck-enabled no] ] {
r config set sanitize-dump-payload yes
r debug set-skip-checksum-validation 1
r RESTORE _hash 0 " \x 0 D \x 1 9 \x 1 9 \x 0 0 \x 0 0 \x 0 0 \x 1 6 \x 0 0 \x 0 0 \x 0 0 \x 0 6 \x 0 0 \x 0 0 \x F 1 \x 0 2 \x F 1 \x 0 2 \x F 2 \x 0 2 \x 0 2 \x 5 F \x 3 1 \x 0 4 \x F 3 \x 0 2 \x F 3 \x F F \x 0 9 \x 0 0 \x 3 8 \x B 8 \x 1 0 \x C 0 \x 8 A \x F 9 \x 1 6 \x D F "
r HSET _hash 394891450 1635910264
r HMGET _hash 887312884855
}
}
test { corrupt payload: fuzzer findings - uneven entry count in hash} {
start_server [ list overrides [ list loglevel verbose use-exit-on-panic yes crash-memcheck-enabled no] ] {
r debug set-skip-checksum-validation 1
2021-08-10 02:18:49 -04:00
catch {
r RESTORE _hashbig 0 " \x 0 D \x 3 D \x 3 D \x 0 0 \x 0 0 \x 0 0 \x 3 8 \x 0 0 \x 0 0 \x 0 0 \x 1 4 \x 0 0 \x 0 0 \x F 2 \x 0 2 \x 0 2 \x 5 F \x 3 1 \x 0 4 \x 1 C \x 0 2 \x F 7 \x 0 2 \x F 1 \x 0 2 \x F 1 \x 0 2 \x F 5 \x 0 2 \x F 5 \x 0 2 \x F 4 \x 0 2 \x 0 2 \x 5 F \x 3 3 \x 0 4 \x F 6 \x 0 2 \x 0 2 \x 5 F \x 3 5 \x 0 4 \x F 8 \x 0 2 \x 0 2 \x 5 F \x 3 7 \x 0 4 \x F 9 \x 0 2 \x F 9 \x 0 2 \x F 3 \x 0 2 \x F 3 \x 0 2 \x F A \x 0 2 \x 0 2 \x 5 F \x 3 9 \x F F \x 0 9 \x 0 0 \x 7 3 \x B 7 \x 6 8 \x C 8 \x 9 7 \x 2 4 \x 8 E \x 8 8 "
} err
assert_match " * B a d d a t a f o r m a t * " $err
verify_log_message 0 " * i n t e g r i t y c h e c k f a i l e d * " 0
2020-08-14 09:05:34 -04:00
}
}
test { corrupt payload: fuzzer findings - invalid read in lzf_decompress} {
start_server [ list overrides [ list loglevel verbose use-exit-on-panic yes crash-memcheck-enabled no] ] {
r config set sanitize-dump-payload no
r debug set-skip-checksum-validation 1
catch { r RESTORE _setbig 0 " \x 0 2 \x 0 3 \x 0 2 \x 5 F \x 3 1 \x C 0 \x 0 2 \x C 3 \x 0 0 \x 0 9 \x 0 0 \x E 6 \x D C \x 7 6 \x 4 4 \x F F \x E B \x 3 D \x F E " } err
assert_match " * B a d d a t a f o r m a t * " $err
}
}
test { corrupt payload: fuzzer findings - leak in rdbloading due to dup entry in set} {
start_server [ list overrides [ list loglevel verbose use-exit-on-panic yes crash-memcheck-enabled no] ] {
r config set sanitize-dump-payload no
r debug set-skip-checksum-validation 1
catch { r RESTORE _setbig 0 " \x 0 2 \x 0 A \x 0 2 \x 5 F \x 3 9 \x C 0 \x 0 6 \x 0 2 \x 5 F \x 3 1 \x C 0 \x 0 0 \x C 0 \x 0 4 \x 0 2 \x 5 F \x 3 5 \x C 0 \x 0 2 \x C 0 \x 0 8 \x 0 2 \x 5 F \x 3 1 \x 0 2 \x 5 F \x 3 3 \x 0 9 \x 0 0 \x 7 A \x 5 A \x F B \x 9 0 \x 3 A \x E 9 \x 3 C \x B E " } err
assert_match " * B a d d a t a f o r m a t * " $err
}
}
2021-08-05 15:42:20 -04:00
test { corrupt payload: fuzzer findings - empty intset} {
2020-08-14 09:05:34 -04:00
start_server [ list overrides [ list loglevel verbose use-exit-on-panic yes crash-memcheck-enabled no] ] {
r config set sanitize-dump-payload no
r debug set-skip-checksum-validation 1
2021-08-05 15:42:20 -04:00
catch { r RESTORE _setbig 0 " \x 0 2 \x C 0 \x C 0 \x 0 6 \x 0 2 \x 5 F \x 3 9 \x C 0 \x 0 2 \x 0 2 \x 5 F \x 3 3 \x C 0 \x 0 0 \x 0 2 \x 5 F \x 3 1 \x C 0 \x 0 4 \x C 0 \x 0 8 \x 0 2 \x 5 F \x 3 7 \x 0 2 \x 5 F \x 3 5 \x 0 9 \x 0 0 \x C 5 \x D 4 \x 6 D \x B A \x A D \x 1 4 \x B 7 \x E 7 " } err
assert_match " * B a d d a t a f o r m a t * " $err
r ping
2020-08-14 09:05:34 -04:00
}
}
2021-09-09 11:18:53 -04:00
test { corrupt payload: fuzzer findings - zset ziplist entry lensize is 0 } {
2020-08-14 09:05:34 -04:00
start_server [ list overrides [ list loglevel verbose use-exit-on-panic yes crash-memcheck-enabled no] ] {
r config set sanitize-dump-payload no
r debug set-skip-checksum-validation 1
2021-09-09 11:18:53 -04:00
catch { r RESTORE _zsetbig 0 " \x 0 C \x 3 D \x 3 D \x 0 0 \x 0 0 \x 0 0 \x 3 A \x 0 0 \x 0 0 \x 0 0 \x 1 4 \x 0 0 \x 0 0 \x F 1 \x 0 2 \x F 1 \x 0 2 \x 0 2 \x 5 F \x 3 1 \x 0 4 \x F 2 \x 0 2 \x F 3 \x 0 2 \x F 3 \x 0 2 \x 0 2 \x 5 F \x 3 3 \x 0 4 \x F 4 \x 0 2 \x E E \x 0 2 \x F 5 \x 0 2 \x 0 2 \x 5 F \x 3 5 \x 0 4 \x F 6 \x 0 2 \x F 7 \x 0 2 \x F 7 \x 0 2 \x 0 2 \x 5 F \x 3 7 \x 0 4 \x F 8 \x 0 2 \x F 9 \x 0 2 \x F 9 \x 0 2 \x 0 2 \x 5 F \x 3 9 \x 0 4 \x F A \x F F \x 0 9 \x 0 0 \x A E \x F 9 \x 7 7 \x 2 A \x 4 7 \x 2 4 \x 3 3 \x F 6 " } err
assert_match " * B a d d a t a f o r m a t * " $err
verify_log_message 0 " * Z s e t z i p l i s t i n t e g r i t y c h e c k f a i l e d * " 0
2020-08-14 09:05:34 -04:00
}
}
test { corrupt payload: fuzzer findings - valgrind ziplist prevlen reaches outside the ziplist} {
start_server [ list overrides [ list loglevel verbose use-exit-on-panic yes crash-memcheck-enabled no] ] {
r config set sanitize-dump-payload no
r debug set-skip-checksum-validation 1
2021-11-24 06:34:13 -05:00
catch { r RESTORE _listbig 0 " \x 0 E \x 0 2 \x 1 B \x 1 B \x 0 0 \x 0 0 \x 0 0 \x 1 6 \x 0 0 \x 0 0 \x 0 0 \x 0 5 \x 0 0 \x 0 0 \x 0 2 \x 5 F \x 3 9 \x 0 4 \x F 9 \x 0 2 \x 0 2 \x 5 F \x 3 7 \x 0 4 \x F 7 \x 0 2 \x 0 2 \x 5 F \x 3 5 \x F F \x 1 9 \x 1 9 \x 0 0 \x 0 0 \x 0 0 \x 1 6 \x 0 0 \x 0 0 \x 0 0 \x 0 5 \x 0 0 \x 0 0 \x F 5 \x 0 2 \x 0 2 \x 5 F \x 3 3 \x 0 4 \x F 3 \x 9 5 \x 0 2 \x 5 F \x 3 1 \x 0 4 \x F 1 \x F F \x 0 9 \x 0 0 \x 0 C \x F C \x 9 9 \x 2 C \x 2 3 \x 4 5 \x 1 5 \x 6 0 " } err
assert_match " * B a d d a t a f o r m a t * " $err
verify_log_message 0 " * i n t e g r i t y c h e c k f a i l e d * " 0
2020-08-14 09:05:34 -04:00
}
}
test { corrupt payload: fuzzer findings - valgrind - bad rdbLoadDoubleValue} {
start_server [ list overrides [ list loglevel verbose use-exit-on-panic yes crash-memcheck-enabled no] ] {
r config set sanitize-dump-payload no
r debug set-skip-checksum-validation 1
catch { r RESTORE _list 0 " \x 0 3 \x 0 1 \x 1 1 \x 1 1 \x 0 0 \x 0 0 \x 0 0 \x 0 A \x 0 0 \x 0 0 \x 0 0 \x 0 1 \x 0 0 \x 0 0 \x D 0 \x 0 7 \x 1 A \x E 9 \x 0 2 \x F F \x 0 9 \x 0 0 \x 1 A \x 0 6 \x 0 7 \x 3 2 \x 4 1 \x 2 8 \x 3 A \x 4 6 " } err
assert_match " * B a d d a t a f o r m a t * " $err
}
}
test { corrupt payload: fuzzer findings - valgrind ziplist prev too big} {
start_server [ list overrides [ list loglevel verbose use-exit-on-panic yes crash-memcheck-enabled no] ] {
r config set sanitize-dump-payload no
r debug set-skip-checksum-validation 1
2021-11-24 06:34:13 -05:00
catch { r RESTORE _list 0 " \x 0 E \x 0 1 \x 1 3 \x 1 3 \x 0 0 \x 0 0 \x 0 0 \x 1 0 \x 0 0 \x 0 0 \x 0 0 \x 0 3 \x 0 0 \x 0 0 \x F 3 \x 0 2 \x 0 2 \x 5 F \x 3 1 \x C 1 \x F 1 \x F F \x 0 9 \x 0 0 \x C 9 \x 4 B \x 3 1 \x F E \x 6 1 \x C 0 \x 9 6 \x F E " } err
assert_match " * B a d d a t a f o r m a t * " $err
verify_log_message 0 " * i n t e g r i t y c h e c k f a i l e d * " 0
2020-08-14 09:05:34 -04:00
}
}
test { corrupt payload: fuzzer findings - lzf decompression fails, avoid valgrind invalid read} {
start_server [ list overrides [ list loglevel verbose use-exit-on-panic yes crash-memcheck-enabled no] ] {
r config set sanitize-dump-payload no
r debug set-skip-checksum-validation 1
catch { r RESTORE _stream 0 " \x 0 F \x 0 2 \x 1 0 \x 0 0 \x 0 0 \x 0 1 \x 7 3 \x D D \x A A \x 2 A \x B 9 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x C 3 \x 4 0 \x 4 B \x 4 0 \x 5 C \x 1 8 \x 5 C \x 0 0 \x 0 0 \x 0 0 \x 2 4 \x 0 0 \x 0 5 \x 0 1 \x 0 0 \x 0 1 \x 0 2 \x 0 1 \x 8 4 \x 6 9 \x 7 4 \x 6 5 \x 6 D \x 0 5 \x 8 5 \x 7 6 \x 6 1 \x 6 C \x 7 5 \x 6 5 \x 0 6 \x 4 0 \x 1 0 \x 0 0 \x 0 0 \x 2 0 \x 0 1 \x 0 0 \x 0 1 \x 2 0 \x 0 3 \x 0 0 \x 0 5 \x 2 0 \x 1 C \x 4 0 \x 0 7 \x 0 5 \x 0 1 \x 0 1 \x 8 2 \x 5 F \x 3 1 \x 0 3 \x 8 0 \x 0 D \x 4 0 \x 0 0 \x 0 0 \x 0 2 \x 6 0 \x 1 9 \x 4 0 \x 2 7 \x 4 0 \x 1 9 \x 0 0 \x 3 3 \x 6 0 \x 1 9 \x 4 0 \x 2 9 \x 0 2 \x 0 1 \x 0 1 \x 0 4 \x 2 0 \x 1 9 \x 0 0 \x F F \x 1 0 \x 0 0 \x 0 0 \x 0 1 \x 7 3 \x D D \x A A \x 2 A \x B C \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x C 3 \x 4 0 \x 4 D \x 4 0 \x 5 E \x 1 8 \x 5 E \x 0 0 \x 0 0 \x 0 0 \x 2 4 \x 0 0 \x 0 5 \x 0 1 \x 0 0 \x 0 1 \x 0 2 \x 0 1 \x 8 4 \x 6 9 \x 7 4 \x 6 5 \x 6 D \x 0 5 \x 8 5 \x 7 6 \x 6 1 \x 6 C \x 7 5 \x 6 5 \x 0 6 \x 4 0 \x 1 0 \x 0 0 \x 0 0 \x 2 0 \x 0 1 \x 0 6 \x 0 1 \x 0 1 \x 8 2 \x 5 F \x 3 5 \x 0 3 \x 0 5 \x 2 0 \x 1 E \x 1 7 \x 0 B \x 0 3 \x 0 1 \x 0 1 \x 0 6 \x 0 1 \x 4 0 \x 0 B \x 0 0 \x 0 1 \x 6 0 \x 0 D \x 0 2 \x 8 2 \x 5 F \x 3 7 \x 6 0 \x 1 9 \x 8 0 \x 0 0 \x 0 0 \x 0 8 \x 6 0 \x 1 9 \x 8 0 \x 2 7 \x 0 2 \x 8 2 \x 5 F \x 3 9 \x 2 0 \x 1 9 \x 0 0 \x F F \x 0 A \x 8 1 \x 0 0 \x 0 0 \x 0 1 \x 7 3 \x D D \x A A \x 2 A \x B E \x 0 0 \x 0 0 \x 0 9 \x 0 0 \x 2 1 \x 8 5 \x 7 7 \x 4 3 \x 7 1 \x 7 B \x 1 7 \x 8 8 " } err
assert_match " * B a d d a t a f o r m a t * " $err
}
}
test { corrupt payload: fuzzer findings - stream bad lp_count} {
start_server [ list overrides [ list loglevel verbose use-exit-on-panic yes crash-memcheck-enabled no] ] {
r config set sanitize-dump-payload yes
r debug set-skip-checksum-validation 1
catch { r RESTORE _stream 0 " \x 0 F \x 0 1 \x 1 0 \x 0 0 \x 0 0 \x 0 1 \x 7 3 \x D E \x D F \x 7 D \x 9 B \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 4 0 \x 4 2 \x 4 2 \x 0 0 \x 0 0 \x 0 0 \x 1 8 \x 0 0 \x 0 3 \x 0 1 \x 0 0 \x 0 1 \x 0 2 \x 0 1 \x 8 4 \x 6 9 \x 7 4 \x 6 5 \x 6 D \x 0 5 \x 8 5 \x 7 6 \x 6 1 \x 6 C \x 7 5 \x 6 5 \x 0 6 \x 0 0 \x 0 1 \x 0 2 \x 0 1 \x 0 0 \x 0 1 \x 0 0 \x 0 1 \x 0 1 \x 0 1 \x 0 0 \x 0 1 \x 5 6 \x 0 1 \x 0 2 \x 0 1 \x 2 2 \x 0 1 \x 0 0 \x 0 1 \x 0 1 \x 0 1 \x 8 2 \x 5 F \x 3 1 \x 0 3 \x 0 5 \x 0 1 \x 0 2 \x 0 1 \x 2 C \x 0 1 \x 0 0 \x 0 1 \x 0 1 \x 0 1 \x 0 2 \x 0 1 \x 0 5 \x 0 1 \x F F \x 0 3 \x 8 1 \x 0 0 \x 0 0 \x 0 1 \x 7 3 \x D E \x D F \x 7 D \x C 7 \x 0 0 \x 0 1 \x 0 7 \x 6 D \x 7 9 \x 6 7 \x 7 2 \x 6 F \x 7 5 \x 7 0 \x 8 1 \x 0 0 \x 0 0 \x 0 1 \x 7 3 \x D E \x D F \x 7 D \x 9 B \x 0 0 \x 0 1 \x 0 0 \x 0 0 \x 0 1 \x 7 3 \x D E \x D F \x 7 D \x 9 B \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x F 9 \x 7 D \x D F \x D E \x 7 3 \x 0 1 \x 0 0 \x 0 0 \x 0 1 \x 0 1 \x 0 5 \x 4 1 \x 6 C \x 6 9 \x 6 3 \x 6 5 \x E B \x 7 D \x D F \x D E \x 7 3 \x 0 1 \x 0 0 \x 0 0 \x 0 1 \x 0 0 \x 0 0 \x 0 1 \x 7 3 \x D E \x D F \x 7 D \x 9 B \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 9 \x 0 0 \x B 2 \x A 8 \x A 7 \x 5 F \x 1 B \x 6 1 \x 7 2 \x D 5 " } err
assert_match " * B a d d a t a f o r m a t * " $err
r ping
}
}
test { corrupt payload: fuzzer findings - stream bad lp_count - unsanitized} {
start_server [ list overrides [ list loglevel verbose use-exit-on-panic yes crash-memcheck-enabled no] ] {
r config set sanitize-dump-payload no
r debug set-skip-checksum-validation 1
r RESTORE _stream 0 " \x 0 F \x 0 1 \x 1 0 \x 0 0 \x 0 0 \x 0 1 \x 7 3 \x D E \x D F \x 7 D \x 9 B \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 4 0 \x 4 2 \x 4 2 \x 0 0 \x 0 0 \x 0 0 \x 1 8 \x 0 0 \x 0 3 \x 0 1 \x 0 0 \x 0 1 \x 0 2 \x 0 1 \x 8 4 \x 6 9 \x 7 4 \x 6 5 \x 6 D \x 0 5 \x 8 5 \x 7 6 \x 6 1 \x 6 C \x 7 5 \x 6 5 \x 0 6 \x 0 0 \x 0 1 \x 0 2 \x 0 1 \x 0 0 \x 0 1 \x 0 0 \x 0 1 \x 0 1 \x 0 1 \x 0 0 \x 0 1 \x 5 6 \x 0 1 \x 0 2 \x 0 1 \x 2 2 \x 0 1 \x 0 0 \x 0 1 \x 0 1 \x 0 1 \x 8 2 \x 5 F \x 3 1 \x 0 3 \x 0 5 \x 0 1 \x 0 2 \x 0 1 \x 2 C \x 0 1 \x 0 0 \x 0 1 \x 0 1 \x 0 1 \x 0 2 \x 0 1 \x 0 5 \x 0 1 \x F F \x 0 3 \x 8 1 \x 0 0 \x 0 0 \x 0 1 \x 7 3 \x D E \x D F \x 7 D \x C 7 \x 0 0 \x 0 1 \x 0 7 \x 6 D \x 7 9 \x 6 7 \x 7 2 \x 6 F \x 7 5 \x 7 0 \x 8 1 \x 0 0 \x 0 0 \x 0 1 \x 7 3 \x D E \x D F \x 7 D \x 9 B \x 0 0 \x 0 1 \x 0 0 \x 0 0 \x 0 1 \x 7 3 \x D E \x D F \x 7 D \x 9 B \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x F 9 \x 7 D \x D F \x D E \x 7 3 \x 0 1 \x 0 0 \x 0 0 \x 0 1 \x 0 1 \x 0 5 \x 4 1 \x 6 C \x 6 9 \x 6 3 \x 6 5 \x E B \x 7 D \x D F \x D E \x 7 3 \x 0 1 \x 0 0 \x 0 0 \x 0 1 \x 0 0 \x 0 0 \x 0 1 \x 7 3 \x D E \x D F \x 7 D \x 9 B \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 9 \x 0 0 \x B 2 \x A 8 \x A 7 \x 5 F \x 1 B \x 6 1 \x 7 2 \x D 5 "
catch { r XREVRANGE _stream 638932639 738 }
assert_equal [ count_log_message 0 " c r a s h e d b y s i g n a l " ] 0
assert_equal [ count_log_message 0 " A S S E R T I O N F A I L E D " ] 1
}
}
test { corrupt payload: fuzzer findings - stream integrity check issue} {
start_server [ list overrides [ list loglevel verbose use-exit-on-panic yes crash-memcheck-enabled no] ] {
r config set sanitize-dump-payload yes
r debug set-skip-checksum-validation 1
catch { r RESTORE _stream 0 " \x 0 F \x 0 2 \x 1 0 \x 0 0 \x 0 0 \x 0 1 \x 7 5 \x 2 D \x A 2 \x 9 0 \x 6 7 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x C 3 \x 4 0 \x 4 F \x 4 0 \x 5 C \x 1 8 \x 5 C \x 0 0 \x 0 0 \x 0 0 \x 2 4 \x 0 0 \x 0 5 \x 0 1 \x 0 0 \x 0 1 \x 4 A \x 0 1 \x 8 4 \x 6 9 \x 7 4 \x 6 5 \x 6 D \x 0 5 \x 8 5 \x 7 6 \x 6 1 \x 6 C \x 7 5 \x 6 5 \x 0 6 \x 4 0 \x 1 0 \x 0 0 \x 0 0 \x 2 0 \x 0 1 \x 0 0 \x 0 1 \x 2 0 \x 0 3 \x 0 0 \x 0 5 \x 2 0 \x 1 C \x 4 0 \x 0 9 \x 0 5 \x 0 1 \x 0 1 \x 8 2 \x 5 F \x 3 1 \x 0 3 \x 8 0 \x 0 D \x 0 0 \x 0 2 \x 2 0 \x 0 D \x 0 0 \x 0 2 \x A 0 \x 1 9 \x 0 0 \x 0 3 \x 2 0 \x 0 B \x 0 2 \x 8 2 \x 5 F \x 3 3 \x A 0 \x 1 9 \x 0 0 \x 0 4 \x 2 0 \x 0 D \x 0 0 \x 0 4 \x 2 0 \x 1 9 \x 0 0 \x F F \x 1 0 \x 0 0 \x 0 0 \x 0 1 \x 7 5 \x 2 D \x A 2 \x 9 0 \x 6 7 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 5 \x C 3 \x 4 0 \x 5 6 \x 4 0 \x 6 0 \x 1 8 \x 6 0 \x 0 0 \x 0 0 \x 0 0 \x 2 4 \x 0 0 \x 0 5 \x 0 1 \x 0 0 \x 0 1 \x 0 2 \x 0 1 \x 8 4 \x 6 9 \x 7 4 \x 6 5 \x 6 D \x 0 5 \x 8 5 \x 7 6 \x 6 1 \x 6 C \x 7 5 \x 6 5 \x 0 6 \x 4 0 \x 1 0 \x 0 0 \x 0 0 \x 2 0 \x 0 1 \x 0 6 \x 0 1 \x 0 1 \x 8 2 \x 5 F \x 3 5 \x 0 3 \x 0 5 \x 2 0 \x 1 E \x 4 0 \x 0 B \x 0 3 \x 0 1 \x 0 1 \x 0 6 \x 0 1 \x 8 0 \x 0 B \x 0 0 \x 0 2 \x 2 0 \x 0 B \x 0 2 \x 8 2 \x 5 F \x 3 7 \x 6 0 \x 1 9 \x 0 3 \x 0 1 \x 0 1 \x D F \x F B \x 2 0 \x 0 5 \x 0 0 \x 0 8 \x 6 0 \x 1 A \x 2 0 \x 0 C \x 0 0 \x F C \x 2 0 \x 0 5 \x 0 2 \x 8 2 \x 5 F \x 3 9 \x 2 0 \x 1 B \x 0 0 \x F F \x 0 A \x 8 1 \x 0 0 \x 0 0 \x 0 1 \x 7 5 \x 2 D \x A 2 \x 9 0 \x 6 8 \x 0 1 \x 0 0 \x 0 9 \x 0 0 \x 1 D \x 6 F \x C 0 \x 6 9 \x 8 A \x D E \x F 7 \x 9 2 " } err
assert_match " * B a d d a t a f o r m a t * " $err
}
}
test { corrupt payload: fuzzer findings - infinite loop} {
start_server [ list overrides [ list loglevel verbose use-exit-on-panic yes crash-memcheck-enabled no] ] {
r config set sanitize-dump-payload no
r debug set-skip-checksum-validation 1
r RESTORE _stream 0 " \x 0 F \x 0 1 \x 1 0 \x 0 0 \x 0 0 \x 0 1 \x 7 5 \x 3 A \x A 6 \x D 0 \x 9 3 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 4 0 \x 4 2 \x 4 2 \x 0 0 \x 0 0 \x 0 0 \x 1 8 \x 0 0 \x 0 3 \x 0 1 \x 0 0 \x 0 1 \x 0 2 \x 0 1 \x 8 4 \x 6 9 \x 7 4 \x 6 5 \x 6 D \x 0 5 \x 8 5 \x 7 6 \x 6 1 \x 6 C \x 7 5 \x 6 5 \x 0 6 \x 0 0 \x 0 1 \x 0 2 \x 0 1 \x 0 0 \x 0 1 \x 0 0 \x 0 1 \x 0 1 \x 0 1 \x 0 0 \x 0 1 \x 0 5 \x 0 1 \x 0 2 \x 0 1 \x 0 0 \x 0 1 \x 0 1 \x 0 1 \x 0 1 \x 0 1 \x 8 2 \x 5 F \x 3 1 \x 0 3 \x F D \x 0 1 \x 0 2 \x 0 1 \x 0 0 \x 0 1 \x 0 2 \x 0 1 \x 0 1 \x 0 1 \x 0 2 \x 0 1 \x 0 5 \x 0 1 \x F F \x 0 3 \x 8 1 \x 0 0 \x 0 0 \x 0 1 \x 7 5 \x 3 A \x A 6 \x D 0 \x 9 3 \x 0 2 \x 0 1 \x 0 7 \x 6 D \x 7 9 \x 6 7 \x 7 2 \x 6 F \x 7 5 \x 7 0 \x 8 1 \x 0 0 \x 0 0 \x 0 1 \x 7 5 \x 3 A \x A 6 \x D 0 \x 9 3 \x 0 0 \x 0 1 \x 0 0 \x 0 0 \x 0 1 \x 7 5 \x 3 A \x A 6 \x D 0 \x 9 3 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 9 4 \x D 0 \x A 6 \x 3 A \x 7 5 \x 0 1 \x 0 0 \x 0 0 \x 0 1 \x 0 1 \x 0 5 \x 4 1 \x 6 C \x 6 9 \x 6 3 \x 6 5 \x 9 4 \x D 0 \x A 6 \x 3 A \x 7 5 \x 0 1 \x 0 0 \x 0 0 \x 0 1 \x 0 0 \x 0 0 \x 0 1 \x 7 5 \x 3 A \x A 6 \x D 0 \x 9 3 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 9 \x 0 0 \x C 4 \x 0 9 \x A D \x 6 9 \x 7 E \x E E \x A 6 \x 2 F "
catch { r XREVRANGE _stream 288270516 971031845 }
assert_equal [ count_log_message 0 " c r a s h e d b y s i g n a l " ] 0
assert_equal [ count_log_message 0 " A S S E R T I O N F A I L E D " ] 1
}
}
2021-08-10 02:18:49 -04:00
test { corrupt payload: fuzzer findings - hash ziplist too long entry len} {
2020-11-02 02:35:37 -05:00
start_server [ list overrides [ list loglevel verbose use-exit-on-panic yes crash-memcheck-enabled no] ] {
r debug set-skip-checksum-validation 1
2021-08-10 02:18:49 -04:00
catch {
r RESTORE _hash 0 " \x 0 D \x 3 D \x 3 D \x 0 0 \x 0 0 \x 0 0 \x 3 A \x 0 0 \x 0 0 \x 0 0 \x 1 4 \x 1 3 \x 0 0 \x F 5 \x 0 2 \x F 5 \x 0 2 \x F 2 \x 0 2 \x 5 3 \x 5 F \x 3 1 \x 0 4 \x F 3 \x 0 2 \x F 3 \x 0 2 \x F 7 \x 0 2 \x F 7 \x 0 2 \x F 8 \x 0 2 \x 0 2 \x 5 F \x 3 7 \x 0 4 \x F 1 \x 0 2 \x F 1 \x 0 2 \x F 6 \x 0 2 \x 0 2 \x 5 F \x 3 5 \x 0 4 \x F 4 \x 0 2 \x 0 2 \x 5 F \x 3 3 \x 0 4 \x F A \x 0 2 \x 0 2 \x 5 F \x 3 9 \x 0 4 \x F 9 \x 0 2 \x F 9 \x F F \x 0 9 \x 0 0 \x B 5 \x 4 8 \x D E \x 6 2 \x 3 1 \x D 0 \x E 5 \x 6 3 "
} err
assert_match " * B a d d a t a f o r m a t * " $err
verify_log_message 0 " * i n t e g r i t y c h e c k f a i l e d * " 0
2020-11-02 02:35:37 -05:00
}
}
2021-08-18 09:00:22 -04:00
if { $run_oom_tests } {
2020-11-22 14:22:49 -05:00
test { corrupt payload: OOM in rdbGenericLoadStringObject} {
start_server [ list overrides [ list loglevel verbose use-exit-on-panic yes crash-memcheck-enabled no] ] {
r config set sanitize-dump-payload no
catch { r RESTORE x 0 " \x 0 A \x 8 1 \x 7 F \x F F \x F F \x F F \x F F \x F F \x F F \x F F \x 1 3 \x 0 0 \x 0 0 \x 0 0 \x 0 E \x 0 0 \x 0 0 \x 0 0 \x 0 2 \x 0 0 \x 0 0 \x 0 2 \x 6 1 \x 0 0 \x 0 4 \x 0 2 \x 6 2 \x 0 0 \x F F \x 0 9 \x 0 0 \x 5 7 \x 0 4 \x E 5 \x C D \x D 4 \x 3 7 \x 6 C \x 5 7 " } err
assert_match " * B a d d a t a f o r m a t * " $err
r ping
}
}
test { corrupt payload: fuzzer findings - OOM in dictExpand} {
start_server [ list overrides [ list loglevel verbose use-exit-on-panic yes crash-memcheck-enabled no] ] {
r config set sanitize-dump-payload no
r debug set-skip-checksum-validation 1
catch { r RESTORE x 0 " \x 0 2 \x 8 1 \x 0 2 \x 5 F \x 3 1 \x C 0 \x 0 0 \x C 0 \x 0 2 \x 0 9 \x 0 0 \x C D \x 8 4 \x 2 C \x B 7 \x E 8 \x A 4 \x 4 9 \x 5 7 " } err
assert_match " * B a d d a t a f o r m a t * " $err
r ping
}
}
2021-08-18 09:00:22 -04:00
}
2021-09-09 11:18:53 -04:00
test { corrupt payload: fuzzer findings - zset ziplist invalid tail offset} {
2020-08-14 09:05:34 -04:00
start_server [ list overrides [ list loglevel verbose use-exit-on-panic yes crash-memcheck-enabled no] ] {
r config set sanitize-dump-payload no
r debug set-skip-checksum-validation 1
2021-09-09 11:18:53 -04:00
catch { r RESTORE _zset 0 " \x 0 C \x 1 9 \x 1 9 \x 0 0 \x 0 0 \x 0 0 \x 0 2 \x 0 0 \x 0 0 \x 0 0 \x 0 6 \x 0 0 \x 0 0 \x F 1 \x 0 2 \x F 1 \x 0 2 \x 0 2 \x 5 F \x 3 1 \x 0 4 \x F 2 \x 0 2 \x F 3 \x 0 2 \x F 3 \x F F \x 0 9 \x 0 0 \x 4 D \x 7 2 \x 7 B \x 9 7 \x C D \x 9 A \x 7 0 \x C 1 " } err
assert_match " * B a d d a t a f o r m a t * " $err
verify_log_message 0 " * Z s e t z i p l i s t i n t e g r i t y c h e c k f a i l e d * " 0
2020-08-14 09:05:34 -04:00
}
}
test { corrupt payload: fuzzer findings - negative reply length} {
start_server [ list overrides [ list loglevel verbose use-exit-on-panic yes crash-memcheck-enabled no] ] {
r config set sanitize-dump-payload no
r debug set-skip-checksum-validation 1
r RESTORE _stream 0 " \x 0 F \x 0 1 \x 1 0 \x 0 0 \x 0 0 \x 0 1 \x 7 5 \x C F \x A 1 \x 1 6 \x A 7 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 4 0 \x 4 2 \x 4 2 \x 0 0 \x 0 0 \x 0 0 \x 1 8 \x 0 0 \x 0 3 \x 0 1 \x 0 0 \x 0 1 \x 0 2 \x 0 1 \x 8 4 \x 6 9 \x 7 4 \x 6 5 \x 6 D \x 0 5 \x 8 5 \x 7 6 \x 6 1 \x 6 C \x 7 5 \x 6 5 \x 0 6 \x 0 0 \x 0 1 \x 0 2 \x 0 1 \x 0 0 \x 0 1 \x 0 0 \x 0 1 \x 0 1 \x 0 1 \x 0 0 \x 0 1 \x 0 5 \x 0 1 \x 0 2 \x 0 1 \x 0 0 \x 0 1 \x 0 1 \x 0 1 \x 0 1 \x 0 1 \x 1 4 \x 5 F \x 3 1 \x 0 3 \x 0 5 \x 0 1 \x 0 2 \x 0 1 \x 0 0 \x 0 1 \x 0 2 \x 0 1 \x 0 1 \x 0 1 \x 0 2 \x 0 1 \x 0 5 \x 0 1 \x F F \x 0 3 \x 8 1 \x 0 0 \x 0 0 \x 0 1 \x 7 5 \x C F \x A 1 \x 1 6 \x A 7 \x 0 2 \x 0 1 \x 0 7 \x 6 D \x 7 9 \x 6 7 \x 7 2 \x 6 F \x 7 5 \x 7 0 \x 8 1 \x 0 0 \x 0 0 \x 0 1 \x 7 5 \x C F \x A 1 \x 1 6 \x A 7 \x 0 1 \x 0 1 \x 0 0 \x 0 0 \x 0 1 \x 7 5 \x C F \x A 1 \x 1 6 \x A 7 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 1 \x A 7 \x 1 6 \x A 1 \x C F \x 7 5 \x 0 1 \x 0 0 \x 0 0 \x 0 1 \x 0 1 \x 0 5 \x 4 1 \x 6 C \x 6 9 \x 6 3 \x 6 5 \x A 7 \x 1 6 \x A 1 \x C F \x 7 5 \x 0 1 \x 0 0 \x 0 0 \x 0 1 \x 0 0 \x 0 0 \x 0 1 \x 7 5 \x C F \x A 1 \x 1 6 \x A 7 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 1 \x 0 9 \x 0 0 \x 1 B \x 4 2 \x 5 2 \x B 8 \x D D \x 5 C \x E 5 \x 4 E "
catch { r XADD _stream * - 956 - 2601503852 }
catch { r XINFO STREAM _stream FULL}
assert_equal [ count_log_message 0 " c r a s h e d b y s i g n a l " ] 0
assert_equal [ count_log_message 0 " A S S E R T I O N F A I L E D " ] 1
}
}
test { corrupt payload: fuzzer findings - valgrind negative malloc} {
start_server [ list overrides [ list loglevel verbose use-exit-on-panic yes crash-memcheck-enabled no] ] {
r config set sanitize-dump-payload yes
r debug set-skip-checksum-validation 1
catch { r RESTORE _key 0 " \x 0 E \x 0 1 \x 8 1 \x D 6 \x D 6 \x 0 0 \x 0 0 \x 0 0 \x 0 A \x 0 0 \x 0 0 \x 0 0 \x 0 1 \x 0 0 \x 0 0 \x 4 0 \x C 8 \x 6 F \x 2 F \x 3 6 \x E 2 \x D F \x E 3 \x 2 E \x 2 6 \x 6 4 \x 8 B \x 8 7 \x D 1 \x 7 A \x B D \x F F \x E F \x E F \x 6 3 \x 6 5 \x F 6 \x F 8 \x 8 C \x 4 E \x E C \x 9 6 \x 8 9 \x 5 6 \x 8 8 \x F 8 \x 3 D \x 9 6 \x 5 A \x 3 2 \x B D \x D 1 \x 3 6 \x D 8 \x 0 2 \x E 6 \x 6 6 \x 3 7 \x C B \x 3 4 \x 3 4 \x C 4 \x 5 2 \x A 7 \x 2 A \x D 5 \x 6 F \x 2 F \x 7 E \x E E \x A 2 \x 9 4 \x D 9 \x E B \x A 9 \x 0 9 \x 3 8 \x 3 B \x E 1 \x A 9 \x 6 0 \x B 6 \x 4 E \x 0 9 \x 4 4 \x 1 F \x 7 0 \x 2 4 \x A A \x 4 7 \x A 8 \x 6 E \x 3 0 \x E 1 \x 1 3 \x 4 9 \x 4 E \x A 1 \x 9 2 \x C 4 \x 6 C \x F 0 \x 3 5 \x 8 3 \x D 9 \x 4 F \x D 9 \x 9 C \x 0 A \x 0 D \x 7 A \x E 7 \x B 1 \x 6 1 \x F 5 \x C 1 \x 2 D \x D C \x C 3 \x 0 E \x 8 7 \x A 6 \x 8 0 \x 1 5 \x 1 8 \x B A \x 7 F \x 7 2 \x D D \x 1 4 \x 7 5 \x 4 6 \x 4 4 \x 0 B \x C A \x 9 C \x 8 F \x 1 C \x 3 C \x D 7 \x D A \x 0 6 \x 6 2 \x 1 8 \x 7 E \x 1 5 \x 1 7 \x 2 4 \x A B \x 4 5 \x 2 1 \x 2 7 \x C 2 \x B C \x B B \x 8 6 \x 6 E \x D 8 \x B D \x 8 E \x 5 0 \x E 0 \x E 0 \x 8 8 \x A 4 \x 9 B \x 9 D \x 1 5 \x 2 A \x 9 8 \x F F \x 5 E \x 7 8 \x 6 C \x 8 1 \x F C \x A 8 \x C 9 \x C 8 \x E 6 \x 6 1 \x C 8 \x D 1 \x 4 A \x 7 F \x 8 1 \x D 6 \x A 6 \x 1 A \x A D \x 4 C \x C 1 \x A 2 \x 1 C \x 9 0 \x 6 8 \x 1 5 \x 2 A \x 8 A \x 3 6 \x C 0 \x 5 8 \x C 3 \x C C \x A 6 \x 5 4 \x 1 9 \x 1 2 \x 0 F \x E B \x 4 6 \x F F \x 6 E \x E 3 \x A 7 \x 9 2 \x F 8 \x F F \x 0 9 \x 0 0 \x D 0 \x 7 1 \x F 7 \x 9 F \x F 7 \x 6 A \x D 6 \x 2 E " } err
assert_match " * B a d d a t a f o r m a t * " $err
r ping
}
}
2020-12-14 10:10:31 -05:00
test { corrupt payload: fuzzer findings - valgrind invalid read} {
start_server [ list overrides [ list loglevel verbose use-exit-on-panic yes crash-memcheck-enabled no] ] {
r config set sanitize-dump-payload yes
r debug set-skip-checksum-validation 1
catch { r RESTORE _key 0 " \x 0 5 \x 0 A \x 0 2 \x 5 F \x 3 9 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 2 2 \x 4 0 \x C 0 \x 0 8 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 2 0 \x 4 0 \x 0 2 \x 5 F \x 3 7 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 1 C \x 4 0 \x C 0 \x 0 6 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 1 8 \x 4 0 \x 0 2 \x 5 F \x 3 3 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 1 4 \x 4 0 \x C 0 \x 0 4 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 1 0 \x 4 0 \x 0 2 \x 5 F \x 3 3 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 8 \x 4 0 \x C 0 \x 0 2 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 4 0 \x 0 2 \x 5 F \x 3 1 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x F 0 \x 3 F \x C 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 9 \x 0 0 \x 3 C \x 6 6 \x D 7 \x 1 4 \x A 9 \x D A \x 3 C \x 6 9 " } err
assert_match " * B a d d a t a f o r m a t * " $err
r ping
}
}
2021-08-05 15:42:20 -04:00
test { corrupt payload: fuzzer findings - empty hash ziplist} {
2021-01-31 05:13:45 -05:00
start_server [ list overrides [ list loglevel verbose use-exit-on-panic yes crash-memcheck-enabled no] ] {
r config set sanitize-dump-payload yes
r debug set-skip-checksum-validation 1
2021-08-05 15:42:20 -04:00
catch { r RESTORE _int 0 " \x 0 4 \x C 0 \x 0 1 \x 0 9 \x 0 0 \x F 6 \x 8 A \x B 6 \x 7 A \x 8 5 \x 8 7 \x 7 2 \x 4 D " } err
assert_match " * B a d d a t a f o r m a t * " $err
r ping
2021-01-31 05:13:45 -05:00
}
}
2021-03-24 05:33:49 -04:00
test { corrupt payload: fuzzer findings - stream with no records} {
start_server [ list overrides [ list loglevel verbose use-exit-on-panic yes crash-memcheck-enabled no] ] {
r config set sanitize-dump-payload no
r debug set-skip-checksum-validation 1
r restore _stream 0 " \x 0 F \x 0 1 \x 1 0 \x 0 0 \x 0 0 \x 0 1 \x 7 8 \x 4 D \x 5 5 \x 6 8 \x 0 9 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 4 0 \x 4 2 \x 4 2 \x 0 0 \x 0 0 \x 0 0 \x 1 8 \x 0 0 \x 0 2 \x 0 1 \x 0 1 \x 0 1 \x 0 2 \x 0 1 \x 8 4 \x 6 9 \x 7 4 \x 6 5 \x 6 D \x 0 5 \x 8 5 \x 7 6 \x 6 1 \x 6 C \x 7 5 \x 6 5 \x 0 6 \x 0 0 \x 0 1 \x 0 2 \x 0 1 \x 0 0 \x 0 1 \x 0 0 \x 0 1 \x 0 1 \x 0 1 \x 0 0 \x 0 1 \x 0 5 \x 0 1 \x 0 3 \x 0 1 \x 3 E \x 0 1 \x 0 0 \x 0 1 \x 0 1 \x 0 1 \x 8 2 \x 5 F \x 3 1 \x 0 3 \x 0 5 \x 0 1 \x 0 2 \x 0 1 \x 5 0 \x 0 1 \x 0 0 \x 0 1 \x 0 1 \x 0 1 \x 0 2 \x 0 1 \x 0 5 \x 2 3 \x F F \x 0 2 \x 8 1 \x 0 0 \x 0 0 \x 0 1 \x 7 8 \x 4 D \x 5 5 \x 6 8 \x 5 9 \x 0 0 \x 0 1 \x 0 7 \x 6 D \x 7 9 \x 6 7 \x 7 2 \x 6 F \x 7 5 \x 7 0 \x 8 1 \x 0 0 \x 0 0 \x 0 1 \x 7 8 \x 4 D \x 5 5 \x 6 8 \x 4 7 \x 0 0 \x 0 1 \x 0 0 \x 0 0 \x 0 1 \x 7 8 \x 4 D \x 5 5 \x 6 8 \x 4 7 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 9 F \x 6 8 \x 5 5 \x 4 D \x 7 8 \x 0 1 \x 0 0 \x 0 0 \x 0 1 \x 0 1 \x 0 5 \x 4 1 \x 6 C \x 6 9 \x 6 3 \x 6 5 \x 8 5 \x 6 8 \x 5 5 \x 4 D \x 7 8 \x 0 1 \x 0 0 \x 0 0 \x 0 1 \x 0 0 \x 0 0 \x 0 1 \x 7 8 \x 4 D \x 5 5 \x 6 8 \x 4 7 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 9 \x 0 0 \x F 1 \x C 0 \x 7 2 \x 7 0 \x 3 9 \x 4 0 \x 1 E \x A 9 " replace
catch { r XREAD STREAMS _stream $ }
assert_equal [ count_log_message 0 " c r a s h e d b y s i g n a l " ] 0
assert_equal [ count_log_message 0 " G u r u M e d i t a t i o n " ] 1
}
}
2021-08-05 15:56:14 -04:00
test { corrupt payload: fuzzer findings - quicklist ziplist tail followed by extra data which start with 0xff } {
start_server [ list overrides [ list loglevel verbose use-exit-on-panic yes crash-memcheck-enabled no] ] {
r config set sanitize-dump-payload yes
r debug set-skip-checksum-validation 1
catch {
r restore key 0 " \x 0 E \x 0 1 \x 1 1 \x 1 1 \x 0 0 \x 0 0 \x 0 0 \x 0 A \x 0 0 \x 0 0 \x 0 0 \x 0 1 \x 0 0 \x 0 0 \x F 6 \x F F \x B 0 \x 6 C \x 9 C \x F F \x 0 9 \x 0 0 \x 9 C \x 3 7 \x 4 7 \x 4 9 \x 4 D \x D E \x 9 4 \x F 5 " replace
} err
assert_match " * B a d d a t a f o r m a t * " $err
verify_log_message 0 " * i n t e g r i t y c h e c k f a i l e d * " 0
}
}
test { corrupt payload: fuzzer findings - dict init to huge size} {
start_server [ list overrides [ list loglevel verbose use-exit-on-panic yes crash-memcheck-enabled no] ] {
r config set sanitize-dump-payload no
r debug set-skip-checksum-validation 1
catch { r restore key 0 " \x 0 2 \x 8 1 \x C 0 \x 0 0 \x 0 2 \x 5 F \x 3 1 \x C 0 \x 0 2 \x 0 9 \x 0 0 \x B 2 \x 1 B \x E 5 \x 1 7 \x 2 E \x 1 5 \x F 4 \x 6 C " replace} err
assert_match " * B a d d a t a f o r m a t * " $err
r ping
}
}
test { corrupt payload: fuzzer findings - huge string} {
start_server [ list overrides [ list loglevel verbose use-exit-on-panic yes crash-memcheck-enabled no] ] {
r config set sanitize-dump-payload yes
r debug set-skip-checksum-validation 1
catch { r restore key 0 " \x 0 0 \x 8 1 \x 0 1 \x 0 9 \x 0 0 \x F 6 \x 2 B \x B 6 \x 7 A \x 8 5 \x 8 7 \x 7 2 \x 4 D " } err
assert_match " * B a d d a t a f o r m a t * " $err
r ping
}
}
test { corrupt payload: fuzzer findings - stream PEL without consumer} {
start_server [ list overrides [ list loglevel verbose use-exit-on-panic yes crash-memcheck-enabled no] ] {
r config set sanitize-dump-payload yes
r debug set-skip-checksum-validation 1
catch { r restore _stream 0 " \x 0 F \x 0 1 \x 1 0 \x 0 0 \x 0 0 \x 0 1 \x 7 B \x 0 8 \x F 0 \x B 2 \x 3 4 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x C 3 \x 3 B \x 4 0 \x 4 2 \x 1 9 \x 4 2 \x 0 0 \x 0 0 \x 0 0 \x 1 8 \x 0 0 \x 0 2 \x 0 1 \x 0 1 \x 0 1 \x 0 2 \x 0 1 \x 8 4 \x 6 9 \x 7 4 \x 6 5 \x 6 D \x 0 5 \x 8 5 \x 7 6 \x 6 1 \x 6 C \x 7 5 \x 6 5 \x 0 6 \x 0 0 \x 2 0 \x 1 0 \x 0 0 \x 0 0 \x 2 0 \x 0 1 \x 0 0 \x 0 1 \x 2 0 \x 0 3 \x 0 2 \x 0 5 \x 0 1 \x 0 3 \x 2 0 \x 0 5 \x 4 0 \x 0 0 \x 0 4 \x 8 2 \x 5 F \x 3 1 \x 0 3 \x 0 5 \x 6 0 \x 1 9 \x 8 0 \x 3 2 \x 0 2 \x 0 5 \x 0 1 \x F F \x 0 2 \x 8 1 \x 0 0 \x 0 0 \x 0 1 \x 7 B \x 0 8 \x F 0 \x B 2 \x 3 4 \x 0 2 \x 0 1 \x 0 7 \x 6 D \x 7 9 \x 6 7 \x 7 2 \x 6 F \x 7 5 \x 7 0 \x 8 1 \x 0 0 \x 0 0 \x 0 1 \x 7 B \x 0 8 \x F 0 \x B 2 \x 3 4 \x 0 1 \x 0 1 \x 0 0 \x 0 0 \x 0 1 \x 7 B \x 0 8 \x F 0 \x B 2 \x 3 4 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 1 \x 3 5 \x B 2 \x F 0 \x 0 8 \x 7 B \x 0 1 \x 0 0 \x 0 0 \x 0 1 \x 0 1 \x 1 3 \x 4 1 \x 6 C \x 6 9 \x 6 3 \x 6 5 \x 3 5 \x B 2 \x F 0 \x 0 8 \x 7 B \x 0 1 \x 0 0 \x 0 0 \x 0 1 \x 0 0 \x 0 0 \x 0 1 \x 7 B \x 0 8 \x F 0 \x B 2 \x 3 4 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 1 \x 0 9 \x 0 0 \x 2 8 \x 2 F \x E 0 \x C 5 \x 0 4 \x B B \x A 7 \x 3 1 " } err
assert_match " * B a d d a t a f o r m a t * " $err
r ping
}
}
test { corrupt payload: fuzzer findings - stream listpack valgrind issue} {
start_server [ list overrides [ list loglevel verbose use-exit-on-panic yes crash-memcheck-enabled no] ] {
r config set sanitize-dump-payload no
r debug set-skip-checksum-validation 1
r restore _stream 0 " \x 0 F \x 0 1 \x 1 0 \x 0 0 \x 0 0 \x 0 1 \x 7 B \x 0 9 \x 5 E \x 9 4 \x F F \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 4 0 \x 4 2 \x 4 2 \x 0 0 \x 0 0 \x 0 0 \x 1 8 \x 0 0 \x 0 2 \x 0 1 \x 0 1 \x 0 1 \x 0 2 \x 0 1 \x 8 4 \x 6 9 \x 7 4 \x 6 5 \x 6 D \x 0 5 \x 8 5 \x 7 6 \x 6 1 \x 6 C \x 7 5 \x 6 5 \x 0 6 \x 0 0 \x 0 1 \x 0 2 \x 0 1 \x 0 0 \x 0 1 \x 0 0 \x 0 1 \x 0 1 \x 0 1 \x 0 0 \x 0 1 \x 0 5 \x 0 1 \x 0 3 \x 0 1 \x 2 5 \x 0 1 \x 0 0 \x 0 1 \x 0 1 \x 0 1 \x 8 2 \x 5 F \x 3 1 \x 0 3 \x 0 5 \x 0 1 \x 0 2 \x 0 1 \x 3 2 \x 0 1 \x 0 0 \x 0 1 \x 0 1 \x 0 1 \x 0 2 \x 0 1 \x F 0 \x 0 1 \x F F \x 0 2 \x 8 1 \x 0 0 \x 0 0 \x 0 1 \x 7 B \x 0 9 \x 5 E \x 9 5 \x 3 1 \x 0 0 \x 0 1 \x 0 7 \x 6 D \x 7 9 \x 6 7 \x 7 2 \x 6 F \x 7 5 \x 7 0 \x 8 1 \x 0 0 \x 0 0 \x 0 1 \x 7 B \x 0 9 \x 5 E \x 9 5 \x 2 4 \x 0 0 \x 0 1 \x 0 0 \x 0 0 \x 0 1 \x 7 B \x 0 9 \x 5 E \x 9 5 \x 2 4 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 5 C \x 9 5 \x 5 E \x 0 9 \x 7 B \x 0 1 \x 0 0 \x 0 0 \x 0 1 \x 0 1 \x 0 5 \x 4 1 \x 6 C \x 6 9 \x 6 3 \x 6 5 \x 4 B \x 9 5 \x 5 E \x 0 9 \x 7 B \x 0 1 \x 0 0 \x 0 0 \x 0 1 \x 0 0 \x 0 0 \x 0 1 \x 7 B \x 0 9 \x 5 E \x 9 5 \x 2 4 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 9 \x 0 0 \x 1 9 \x 2 9 \x 9 4 \x D F \x 7 6 \x F 8 \x 1 A \x C 6 "
catch { r XINFO STREAM _stream FULL }
assert_equal [ count_log_message 0 " c r a s h e d b y s i g n a l " ] 0
assert_equal [ count_log_message 0 " A S S E R T I O N F A I L E D " ] 1
}
}
test { corrupt payload: fuzzer findings - stream with bad lpFirst} {
start_server [ list overrides [ list loglevel verbose use-exit-on-panic yes crash-memcheck-enabled no] ] {
r config set sanitize-dump-payload yes
r debug set-skip-checksum-validation 1
catch { r restore _stream 0 " \x 0 F \x 0 1 \x 1 0 \x 0 0 \x 0 0 \x 0 1 \x 7 B \x 0 E \x 5 2 \x D 2 \x E C \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 4 0 \x 4 2 \x 4 2 \x 0 0 \x 0 0 \x 0 0 \x 1 8 \x 0 0 \x 0 2 \x F 7 \x 0 1 \x 0 1 \x 0 2 \x 0 1 \x 8 4 \x 6 9 \x 7 4 \x 6 5 \x 6 D \x 0 5 \x 8 5 \x 7 6 \x 6 1 \x 6 C \x 7 5 \x 6 5 \x 0 6 \x 0 0 \x 0 1 \x 0 2 \x 0 1 \x 0 0 \x 0 1 \x 0 0 \x 0 1 \x 0 1 \x 0 1 \x 0 0 \x 0 1 \x 0 5 \x 0 1 \x 0 3 \x 0 1 \x 0 1 \x 0 1 \x 0 0 \x 0 1 \x 0 1 \x 0 1 \x 8 2 \x 5 F \x 3 1 \x 0 3 \x 0 5 \x 0 1 \x 0 2 \x 0 1 \x 0 1 \x 0 1 \x 0 1 \x 0 1 \x 0 1 \x 0 1 \x 0 2 \x 0 1 \x 0 5 \x 0 1 \x F F \x 0 2 \x 8 1 \x 0 0 \x 0 0 \x 0 1 \x 7 B \x 0 E \x 5 2 \x D 2 \x E D \x 0 1 \x 0 1 \x 0 7 \x 6 D \x 7 9 \x 6 7 \x 7 2 \x 6 F \x 7 5 \x 7 0 \x 8 1 \x 0 0 \x 0 0 \x 0 1 \x 7 B \x 0 E \x 5 2 \x D 2 \x E D \x 0 0 \x 0 1 \x 0 0 \x 0 0 \x 0 1 \x 7 B \x 0 E \x 5 2 \x D 2 \x E D \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x E D \x D 2 \x 5 2 \x 0 E \x 7 B \x 0 1 \x 0 0 \x 0 0 \x 0 1 \x 0 1 \x 0 5 \x 4 1 \x 6 C \x 6 9 \x 6 3 \x 6 5 \x E D \x D 2 \x 5 2 \x 0 E \x 7 B \x 0 1 \x 0 0 \x 0 0 \x 0 1 \x 0 0 \x 0 0 \x 0 1 \x 7 B \x 0 E \x 5 2 \x D 2 \x E D \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 9 \x 0 0 \x A C \x 0 5 \x C 9 \x 9 7 \x 5 D \x 4 5 \x 8 0 \x B 3 " } err
assert_match " * B a d d a t a f o r m a t * " $err
r ping
}
}
test { corrupt payload: fuzzer findings - stream listpack lpPrev valgrind issue} {
start_server [ list overrides [ list loglevel verbose use-exit-on-panic yes crash-memcheck-enabled no] ] {
r config set sanitize-dump-payload no
r debug set-skip-checksum-validation 1
r restore _stream 0 " \x 0 F \x 0 1 \x 1 0 \x 0 0 \x 0 0 \x 0 1 \x 7 B \x 0 E \x A E \x 6 6 \x 3 6 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 4 0 \x 4 2 \x 4 2 \x 0 0 \x 0 0 \x 0 0 \x 1 8 \x 0 0 \x 0 2 \x 0 1 \x 0 1 \x 0 1 \x 0 2 \x 0 1 \x 8 4 \x 6 9 \x 7 4 \x 6 5 \x 6 D \x 0 5 \x 8 5 \x 7 6 \x 6 1 \x 6 C \x 7 5 \x 6 5 \x 0 6 \x 0 0 \x 0 1 \x 0 2 \x 0 1 \x 0 0 \x 0 1 \x 0 0 \x 0 1 \x 0 1 \x 0 1 \x 0 0 \x 0 1 \x 1 D \x 0 1 \x 0 3 \x 0 1 \x 2 4 \x 0 1 \x 0 0 \x 0 1 \x 0 1 \x 6 9 \x 8 2 \x 5 F \x 3 1 \x 0 3 \x 0 5 \x 0 1 \x 0 2 \x 0 1 \x 3 3 \x 0 1 \x 0 0 \x 0 1 \x 0 1 \x 0 1 \x 0 2 \x 0 1 \x 0 5 \x 0 1 \x F F \x 0 2 \x 8 1 \x 0 0 \x 0 0 \x 0 1 \x 7 B \x 0 E \x A E \x 6 6 \x 6 9 \x 0 0 \x 0 1 \x 0 7 \x 6 D \x 7 9 \x 6 7 \x 7 2 \x 6 F \x 7 5 \x 7 0 \x 8 1 \x 0 0 \x 0 0 \x 0 1 \x 7 B \x 0 E \x A E \x 6 6 \x 5 A \x 0 0 \x 0 1 \x 0 0 \x 0 0 \x 0 1 \x 7 B \x 0 E \x A E \x 6 6 \x 5 A \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 9 4 \x 6 6 \x A E \x 0 E \x 7 B \x 0 1 \x 0 0 \x 0 0 \x 0 1 \x 0 1 \x 0 5 \x 4 1 \x 6 C \x 6 9 \x 6 3 \x 6 5 \x 8 3 \x 6 6 \x A E \x 0 E \x 7 B \x 0 1 \x 0 0 \x 0 0 \x 0 1 \x 0 0 \x 0 0 \x 0 1 \x 7 B \x 0 E \x A E \x 6 6 \x 5 A \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 9 \x 0 0 \x D 5 \x D 7 \x A 5 \x 5 C \x 6 3 \x 1 C \x 0 9 \x 4 0 "
catch { r XREVRANGE _stream 1618622681 606195012389 }
assert_equal [ count_log_message 0 " c r a s h e d b y s i g n a l " ] 0
assert_equal [ count_log_message 0 " A S S E R T I O N F A I L E D " ] 1
}
}
test { corrupt payload: fuzzer findings - stream with non-integer entry id} {
start_server [ list overrides [ list loglevel verbose use-exit-on-panic yes crash-memcheck-enabled no] ] {
r config set sanitize-dump-payload yes
r debug set-skip-checksum-validation 1
catch { r restore _streambig 0 " \x 0 F \x 0 3 \x 1 0 \x 0 0 \x 0 0 \x 0 1 \x 7 B \x 1 3 \x 3 4 \x C 3 \x B 2 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x C 3 \x 4 0 \x 4 F \x 4 0 \x 5 C \x 1 8 \x 5 C \x 0 0 \x 0 0 \x 0 0 \x 2 4 \x 0 0 \x 0 5 \x 0 1 \x 0 0 \x 0 1 \x 0 2 \x 0 1 \x 8 4 \x 6 9 \x 7 4 \x 6 5 \x 6 D \x 0 5 \x 8 5 \x 7 6 \x 6 1 \x 6 C \x 7 5 \x 6 5 \x 0 6 \x 4 0 \x 1 0 \x 0 0 \x 8 0 \x 2 0 \x 0 1 \x 0 0 \x 0 1 \x 2 0 \x 0 3 \x 0 0 \x 0 5 \x 2 0 \x 1 C \x 4 0 \x 0 9 \x 0 5 \x 0 1 \x 0 1 \x 8 2 \x 5 F \x 3 1 \x 0 3 \x 8 0 \x 0 D \x 0 0 \x 0 2 \x 2 0 \x 0 D \x 0 0 \x 0 2 \x A 0 \x 1 9 \x 0 0 \x 0 3 \x 2 0 \x 0 B \x 0 2 \x 8 2 \x 5 F \x 3 3 \x A 0 \x 1 9 \x 0 0 \x 0 4 \x 2 0 \x 0 D \x 0 0 \x 0 4 \x 2 0 \x 1 9 \x 0 0 \x F F \x 1 0 \x 0 0 \x 0 0 \x 0 1 \x 7 B \x 1 3 \x 3 4 \x C 3 \x B 2 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 5 \x C 3 \x 4 0 \x 5 6 \x 4 0 \x 6 1 \x 1 8 \x 6 1 \x 0 0 \x 0 0 \x 0 0 \x 2 4 \x 0 0 \x 0 5 \x 0 1 \x 0 0 \x 0 1 \x 0 2 \x 0 1 \x 8 4 \x 6 9 \x 7 4 \x 6 5 \x 6 D \x 0 5 \x 8 5 \x 7 6 \x 6 1 \x 6 C \x 7 5 \x 6 5 \x 0 6 \x 4 0 \x 1 0 \x 0 0 \x 0 0 \x 2 0 \x 0 1 \x 0 6 \x 0 1 \x 0 1 \x 8 2 \x 5 F \x 3 5 \x 0 3 \x 0 5 \x 2 0 \x 1 E \x 4 0 \x 0 B \x 0 3 \x 0 1 \x 0 1 \x 0 6 \x 0 1 \x 4 0 \x 0 B \x 0 3 \x 0 1 \x 0 1 \x D F \x F B \x 2 0 \x 0 5 \x 0 2 \x 8 2 \x 5 F \x 3 7 \x 6 0 \x 1 A \x 2 0 \x 0 E \x 0 0 \x F C \x 2 0 \x 0 5 \x 0 0 \x 0 8 \x C 0 \x 1 B \x 0 0 \x F D \x 2 0 \x 0 C \x 0 2 \x 8 2 \x 5 F \x 3 9 \x 2 0 \x 1 B \x 0 0 \x F F \x 1 0 \x 0 0 \x 0 0 \x 0 1 \x 7 B \x 1 3 \x 3 4 \x C 3 \x B 3 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 3 \x C 3 \x 3 D \x 4 0 \x 4 A \x 1 8 \x 4 A \x 0 0 \x 0 0 \x 0 0 \x 1 5 \x 0 0 \x 0 2 \x 0 1 \x 0 0 \x 0 1 \x 0 2 \x 0 1 \x 8 4 \x 6 9 \x 7 4 \x 6 5 \x 6 D \x 0 5 \x 8 5 \x 7 6 \x 6 1 \x 6 C \x 7 5 \x 6 5 \x 0 6 \x 4 0 \x 1 0 \x 0 0 \x 0 0 \x 2 0 \x 0 1 \x 4 0 \x 0 0 \x 0 0 \x 0 5 \x 6 0 \x 0 7 \x 0 2 \x D F \x F D \x 0 2 \x C 0 \x 2 3 \x 0 9 \x 0 1 \x 0 1 \x 8 6 \x 7 5 \x 6 E \x 6 9 \x 7 1 \x 7 5 \x 6 5 \x 0 7 \x A 0 \x 2 D \x 0 2 \x 0 8 \x 0 1 \x F F \x 0 C \x 8 1 \x 0 0 \x 0 0 \x 0 1 \x 7 B \x 1 3 \x 3 4 \x C 3 \x B 4 \x 0 0 \x 0 0 \x 0 9 \x 0 0 \x 9 D \x B D \x D 5 \x B 9 \x 3 3 \x C 4 \x C 5 \x F F " } err
assert_match " * B a d d a t a f o r m a t * " $err
r ping
}
}
2021-08-05 15:42:20 -04:00
test { corrupt payload: fuzzer findings - empty quicklist} {
start_server [ list overrides [ list loglevel verbose use-exit-on-panic yes crash-memcheck-enabled no] ] {
r config set sanitize-dump-payload yes
r debug set-skip-checksum-validation 1
catch {
r restore key 0 " \x 0 E \x C 0 \x 2 B \x 1 5 \x 0 0 \x 0 0 \x 0 0 \x 0 A \x 0 0 \x 0 0 \x 0 0 \x 0 1 \x 0 0 \x 0 0 \x E 0 \x 6 2 \x 5 8 \x E A \x D F \x 2 2 \x 0 0 \x 0 0 \x 0 0 \x F F \x 0 9 \x 0 0 \x D F \x 3 5 \x D 2 \x 6 7 \x D C \x 0 E \x 8 9 \x A B " replace
} err
assert_match " * B a d d a t a f o r m a t * " $err
r ping
}
}
test { corrupt payload: fuzzer findings - empty zset} {
start_server [ list overrides [ list loglevel verbose use-exit-on-panic yes crash-memcheck-enabled no] ] {
r config set sanitize-dump-payload yes
r debug set-skip-checksum-validation 1
catch { r restore key 0 " \x 0 5 \x C 0 \x 0 1 \x 0 9 \x 0 0 \x F 6 \x 8 A \x B 6 \x 7 A \x 8 5 \x 8 7 \x 7 2 \x 4 D " } err
assert_match " * B a d d a t a f o r m a t * " $err
r ping
}
}
test { corrupt payload: fuzzer findings - hash with len of 0 } {
start_server [ list overrides [ list loglevel verbose use-exit-on-panic yes crash-memcheck-enabled no] ] {
r config set sanitize-dump-payload yes
r debug set-skip-checksum-validation 1
catch { r restore key 0 " \x 0 4 \x C 0 \x 2 1 \x 0 9 \x 0 0 \x F 6 \x 8 A \x B 6 \x 7 A \x 8 5 \x 8 7 \x 7 2 \x 4 D " } err
assert_match " * B a d d a t a f o r m a t * " $err
r ping
}
}
2021-08-10 02:18:49 -04:00
test { corrupt payload: fuzzer findings - hash listpack first element too long entry len} {
start_server [ list overrides [ list loglevel verbose use-exit-on-panic yes crash-memcheck-enabled no] ] {
r debug set-skip-checksum-validation 1
r config set sanitize-dump-payload yes
catch { r restore _hash 0 " \x 1 0 \x 1 5 \x 1 5 \x 0 0 \x 0 0 \x 0 0 \x 0 6 \x 0 0 \x F 0 \x 0 1 \x 0 0 \x 0 1 \x 0 1 \x 0 1 \x 8 2 \x 5 F \x 3 1 \x 0 3 \x 0 2 \x 0 1 \x 0 2 \x 0 1 \x F F \x 0 A \x 0 0 \x 9 4 \x 2 1 \x 0 A \x F A \x 0 6 \x 5 2 \x 9 F \x 4 4 " replace } err
assert_match " * B a d d a t a f o r m a t * " $err
verify_log_message 0 " * i n t e g r i t y c h e c k f a i l e d * " 0
}
}
2021-08-20 03:37:45 -04:00
test { corrupt payload: fuzzer findings - stream double free listpack when insert dup node to rax returns 0 } {
start_server [ list overrides [ list loglevel verbose use-exit-on-panic yes crash-memcheck-enabled no] ] {
r debug set-skip-checksum-validation 1
r config set sanitize-dump-payload yes
catch { r restore _stream 0 " \x 0 F \x 0 3 \x 1 0 \x 0 0 \x 0 0 \x 0 1 \x 7 B \x 6 0 \x 5 A \x 2 3 \x 7 9 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x C 3 \x 4 0 \x 4 F \x 4 0 \x 5 C \x 1 8 \x 5 C \x 0 0 \x 0 0 \x 0 0 \x 2 4 \x 0 0 \x 0 5 \x 0 1 \x 0 0 \x 0 1 \x 0 2 \x 0 1 \x 8 4 \x 6 9 \x 7 4 \x 6 5 \x 6 D \x 0 5 \x 8 5 \x 7 6 \x 6 1 \x 6 C \x 7 5 \x 6 5 \x 0 6 \x 4 0 \x 1 0 \x 0 0 \x 0 0 \x 2 0 \x 0 1 \x 0 0 \x 0 1 \x 2 0 \x 0 3 \x 0 0 \x 0 5 \x 2 0 \x 1 C \x 4 0 \x 0 9 \x 0 5 \x 0 1 \x 0 1 \x 8 2 \x 5 F \x 3 1 \x 0 3 \x 8 0 \x 0 D \x 0 0 \x 0 2 \x 2 0 \x 0 D \x 0 0 \x 0 2 \x A 0 \x 1 9 \x 0 0 \x 0 3 \x 2 0 \x 0 B \x 0 2 \x 8 2 \x 5 F \x 3 3 \x A 0 \x 1 9 \x 0 0 \x 0 4 \x 2 0 \x 0 D \x 0 0 \x 0 4 \x 2 0 \x 1 9 \x 0 0 \x F F \x 1 0 \x 0 0 \x 0 0 \x 0 1 \x 7 B \x 6 0 \x 5 A \x 2 3 \x 7 9 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 5 \x C 3 \x 4 0 \x 5 1 \x 4 0 \x 5 E \x 1 8 \x 5 E \x 0 0 \x 0 0 \x 0 0 \x 2 4 \x 0 0 \x 0 5 \x 0 1 \x 0 0 \x 0 1 \x 0 2 \x 0 1 \x 8 4 \x 6 9 \x 7 4 \x 6 5 \x 6 D \x 0 5 \x 8 5 \x 7 6 \x 6 1 \x 6 C \x 7 5 \x 6 5 \x 0 6 \x 4 0 \x 1 0 \x 0 0 \x 0 0 \x 2 0 \x 0 1 \x 0 6 \x 0 1 \x 0 1 \x 8 2 \x 5 F \x 3 5 \x 0 3 \x 0 5 \x 2 0 \x 1 E \x 4 0 \x 0 B \x 0 3 \x 0 1 \x 0 1 \x 0 6 \x 0 1 \x 8 0 \x 0 B \x 0 0 \x 0 2 \x 2 0 \x 0 B \x 0 2 \x 8 2 \x 5 F \x 3 7 \x A 0 \x 1 9 \x 0 0 \x 0 3 \x 2 0 \x 0 D \x 0 0 \x 0 8 \x A 0 \x 1 9 \x 0 0 \x 0 4 \x 2 0 \x 0 B \x 0 2 \x 8 2 \x 5 F \x 3 9 \x 2 0 \x 1 9 \x 0 0 \x F F \x 1 0 \x 0 0 \x 0 0 \x 0 1 \x 7 B \x 6 0 \x 5 A \x 2 3 \x 7 9 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x C 3 \x 3 B \x 4 0 \x 4 9 \x 1 8 \x 4 9 \x 0 0 \x 0 0 \x 0 0 \x 1 5 \x 0 0 \x 0 2 \x 0 1 \x 0 0 \x 0 1 \x 0 2 \x 0 1 \x 8 4 \x 6 9 \x 7 4 \x 6 5 \x 6 D \x 0 5 \x 8 5 \x 7 6 \x 6 1 \x 6 C \x 7 5 \x 6 5 \x 0 6 \x 4 0 \x 1 0 \x 0 0 \x 0 0 \x 2 0 \x 0 1 \x 4 0 \x 0 0 \x 0 0 \x 0 5 \x 2 0 \x 0 7 \x 4 0 \x 0 9 \x C 0 \x 2 2 \x 0 9 \x 0 1 \x 0 1 \x 8 6 \x 7 5 \x 6 E \x 6 9 \x 7 1 \x 7 5 \x 6 5 \x 0 7 \x A 0 \x 2 C \x 0 2 \x 0 8 \x 0 1 \x F F \x 0 C \x 8 1 \x 0 0 \x 0 0 \x 0 1 \x 7 B \x 6 0 \x 5 A \x 2 3 \x 7 A \x 0 1 \x 0 0 \x 0 A \x 0 0 \x 9 C \x 8 F \x 1 E \x B F \x 2 E \x 0 5 \x 5 9 \x 0 9 " replace } err
assert_match " * B a d d a t a f o r m a t * " $err
r ping
}
}
2021-11-21 07:30:20 -05:00
test { corrupt payload: fuzzer findings - LCS OOM} {
start_server [ list overrides [ list loglevel verbose use-exit-on-panic yes crash-memcheck-enabled no] ] {
r SETRANGE _int 423324 1450173551
catch { r LCS _int _int} err
assert_match " * I n s u f f i c i e n t m e m o r y * " $err
r ping
}
}
2021-11-21 11:47:10 -05:00
test { corrupt payload: fuzzer findings - gcc asan reports false leak on assert} {
start_server [ list overrides [ list loglevel verbose use-exit-on-panic yes crash-memcheck-enabled no] ] {
r debug set-skip-checksum-validation 1
r config set sanitize-dump-payload no
catch { r restore _list 0 " \x 1 2 \x 0 1 \x 0 2 \x 1 3 \x 1 3 \x 0 0 \x 0 0 \x 0 0 \x 1 0 \x 0 0 \x 0 0 \x 0 0 \x 0 3 \x 0 0 \x 0 0 \x F 3 \x F E \x 0 2 \x 5 F \x 3 1 \x 0 4 \x F 1 \x F F \x 0 A \x 0 0 \x 1 9 \x 8 D \x 3 D \x 7 4 \x 8 5 \x 9 4 \x 2 9 \x B D " }
catch { r LPOP _list } err
assert_equal [ count_log_message 0 " c r a s h e d b y s i g n a l " ] 0
assert_equal [ count_log_message 0 " A S S E R T I O N F A I L E D " ] 1
}
}
2021-12-26 04:40:11 -05:00
test { corrupt payload: fuzzer findings - lpFind invalid access} {
2021-11-22 08:30:00 -05:00
start_server [ list overrides [ list loglevel verbose use-exit-on-panic yes crash-memcheck-enabled no] ] {
r debug set-skip-checksum-validation 1
r config set sanitize-dump-payload no
r restore _hashbig 0 " \x 1 0 \x 3 9 \x 3 9 \x 0 0 \x 0 0 \x 0 0 \x 1 4 \x 0 0 \x 0 6 \x 0 1 \x 0 6 \x 0 1 \x 0 3 \x 0 1 \x 8 2 \x 5 F \x 3 3 \x 0 3 \x 0 7 \x 0 1 \x 8 2 \x 5 F \x 3 7 \x 0 3 \x 0 0 \x 0 1 \x 0 0 \x 0 1 \x 0 4 \x 0 1 \x 0 4 \x 0 1 \x 0 9 \x 0 1 \x 8 2 \x 5 F \x 3 9 \x 0 3 \x 0 5 \x 0 1 \x 8 2 \x 5 F \x 3 5 \x 0 3 \x 0 8 \x 0 1 \x 0 8 \x 0 1 \x 0 1 \x 0 1 \x 8 2 \x 5 F \x 3 1 \x 0 3 \x 0 2 \x 0 1 \x F 0 \x 0 1 \x F F \x 0 A \x 0 0 \x 2 9 \x D 7 \x E 4 \x 5 2 \x 7 9 \x 7 A \x 9 5 \x 8 2 "
catch { r HLEN _hashbig }
catch { r HSETNX _hashbig 513072881620 " \x 9 A \x 4 B \x 1 F \x F 2 \x 9 9 \x 7 4 \x 6 E \x 9 6 \x 8 4 \x 7 F \x B 9 \x 8 5 \x B E \x D 6 \x 1 A \x 9 3 \x 0 A \x E D \x A E \x 1 9 \x A 0 \x 5 A \x 6 7 \x D 6 \x 8 9 \x A 8 \x F 9 \x F 2 \x B 8 \x B D \x 3 E \x 5 A \x C F \x D 2 \x 5 B \x 1 7 \x A 4 \x B B \x B 2 \x A 9 \x 5 6 \x 6 7 \x 6 E \x 0 B \x E D \x C D \x 3 6 \x 4 9 \x C 6 \x 8 4 \x F F \x C 2 \x 7 6 \x 9 B \x F 3 \x 4 9 \x 8 8 \x 9 7 \x 9 2 \x D 2 \x 5 4 \x E 9 \x 0 8 \x 1 9 \x 8 6 \x 4 0 \x 9 6 \x 2 4 \x 6 8 \x 2 5 \x 9 D \x F 7 \x 0 E \x B 7 \x 3 6 \x 8 5 \x 6 8 \x 6 B \x 2 A \x 9 7 \x 6 4 \x 3 0 \x E 6 \x F F \x 9 A \x 2 A \x 4 2 \x 2 B \x 3 1 \x 0 1 \x 3 2 \x B 3 \x E E \x 7 8 \x 1 A \x 2 6 \x 9 4 \x E 2 \x 0 7 \x 3 4 \x 5 0 \x 8 A \x F F \x F 9 \x A E \x E A \x E C \x 5 9 \x 4 2 \x F 5 \x 3 9 \x 4 0 \x 6 5 \x D E \x 5 5 \x C C \x 7 7 \x 1 B \x 3 2 \x 0 2 \x 1 9 \x E E \x 3 C \x D 4 \x 7 9 \x 4 8 \x 0 1 \x 4 F \x 5 1 \x F E \x 2 2 \x E 0 \x 0 C \x F 4 \x 0 7 \x 0 6 \x C D \x 5 5 \x 3 0 \x C 0 \x 2 4 \x 3 2 \x D 4 \x C C \x A F \x 8 2 \x 0 5 \x 4 8 \x 1 4 \x 1 0 \x 5 5 \x A 1 \x 3 D \x F 6 \x 8 1 \x 4 5 \x 5 4 \x E A \x 7 1 \x 2 4 \x 2 7 \x 0 6 \x D C \x F A \x E 4 \x E 4 \x 8 7 \x C C \x 8 1 \x A 0 \x 4 7 \x A 5 \x A F \x D 1 \x 8 9 \x E 7 \x 4 2 \x C 3 \x 2 4 \x D 0 \x 3 2 \x 7 A \x D E \x 4 4 \x 4 7 \x 6 E \x 1 F \x C B \x E E \x A 6 \x 4 6 \x D E \x 0 D \x E 6 \x D 5 \x 1 6 \x 0 3 \x 2 A \x D 6 \x 9 E \x F D \x 9 4 \x 0 2 \x 2 C \x D B \x 1 F \x D 0 \x B E \x 9 8 \x 1 0 \x E 3 \x E B \x E A \x B E \x E 5 \x D 1 " }
}
}
2021-11-23 07:56:52 -05:00
test { corrupt payload: fuzzer findings - invalid access in ziplist tail prevlen decoding} {
start_server [ list overrides [ list loglevel verbose use-exit-on-panic yes crash-memcheck-enabled no] ] {
r debug set-skip-checksum-validation 1
r config set sanitize-dump-payload no
2021-11-24 06:34:13 -05:00
catch { r restore _listbig 0 " \x 0 e \x 0 2 \x 1 B \x 1 B \x 0 0 \x 0 0 \x 0 0 \x 1 6 \x 0 0 \x 0 0 \x 0 0 \x 0 5 \x 0 0 \x 0 0 \x 0 2 \x 5 F \x 3 9 \x 0 4 \x F 9 \x 0 2 \x 0 2 \x 5 F \x 3 7 \x 0 4 \x F 7 \x 0 2 \x 0 2 \x 5 F \x 3 5 \x F F \x 1 9 \x 1 9 \x 0 0 \x 0 0 \x 0 0 \x 1 6 \x 0 0 \x 0 0 \x 0 0 \x 0 5 \x 0 0 \x 0 0 \x F 5 \x 0 2 \x 0 2 \x 5 F \x 3 3 \x 0 4 \x F 3 \x 0 2 \x 0 2 \x 5 F \x 3 1 \x F E \x F 1 \x F F \x 0 A \x 0 0 \x 6 B \x 4 3 \x 3 2 \x 2 F \x B B \x 2 9 \x 0 a \x B E " } err
assert_match " * B a d d a t a f o r m a t * " $err
r ping
2021-11-23 07:56:52 -05:00
}
}
2021-12-26 04:40:11 -05:00
test { corrupt payload: fuzzer findings - zset zslInsert with a NAN score} {
start_server [ list overrides [ list loglevel verbose use-exit-on-panic yes crash-memcheck-enabled no] ] {
r config set sanitize-dump-payload no
r debug set-skip-checksum-validation 1
catch { r restore _nan_zset 0 " \x 0 5 \x 0 A \x 0 2 \x 5 F \x 3 9 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 2 2 \x 4 0 \x C 0 \x 0 8 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 2 0 \x 4 0 \x 0 2 \x 5 F \x 3 7 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 1 C \x 4 0 \x C 0 \x 0 6 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 1 8 \x 4 0 \x 0 2 \x 5 F \x 3 5 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 1 4 \x 4 0 \x C 0 \x 0 4 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 1 0 \x 4 0 \x 0 2 \x 5 F \x 3 3 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 8 \x 4 0 \x C 0 \x 0 2 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 4 0 \x 0 2 \x 5 F \x 3 1 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 5 5 \x F 0 \x 7 F \x C 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 A \x 0 0 \x E C \x 9 4 \x 8 6 \x D 8 \x F D \x 5 C \x 5 F \x D 8 " } err
assert_match " * B a d d a t a f o r m a t * " $err
r ping
}
}
2022-04-14 01:29:35 -04:00
test { corrupt payload: fuzzer findings - streamLastValidID panic} {
start_server [ list overrides [ list loglevel verbose use-exit-on-panic yes crash-memcheck-enabled no] ] {
r config set sanitize-dump-payload yes
r debug set-skip-checksum-validation 1
catch { r restore _streambig 0 " \x 1 3 \x C 0 \x 1 0 \x 0 0 \x 0 0 \x 0 1 \x 8 0 \x 2 0 \x 4 8 \x A 0 \x 3 3 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x C 3 \x 4 0 \x 4 F \x 4 0 \x 5 C \x 1 8 \x 5 C \x 0 0 \x 0 0 \x 0 0 \x 2 4 \x 0 0 \x 0 5 \x 0 1 \x 0 0 \x 0 1 \x 0 2 \x 0 1 \x 8 4 \x 6 9 \x 7 4 \x 6 5 \x 6 D \x 0 5 \x 8 5 \x 7 6 \x 6 1 \x 6 C \x 7 5 \x 6 5 \x 0 6 \x 4 0 \x 1 0 \x 0 0 \x 0 0 \x 2 0 \x 0 1 \x 0 0 \x 0 1 \x 2 0 \x 0 3 \x 0 0 \x 0 5 \x 2 0 \x 1 C \x 4 0 \x 0 9 \x 0 5 \x 0 1 \x 0 1 \x 8 2 \x 5 F \x 3 1 \x 0 3 \x 8 0 \x 0 D \x 0 0 \x 0 2 \x 2 0 \x 0 D \x 0 0 \x 0 2 \x A 0 \x 1 9 \x 0 0 \x 0 3 \x 2 0 \x 0 B \x 0 2 \x 8 2 \x 5 F \x 3 3 \x 6 0 \x 1 9 \x 4 0 \x 2 F \x 0 2 \x 0 1 \x 0 1 \x 0 4 \x 2 0 \x 1 9 \x 0 0 \x F F \x 1 0 \x 0 0 \x 0 0 \x 0 1 \x 8 0 \x 2 0 \x 4 8 \x A 0 \x 3 4 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 1 \x C 3 \x 4 0 \x 5 1 \x 4 0 \x 5 E \x 1 8 \x 5 E \x 0 0 \x 0 0 \x 0 0 \x 2 4 \x 0 0 \x 0 5 \x 0 1 \x 0 0 \x 0 1 \x 0 2 \x 0 1 \x 8 4 \x 6 9 \x 7 4 \x 6 5 \x 6 D \x 0 5 \x 8 5 \x 7 6 \x 6 1 \x 6 C \x 7 5 \x 6 5 \x 0 6 \x 4 0 \x 1 0 \x 0 0 \x 0 0 \x 2 0 \x 0 1 \x 0 6 \x 0 1 \x 0 1 \x 8 2 \x 5 F \x 3 5 \x 0 3 \x 0 5 \x 2 0 \x 1 E \x 4 0 \x 0 B \x 0 3 \x 0 1 \x 0 1 \x 0 6 \x 0 1 \x 8 0 \x 0 B \x 0 0 \x 0 2 \x 2 0 \x 0 B \x 0 2 \x 8 2 \x 5 F \x 3 7 \x A 0 \x 1 9 \x 0 0 \x 0 3 \x 2 0 \x 0 D \x 0 0 \x 0 8 \x A 0 \x 1 9 \x 0 0 \x 0 4 \x 2 0 \x 0 B \x 0 2 \x 8 2 \x 5 F \x 3 9 \x 2 0 \x 1 9 \x 0 0 \x F F \x 1 0 \x 0 0 \x 0 0 \x 0 1 \x 8 0 \x 2 0 \x 4 8 \x A 0 \x 3 4 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 6 \x C 3 \x 3 D \x 4 0 \x 4 A \x 1 8 \x 4 A \x 0 0 \x 0 0 \x 0 0 \x 1 5 \x 0 0 \x 0 2 \x 0 1 \x 0 0 \x 0 1 \x 0 2 \x 0 1 \x 8 4 \x 6 9 \x 7 4 \x 6 5 \x 6 D \x 0 5 \x 8 5 \x 7 6 \x 6 1 \x 6 C \x 7 5 \x 6 5 \x 0 6 \x 4 0 \x 1 0 \x 0 0 \x 0 0 \x 2 0 \x 0 1 \x 4 0 \x 0 0 \x 0 0 \x 0 5 \x 6 0 \x 0 7 \x 0 2 \x D F \x F A \x 0 2 \x C 0 \x 2 3 \x 0 9 \x 0 1 \x 0 1 \x 8 6 \x 7 5 \x 6 E \x 6 9 \x 7 1 \x 7 5 \x 6 5 \x 0 7 \x A 0 \x 2 D \x 0 2 \x 0 8 \x 0 1 \x F F \x 0 C \x 8 1 \x 0 0 \x 0 0 \x 0 1 \x 8 0 \x 2 0 \x 4 8 \x A 0 \x 3 5 \x 0 0 \x 8 1 \x 0 0 \x 0 0 \x 0 1 \x 8 0 \x 2 0 \x 4 8 \x A 0 \x 3 3 \x 0 0 \x 0 0 \x 0 0 \x 0 C \x 0 0 \x 0 A \x 0 0 \x 3 4 \x 8 B \x 0 E \x 5 B \x 4 2 \x C D \x D 6 \x 0 8 " } err
assert_match " * B a d d a t a f o r m a t * " $err
r ping
}
}
2022-07-13 02:14:38 -04:00
test { corrupt payload: fuzzer findings - valgrind fishy value warning} {
start_server [ list overrides [ list loglevel verbose use-exit-on-panic yes crash-memcheck-enabled no] ] {
r config set sanitize-dump-payload yes
r debug set-skip-checksum-validation 1
catch { r restore _key 0 " \x 1 3 \x 0 1 \x 1 0 \x 0 0 \x 0 0 \x 0 1 \x 8 1 \x C C \x 0 7 \x D C \x F 2 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 4 0 \x 4 2 \x 4 2 \x 0 0 \x 0 0 \x 0 0 \x 1 8 \x 0 0 \x 0 2 \x 0 1 \x 0 1 \x 0 1 \x 0 2 \x 0 1 \x 8 4 \x 6 9 \x 7 4 \x 6 5 \x 6 D \x 0 5 \x 8 5 \x 7 6 \x 6 1 \x 6 C \x 7 5 \x 6 5 \x 0 6 \x 0 0 \x 0 1 \x 0 2 \x 0 1 \x 0 0 \x 0 1 \x 0 0 \x 0 1 \x 0 1 \x 0 1 \x 0 0 \x 0 1 \x 0 5 \x 0 1 \x 0 3 \x 0 1 \x 2 C \x 0 1 \x 0 0 \x 0 1 \x 0 1 \x 0 1 \x 8 2 \x 5 F \x 3 1 \x 0 3 \x 0 5 \x 0 1 \x 0 2 \x 0 1 \x 3 C \x 0 1 \x 0 0 \x 0 1 \x 0 1 \x 0 1 \x 0 2 \x 0 1 \x 0 5 \x 0 1 \x F F \x 0 2 \x D 0 \x 0 0 \x 0 0 \x 0 1 \x 8 1 \x C C \x 0 7 \x D D \x 2 E \x 0 0 \x 8 1 \x 0 0 \x 0 0 \x 0 1 \x 8 1 \x C C \x 0 7 \x D C \x F 2 \x 0 0 \x 8 1 \x 0 0 \x 0 0 \x 0 1 \x 8 1 \x C C \x 0 7 \x D D \x 1 E \x 0 0 \x 0 3 \x 0 1 \x 0 7 \x 6 D \x 7 9 \x 6 7 \x 7 2 \x 6 F \x 7 5 \x 7 0 \x 8 1 \x 0 0 \x 0 0 \x 0 1 \x 8 1 \x C C \x 0 7 \x D D \x 1 E \x 0 0 \x 0 2 \x 0 1 \x 0 0 \x 0 0 \x 0 1 \x 8 1 \x C C \x 0 7 \x D D \x 1 E \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 7 1 \x D D \x 0 7 \x C C \x 8 1 \x 0 1 \x 0 0 \x 0 0 \x 0 1 \x 0 1 \x 0 5 \x 4 1 \x 6 C \x 6 9 \x 6 3 \x 6 5 \x 5 8 \x D D \x 0 7 \x C C \x 8 1 \x 0 1 \x 0 0 \x 0 0 \x 0 1 \x 0 0 \x 0 0 \x 0 1 \x 8 1 \x C C \x 0 7 \x D D \x 1 E \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 A \x 0 0 \x 2 F \x B 0 \x D 1 \x 1 5 \x 0 A \x 9 7 \x 8 7 \x 6 B " } err
assert_match " * B a d d a t a f o r m a t * " $err
r ping
}
}
2022-11-20 05:12:15 -05:00
test { corrupt payload: fuzzer findings - empty set listpack} {
start_server [ list overrides [ list loglevel verbose use-exit-on-panic yes crash-memcheck-enabled no] ] {
r config set sanitize-dump-payload no
r debug set-skip-checksum-validation 1
catch { r restore _key 0 " \x 1 4 \x 2 5 \x 2 5 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 0 \x 0 2 \x 0 1 \x 8 2 \x 5 F \x 3 7 \x 0 3 \x 0 6 \x 0 1 \x 8 2 \x 5 F \x 3 5 \x 0 3 \x 8 2 \x 5 F \x 3 3 \x 0 3 \x 0 0 \x 0 1 \x 8 2 \x 5 F \x 3 1 \x 0 3 \x 8 2 \x 5 F \x 3 9 \x 0 3 \x 0 4 \x A 9 \x 0 8 \x 0 1 \x F F \x 0 B \x 0 0 \x A 3 \x 2 6 \x 4 9 \x B 4 \x 8 6 \x B 0 \x 0 F \x 4 1 " } err
assert_match " * B a d d a t a f o r m a t * " $err
r ping
}
}
2022-07-13 02:14:38 -04:00
2022-11-22 04:20:24 -05:00
test { corrupt payload: fuzzer findings - set with duplicate elements causes sdiff to hang} {
start_server [ list overrides [ list loglevel verbose use-exit-on-panic yes crash-memcheck-enabled no] ] {
r config set sanitize-dump-payload yes
r debug set-skip-checksum-validation 1
catch { r restore _key 0 " \x 1 4 \x 2 5 \x 2 5 \x 0 0 \x 0 0 \x 0 0 \x 0 A \x 0 0 \x 0 6 \x 0 1 \x 8 2 \x 5 F \x 3 5 \x 0 3 \x 0 4 \x 0 1 \x 8 2 \x 5 F \x 3 1 \x 0 3 \x 8 2 \x 5 F \x 3 3 \x 0 3 \x 0 0 \x 0 1 \x 8 2 \x 5 F \x 3 9 \x 0 3 \x 8 2 \x 5 F \x 3 3 \x 0 3 \x 0 8 \x 0 1 \x 0 2 \x 0 1 \x F F \x 0 B \x 0 0 \x 3 1 \x B E \x 7 D \x 4 1 \x 0 1 \x 0 3 \x 5 B \x E C " replace} err
assert_match " * B a d d a t a f o r m a t * " $err
r ping
# In the past, it generated a broken protocol and left the client hung in sdiff
r config set sanitize-dump-payload no
assert_equal { OK } [ r restore _key 0 " \x 1 4 \x 2 5 \x 2 5 \x 0 0 \x 0 0 \x 0 0 \x 0 A \x 0 0 \x 0 6 \x 0 1 \x 8 2 \x 5 F \x 3 5 \x 0 3 \x 0 4 \x 0 1 \x 8 2 \x 5 F \x 3 1 \x 0 3 \x 8 2 \x 5 F \x 3 3 \x 0 3 \x 0 0 \x 0 1 \x 8 2 \x 5 F \x 3 9 \x 0 3 \x 8 2 \x 5 F \x 3 3 \x 0 3 \x 0 8 \x 0 1 \x 0 2 \x 0 1 \x F F \x 0 B \x 0 0 \x 3 1 \x B E \x 7 D \x 4 1 \x 0 1 \x 0 3 \x 5 B \x E C " replace]
assert_type set _key
assert_encoding listpack _key
assert_equal 10 [ r scard _key]
assert_equal { 0 2 4 6 8 _1 _3 _3 _5 _9} [ lsort [ r smembers _key] ]
assert_equal { 0 2 4 6 8 _1 _3 _5 _9} [ lsort [ r sdiff _key] ]
}
Add reply_schema to command json files (internal for now) (#10273)
Work in progress towards implementing a reply schema as part of COMMAND DOCS, see #9845
Since ironing the details of the reply schema of each and every command can take a long time, we
would like to merge this PR when the infrastructure is ready, and let this mature in the unstable branch.
Meanwhile the changes of this PR are internal, they are part of the repo, but do not affect the produced build.
### Background
In #9656 we add a lot of information about Redis commands, but we are missing information about the replies
### Motivation
1. Documentation. This is the primary goal.
2. It should be possible, based on the output of COMMAND, to be able to generate client code in typed
languages. In order to do that, we need Redis to tell us, in detail, what each reply looks like.
3. We would like to build a fuzzer that verifies the reply structure (for now we use the existing
testsuite, see the "Testing" section)
### Schema
The idea is to supply some sort of schema for the various replies of each command.
The schema will describe the conceptual structure of the reply (for generated clients), as defined in RESP3.
Note that the reply structure itself may change, depending on the arguments (e.g. `XINFO STREAM`, with
and without the `FULL` modifier)
We decided to use the standard json-schema (see https://json-schema.org/) as the reply-schema.
Example for `BZPOPMIN`:
```
"reply_schema": {
"oneOf": [
{
"description": "Timeout reached and no elements were popped.",
"type": "null"
},
{
"description": "The keyname, popped member, and its score.",
"type": "array",
"minItems": 3,
"maxItems": 3,
"items": [
{
"description": "Keyname",
"type": "string"
},
{
"description": "Member",
"type": "string"
},
{
"description": "Score",
"type": "number"
}
]
}
]
}
```
#### Notes
1. It is ok that some commands' reply structure depends on the arguments and it's the caller's responsibility
to know which is the relevant one. this comes after looking at other request-reply systems like OpenAPI,
where the reply schema can also be oneOf and the caller is responsible to know which schema is the relevant one.
2. The reply schemas will describe RESP3 replies only. even though RESP3 is structured, we want to use reply
schema for documentation (and possibly to create a fuzzer that validates the replies)
3. For documentation, the description field will include an explanation of the scenario in which the reply is sent,
including any relation to arguments. for example, for `ZRANGE`'s two schemas we will need to state that one
is with `WITHSCORES` and the other is without.
4. For documentation, there will be another optional field "notes" in which we will add a short description of
the representation in RESP2, in case it's not trivial (RESP3's `ZRANGE`'s nested array vs. RESP2's flat
array, for example)
Given the above:
1. We can generate the "return" section of all commands in [redis-doc](https://redis.io/commands/)
(given that "description" and "notes" are comprehensive enough)
2. We can generate a client in a strongly typed language (but the return type could be a conceptual
`union` and the caller needs to know which schema is relevant). see the section below for RESP2 support.
3. We can create a fuzzer for RESP3.
### Limitations (because we are using the standard json-schema)
The problem is that Redis' replies are more diverse than what the json format allows. This means that,
when we convert the reply to a json (in order to validate the schema against it), we lose information (see
the "Testing" section below).
The other option would have been to extend the standard json-schema (and json format) to include stuff
like sets, bulk-strings, error-string, etc. but that would mean also extending the schema-validator - and that
seemed like too much work, so we decided to compromise.
Examples:
1. We cannot tell the difference between an "array" and a "set"
2. We cannot tell the difference between simple-string and bulk-string
3. we cannot verify true uniqueness of items in commands like ZRANGE: json-schema doesn't cover the
case of two identical members with different scores (e.g. `[["m1",6],["m1",7]]`) because `uniqueItems`
compares (member,score) tuples and not just the member name.
### Testing
This commit includes some changes inside Redis in order to verify the schemas (existing and future ones)
are indeed correct (i.e. describe the actual response of Redis).
To do that, we added a debugging feature to Redis that causes it to produce a log of all the commands
it executed and their replies.
For that, Redis needs to be compiled with `-DLOG_REQ_RES` and run with
`--reg-res-logfile <file> --client-default-resp 3` (the testsuite already does that if you run it with
`--log-req-res --force-resp3`)
You should run the testsuite with the above args (and `--dont-clean`) in order to make Redis generate
`.reqres` files (same dir as the `stdout` files) which contain request-response pairs.
These files are later on processed by `./utils/req-res-log-validator.py` which does:
1. Goes over req-res files, generated by redis-servers, spawned by the testsuite (see logreqres.c)
2. For each request-response pair, it validates the response against the request's reply_schema
(obtained from the extended COMMAND DOCS)
5. In order to get good coverage of the Redis commands, and all their different replies, we chose to use
the existing redis test suite, rather than attempt to write a fuzzer.
#### Notes about RESP2
1. We will not be able to use the testing tool to verify RESP2 replies (we are ok with that, it's time to
accept RESP3 as the future RESP)
2. Since the majority of the test suite is using RESP2, and we want the server to reply with RESP3
so that we can validate it, we will need to know how to convert the actual reply to the one expected.
- number and boolean are always strings in RESP2 so the conversion is easy
- objects (maps) are always a flat array in RESP2
- others (nested array in RESP3's `ZRANGE` and others) will need some special per-command
handling (so the client will not be totally auto-generated)
Example for ZRANGE:
```
"reply_schema": {
"anyOf": [
{
"description": "A list of member elements",
"type": "array",
"uniqueItems": true,
"items": {
"type": "string"
}
},
{
"description": "Members and their scores. Returned in case `WITHSCORES` was used.",
"notes": "In RESP2 this is returned as a flat array",
"type": "array",
"uniqueItems": true,
"items": {
"type": "array",
"minItems": 2,
"maxItems": 2,
"items": [
{
"description": "Member",
"type": "string"
},
{
"description": "Score",
"type": "number"
}
]
}
}
]
}
```
### Other changes
1. Some tests that behave differently depending on the RESP are now being tested for both RESP,
regardless of the special log-req-res mode ("Pub/Sub PING" for example)
2. Update the history field of CLIENT LIST
3. Added basic tests for commands that were not covered at all by the testsuite
### TODO
- [x] (maybe a different PR) add a "condition" field to anyOf/oneOf schemas that refers to args. e.g.
when `SET` return NULL, the condition is `arguments.get||arguments.condition`, for `OK` the condition
is `!arguments.get`, and for `string` the condition is `arguments.get` - https://github.com/redis/redis/issues/11896
- [x] (maybe a different PR) also run `runtest-cluster` in the req-res logging mode
- [x] add the new tests to GH actions (i.e. compile with `-DLOG_REQ_RES`, run the tests, and run the validator)
- [x] (maybe a different PR) figure out a way to warn about (sub)schemas that are uncovered by the output
of the tests - https://github.com/redis/redis/issues/11897
- [x] (probably a separate PR) add all missing schemas
- [x] check why "SDOWN is triggered by misconfigured instance replying with errors" fails with --log-req-res
- [x] move the response transformers to their own file (run both regular, cluster, and sentinel tests - need to
fight with the tcl including mechanism a bit)
- [x] issue: module API - https://github.com/redis/redis/issues/11898
- [x] (probably a separate PR): improve schemas: add `required` to `object`s - https://github.com/redis/redis/issues/11899
Co-authored-by: Ozan Tezcan <ozantezcan@gmail.com>
Co-authored-by: Hanna Fadida <hanna.fadida@redislabs.com>
Co-authored-by: Oran Agra <oran@redislabs.com>
Co-authored-by: Shaya Potter <shaya@redislabs.com>
2023-03-11 03:14:16 -05:00
} { } { logreqres : skip} ; # This test violates {"uniqueItems": true}
2022-11-22 04:20:24 -05:00
Sanitize dump payload: ziplist, listpack, zipmap, intset, stream
When loading an encoded payload we will at least do a shallow validation to
check that the size that's encoded in the payload matches the size of the
allocation.
This let's us later use this encoded size to make sure the various offsets
inside encoded payload don't reach outside the allocation, if they do, we'll
assert/panic, but at least we won't segfault or smear memory.
We can also do 'deep' validation which runs on all the records of the encoded
payload and validates that they don't contain invalid offsets. This lets us
detect corruptions early and reject a RESTORE command rather than accepting
it and asserting (crashing) later when accessing that payload via some command.
configuration:
- adding ACL flag skip-sanitize-payload
- adding config sanitize-dump-payload [yes/no/clients]
For now, we don't have a good way to ensure MIGRATE in cluster resharding isn't
being slowed down by these sanitation, so i'm setting the default value to `no`,
but later on it should be set to `clients` by default.
changes:
- changing rdbReportError not to `exit` in RESTORE command
- adding a new stat to be able to later check if cluster MIGRATE isn't being
slowed down by sanitation.
2020-08-13 09:41:05 -04:00
} ; # tags
