This change only affects keys with expiry time.
For SETEX, the average improvement is 5%, and for GET with
expiation key, we gain a improvement of 13%.
When keys have expiration time, Redis has an assertion to look
up the main dict every time when it touches the expires.
This comes with a performance const, especially during rehash.
the damage will be double.
It looks like that assert was added some ten years old, maybe out
of paranoia, and there's probably no reason to keep it at that cost.
For the XREADGROUP BLOCK > scenario, there is an endless loop.
Due to #11012, it keep going, reprocess command -> blockForKeys -> reprocess command
The right fix is to avoid an endless loop in handleClientsBlockedOnKey and handleClientsBlockedOnKeys,
looks like there was some attempt in handleClientsBlockedOnKeys but maybe not sufficiently good,
and it looks like using a similar trick in handleClientsBlockedOnKey is complicated.
i.e. stashing the list on the stack and iterating on it after creating a fresh one for future use,
is problematic since the code keeps accessing the global list.
Co-authored-by: Oran Agra <oran@redislabs.com>
This will increase the size of an already large COB (one already passed
the threshold for disconnection)
This could also mean that we'll attempt to write that data to the socket
and the replica will manage to read it, which will result in an
undesired partial sync (undesired for the test)
In 7.2, After 971b177fa we make sure (assert) that
the duration has been recorded when resetting the client.
This is not true for rejected commands.
The use case I found is a blocking command that an ACL rule changed before
it was unblocked, and while reprocessing it, the command rejected and triggered the assert.
The PR reset the command duration inside rejectCommand / rejectCommandSds.
Co-authored-by: Oran Agra <oran@redislabs.com>
In #11963, some new tests about eventloop duration were added, which includes time measurement in TCL scripts. This has caused some unexpected CI failures, such as #12169 and #12177, due to slow test servers or some performance jittering.
Added missing test case coverage for below scenarios:
1. The command only works if all the specified slots are, from
the point of view of the node receiving the command, currently
not assigned. A node will refuse to take ownership for slots that
already belong to some other node (including itself).
2. The command fails if the same slot is specified multiple times.
This leak will only happen in loadServerConfigFromString,
that is, when we are loading a redis.conf, and the user is wrong.
Because it happens in loadServerConfigFromString, redis will
exit if there is an error, so this is actually just a cleanup.
b6a052fe0 Helper for setting TCP_USER_TIMEOUT socket option (#1188)
3fa9b6944 Add RedisModule adapter (#1182)
d13c091e9 Fix wincrypt symbols conflict
5d84c8cfd Add a test ensuring we don't clobber connection error.
3f95fcdae Don't attempt to set a timeout if we are in an error state.
aacb84b8d Fix typo in makefile.
563b062e3 Accept -nan per the RESP3 spec recommendation.
04c1b5b02 Fix colliding option values
4ca8e73f6 Rework searching for openssl
cd208812f Attempt to find the correct path for openssl.
011f7093c Allow specifying the keepalive interval
e9243d4f7 Cmake static or shared (#1160)
1cbd5bc76 Write a version file for the CMake package (#1165)
6f5bae8c6 fix typo
acd09461d CMakeLists.txt: respect BUILD_SHARED_LIBS
97fcf0fd1 Add sdevent adapter
ccff093bc Bump dev version for the next release cycle.
c14775b4e Prepare for v1.1.0 GA
f0bdf8405 Add support for nan in RESP3 double (#1133)
991b0b0b3 Add an example that calls redisCommandArgv (#1140)
a36686f84 CI updates (#1139)
8ad4985e9 fix flag reference
7583ebb1b Make freeing a NULL redisAsyncContext a no op.
2c53dea7f Update version in dev branch.
f063370ed Prepare for v1.1.0-rc1
2b069573a CI fixes in preparation of release
e1e9eb40d Add author information to release-drafter template.
afc29ee1a Update for mingw cross compile
ceb8a8815 fixed cpp build error with adapters/libhv.h
3b15a04b5 Fixup of PR734: Coverage of hiredis.c (#1124)
c245df9fb CMake corrections for building on Windows (#1122)
9c338a598 Fix PUSH handler tests for Redis >= 7.0.5
6d5c3ee74 Install on windows fixes (#1117)
68b29e1ad Add timeout support to libhv adapter. (#1109)
722e3409c Additional include directory given by pkg-config (#1118)
bd9ccb8c4 Use __attribute__ when building with clang on windows
5392adc26 set default SSL certificate directory
560e66486 Minor refactor
d756f68a5 Add libhv example to our standard Makefile
a66916719 Add adapters/libhv
855b48a81 Fix pkgconfig for hiredis_ssl
79ae5ffc6 Fix protocol error (#1106)
61b5b299f Use a windows specific keepalive function. (#1104)
fce8abc1c Introduce .close method for redisContextFuncs
cfb6ca881 Add REDIS_OPT_PREFER_UNSPEC (#1101)
cc7c35ce6 Update documentation to explain redisConnectWithOptions.
bc8d837b7 fix heap-buffer-overflow (#957)
ca4a0e850 uvadapter: reduce number of uv_poll_start calls
35d398c90 Fix cmake config path on Linux. CMake config files were installed to `/usr/local/share/hiredis`, which is not recognizable by `find_package()`. I'm not sure why it was set that way. Given the commit introducing it is for Windows, I keep that behavior consistent there, but fix the rest.
10c78c6e1 Add possibility to prefer IPv6, IPv4 or unspecified
1abe0c828 fuzzer: No alloc in redisFormatCommand() when fail
329eaf9ba Fix heap-buffer-overflow issue in redisvFormatCommad
eaae7321c Polling adapter requires sockcompat.h
0a5fa3dde Regression test for off-by-one parsing error
9e174e8f7 Add do while(0) protection for macros
4ad99c69a Rework asSleep to be a generic millisleep function.
75cb6c1ea Do store command timeout in the context for redisSetTimeout (#593)
c57cad658 CMake: remove dict.c form hiredis_sources
8491a65a9 Add Github Actions CI workflow for hiredis: Arm, Arm64, 386, windows. (#943)
77e4f09ea Merge pull request #964 from afcidk/fix-createDoubleObject
9219f7e7c Merge pull request #901 from devnexen/illumos_test_fix
810cc6104 Merge pull request #905 from sundb/master
df8b74d69 Merge pull request #1091 from redis/ssl-error-ub-fix
0ed6cdec3 Fix some undefined behaviour
507a6dcaa Merge pull request #1090 from Nordix/subscribe-oom-error
b044eaa6a Copy error to redisAsyncContext when finding subscribe cb
e0200b797 Merge pull request #1087 from redis/const-and-non-const-callback
6a3e96ad2 Maintain backward compatibiliy withour onConnect callback.
e7afd998f Merge pull request #1079 from SukkaW/drop-macos-10.15-runner
17c8fe079 Merge pull request #931 from kristjanvalur/pr2
b808c0c20 Merge pull request #1083 from chayim/ck-drafter
367a82bf0 Merge pull request #1085 from stanhu/ssl-improve-options-setting
71119a71d Make it possible to set SSL verify mode
dd7979ac1 Merge pull request #1084 from stanhu/sh-improve-ssl-docs
c71116178 Improve example for SSL initialization in README.md
5c9b6b571 Release drafter
a606ccf2a CI: use recommended `vmactions/freebsd-vm@v0`
0865c115b Merge pull request #1080 from Nordix/readme-corrections
f6cee7142 Fix README typos
06be7ff31 Merge pull request #1050 from smmir-cent/fix-cmake-version
7dd833d54 CI: bump macos runner version
f69fac769 Drop `const` on redisAsyncContext in redisConnectCallback Since the callback is now re-entrant, it can call apis such as redisAsyncDisconnect()
005d7edeb Support calling redisAsyncDisconnect from the onConnected callback, by deferring context deletion
6ed060920 Add async regression test for issue #931
eaa2a7ee7 Merge pull request #932 from kristjanvalur/pr3
2ccef30f3 Add regression test for issue #945
4b901d44a Initial async tests
31c91408e Polling adapter and example
8a15f4d65 Merge pull request #1057 from orgads/static-name
902dd047f Merge pull request #1054 from kristjanvalur/pr08
c78d0926b Merge pull request #1074 from michael-grunder/kristjanvalur-pr4
2b115d56c Whitespace
1343988ce Fix typos
47b57aa24 Add some documentation on connect/disconnect callbacks and command callbacks
a890d9ce2 Merge pull request #1073 from michael-grunder/kristjanvalur-pr1
f246ee433 Whitespace, style
94c1985bd Use correct type for getsockopt()
5e002bc21 Support failed async connects on windows.
5d68ad2f4 Merge pull request #1072 from michael-grunder/fix-redis7-unit-tests
f4b6ed289 Fix tests so they work for Redis 7.0
95a0c1283 Merge pull request #1058 from orgads/win64
eedb37a65 Fix warnings on Win64
47c3ecefc Merge pull request #1062 from yossigo/fix-push-notification-order
e23d91c97 Merge pull request #1061 from yossigo/update-redis-apt
34211ad54 Merge pull request #1063 from redis/fix-windows-tests
9957af7e3 Whitelist hiredis repo path in cygwin
b455b3381 Handle push notifications before or after reply.
aed9ce446 Use official repository for redis package.
d7683f35a Merge pull request #1047 from Nordix/unsubscribe-handling
7c44a9d7e Merge pull request #1045 from Nordix/sds-updates
dd4bf9783 Use the same name for static and shared libraries
ff57c18b9 Embed debug information in windows static lib, rather than create a .pdb file
8310ad4f5 fix cmake version
7123b87f6 Handle any pipelined unsubscribe in async
b6fb548fc Ignore pubsub replies without a channel/pattern
00b82683b Handle overflows as errors instead of asserting
64062a1d4 Catch size_t overflows in sds.c
066c6de79 Use size_t/long to avoid truncation
c6657ef65 Merge branch 'redis:master' into master
50cdcab49 Fix potential fault at createDoubleObject
fd033e983 Remove semicolon after do-while in _EL_CLEANUP
664c415e7 Illumos test fixes, error message difference fot bad hostname test.
git-subtree-dir: deps/hiredis
git-subtree-split: b6a052fe0959dae69e16b9d74449faeb1b70dbe1
This test was introduced in #12079, it works well most of the time, but
occasionally fails:
```
00:34:45> SENTINEL SIMULATE-FAILURE crash-after-election works: OK
00:34:45> SENTINEL SIMULATE-FAILURE crash-after-promotion works: FAILED: Sentinel set crash-after-promotion but did not exit
```
Don't know the reason, it may be affected by the exit of the previous
crash-after-election test. Because it doesn't really make much sense to
go deeper into it now, we re-source init-tests to get a clean environment
before each test, to try to fix this.
After applying this change, we found a new error:
```
16:39:33> SENTINEL SIMULATE-FAILURE crash-after-election works: FAILED: caught an error in the test couldn't open socket: connection refused
couldn't open socket: connection refused
```
I am guessing the sentinel triggers failover and exits before SENTINEL FAILOVER,
added a new || condition in wait_for_condition to fix it.
We check lazyfree_lazy_server_del in sunionDiffGenericCommand
to see if we need to lazyfree the temp set. Now do the same in
zunionInterDiffGenericCommand to lazyfree the temp zset.
This is a minor change, follow #5903. Also improved the comments.
Additionally, avoid creating unused zset object in ZINTERCARD,
results in some 10% performance improvement.
So far clients being blocked and unblocked by a module command would
update the c->woff variable and so WAIT was ineffective and got released
without waiting for the command actions to propagate.
This seems to have existed since forever, but not for RM_BlockClientOnKeys.
It is problematic though to know if the module did or didn't propagate
anything in that command, so for now, instead of adding an API, we'll
just update the woff to the latest offset when unblocking, this will
cause the client to possibly wait excessively, but that's not that bad.
XREAD only supports a special ID of $ and XREADGROUP only supports ^.
make sure not to suggest the wrong one when rerunning an error about unbalanced ID arguments
Co-authored-by: Oran Agra <oran@redislabs.com>
Rather than a fixed iovcnt for connWritev, support maxiov per connection type instead.
A minor change to reduce memory for struct connection.
Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
It was missing in #12223, and the reply-schemas daily
was failing:
```
jsonschema.exceptions.ValidationError: 'nothing' is not valid under any of the given schemas
Failed validating 'oneOf' in schema[0]['properties']['loglevel']:
{'oneOf': [{'const': 'debug'},
{'const': 'verbose'},
{'const': 'notice'},
{'const': 'warning'},
{'const': 'unknown'}]}
On instance['loglevel']:
'nothing'
```
The light version only shows the table sizes, while the pre-existing
version that shows chain length stats is reachable with the `full` argument.
This should allow looking into rehashing state, even on huge dicts, on
which we're afraid to run the command for fear of causing a server freeze.
Also, fix a possible overflow in dictGetStats.
This pr can get two performance benefits:
1. Stop redundant initialization when most robj objects are created
2. LRU_CLOCK will no longer be called in io threads, so we can avoid the `atomicGet`
Another code optimization:
deleted the redundant judgment in dbSetValue, no matter in LFU or LRU, the lru field inold
robj is always the freshest (it is always updated in lookupkey), so we don't need to judge if in LFU
We add a new loglevel 'nothing' to disable logging in #12133.
This PR syncs that config change to sentinel. Because in #11214
we support modifying loglevel in runtime.
Although I think sentinel doesn't need this nothing config,
it's better to be consistent.
In #12166, we removed a call to CLUSTER SLAVES, which
then caused reply-schemas ci to fail:
```
WARNING! The following commands were not hit at all:
cluster|slaves
ERROR! at least one command was not hit by the tests
```
Because we already have command output that cover CLUSTER REPLICAS
elsewhere, here we simply add some dummy tests to fix the ci.
This commit excludes aux fields from the output of the `cluster nodes` and `cluster replicas` command.
We may decide to re-introduce them in some form or another in the future, but not in v7.2.
Users can record logs of different levels by setting the `loglevel`.
However, sometimes there are many logs even at the warning level,
which can affect the performance of Redis.
For example, when a user accesses the tls-port using a non-encrypted link,
Redis will log lots of "# Error accepting a client connection: ...".
We can provide the ability to disable logging so that users can temporarily turn
off logging and turn it back on after the problem is resolved.
previously the argv wasn't freed so would leak. not a common case, but should be handled.
Solution: move RUN_AS_USER setup and error exit to the right place.
this way, when we do `goto cleanup` (instead of return) it'll automatically do the right thing (including autoMemoryAdd)
Removed the user argument from moduleAllocTempClient (reverted to the state before 6e993a5)
Co-authored-by: Oran Agra <oran@redislabs.com>
Optimized HRANDFIELD and ZRANDMEMBER commands as in #8444,
CASE 3 under listpack encoding. Boost optimization to CASE 2.5.
CASE 2.5 listpack only. Sampling unique elements, in non-random order.
Listpack encoded hashes / zsets are meant to be relatively small, so
HRANDFIELD_SUB_STRATEGY_MUL / ZRANDMEMBER_SUB_STRATEGY_MUL
isn't necessary and we rather not make copies of the entries. Instead, we
emit them directly to the output buffer.
Simple benchmarks shows it provides some 400% improvement in HRANDFIELD
and ZRANGESTORE both in CASE 3.
Unrelated changes: remove useless setTypeRandomElements and fix a typo.
A single SPOP with command with count argument resulted in many SPOP
commands being propagated to the replica.
This is inefficient because the key name is repeated many times, and is also
being looked-up many times.
also it results in high QPS metrics on the replica.
To solve that, we flush batches of 1024 fields per SPOP command.
Co-authored-by: zhaozhao.zz <zhaozhao.zz@alibaba-inc.com>
For zsets that will eventually be stored as the skiplist encoding (has a dict),
we can convert it to skiplist ahead of time. This change checks the number
of arguments in the ZADD command, and converts the data-structure
if the number of new entries exceeds the listpack-max-entries configuration.
This can cause us to over-allocate memory if there are duplicate entries in the
input, which is unexpected.
For ZRANGESTORE, we know the size of the zset, so we can expand
the dict in advance, to avoid the temporary dict from being rehashed
while it grows.
Simple benchmarks shows it provides some 4% improvement in ZADD and 20% in ZRANGESTORE
Current tests for BITFIELD_RO command are skipped in the external mode,
and therefore reply-schemas-validator reports a coverage error.
This PR adds basic tests to increase coverage.
When `replicationFeedSlaves()` serializes a command, it repeatedly calls
`feedReplicationBuffer()` to feed it to the replication backlog piece by piece.
It is unnecessary to call `incrementalTrimReplicationBacklog()` for every small
amount of data added with `feedReplicationBuffer()` as the chance of the conditions
being met for trimming are very low and these frequent calls add up to a notable
performance cost. Instead, we will only attempt trimming when a new block is added
to the replication backlog.
Using redis-benchmark to saturate a local redis server indicated a performance
improvement of around 3-3.5% for 100 byte SET commands with this change.
Extend SENTINEL CONFIG SET and SENTINEL CONFIG GET to be
compatible with variadic CONFIG SET and CONFIG GET and allow multiple
parameters to be modified in a single call atomically.
Co-authored-by: Oran Agra <oran@redislabs.com>
The measured latency(duration) includes the list below, which can be shown by `INFO STATS`.
```
eventloop_cycles // ever increasing counter
eventloop_duration_sum // cumulative duration of eventloop in microseconds
eventloop_duration_cmd_sum // cumulative duration of executing commands in microseconds
instantaneous_eventloop_cycles_per_sec // average eventloop count per second in recent 1.6s
instantaneous_eventloop_duration_usec // average single eventloop duration in recent 1.6s
```
Also added some experimental metrics, which are shown only when `INFO DEBUG` is called.
This section isn't included in the default INFO, or even in `INFO ALL` and the fields in this section
can change in the future without considering backwards compatibility.
```
eventloop_duration_aof_sum // cumulative duration of writing AOF
eventloop_duration_cron_sum // cumulative duration cron jobs (serverCron, beforeSleep excluding IO and AOF)
eventloop_cmd_per_cycle_max // max number of commands executed in one eventloop
eventloop_duration_max // max duration of one eventloop
```
All of these are being reset by CONFIG RESETSTAT
When using scan in redis-cli, the SCAN COUNT is fixed, which means the
full scan can take a long time if there are a lot of keys, this will let users specify
a bigger COUNT option.
This pattern is from COMMAND INFO:
Returns information about one, multiple or all commands.
Also re-generate commands.def, the GEO change was missing in #12151.
