caleb-brown/redict
1
0
Fork 0
mirror of https://codeberg.org/redict/redict.git synced 2025-01-22 08:08:53 -05:00
Code Issues Actions Packages Projects Releases Wiki Activity
f3f6f7c0d6
redict/src
History
Viktor Söderqvist f3f6f7c0d6
Key as dict entry - memory optimization for sets (#11595) 
If a dict has only keys, and no use of values, then a key can be stored directly in a
dict's hashtable. The key replaces the dictEntry. To distinguish between a key and
a dictEntry, we only use this optimization if the key is odd, i.e. if the key has the least
significant bit set. This is true for sds strings, since the sds header is always an odd
number of bytes.

Dict entries are used as a fallback when there is a hash collision. A special dict entry
without a value (only key and next) is used so we save one word in this case too.

This saves 24 bytes per set element for larges sets, and also gains some speed improvement
as a side effect (less allocations and cache misses).

A quick test adding 1M elements to a set using the command below resulted in memory
usage of 28.83M, compared to 46.29M on unstable.
That's 18 bytes per set element on average.

    eval 'for i=1,1000000,1 do redis.call("sadd", "myset", "x"..i) end' 0

Other changes:

Allocations are ensured to have at least 8 bits alignment on all systems. This affects 32-bit
builds compiled without HAVE_MALLOC_SIZE (not jemalloc or glibc) in which Redis
stores the size of each allocation, after this change in 8 bytes instead of previously 4 bytes
per allocation. This is done so we can reliably use the 3 least significant bits in a pointer to
encode stuff.
2023-01-20 18:45:29 +02:00
..
commands Add minimum version information to new xsetid arguments (#11694) 2023-01-10 09:09:51 +02:00
modules use $^ instead of $< for linker in module makefile (#10530) 2022-04-05 17:08:27 +03:00
.gitignore Ignore gcov/lcov artifacts 2012-04-13 17:52:33 -07:00
acl.c Retain ACL categories used to generate ACL for displaying them later (#11224) 2022-11-03 10:14:56 -07:00
adlist.c optimize unwatchAllKeys() (#11511) 2022-11-23 17:39:08 +02:00
adlist.h optimize unwatchAllKeys() (#11511) 2022-11-23 17:39:08 +02:00
ae_epoll.c Fail fast when systemic error occurs in poll (#8749) 2021-04-26 15:52:06 +03:00
ae_evport.c Fix cluster bus extensions backwards compatibility (#10206) 2022-01-30 19:43:37 +02:00
ae_kqueue.c Fix the timing of read and write events under kqueue (#9416) 2021-09-02 11:07:51 +03:00
ae_select.c fix unused argument warning in ae_select.c (#10824) 2022-06-07 14:47:09 +03:00
ae.c Avoid spurious wakeup on deleted timer event (#11069) 2022-11-25 20:36:33 -08:00
ae.h Add event loop support to the module API (#10001) 2022-01-18 13:10:07 +02:00
anet.c Introduce connAddr 2022-08-22 15:01:40 +08:00
anet.h Introduce connAddr 2022-08-22 15:01:40 +08:00
aof.c Add explicit error log message for AOF_TRUNCATED status when server load AOF file (#11484) 2022-11-22 16:18:36 +02:00
asciilogo.h Changes http to https in texts (#8495) 2021-03-10 19:11:16 +02:00
atomicvar.h Adding parentheses and do-while(0) to macros (#11080) 2022-08-03 19:38:08 +03:00
bio.c Removing old redundant code from bio.c (#11136) 2022-08-26 09:09:23 -07:00
bio.h Removing old redundant code from bio.c (#11136) 2022-08-26 09:09:23 -07:00
bitops.c Change compiler optimizations to -O3 -flto (#11207) 2022-10-02 15:15:14 +03:00
blocked.c reprocess command when client is unblocked on keys (#11012) 2023-01-01 23:35:42 +02:00
call_reply.c Fix broken protocol in MISCONF error, RM_Yield bugs, RM_Call(EVAL) OOM check bug, and new RM_Call checks. (#10786) 2022-06-01 13:04:22 +03:00
call_reply.h Add new RM_Call flags for script mode, no writes, and error replies. (#10372) 2022-03-22 14:13:28 +02:00
childinfo.c fixes for fork child exit and test: #11463 (#11499) 2022-11-12 20:35:34 +02:00
cli_common.c Fix error/warning on Arm due to unsigned char. (#10572) 2022-04-12 18:55:11 +03:00
cli_common.h redis-cli: Better --json Unicode support and --quoted-json (#10286) 2022-03-05 21:25:52 +02:00
cluster.c Increase frequency of failover log and emit the status of the election to help debugging (#11665) 2023-01-11 16:42:23 -08:00
cluster.h Increase frequency of failover log and emit the status of the election to help debugging (#11665) 2023-01-11 16:42:23 -08:00
commands.c Add minimum version information to new xsetid arguments (#11694) 2023-01-10 09:09:51 +02:00
config.c Make dictEntry opaque 2023-01-11 09:59:24 +01:00
config.h register debug support on illumos/solaris. (#11335) 2022-10-02 16:36:31 +03:00
connection.c Use cached value correctly inside connectionTypeTls() (#11236) 2022-09-06 09:04:33 +03:00
connection.h Introduce .is_local method for connection layer (#11672) 2023-01-04 10:52:56 +02:00
connhelpers.h Fixed some typos, add a spell check ci and others minor fix (#8890) 2021-06-10 15:39:33 +03:00
crc16_slottable.h Added basic support for clusters to redis-benchmark. 2019-03-01 17:53:14 +01:00
crc16.c RDMF (Redis/Disque merge friendlyness) refactoring WIP 1. 2015-07-26 15:17:18 +02:00
crc64.c Add --large-memory flag for REDIS_TEST to enable tests that consume more than 100mb (#9784) 2021-11-16 08:55:10 +02:00
crc64.h Add --large-memory flag for REDIS_TEST to enable tests that consume more than 100mb (#9784) 2021-11-16 08:55:10 +02:00
crcspeed.c Fixed some typos, add a spell check ci and others minor fix (#8890) 2021-06-10 15:39:33 +03:00
crcspeed.h Added crcspeed library 2020-04-24 17:11:21 -07:00
db.c Obuf limit, exit during loop in *RAND* commands and KEYS (#11676) 2023-01-16 13:51:18 +02:00
debug.c Make dictEntry opaque 2023-01-11 09:59:24 +01:00
debugmacro.h Supplement define guards to prevent multiple inclusion (#10246) 2022-02-06 20:13:34 -08:00
defrag.c Key as dict entry - memory optimization for sets (#11595) 2023-01-20 18:45:29 +02:00
dict.c Key as dict entry - memory optimization for sets (#11595) 2023-01-20 18:45:29 +02:00
dict.h Key as dict entry - memory optimization for sets (#11595) 2023-01-20 18:45:29 +02:00
endianconv.c Avoid using unsafe C functions (#10932) 2022-07-18 10:56:26 +03:00
endianconv.h Add --large-memory flag for REDIS_TEST to enable tests that consume more than 100mb (#9784) 2021-11-16 08:55:10 +02:00
eval.c Make dictEntry opaque 2023-01-11 09:59:24 +01:00
evict.c Cleanup: Get rid of server.core_propagates (#11572) 2022-12-20 09:51:50 +02:00
expire.c Remove the bucket-cb from dictScan and move dictEntry defrag to dictScanDefrag 2023-01-11 10:25:20 +01:00
fmacros.h Fix failed tests on Linux Alpine and add a CI job. (#8532) 2021-02-23 12:57:45 +02:00
function_lua.c Add missing lua_pop in luaGetFromRegistry (#11097) 2022-08-14 11:50:18 +03:00
functions.c Make dictEntry opaque 2023-01-11 09:59:24 +01:00
functions.h Functions: Move library meta data to be part of the library payload. (#10500) 2022-04-05 10:27:24 +03:00
geo.c Speedup GEODIST with fixedpoint_d2string as an optimized version of snprintf %.4f (#11552) 2022-12-04 10:11:38 +02:00
geo.h RDMF (Redis/Disque merge friendlyness) refactoring WIP 1. 2015-07-26 15:17:18 +02:00
geohash_helper.c GEOSEARCH BYBOX: Simplified haversine distance formula when longitude diff is 0 (#11579) 2022-12-05 15:45:04 +02:00
geohash_helper.h Delete some unimplemented prototype. (#8882) 2021-04-29 08:25:10 +03:00
geohash.c Fix mistake / outdated doc comment (#10521) 2022-04-04 15:35:49 +03:00
geohash.h Remove duplicate header file include (#10264) 2022-02-08 16:49:47 +02:00
help.h Make PFMERGE source key optional in docs, add tests with one input key, add tests on missing source keys (#11205) 2022-10-22 20:41:17 +03:00
hyperloglog.c Hyperloglog avoid allocate more than 'server.hll_sparse_max_bytes' bytes of memory for sparse representation (#11438) 2022-11-28 17:35:31 +02:00
intset.c Listpack encoding for sets (#11290) 2022-11-09 19:50:07 +02:00
intset.h Listpack encoding for sets (#11290) 2022-11-09 19:50:07 +02:00
latency.c Add warning for suspected slow system clocksource setting (#10636) 2022-05-22 17:10:31 +03:00
latency.h Add warning for suspected slow system clocksource setting (#10636) 2022-05-22 17:10:31 +03:00
lazyfree.c Add listpack encoding for list (#11303) 2022-11-16 20:29:46 +02:00
listpack_malloc.h Optimize listpack for stream usage to avoid repeated reallocs (#6281) 2021-02-16 16:17:38 +02:00
listpack.c When converting a set to dict, presize for one more element to be added (#11559) 2022-12-06 11:25:51 +02:00
listpack.h When converting a set to dict, presize for one more element to be added (#11559) 2022-12-06 11:25:51 +02:00
localtime.c fix typos (#10402) 2022-03-09 13:58:23 +02:00
lolwut5.c Fixed some typos, add a spell check ci and others minor fix (#8890) 2021-06-10 15:39:33 +03:00
lolwut6.c Fixed some typos, add a spell check ci and others minor fix (#8890) 2021-06-10 15:39:33 +03:00
lolwut.c Fixed some typos, add a spell check ci and others minor fix (#8890) 2021-06-10 15:39:33 +03:00
lolwut.h add include guard for lolwut.h 2020-05-05 23:35:08 -04:00
lzf_c.c Change lzf to handle values larger than UINT32_MAX (#9776) 2021-11-16 13:12:25 +02:00
lzf_d.c Change lzf to handle values larger than UINT32_MAX (#9776) 2021-11-16 13:12:25 +02:00
lzf.h Change lzf to handle values larger than UINT32_MAX (#9776) 2021-11-16 13:12:25 +02:00
lzfP.h Change lzf to handle values larger than UINT32_MAX (#9776) 2021-11-16 13:12:25 +02:00
Makefile Use jemalloc by default also on ARM (#11407) 2022-11-07 19:11:12 +02:00
memtest.c Add sanitizer support and clean up sanitizer findings (#9601) 2021-11-11 13:51:33 +02:00
mkreleasehdr.sh Build TLS as a loadable module 2022-08-23 12:37:56 +03:00
module.c Move stat_active_defrag_hits increment to activeDefragAlloc 2023-01-11 10:25:20 +01:00
monotonic.c Optimization: Use either monotonic or wall-clock to measure command execution time, to regain up to 4% execution time (#10502) 2022-04-20 14:00:30 +03:00
monotonic.h Optimization: Use either monotonic or wall-clock to measure command execution time, to regain up to 4% execution time (#10502) 2022-04-20 14:00:30 +03:00
mt19937-64.c Fix random element selection for large hash tables. (#8133) 2020-12-23 15:52:07 +02:00
mt19937-64.h Fix random element selection for large hash tables. (#8133) 2020-12-23 15:52:07 +02:00
multi.c Add redis_ prefix for member2struct, avoid redefined warning in FreeBSD (#11549) 2022-11-27 10:18:48 +02:00
networking.c Include peer-addr:port and local-addr:port when logging accept errors (#11622) 2023-01-04 16:09:27 -08:00
notify.c Add RM_PublishMessageShard (#10543) 2022-04-17 15:43:22 +03:00
object.c Remove the bucket-cb from dictScan and move dictEntry defrag to dictScanDefrag 2023-01-11 10:25:20 +01:00
pqsort.c Fix null pointer subtraction warning (#10498) 2022-04-04 18:38:18 +03:00
pqsort.h BSD license added to every C source and header file. 2012-11-08 18:31:32 +01:00
pubsub.c Make dictEntry opaque 2023-01-11 09:59:24 +01:00
quicklist.c Add listpack encoding for list (#11303) 2022-11-16 20:29:46 +02:00
quicklist.h Add listpack encoding for list (#11303) 2022-11-16 20:29:46 +02:00
rand.c Use 'void' for zero-argument functions 2014-08-08 10:05:32 +02:00
rand.h BSD license added to every C source and header file. 2012-11-08 18:31:32 +01:00
rax_malloc.h Cluster: hash slots tracking using a radix tree. 2017-03-27 16:37:22 +02:00
rax.c code, typo and comment cleanups (#11280) 2022-10-02 13:56:45 +03:00
rax.h Squash merging 125 typo/grammar/comment/doc PRs (#7773) 2020-09-10 13:43:38 +03:00
rdb.c Fix zuiFind crash / RM_ScanKey hang on SET object listpack encoding (#11581) 2022-12-09 17:08:01 +02:00
rdb.h Stream consumers: Re-purpose seen-time, add active-time (#11099) 2022-11-30 14:21:31 +02:00
redis-benchmark.c benchmark getRedisConfig exit only when meet NOAUTH error (#11096) 2022-11-28 20:51:25 +08:00
redis-check-aof.c Unify repeated code in redis-check-aof (#11456) 2022-11-06 14:49:55 +02:00
redis-check-rdb.c Listpack encoding for sets (#11290) 2022-11-09 19:50:07 +02:00
redis-cli.c Make dictEntry opaque 2023-01-11 09:59:24 +01:00
redis-trib.rb Redis-trib deprecated: it no longer works and it 2018-07-13 10:51:58 +02:00
redisassert.c Add sanitizer support and clean up sanitizer findings (#9601) 2021-11-11 13:51:33 +02:00
redisassert.h Fix a prototype inconsitency of _serverAssert between redisassert.h and redis.h (#10872) 2022-06-19 08:42:12 +03:00
redismodule.h Add a special notification unlink available only for modules (#9406) 2022-11-30 11:56:36 +02:00
release.c Build TLS as a loadable module 2022-08-23 12:37:56 +03:00
replication.c reprocess command when client is unblocked on keys (#11012) 2023-01-01 23:35:42 +02:00
resp_parser.c Unified Lua and modules reply parsing and added RESP3 support to RM_Call (#9202) 2021-08-04 16:28:07 +03:00
resp_parser.h Fix an mistake in comment (#10560) 2022-04-10 09:29:50 +03:00
rio.c optimizing d2string() and addReplyDouble() with grisu2: double to string conversion based on Florian Loitsch's Grisu-algorithm (#10587) 2022-10-15 12:17:41 +03:00
rio.h Adds isolated netstats for replication. (#10062) 2022-05-31 08:07:33 +03:00
script_lua.c Fix potential issue with Lua argv caching, module command filter and libc realloc (#11652) 2023-01-04 11:03:55 +02:00
script_lua.h Protect any table which is reachable from globals and added globals white list. 2022-04-27 00:37:40 +03:00
script.c reprocess command when client is unblocked on keys (#11012) 2023-01-01 23:35:42 +02:00
script.h Reintroduce lua argument cache in luaRedisGenericCommand removed in v7.0 (#11541) 2022-12-05 08:33:53 +02:00
sds.c Fix additional AOF filename issues. (#10110) 2022-01-18 12:52:27 +02:00
sds.h Fix additional AOF filename issues. (#10110) 2022-01-18 12:52:27 +02:00
sdsalloc.h Sanitize dump payload: fail RESTORE if memory allocation fails 2020-12-06 14:54:34 +02:00
sentinel.c Fix sentinel issue if replica changes IP (#11590) 2022-12-08 19:14:21 +02:00
server.c Key as dict entry - memory optimization for sets (#11595) 2023-01-20 18:45:29 +02:00
server.h Move stat_active_defrag_hits increment to activeDefragAlloc 2023-01-11 10:25:20 +01:00
setcpuaffinity.c cpu affinity: DragonFlyBSD support (#7956) 2020-10-25 14:14:05 +02:00
setproctitle.c Fix failed tests on Linux Alpine and add a CI job. (#8532) 2021-02-23 12:57:45 +02:00
sha1.c Ignore -Wstringop-overread warning for SHA1Transform() on GCC 12 (#11538) 2022-11-24 15:27:16 +02:00
sha1.h Add --large-memory flag for REDIS_TEST to enable tests that consume more than 100mb (#9784) 2021-11-16 08:55:10 +02:00
sha256.c Add sanitizer support and clean up sanitizer findings (#9601) 2021-11-11 13:51:33 +02:00
sha256.h fix explanation of sha256 (#9220) 2021-07-10 10:04:54 -05:00
siphash.c Add sanitizer support and clean up sanitizer findings (#9601) 2021-11-11 13:51:33 +02:00
slowlog.c slowlog get command supports passing in -1 to get all logs. (#9018) 2021-06-14 16:46:45 +03:00
slowlog.h Auto-generate the command table from JSON files (#9656) 2021-12-15 21:23:15 +02:00
socket.c Introduce .is_local method for connection layer (#11672) 2023-01-04 10:52:56 +02:00
solarisfixes.h Check for __sun macro in solarisfixes.h, not in includers. 2015-01-09 11:23:22 +01:00
sort.c Avoid integer overflows in SETRANGE and SORT (CVE-2022-35977) (#11720) 2023-01-16 13:49:30 +02:00
sparkline.c Squash merging 125 typo/grammar/comment/doc PRs (#7773) 2020-09-10 13:43:38 +03:00
sparkline.h LATENCY GRAPH implemented. 2014-07-02 16:31:22 +02:00
stream.h Stream consumers: Re-purpose seen-time, add active-time (#11099) 2022-11-30 14:21:31 +02:00
strl.c Avoid using unsafe C functions (#10932) 2022-07-18 10:56:26 +03:00
syncio.c syncWithMaster(): non blocking state machine. 2015-08-06 18:12:20 +02:00
syscheck.c Fixes typo (double word) in memory overcommit message (#11675) 2023-01-10 16:06:24 +02:00
syscheck.h Add warning for suspected slow system clocksource setting (#10636) 2022-05-22 17:10:31 +03:00
t_hash.c Obuf limit, exit during loop in *RAND* commands and KEYS (#11676) 2023-01-16 13:51:18 +02:00
t_list.c reprocess command when client is unblocked on keys (#11012) 2023-01-01 23:35:42 +02:00
t_set.c Key as dict entry - memory optimization for sets (#11595) 2023-01-20 18:45:29 +02:00
t_stream.c reprocess command when client is unblocked on keys (#11012) 2023-01-01 23:35:42 +02:00
t_string.c Avoid integer overflows in SETRANGE and SORT (CVE-2022-35977) (#11720) 2023-01-16 13:49:30 +02:00
t_zset.c Obuf limit, exit during loop in *RAND* commands and KEYS (#11676) 2023-01-16 13:51:18 +02:00
testhelp.h Add --large-memory flag for REDIS_TEST to enable tests that consume more than 100mb (#9784) 2021-11-16 08:55:10 +02:00
timeout.c Blocking command with a 0.001 seconds timeout blocks indefinitely (#11688) 2023-01-08 01:02:48 -08:00
tls.c Introduce .is_local method for connection layer (#11672) 2023-01-04 10:52:56 +02:00
tracking.c Cleanup: Get rid of server.core_propagates (#11572) 2022-12-20 09:51:50 +02:00
unix.c Introduce .is_local method for connection layer (#11672) 2023-01-04 10:52:56 +02:00
util.c Fixed small distance replies on GEODIST and GEO commands WITHDIST (#11631) 2022-12-15 22:25:38 +02:00
util.h Speedup GEODIST with fixedpoint_d2string as an optimized version of snprintf %.4f (#11552) 2022-12-04 10:11:38 +02:00
valgrind.sup Sanitize dump payload: fuzz tester and fixes for segfaults and leaks it exposed 2020-12-06 14:54:34 +02:00
version.h Add Module API for version and compatibility checks (#7865) 2020-10-11 17:21:58 +03:00
ziplist.c fix arm build warning due to new compiler optimizations (#11362) 2022-10-07 21:24:54 +03:00
ziplist.h Add --large-memory flag for REDIS_TEST to enable tests that consume more than 100mb (#9784) 2021-11-16 08:55:10 +02:00
zipmap.c Add --large-memory flag for REDIS_TEST to enable tests that consume more than 100mb (#9784) 2021-11-16 08:55:10 +02:00
zipmap.h Add --large-memory flag for REDIS_TEST to enable tests that consume more than 100mb (#9784) 2021-11-16 08:55:10 +02:00
zmalloc.c Key as dict entry - memory optimization for sets (#11595) 2023-01-20 18:45:29 +02:00
zmalloc.h zmalloc api set malloc attributes for api giving non aliased pointers. (#11196) 2022-09-05 16:09:28 +03:00