caleb-brown/redict
1
0
Fork 0
mirror of https://codeberg.org/redict/redict.git synced 2025-01-24 00:59:02 -05:00
Code Issues Actions Packages Projects Releases Wiki Activity
7,420 Commits 4 Branches 5 Tags 115 MiB
5f3adbee33
Commit Graph

192 Commits

Author SHA1 Message Date
antirez
33e1db36fa Four new persistence fields in INFO. A few renamed. 
The 'persistence' section of INFO output now contains additional four
fields related to RDB and AOF persistence:

 rdb_last_bgsave_time_sec       Duration of latest BGSAVE in sec.
 rdb_current_bgsave_time_sec    Duration of current BGSAVE in sec.
 aof_last_rewrite_time_sec      Duration of latest AOF rewrite in sec.
 aof_current_rewrite_time_sec   Duration of current AOF rewrite in sec.

The 'current' fields are set to -1 if a BGSAVE / AOF rewrite is not in
progress. The 'last' fileds are set to -1 if no previous BGSAVE / AOF
rewrites were performed.

Additionally a few fields in the persistence section were renamed for
consistency:

 changes_since_last_save -> rdb_changes_since_last_save
 bgsave_in_progress -> rdb_bgsave_in_progress
 last_save_time -> rdb_last_save_time
 last_bgsave_status -> rdb_last_bgsave_status
 bgrewriteaof_in_progress -> aof_rewrite_in_progress
 bgrewriteaof_scheduled -> aof_rewrite_scheduled

After the renaming, fields in the persistence section start with rdb_ or
aof_ prefix depending on the persistence method they describe.
The field 'loading' and related fields are not prefixed because they are
unique for both the persistence methods.
 2012-05-25 12:11:30 +02:00
antirez
47ca4b6e28 Allow an AOF rewrite buffer > 2GB (Fix for issue #504). 
During the AOF rewrite process, the parent process needs to accumulate
the new writes in an in-memory buffer: when the child will terminate the
AOF rewriting process this buffer (that ist the difference between the
dataset when the rewrite was started, and the current dataset) is
flushed to the new AOF file.

We used to implement this buffer using an sds.c string, but sds.c has a
2GB limit. Sometimes the dataset can be big enough, the amount of writes
so high, and the rewrite process slow enough that we overflow the 2GB
limit, causing a crash, documented on github by issue #504.

In order to prevent this from happening, this commit introduces a new
system to accumulate writes, implemented by a linked list of blocks of
10 MB each, so that we also avoid paying the reallocation cost.

Note that theoretically modern operating systems may implement realloc()
simply as a remaping of the old pages, thus with very good performances,
see for instance the mremap() syscall on Linux. However this is not
always true, and jemalloc by default avoids doing this because there are
issues with the current implementation of mremap().

For this reason we are using a linked list of blocks instead of a single
block that gets reallocated again and again.

The changes in this commit lacks testing, that will be performed before
merging into the unstable branch. This fix will not enter 2.4 because it
is too invasive. However 2.4 will log a warning when the AOF rewrite
buffer is near to the 2GB limit.
 2012-05-24 15:19:15 +02:00
antirez
5a55999304 Use comments to split aof.c into sections. 
This makes the code more readable, it is still not the case to split the
file itself into three different files, but the logical separation
improves the readability especially since new commits are going to
introduce an additional section.
 2012-05-21 16:50:05 +02:00
antirez
2cbdab903f For coverage testing use exit() instead of _exit() when termiating saving children. 2012-04-07 12:11:23 +02:00
Premysl Hruby
d194905449 use server.unixtime instead of time(NULL) where possible (cluster.c not checked though) 2012-03-27 17:39:58 +02:00
antirez
c1d01b3c57 New INFO field aof_delayed_fsync introduced. 
This new field counts all the times Redis is configured with AOF enabled and
fsync policy 'everysec', but the previous fsync performed by the
background thread was not able to complete within two seconds, forcing
Redis to perform a write against the AOF file while the fsync is still
in progress (likely a blocking operation).
 2012-03-25 11:27:35 +02:00
antirez
ae22bf1ef6 Reclaim space from the client querybuf if needed. 2012-03-14 15:32:30 +01:00
antirez
addc032756 Added a top-function comment to rioWriteHashIteratorCursor() to better specify what the function does. Not immediately clear from the name. 2012-03-10 10:36:51 +01:00
antirez
8562798308 Merge conflicts resolved. 2012-03-09 22:07:45 +01:00
antirez
2f0f0d95c0 lenght -> length 2012-01-24 15:33:15 +01:00
antirez
7eac2a75a4 Implementation of the internals that make possible to terminate clients overcoming configured output buffer (soft and hard) limits. 2012-01-23 16:12:37 +01:00
antirez
3853c16839 Track the length of the client pending output buffers (still to transfer) in a new field in the client structure. 2012-01-17 12:23:25 +01:00
Pieter Noordhuis
ebd85e9a45 Encode small hashes with a ziplist 2012-01-02 22:14:10 -08:00
antirez
e51b79f315 sligthly better AOF related logs 2011-12-21 17:12:23 +01:00
antirez
b941417c1b Log when the child rewriting the AOF is killed by stopAppendOnly() 2011-12-21 16:37:22 +01:00
antirez
f48cd4b90c some RDB server struct fields renamed. 2011-12-21 12:22:13 +01:00
antirez
ff2145adac more AOF server struct fields renamed. 2011-12-21 12:17:02 +01:00
antirez
2c915bcf6d AOF fileds in the global server state, and define names, renamed with more consistent names. More work to do. 2011-12-21 11:58:42 +01:00
antirez
e394114d95 AOF refactoring, now with three states: ON, OFF, WAIT_REWRITE. 2011-12-21 10:31:34 +01:00
antirez
af77acc310 startAppendOnly() fixed to set server.appendonly to 0 if the start failed. 2011-12-20 17:00:50 +01:00
antirez
087f414090 Fix: when aof_write_rewrite is true don't append on the AOF buffer but accumulate the differences for the rewrite. 2011-12-15 20:03:28 +01:00
antirez
e7a2e7c1f7 AOF fixes in the context of replicaiton (when AOF is used by slave) and CONFIG SET appendonly yes/no. 2011-12-15 16:07:49 +01:00
antirez
54ecc0e777 Hash type AOF rewrite using HMSET. 2011-12-13 11:10:51 +01:00
antirez
7df9b1412b Sorted Set type AOF rewrite using variadic ZADD. 2011-12-13 11:10:46 +01:00
antirez
8d875ccb7a Set type AOF rewrite using variadic SADD. 2011-12-13 11:10:41 +01:00
antirez
38c06fa037 List type AOF rewrite using variadic RPUSH for the linked list encoding. 2011-12-13 11:10:31 +01:00
antirez
5b25009656 Lists AOF rewrite using variadic RPUSH (work in progress) 2011-12-13 11:10:21 +01:00
antirez
4be855e757 Fixed issues with expire introduced with latest millisecond resolution feature. Many time_t were not converted to long long, and one time() call was not replaced with mstime(). 2011-11-12 01:04:27 +01:00
antirez
12d293ca6e high resolution expires API modified to use separated commands. AOF transation to PEXPIREAT of all the expire-style commands fixed. 2011-11-10 17:52:02 +01:00
antirez
b0b74486e1 use "EXPIREAT ... ms" form when rewriting the AOF 2011-11-09 17:20:14 +01:00
antirez
7dcc10b65e Initial support for key expire times with millisecond resolution. RDB version is now 3, new opcoded added for high resolution times. Redis is still able to correctly load RDB version 2. Tests passing but still a work in progress. API to specify milliseconds expires still missing, but the precision of normal expires is now already improved and working. 2011-11-09 16:51:19 +01:00
antirez
c0ba9ebe13 dict.c API names modified to be more coincise and consistent. 2011-11-08 17:07:55 +01:00
antirez
b508aeb994 Clear the AOF rewrite scheduled flag once an AOF rewrite is triggered. Fix for issue #161, probably fixing 159 as well. 2011-10-26 12:53:30 +02:00
antirez
ff15dba093 not used parameter for redisLog() removed. Error message grammar fixed. 
Thanks to @ardsrk.
 2011-10-08 10:59:04 +02:00
antirez
f96a8a8054 rioInitWithFile nad rioInitWithBuffer functions now take a rio structure pointer to avoid copying a structure to return value to the caller. 2011-09-22 16:00:40 +02:00
antirez
f9c6f39b2b merge conflicts resolved 2011-09-22 15:15:26 +02:00
Salvatore Sanfilippo
18be93e0a4 Merge pull request #74 from kmerenkov/issue_620 
[issue 620] don't segfault if number of arguments is less than 1
 2011-09-20 01:55:34 -07:00
antirez
e7aec180e8 Fixed comment typo 2011-09-19 17:50:07 +02:00
antirez
77ca5fcba6 Emit a log message when AOF fsync is still in progress but we are forced to write from the main thread since two seconds already elapsed. 2011-09-19 16:52:13 +02:00
antirez
af4707a80c Merge remote-tracking branch 'origin/unstable' into bg-aof-2 2011-09-16 12:36:33 +02:00
antirez
db3c2a4fb4 postpone the AOF fsync if policy is everysec and there is a background fsync already going. 2011-09-16 12:35:12 +02:00
antirez
9a35eb22bf Fixed typo preventing compilation 2011-09-16 12:11:48 +02:00
Pieter Noordhuis
5f54a5e615 Fix AOF race that may duplicate commands 2011-09-16 11:12:31 +02:00
antirez
4b77700a33 Added aof_backgronud_fsync() function, and use it in the bacground rewrite done handler when the fsync policy is everysec. 2011-09-16 11:08:39 +02:00
antirez
50be9b97bc Use a different thread for every different type of background job 2011-09-15 18:21:16 +02:00
antirez
986630afad use bio.c instead of libeio for closing the AOF file in background. Some comment added and other minor changes. 2011-09-13 18:27:08 +02:00
Pieter Noordhuis
b454056d94 Asynchronously close old file after BGREWRITEAOF 2011-09-13 18:17:06 +02:00
Pieter Noordhuis
d1ec6c8b22 Speed up protocol synthesis for AOF 2011-09-13 12:23:50 +02:00
Pieter Noordhuis
f990782f4d Re-use AOF buffer when it is small enough 2011-09-13 12:22:54 +02:00
Pieter Noordhuis
a57225c2cf Fix indent 2011-09-13 12:22:39 +02:00
Pieter Noordhuis
297322481d Avoid unnecessary calls to time(3) 2011-09-13 12:18:55 +02:00
Konstantin Merenkov
be6f639564 [issue 620] don't segfault if number of arguments is less than 1 2011-08-02 17:05:04 +04:00
antirez
ef67a2fc3c Partial fix for a problem with AOF and BRPOP. We have at least a single evidence of an AOF that ended with a BRPOP call against an empty list, causing later issues at run time. Now the problem is detected while loading the AOF at startup, even if it is still to understand how the corrupted AOF was generated. 2011-06-29 16:11:23 +02:00
antirez
c9d0c3623a diskstore removed 2011-06-25 12:22:03 +02:00
antirez
591f29e0f5 Use a safe iterator while saving the DB, since the getExpire() function will access the iterating dictionary. 2011-06-17 15:41:22 +02:00
antirez
9e40bce3fa different message on BGREWRITEAOF when it is just scheduled and not started. 2011-06-10 18:35:16 +02:00
antirez
c66bf1fa32 correctly set AOF base size field in server structure 2011-06-10 14:56:45 +02:00
antirez
b333e23997 automatic AOF rewrite first implementation. Still to be tested. 2011-06-10 12:39:23 +02:00
antirez
615e414c5d INFO now contains the time (in usecs) needed to fork() in order to persist. The info is available in the stats section of INFO. 2011-05-29 15:17:29 +02:00
Pieter Noordhuis
7271198cf0 Use rio.h functions in aof.c 2011-05-14 12:36:22 +02:00
antirez
6901fe7729 Useless statement removed 2011-05-10 10:07:04 +02:00
Pieter Noordhuis
45b0f6fb14 Use correct argc/argv for cleanup when loading AOF 2011-04-22 09:44:06 +02:00
Pieter Noordhuis
100ed062c0 Test for ENCODING_SKIPLIST instead of ENCODING_RAW 2011-04-06 16:17:07 +02:00
Pieter Noordhuis
dddf5335f4 Fix DEBUG DIGEST, SORT and AOF rewrite 2011-03-14 13:30:06 +01:00
antirez
4aec2ec8c4 AOF file descriptor leak fixed 2011-03-04 16:13:54 +01:00
antirez
69bfffb4a7 test adapted to run with diskstore, and a few bugs fixed 2011-01-09 18:25:34 +01:00
antirez
5b8ce85378 more work towards diskstore bgsave 2011-01-07 19:31:42 +01:00
antirez
36c17a53b6 source reshaped a bit to play well with a bgsaving thread, still work to do, does not compile. 2011-01-07 18:15:14 +01:00
antirez
16d778780e a lot of code reworked/removed to implement object caching 2010-12-28 18:06:40 +01:00
antirez
97e7f8aec3 non blocking loading of the DB / AOF with informations and ETA in INFO output 2010-11-08 11:52:03 +01:00
antirez
0a546fc017 Merge remote branch 'pietern/unixsocket' 2010-11-02 23:47:52 +01:00
antirez
ca734d17ad Revert "Now maxmemory, VM, and everything else uses the fast RSS memory used estimation instead of raw memory reported by zmalloc(). This means that setting maxmemory to 2GB will really have the effect of using up to 2GB of memory." 
This reverts commit a3e60027e7.
 2010-11-02 12:09:59 +01:00
antirez
a3e60027e7 Now maxmemory, VM, and everything else uses the fast RSS memory used estimation instead of raw memory reported by zmalloc(). This means that setting maxmemory to 2GB will really have the effect of using up to 2GB of memory. 2010-11-02 11:50:55 +01:00
antirez
21dbc6499a merge conflict resolved 2010-10-28 22:59:47 +02:00
antirez
d08fac3eb9 more generally usable i/o functions moved to syncio.c 2010-10-25 10:53:28 +02:00
Pieter Noordhuis
75b41de8ca Convert objects in the command procs instead of the protocol code 2010-10-17 17:21:41 +02:00
Pieter Noordhuis
b04ce2a35c Merge master with resolved conflict in src/redis-cli.c 2010-10-13 18:55:46 +02:00
antirez
89f9f83769 Merge remote branch 'pietern/networking-perf' 2010-09-16 11:38:40 +02:00
antirez
412e457c27 fixed typo in the latest commit 2010-09-14 15:18:18 +02:00
antirez
1d18f50458 Advertise the existence of redis-check dump --fix when logging an error about corrupted AOF file 2010-09-14 15:09:37 +02:00
Pieter Noordhuis
3ab203762f Use specialized function to add status and error replies 2010-09-02 23:33:06 +02:00
Pieter Noordhuis
2403fc9fde Intialize bufpos in the fake client 2010-09-02 19:52:04 +02:00
Pieter Noordhuis
57b0738011 Don't build a reply when replaying the AOF 2010-08-30 16:51:39 +02:00
antirez
23c64fe50d translated a few long logn into int64_t for correctness and to avoid compilation warnings as well 2010-08-26 18:11:26 +02:00
Pieter Noordhuis
aaada3f962 Merge branch 'master' into intset-split 
Conflicts:
	src/Makefile
	src/t_set.c
 2010-08-20 12:40:55 +02:00
Pieter Noordhuis
a5639e7dd9 Change initialization to allow listening on both a port and socket 2010-08-03 13:33:12 +02:00
antirez
b67d234563 Fixed a crash loading the AOF file containing MULTI/EXEC, a result of WATCH implementation. Test needed... 2010-07-05 20:06:54 +02:00
Pieter Noordhuis
2767f1c0c6 fix aof and digest code to work with dual set encoding 2010-07-02 20:42:20 +02:00
antirez
3688d7f308 Compilation fixed on Linux after the source code split 2010-07-01 21:13:38 +02:00
antirez
5bd09cd4c5 Fix the AOF fix of the latest commit ;) 2010-07-01 20:18:48 +02:00
antirez
daf2049d0d fixed error code checking for *write operations and return value in AOF rewriting function 2010-07-01 20:13:33 +02:00
antirez
e2641e09cc redis.c split into many different C files. 
networking related stuff moved into networking.c

moved more code

more work on layout of source code

SDS instantaneuos memory saving. By Pieter and Salvatore at VMware ;)

cleanly compiling again after the first split, now splitting it in more C files

moving more things around... work in progress

split replication code

splitting more

Sets split

Hash split

replication split

even more splitting

more splitting

minor change
 2010-07-01 14:38:51 +02:00