Commit Graph

381 Commits

Author SHA1 Message Date
antirez
ff024e3064 Log RDB deletion in persistence-less instances. 2020-03-04 11:19:55 +01:00
antirez
f0acdee4c5 Introduce bg_unlink(). 2020-03-04 11:10:54 +01:00
antirez
5e2319c326 Remove RDB files used for replication in persistence-less instances. 2020-03-03 14:58:15 +01:00
Hengjian Tang
1ab5508324 modify the read buf size according to the write buf size PROTO_IOBUF_LEN defined before 2020-02-25 15:55:28 +08:00
Salvatore Sanfilippo
9c00bdd86e
Merge pull request #6822 from guybe7/diskless_load_module_hook_fix
Diskless-load emptyDb-related fixes
2020-02-06 13:10:00 +01:00
Guy Benoish
92dc5e1fa4 Diskless-load emptyDb-related fixes
1. Call emptyDb even in case of diskless-load: We want modules
   to get the same FLUSHDB event as disk-based replication.
2. Do not fire any module events when flushing the backups array.
3. Delete redundant call to signalFlushedDb (Called from emptyDb).
2020-02-06 16:48:02 +05:30
Salvatore Sanfilippo
5558c0e4cf
Merge pull request #6848 from oranagra/opt_use_diskless_load_calls
reduce repeated calls to use_diskless_load
2020-02-06 10:30:39 +01:00
Oran Agra
aac6a4cf13 move restartAOFAfterSYNC from replicaofCommand to replicationUnsetMaster
replicationUnsetMaster can be called from other places, not just
replicaofCOmmand, and all of these need to restart AOF
2020-02-06 10:14:32 +02:00
Oran Agra
485d5d4a18 reduce repeated calls to use_diskless_load
this function possibly iterates on the module list
2020-02-06 09:41:45 +02:00
ShooterIT
2bc8db9ca5 Rename rdb asynchronously 2019-12-31 21:45:32 +08:00
Johannes Truschnigg
dfb598cf33 Signal systemd readiness atfer Partial Resync
"Partial Resynchronization" is a special variant of replication success
that we have to tell systemd about if it is managing redis-server via a
Type=Notify service unit.
2019-12-19 21:47:24 +01:00
Johannes Truschnigg
641c64ada1 Use libsystemd's sd_notify for communicating redis status to systemd
Instead of replicating a subset of libsystemd's sd_notify(3) internally,
use the dynamic library provided by systemd to communicate with the
service manager.

When systemd supervision was auto-detected or configured, communicate
the actual server status (i.e. "Loading dataset", "Waiting for
master<->replica sync") to systemd, instead of declaring readiness right
after initializing the server process.
2019-11-19 18:55:44 +02:00
Oran Agra
51c3ff8d75 Modules hooks: complete missing hooks for the initial set of hooks
* replication hooks: role change, master link status, replica online/offline
* persistence hooks: saving, loading, loading progress
* misc hooks: cron loop, shutdown, module loaded/unloaded
* change the way hooks test work, and add tests for all of the above

startLoading() now gets flag indicating what is loaded.
stopLoading() now gets an indication of success or failure.
adding startSaving() and stopSaving() with similar args and role.
2019-10-29 17:59:09 +02:00
Wander Hillen
fb1f4f4e7e
Merge branch 'unstable' into minor-typos 2019-10-25 10:18:26 +02:00
Yossi Gottlieb
0db3b0a0ff Merge remote-tracking branch 'upstream/unstable' into tls 2019-10-16 17:08:07 +03:00
Salvatore Sanfilippo
8e3d910d4e
Merge pull request #6429 from charsyam/feature/typo-slave
[trivial] fix typos salves to slaves in replication.c
2019-10-10 14:56:43 +02:00
antirez
747be463d2 Cluster: fix memory leak of cached master.
This is what happened:

1. Instance starts, is a slave in the cluster configuration, but
actually server.masterhost is not set, so technically the instance
is acting like a master.

2. loadDataFromDisk() calls replicationCacheMasterUsingMyself() even if
the instance is a master, in the case it is logically a slave and the
cluster is enabled. So now we have a cached master even if the instance
is practically configured as a master (from the POV of
server.masterhost value and so forth).

3. clusterCron() sees that the instance requires to replicate from its
master, because logically it is a slave, so it calls
replicationSetMaster() that will in turn call
replicationCacheMasterUsingMyself(): before this commit, this call would
overwrite the old cached master, creating a memory leak.
2019-10-10 10:23:34 +02:00
Yossi Gottlieb
61733ded14 TLS: Configuration options.
Add configuration options for TLS protocol versions, ciphers/cipher
suites selection, etc.
2019-10-07 21:07:27 +03:00
Oran Agra
5a47794606 diskless replication rdb transfer uses pipe, and writes to sockets form the parent process.
misc:
- handle SSL_has_pending by iterating though these in beforeSleep, and setting timeout of 0 to aeProcessEvents
- fix issue with epoll signaling EPOLLHUP and EPOLLERR only to the write handlers. (needed to detect the rdb pipe was closed)
- add key-load-delay config for testing
- trim connShutdown which is no longer needed
- rioFdsetWrite -> rioFdWrite - simplified since there's no longer need to write to multiple FDs
- don't detect rdb child exited (don't call wait3) until we detect the pipe is closed
- Cleanup bad optimization from rio.c, add another one
2019-10-07 21:06:30 +03:00
Yossi Gottlieb
b087dd1db6 TLS: Connections refactoring and TLS support.
* Introduce a connection abstraction layer for all socket operations and
integrate it across the code base.
* Provide an optional TLS connections implementation based on OpenSSL.
* Pull a newer version of hiredis with TLS support.
* Tests, redis-cli updates for TLS support.
2019-10-07 21:06:13 +03:00
charsyam
bea0384f5d fix type salves to slaves 2019-10-07 23:48:11 +09:00
antirez
de1f82aa33 Function renamed hasForkChild() -> hasActiveChildProcess(). 2019-09-27 12:03:09 +02:00
Salvatore Sanfilippo
6129758558
Merge branch 'unstable' into modules_fork 2019-09-27 11:24:06 +02:00
Salvatore Sanfilippo
959fb5cf68
Merge pull request #6235 from oranagra/module_rdb_load_errors
Allow modules to handle RDB loading errors.
2019-09-26 11:52:42 +02:00
antirez
0e9b5adbd3 Replication: clarify why repl_put_online_on_ack exists at all. 2019-08-05 17:38:15 +02:00
Oran Agra
40c4183196 Log message when modules prevent diskless-load 2019-07-30 16:32:58 +03:00
Oran Agra
4339706e07 Avoid diskelss-load if modules did not declare they handle read errors 2019-07-30 15:11:57 +03:00
Oran Agra
56258c6b7d Module API for Forking
* create module API for forking child processes.
* refactor duplicate code around creating and tracking forks by AOF and RDB.
* child processes listen to SIGUSR1 and dies exitFromChild in order to
  eliminate a valgrind warning of unhandled signal.
* note that BGSAVE error reply has changed.

valgrind error is:
  Process terminating with default action of signal 10 (SIGUSR1)
2019-07-17 16:40:24 +03:00
antirez
7c9f6ebc8f Diskless replica: fix disklessLoadRestoreBackups() bug. 2019-07-10 12:36:26 +02:00
antirez
3bbb9a1413 Diskless replica: refactoring of DBs backups. 2019-07-10 11:42:26 +02:00
antirez
81b18fa3a0 Diskless replica: a few aesthetic changes to replication.c. 2019-07-08 18:32:47 +02:00
Oran Agra
2de544cfcc diskless replication on slave side (don't store rdb to file), plus some other related fixes
The implementation of the diskless replication was currently diskless only on the master side.
The slave side was still storing the received rdb file to the disk before loading it back in and parsing it.

This commit adds two modes to load rdb directly from socket:
1) when-empty
2) using "swapdb"
the third mode of using diskless slave by flushdb is risky and currently not included.

other changes:
--------------
distinguish between aof configuration and state so that we can re-enable aof only when sync eventually
succeeds (and not when exiting from readSyncBulkPayload after a failed attempt)
also a CONFIG GET and INFO during rdb loading would have lied

When loading rdb from the network, don't kill the server on short read (that can be a network error)

Fix rdb check when performed on preamble AOF

tests:
run replication tests for diskless slave too
make replication test a bit more aggressive
Add test for diskless load swapdb
2019-07-08 15:37:48 +03:00
antirez
074d24df1e Narrow the effects of PR #6029 to the exact state.
CLIENT PAUSE may be used, in other contexts, for a long time making all
the slaves time out. Better for now to be more specific about what
should disable senidng PINGs.

An alternative to that would be to virtually refresh the slave
interactions when clients are paused, however for now I went for this
more conservative solution.
2019-05-15 12:16:43 +02:00
Salvatore Sanfilippo
caf74e507e
Merge pull request #6029 from chendq8/clientpause
fix cluster failover time out
2019-05-15 12:03:19 +02:00
chendianqiang
11f2c6b115 stop ping when client pause 2019-04-17 21:20:10 +08:00
Salvatore Sanfilippo
fcac342955
Merge pull request #3830 from oranagra/diskless_capa_pr
several bugfixes to diskless replication
2019-03-22 17:41:40 +01:00
antirez
b3408e9a9b More sensible name for function: restartAOFAfterSYNC().
Related to #3829.
2019-03-21 17:21:29 +01:00
antirez
9588fd52ac Mostly aesthetic changes to restartAOF().
See #3829.
2019-03-21 17:18:24 +01:00
Oran Agra
b2e03f8329 diskless replication - notify slave when rdb transfer failed
in diskless replication - master was not notifing the slave that rdb transfer
terminated on error, and lets slave wait for replication timeout
2019-03-20 17:46:19 +02:00
oranagra
c9e2900efc bugfix to restartAOF, exit will never happen since retry will get negative.
also reduce an excess sleep
2019-03-20 17:20:07 +02:00
antirez
14b17c3615 replicaofCommand() refactoring: stay into 80 cols. 2019-03-18 11:34:40 +01:00
antirez
8a46d32be2 Make comment in #5911 stay inside 80 cols. 2019-03-10 09:48:06 +01:00
John Sully
5b52bc738b Replicas aren't allowed to run the replicaof command 2019-03-09 11:04:48 -05:00
zhaozhao.zz
ea9d3aefec ACL: add masteruser configuration for replication
In mostly production environment, normal user's behavior should be
limited.

Now in redis ACL mechanism we can do it like that:

    user default on +@all ~* -@dangerous nopass
    user admin on +@all ~* >someSeriousPassword

Then the default normal user can not execute dangerous commands like
FLUSHALL/KEYS.

But some admin commands are in dangerous category too like PSYNC,
and the configurations above will forbid replica from sync with master.

Finally I think we could add a new configuration for replication,
it is masteruser option, like this:

    masteruser admin
    masterauth someSeriousPassword

Then replica will try AUTH admin someSeriousPassword and get privilege
to execute PSYNC. If masteruser is NULL, replica would AUTH with only
masterauth like before.
2019-02-12 17:12:37 +08:00
ArkayZheng
76f20729fc Fix the output bug in rename exceptions. 2019-01-25 21:48:23 +08:00
antirez
4dc69497f5 Refactoring: always kill AOF/RDB child via helper functions. 2019-01-21 11:28:44 +01:00
Salvatore Sanfilippo
adfaf548e3
Merge branch 'unstable' into fixChildInfoPipeFdLeak 2019-01-21 11:20:56 +01:00
Salvatore Sanfilippo
9f939610f3
Merge pull request #5797 from trevor211/fixUpdateDictResizePolicy
Fix update dict resize policy
2019-01-21 11:14:48 +01:00
WuYunlong
440385de14 Fix child info pipe fd leak when child process gets killed. 2019-01-21 17:48:45 +08:00
WuYunlong
f004a3e7ff Update dict resize policy when rdb child process gets killed. 2019-01-21 17:33:18 +08:00
antirez
2c66c525f9 ACL: configure the master connection without user. 2019-01-17 18:33:36 +01:00
antirez
709a6612eb RESP3: addReplyString() -> addReplyProto().
The function naming was totally nuts. Let's fix it as we break PRs
anyway with RESP3 refactoring and changes.
2019-01-09 17:00:30 +01:00
antirez
07bce54093 RESP3: Use new deferred len API in replication.c. 2019-01-09 17:00:29 +01:00
antirez
06a4acb7d3 When replica kills a pending RDB save during SYNC, log it.
This logs what happens in the context of the fix in PR #5367.
2018-10-31 11:47:10 +01:00
Salvatore Sanfilippo
6204d8c139
Merge pull request #5367 from nUl1/fullresync-stopbgsave
Prevent RDB autosave from overwriting full resync results
2018-10-31 11:42:04 +01:00
antirez
3d07ed983e Fix typo in replicationCron() comment. 2018-10-05 18:30:45 +02:00
Andrey Bugaevskiy
466c277b4f Move child termination to readSyncBulkPayload 2018-09-27 19:38:58 +03:00
Andrey Bugaevskiy
98a64523c4 Prevent RDB autosave from overwriting full resync results
During the full database resync we may still have unsaved changes
on the receiving side. This causes a race condition between
synced data rename/load and the rename of rdbSave tempfile.
2018-09-19 19:58:39 +03:00
antirez
61b7a176ef Slave removal: replication.c logs fixed. 2018-09-11 15:32:28 +02:00
antirez
ef2c7a5bbb Slave removal: SLAVEOF -> REPLICAOF. SLAVEOF is now an alias. 2018-09-11 15:32:28 +02:00
Oran Agra
d55598988b fix rare replication stream corruption with disk-based replication
The slave sends \n keepalive messages to the master while parsing the rdb,
and later sends REPLCONF ACK once a second. rarely, the master recives both
a linefeed char and a REPLCONF in the same read, \n*3\r\n$8\r\nREPLCONF\r\n...
and it tries to trim two chars (\r\n) from the query buffer,
trimming the '*' from *3\r\n$8\r\nREPLCONF\r\n...

then the master tries to process a command starting with '3' and replies to
the slave a bunch of -ERR and one +OK.
although the slave silently ignores these (prints a log message), this corrupts
the replication offset at the slave since the slave increases the replication
offset, and the master did not.

other than the fix in processInlineBuffer, i did several other improvments
while hunting this very rare bug.

- when redis replies with "unknown command" it includes a portion of the
  arguments, not just the command name. so it would be easier to understand
  what was recived, in my case, on the slave side,  it was -ERR, but
  the "arguments" were the interesting part (containing info on the error).
- about a year ago i added code in addReplyErrorLength to print the error to
  the log in case of a reply to master (since this string isn't actually
  trasmitted to the master), now changed that block to print a similar log
  message to indicate an error being sent from the master to the slave.
  note that the slave is marked as CLIENT_SLAVE only after PSYNC was received,
  so this will not cause any harm for REPLCONF, and will only indicate problems
  that are gonna corrupt the replication stream anyway.
- two places were c->reply was emptied, and i wanted to reset sentlen
  this is a precaution (i did not actually see such a problem), since a
  non-zero sentlen will cause corruption to be transmitted on the socket.
2018-07-17 12:51:49 +03:00
Oran Agra
bf680b6f8c slave buffers were wasteful and incorrectly counted causing eviction
A) slave buffers didn't count internal fragmentation and sds unused space,
   this caused them to induce eviction although we didn't mean for it.

B) slave buffers were consuming about twice the memory of what they actually needed.
- this was mainly due to sdsMakeRoomFor growing to twice as much as needed each time
  but networking.c not storing more than 16k (partially fixed recently in 237a38737).
- besides it wasn't able to store half of the new string into one buffer and the
  other half into the next (so the above mentioned fix helped mainly for small items).
- lastly, the sds buffers had up to 30% internal fragmentation that was wasted,
  consumed but not used.

C) inefficient performance due to starting from a small string and reallocing many times.

what i changed:
- creating dedicated buffers for reply list, counting their size with zmalloc_size
- when creating a new reply node from, preallocate it to at least 16k.
- when appending a new reply to the buffer, first fill all the unused space of the
  previous node before starting a new one.

other changes:
- expose mem_not_counted_for_evict info field for the benefit of the test suite
- add a test to make sure slave buffers are counted correctly and that they don't cause eviction
2018-07-16 16:43:42 +03:00
Jack Drogon
93238575f7 Fix typo 2018-07-03 18:19:46 +02:00
antirez
677d10b2a8 Set repl_down_since to zero on state change.
PR #5081 fixes an "interesting" bug about Redis Cluster failover but in
general about the updating of repl_down_since, that is used in order to
count the time a slave was left disconnected from its master.

While the fix provided resolves the specific issue, in general the
validity of repl_down_since is limited to states that are different
than the state CONNECTED, and the disconnected time is set when the
state is DISCONNECTED. However from CONNECTED to other states, the state
machine must always go to DISCONNECTED first. So it makes sense to set
the field to zero (since it is meaningless in that context) when the
state is set to CONNECTED.
2018-07-03 12:42:14 +02:00
WuYunlong
2e167f7d0e fix server.repl_down_since resetting, so that slaves could failover
automatically as expected.
2018-06-30 09:39:08 +08:00
antirez
27178a3fde Fix type of argslen in sendSynchronousCommand().
Related to #5037.
2018-06-26 14:38:35 +02:00
antirez
1f1e724f47 Remove black space. 2018-06-26 14:37:22 +02:00
Madelyn Olson
45731edc4b Addressed comments 2018-06-26 00:57:35 +00:00
Madelyn Olson
e8d68b6b72 Fixed replication authentication with whitespace in password 2018-06-26 00:48:37 +00:00
shenlongxing
c85ae56edc Fix write() errno error 2018-06-06 13:06:42 +02:00
Wander Hillen
dcffca0a31 Fix typos, add some periods 2018-03-16 09:59:14 +01:00
Salvatore Sanfilippo
4aa2ecd98b
Merge pull request #4269 from jianqingdu/unstable
fix not call va_end() when syncWrite() failed
2018-01-24 10:55:25 +01:00
antirez
b23927b240 Hopefully more clear comment to explain the change in #4607. 2018-01-16 15:52:13 +01:00
Oran Agra
689b64c3ad PSYNC2 fix - promoted slave should hold on to it's backlog
after a slave is promoted (assuming it has no slaves
and it booted over an hour ago), it will lose it's replication
backlog at the next replication cron, rather than waiting for slaves
to connect to it.
so on a simple master/slave faiover, if the new slave doesn't connect
immediately, it may be too later and PSYNC2 will fail.
2018-01-16 10:10:42 +02:00
antirez
62a4b817c6 add linkClient(): adds the client and caches the list node.
We have this operation in two places: when caching the master and
when linking a new client after the client creation. By having an API
for this we avoid incurring in errors when modifying one of the two
places forgetting the other. The function is also a good place where to
document why we cache the linked list node.

Related to #4497 and #4210.
2017-12-05 16:02:03 +01:00
zhaozhao.zz
43be967690 networking: optimize unlinkClient() in freeClient() 2017-11-30 18:11:05 +08:00
antirez
4d063bb6ba PSYNC2: reorganize comments related to recent fixes.
Related to PR #4412 and issue #4407.
2017-11-24 11:08:29 +01:00
zhaozhao.zz
6ddf0ea293 PSYNC2: safe free backlog when reach the time limit
When we free the backlog, we should use a new
replication ID and clear the ID2. Since without
backlog we can not increment master_repl_offset
even do write commands, that may lead to inconsistency
when we try to connect a "slave-before" master
(if this master is our slave before, our replid
equals the master's replid2). As the master have our
history, so we can match the master's replid2 and
second_replid_offset, that make partial sync work,
but the data is inconsistent.
2017-11-01 17:32:27 +08:00
antirez
bb3b5ddd19 PSYNC2: More refinements related to #4316. 2017-09-20 11:28:13 +02:00
zhaozhao.zz
b541ccef25 PSYNC2: make persisiting replication info more solid
This commit is a reinforcement of commit c1c99e9.

1. Replication information can be stored when the RDB file is
generated by a mater using server.slaveseldb when server.repl_backlog
is not NULL, or set repl_stream_db be -1. That's safe, because
NULL server.repl_backlog will trigger full synchronization,
then master will send SELECT command to replicaiton stream.
2. Only do rdbSave* when rsiptr is not NULL,
if we do rdbSave* without rdbSaveInfo, slave will miss repl-stream-db.
3. Save the replication informations also in the case of
SAVE command, FLUSHALL command and DEBUG reload.
2017-09-20 11:18:10 +02:00
antirez
c1c99e9f4e PSYNC2: Fix the way replication info is saved/loaded from RDB.
This commit attempts to fix a number of bugs reported in #4316.
They are related to the way replication info like replication ID,
offsets, and currently selected DB in the master client, are stored
and loaded by Redis. In order to avoid inconsistencies the changes in
this commit try to enforce that:

1. Replication information are only stored when the RDB file is
generated by a slave that has a valid 'master' client, so that we can
always extract the currently selected DB.
2. When replication informations are persisted in the RDB file, all the
info for a successful PSYNC or nothing is persisted.
3. The RDB replication informations are only loaded if the instance is
configured as a slave, otherwise a master can start with IDs that relate
to a different history of the data set, and stil retain such IDs in the
future while receiving unrelated writes.
2017-09-19 23:03:39 +02:00
antirez
b75ae0bbea PSYNC2: Create backlog on slave partial sync as well.
A slave may be started with an RDB file able to provide enough slave to
perform a successful partial SYNC with its master. However in such a
case, how outlined in issue #4268, the slave backlog will not be
started, since it was only initialized on full syncs attempts. This
creates different problems with successive PSYNC attempts that will
always result in full synchronizations.

Thanks to @fdingiit for discovering the issue.
2017-09-19 10:33:14 +02:00
jianqingdu
498f65ffb7 fix not call va_end when syncWrite() failed
fix not call va_end when syncWrite() failed in sendSynchronousCommand()
2017-08-30 21:20:14 -05:00
antirez
469d6e2b37 PSYNC2: fix master cleanup when caching it.
The master client cleanup was incomplete: resetClient() was missing and
the output buffer of the client was not reset, so pending commands
related to the previous connection could be still sent.

The first problem caused the client argument vector to be, at times,
half populated, so that when the correct replication stream arrived the
protcol got mixed to the arugments creating invalid commands that nobody
called.

Thanks to @yangsiran for also investigating this problem, after
already providing important design / implementation hints for the
original PSYNC2 issues (see referenced Github issue).

Note that this commit adds a new function to the list library of Redis
in order to be able to reset a list without destroying it.

Related to issue #3899.
2017-04-27 17:08:37 +02:00
antirez
189a12afb4 PSYNC2: discard pending transactions from cached master.
During the review of the fix for #3899, @yangsiran identified an
implementation bug: given that the offset is now relative to the applied
part of the replication log, when we cache a master, the successive
PSYNC2 request will be made in order to *include* the transaction that
was not completely processed. This means that we need to discard any
pending transaction from our replication buffer: it will be re-executed.
2017-04-19 14:02:52 +02:00
antirez
22be435efe Fix PSYNC2 incomplete command bug as described in #3899.
This bug was discovered by @kevinmcgehee and constituted a major hidden
bug in the PSYNC2 implementation, caused by the propagation from the
master of incomplete commands to slaves.

The bug had several results:

1. Borrowing from Kevin text in the issue: "Given that slaves blindly
copy over their master's input into their own replication backlog over
successive read syscalls, it's possible that with large commands or
small TCP buffers, partial commands are present in this buffer. If the
master were to fail before successfully propagating the entire command
to a slave, the slaves will never execute the partial command (since the
client is invalidated) but will copy it to replication backlog which may
relay those invalid bytes to its slaves on PSYNC2, corrupting the
backlog and possibly other valid commands that follow the failover.
Simple command boundaries aren't sufficient to capture this, either,
because in the case of a MULTI/EXEC block, if the master successfully
propagates a subset of the commands but not the EXEC, then the
transaction in the backlog becomes corrupt and could corrupt other
slaves that consume this data."

2. As identified by @yangsiran later, there is another effect of the
bug. For the same mechanism of the first problem, a slave having another
slave, could receive a full resynchronization request with an already
half-applied command in the backlog. Once the RDB is ready, it will be
sent to the slave, and the replication will continue sending to the
sub-slave the other half of the command, which is not valid.

The fix, designed by @yangsiran and @antirez, and implemented by
@antirez, uses a secondary buffer in order to feed the sub-masters and
update the replication backlog and offsets, only when a given part of
the query buffer is actually *applied* to the state of the instance,
that is, when the command gets processed and the command is not pending
in the Redis transaction buffer because of CLIENT_MULTI state.

Given that now the backlog and offsets representation are in agreement
with the actual processed commands, both issue 1 and 2 should no longer
be possible.

Thanks to @kevinmcgehee, @yangsiran and @oranagra for their work in
identifying and designing a fix for this problem.
2017-04-19 10:25:45 +02:00
antirez
104584b95e Fix typo in feedReplicationBacklog() top comment. 2017-04-12 12:28:05 +02:00
antirez
76d87f47c7 Don't leak file descriptor on syncWithMaster().
Close #3804.
2017-02-20 10:18:41 +01:00
antirez
8e390a62ad Hopefully improve code comments for issue #3616.
This commit also contains other changes in order to conform the code to
the Redis core style, specifically 80 chars max per line, smart
conditionals in the same line:

    if (that) do_this();
2016-12-16 17:48:38 +01:00
Salvatore Sanfilippo
ca4ca5073e Merge pull request #3616 from oranagra/stop_aofrw_before_rdbload
CoW improvement, stop AOFRW before flushing and parsing slave RDB
2016-12-16 17:43:20 +01:00
antirez
434e6b2da3 PSYNC2: Do not accept WAIT in slave instances.
No longer makes sense since writable slaves only do local writes now:
writes are no longer passed to sub-slaves in the stream.
2016-12-02 10:21:20 +01:00
antirez
6eb720ff2d PSYNC2: Minor memory leak reading -NOMASTERLINK master reply fixed. 2016-11-29 10:25:00 +01:00
antirez
eab865a0a1 PSYNC2: stop sending newlines to sub-slaves when master is down.
This actually includes two changes:

1) No newlines to take the master-slave link up when the upstream master
is down. Doing this is dangerous because the sub-slave often is received
replication protocol for an half-command, so can't receive newlines
without desyncing the replication link, even with the code in order to
cancel out the bytes that PSYNC2 was using. Moreover this is probably
also not needed/sane, because anyway the slave can keep serving
requests, and because if it's configured to don't serve stale data, it's
a good idea, actually, to break the link.

2) When a +CONTINUE with a different ID is received, we now break
connection with the sub-slaves: they need to be notified as well. This
was part of the original specification but for some reason it was not
implemented in the code, and was alter found as a PSYNC2 bug in the
integration testing.
2016-11-28 17:54:04 +01:00
antirez
e09e31b12e PSYNC2: on transient error jump to error, not write_error. 2016-11-24 15:48:18 +01:00
antirez
5b7d42fff3 PSYNC2: bugfixing pre release.
1. Master replication offset was cleared after switching configuration
to some other slave, since it was assumed you can't PSYNC after a
switch. Note the case anymore and when we successfully PSYNC we need to
have our offset untouched.

2. Secondary replication ID was not reset to "000..." pattern at
startup.

3. Master in error state replying -LOADING or other transient errors
forced the slave to discard the cached master and full resync. This is
now fixed.

4. Better logging of what's happening on failed PSYNCs.
2016-11-23 17:36:45 +01:00
Salvatore Sanfilippo
5b83fa482c Merge pull request #3612 from deep011/unstable
fix a possible bug for 'replconf getack'
2016-11-18 10:45:09 +01:00
oranagra
e3a61950a2 when a slave loads an RDB, stop an AOFRW fork before flusing db and parsing rdb file, to avoid a CoW disaster. 2016-11-16 21:30:59 +02:00
deep011
13a92a5bb1 fix a possible bug for 'replconf getack' 2016-11-16 11:04:33 +08:00
antirez
28c96d73b2 PSYNC2: Save replication ID/offset on RDB file.
This means that stopping a slave and restarting it will still make it
able to PSYNC with the master. Moreover the master itself will retain
its ID/offset, in case it gets turned into a slave, or if a slave will
try to PSYNC with it with an exactly updated offset (otherwise there is
no backlog).

This change was possible thanks to PSYNC v2 that makes saving the current
replication state much simpler.
2016-11-10 12:35:29 +01:00
antirez
4e5e366ed2 PSYNC2: Wrap debugging code with if(0) 2016-11-09 15:37:15 +01:00
antirez
2669fb8364 PSYNC2: different improvements to Redis replication.
The gist of the changes is that now, partial resynchronizations between
slaves and masters (without the need of a full resync with RDB transfer
and so forth), work in a number of cases when it was impossible
in the past. For instance:

1. When a slave is promoted to mastrer, the slaves of the old master can
partially resynchronize with the new master.

2. Chained slalves (slaves of slaves) can be moved to replicate to other
slaves or the master itsef, without requiring a full resync.

3. The master itself, after being turned into a slave, is able to
partially resynchronize with the new master, when it joins replication
again.

In order to obtain this, the following main changes were operated:

* Slaves also take a replication backlog, not just masters.

* Same stream replication for all the slaves and sub slaves. The
replication stream is identical from the top level master to its slaves
and is also the same from the slaves to their sub-slaves and so forth.
This means that if a slave is later promoted to master, it has the
same replication backlong, and can partially resynchronize with its
slaves (that were previously slaves of the old master).

* A given replication history is no longer identified by the `runid` of
a Redis node. There is instead a `replication ID` which changes every
time the instance has a new history no longer coherent with the past
one. So, for example, slaves publish the same replication history of
their master, however when they are turned into masters, they publish
a new replication ID, but still remember the old ID, so that they are
able to partially resynchronize with slaves of the old master (up to a
given offset).

* The replication protocol was slightly modified so that a new extended
+CONTINUE reply from the master is able to inform the slave of a
replication ID change.

* REPLCONF CAPA is used in order to notify masters that a slave is able
to understand the new +CONTINUE reply.

* The RDB file was extended with an auxiliary field that is able to
select a given DB after loading in the slave, so that the slave can
continue receiving the replication stream from the point it was
disconnected without requiring the master to insert "SELECT" statements.
This is useful in order to guarantee the "same stream" property, because
the slave must be able to accumulate an identical backlog.

* Slave pings to sub-slaves are now sent in a special form, when the
top-level master is disconnected, in order to don't interfer with the
replication stream. We just use out of band "\n" bytes as in other parts
of the Redis protocol.

An old design document is available here:

https://gist.github.com/antirez/ae068f95c0d084891305

However the implementation is not identical to the description because
during the work to implement it, different changes were needed in order
to make things working well.
2016-11-09 15:37:15 +01:00
charsyam
ca6fc4f031 Simple change just using slaves instead of server.slaves 2016-09-24 15:53:57 +09:00
Qu Chen
d982f44372 Fix a bug to delay bgsave while AOF rewrite in progress for replication 2016-08-02 10:44:33 +02:00
antirez
55385f99de Ability of slave to announce arbitrary ip/port to master.
This feature is useful, especially in deployments using Sentinel in
order to setup Redis HA, where the slave is executed with NAT or port
forwarding, so that the auto-detected port/ip addresses, as listed in
the "INFO replication" output of the master, or as provided by the
"ROLE" command, don't match the real addresses at which the slave is
reachable for connections.
2016-07-27 17:32:15 +02:00
antirez
03f5b508e5 Replication: when possible start RDB saving ASAP.
In a previous commit the replication code was changed in order to
centralize the BGSAVE for replication trigger in replicationCron(),
however after further testings, the 1 second delay imposed by this
change is not acceptable.

So now the BGSAVE is only delayed if the AOF rewriting process is
active. However past comments made sure that replicationCron() is always
able to trigger the BGSAVE when needed, making the code generally more
robust.

The new code is more similar to the initial @oranagra patch where the
BGSAVE was delayed only if an AOF rewrite was in progress.

Trivia: delaying the BGSAVE uncovered a minor Sentinel issue that is now
fixed.
2016-07-22 17:03:18 +02:00
antirez
780a8b1d76 Replication: start BGSAVE for replication always in replicationCron().
This makes the replication code conceptually simpler by removing the
synchronous BGSAVE trigger in syncCommand(). This also means that
socket and disk BGSAVE targets are handled by the same code.
2016-07-21 12:10:56 +02:00
antirez
acc2336fd1 Centralize slave replication handshake aborting.
Now we have a single function to call in any state of the slave
handshake, instead of using different functions for different states
which is error prone. Change performed in the context of issue #2479 but
does not fix it, since should be functionally identical to the past.
Just an attempt to make replication.c simpler to follow.
2015-12-03 10:38:56 +01:00
antirez
ed6228851c PR 2813 fix ported to unstable. 2015-10-15 10:20:09 +02:00
antirez
252cfa0a39 Lazyfree: cond vars to enabled/disable it based on DEL context. 2015-10-02 15:27:57 +02:00
antirez
c69c6c80fb Lazyfree: ability to free whole DBs in background. 2015-10-01 13:02:26 +02:00
antirez
1e7153831d Refactoring: unlinkClient() added to lower freeClient() complexity. 2015-09-30 17:10:03 +02:00
antirez
fdb3be939e Refactoring: new function to test if client has pending output. 2015-09-30 16:41:48 +02:00
antirez
1c7d87df0c Avoid installing the client write handler when possible. 2015-09-30 16:29:41 +02:00
antirez
d036abe27d Log client details on SLAVEOF command having an effect. 2015-08-21 15:29:07 +02:00
antirez
f18e5b634d startBgsaveForReplication(): handle waiting slaves state change.
Before this commit, after triggering a BGSAVE it was up to the caller of
startBgsavForReplication() to handle slaves in WAIT_BGSAVE_START in
order to update them accordingly. However when the replication target is
the socket, this is not possible since the process of updating the
slaves and sending the FULLRESYNC reply must be coupled with the process
of starting an RDB save (the reason is, we need to send the FULLSYNC
command and spawn a child that will start to send RDB data to the slaves
ASAP).

This commit moves the responsibility of handling slaves in
WAIT_BGSAVE_START to startBgsavForReplication() so that for both
diskless and disk-based replication we have the same chain of
responsiblity. In order accomodate such change, the syncCommand() also
needs to put the client in the slave list ASAP (just after the initial
checks) and not at the end, so that startBgsavForReplication() can find
the new slave alrady in the list.

Another related change is what happens if the BGSAVE fails because of
fork() or other errors: we now remove the slave from the list of slaves
and send an error, scheduling the slave connection to be terminated.

As a side effect of this change the following errors found by
Oran Agra are fixed (thanks!):

1. rdbSaveToSlavesSockets() on failed fork will get the slaves cleaned
up, otherwise they remain in a wrong state forever since we setup them
for full resync before actually trying to fork.

2. updateSlavesWaitingBgsave() with replication target set as "socket"
was broken since the function changed the slaves state from
WAIT_BGSAVE_START to WAIT_BGSAVE_END via
replicationSetupSlaveForFullResync(), so later rdbSaveToSlavesSockets()
will not find any slave in the right state (WAIT_BGSAVE_START) to feed.
2015-08-20 17:39:48 +02:00
antirez
bea1259190 slaveTryPartialResynchronization and syncWithMaster: better synergy.
It is simpler if removing the read event handler from the FD is up to
slaveTryPartialResynchronization, after all it is only called in the
context of syncWithMaster.

This commit also makes sure that on error all the event handlers are
removed from the socket before closing it.
2015-08-07 12:04:37 +02:00
antirez
88c716a0f5 syncWithMaster(): non blocking state machine. 2015-08-06 18:12:20 +02:00
antirez
ce5761e061 startBgsaveForReplication(): log what you really do. 2015-08-06 09:49:38 +02:00
antirez
3e6d4d599a Replication: add REPLCONF CAPA EOF support.
Add the concept of slaves capabilities to Redis, the slave now presents
to the Redis master with a set of capabilities in the form:

    REPLCONF capa SOMECAPA capa OTHERCAPA ...

This has the effect of setting slave->slave_capa with the corresponding
SLAVE_CAPA macros that the master can test later to understand if it
the slave will understand certain formats and protocols of the
replication process. This makes it much simpler to introduce new
replication capabilities in the future in a way that don't break old
slaves or masters.

This patch was designed and implemented together with Oran Agra
(@oranagra).
2015-08-06 09:23:23 +02:00
antirez
55ba772703 Fix replication slave pings period.
For PINGs we use the period configured by the user, but for the newlines
of slaves waiting for an RDB to be created (including slaves waiting for
the FULLRESYNC reply) we need to ping with frequency of 1 second, since
the timeout is fixed and needs to be refreshed.
2015-08-05 16:49:16 +02:00
antirez
15de6b108b Make sure we re-emit SELECT after each new slave full sync setup.
In previous commits we moved the FULLRESYNC to the moment we start the
BGSAVE, so that the offset we provide is the right one. However this
also means that we need to re-emit the SELECT statement every time a new
slave starts to accumulate the changes.

To obtian this effect in a more clean way, the function that sends the
FULLRESYNC reply was overloaded with a more important role of also doing
this and chanigng the slave state. So it was renamed to
replicationSetupSlaveForFullResync() to better reflect what it does now.
2015-08-05 13:34:46 +02:00
antirez
a5a06a8ecd Don't send SELECT to slaves in WAIT_BGSAVE_START state. 2015-08-05 11:23:22 +02:00
antirez
62b5c60ead syncCommand() comments improved. 2015-08-05 08:41:57 +02:00
antirez
292fec058a PSYNC initial offset fix.
This commit attempts to fix a bug involving PSYNC and diskless
replication (currently experimental) found by Yuval Inbar from Redis Labs
and that was later found to have even more far reaching effects (the bug also
exists when diskstore is off).

The gist of the bug is that, a Redis master replies with +FULLRESYNC to
a PSYNC attempt that fails and requires a full resynchronization.
However, the baseline offset sent along with FULLRESYNC was always the
current master replication offset. This is not ok, because there are
many reasosn that may delay the RDB file creation. And... guess what,
the master offset we communicate must be the one of the time the RDB
was created. So for example:

1) When the BGSAVE for replication is delayed since there is one
   already but is not good for replication.
2) When the BGSAVE is not needed as we attach one currently ongoing.
3) When because of diskless replication the BGSAVE is delayed.

In all the above cases the PSYNC reply is wrong and the slave may
reconnect later claiming to need a wrong offset: this may cause
data curruption later.
2015-08-04 17:06:10 +02:00
antirez
c1e94b6b9c Force slaves to resync after unsuccessful PSYNC.
Using chained replication where C is slave of B which is in turn slave of
A, if B reconnects the replication link with A but discovers it is no
longer possible to PSYNC, slaves of B must be disconnected and PSYNC
not allowed, since the new B dataset may be completely different after
the synchronization with the master.

Note that there are varius semantical differences in the way this is
handled now compared to the past. In the past the semantics was:

1. When a slave lost connection with its master, disconnected the chained
slaves ASAP. Which is not needed since after a successful PSYNC with the
master, the slaves can continue and don't need to resync in turn.

2. However after a failed PSYNC the replication backlog was not reset, so a
slave was able to PSYNC successfully even if the instance did a full
sync with its master, containing now an entirely different data set.

Now instead chained slaves are not disconnected when the slave lose the
connection with its master, but only when it is forced to full SYNC with
its master. This means that if the slave having chained slaves does a
successful PSYNC all its slaves can continue without troubles.

See issue #2694 for more details.
2015-07-28 16:35:02 +02:00
antirez
278ea9d16b replicationHandleMasterDisconnection() belongs to replication.c. 2015-07-28 14:36:50 +02:00
antirez
32f80e2f1b RDMF: More consistent define names. 2015-07-27 14:37:58 +02:00
antirez
40eb548a80 RDMF: REDIS_OK REDIS_ERR -> C_OK C_ERR. 2015-07-26 23:17:55 +02:00
antirez
2d9e3eb107 RDMF: redisAssert -> serverAssert. 2015-07-26 15:29:53 +02:00
antirez
14ff572482 RDMF: OBJ_ macros for object related stuff. 2015-07-26 15:28:00 +02:00
antirez
554bd0e7bd RDMF: use client instead of redisClient, like Disque. 2015-07-26 15:20:52 +02:00
antirez
424fe9afd9 RDMF: redisLog -> serverLog. 2015-07-26 15:17:43 +02:00
antirez
cef054e868 RDMF (Redis/Disque merge friendlyness) refactoring WIP 1. 2015-07-26 15:17:18 +02:00
antirez
8366907bed Use best effort address binding to connect to the master
We usually want to reach the master using the address of the interface
Redis is bound to (via the "bind" config option). That's useful since
the master will get (and publish) the slave address getting the peer
name of the incoming socket connection from the slave.

However, when this is not possible, for example because the slave is
bound to the loopback interface but repliaces from a master accessed via
an external interface, we want to still connect with the master even
from a different interface: in this case it is not really important that
the master will provide any other address, while it is vital to be able
to replicate correctly.

Related to issues #2609 and #2612.
2015-06-11 14:34:38 +02:00
antirez
6c60526db9 Net: improve prepareClientToWrite() error handling and comments.
When we fail to setup the write handler it does not make sense to take
the client around, it is missing writes: whatever is a client or a slave
anyway the connection should terminated ASAP.

Moreover what the function does exactly with its return value, and in
which case the write handler is installed on the socket, was not clear,
so the functions comment are improved to make the goals of the function
more obvious.

Also related to #2485.
2015-04-01 10:07:45 +02:00
Oran Agra
159875b5a3 fixes to diskless replication.
master was closing the connection if the RDB transfer took long time.
and also sent PINGs to the slave before it got the initial ACK, in which case the slave wouldn't be able to find the EOF marker.
2015-03-31 23:42:08 +03:00
antirez
c3ad70901f Replication: disconnect blocked clients when switching to slave role.
Bug as old as Redis and blocking operations. It's hard to trigger since
only happens on instance role switch, but the results are quite bad
since an inconsistency between master and slave is created.

How to trigger the bug is a good description of the bug itself.

1. Client does "BLPOP mylist 0" in master.
2. Master is turned into slave, that replicates from New-Master.
3. Client does "LPUSH mylist foo" in New-Master.
4. New-Master propagates write to slave.
5. Slave receives the LPUSH, the blocked client get served.

Now Master "mylist" key has "foo", Slave "mylist" key is empty.

Highlights:

* At step "2" above, the client remains attached, basically escaping any
  check performed during command dispatch: read only slave, in that case.
* At step "5" the slave (that was the master), serves the blocked client
  consuming a list element, which is not consumed on the master side.

This scenario is technically likely to happen during failovers, however
since Redis Sentinel already disconnects clients using the CLIENT
command when changing the role of the instance, the bug is avoided in
Sentinel deployments.

Closes #2473.
2015-03-24 16:00:09 +01:00
antirez
c5dd686ecb Replication: put server.master client creation into separated function. 2015-02-04 11:26:20 +01:00
antirez
ce269ad3c5 AnetFormatIP(): renamed, commented, now sticks to IP:port format.
A few code style changes + consistent format: not nice for humans but
better for parsers.
2014-12-11 18:20:30 +01:00
Matt Stancliff
491881e13b Cleanup all IP formatting code
Instead of manually checking for strchr(n,':') everywhere,
we can use our new centralized IP formatting functions.
2014-12-11 10:12:18 -05:00
antirez
1b732c09d0 Network bandwidth tracking + refactoring.
Track bandwidth used by clients and replication (but diskless
replication is not tracked since the actual transfer happens in the
child process).

This includes a refactoring that makes tracking new instantaneous
metrics simpler.
2014-12-03 12:16:25 +01:00
antirez
bb7fea0d5c Diskless SYNC: fix RDB EOF detection.
RDB EOF detection was relying on the final part of the RDB transfer to
be a magic 40 bytes EOF marker. However as the slave is put online
immediately, and because of sockets timeouts, the replication stream is
actually contiguous with the RDB file.

This means that to detect the EOF correctly we should either:

1) Scan all the stream searching for the mark. Sucks CPU-wise.
2) Start to send the replication stream only after an acknowledge.
3) Implement a proper chunked encoding.

For now solution "2" was picked, so the master does not start to send
ASAP the stream of commands in the case of diskless replication. We wait
for the first REPLCONF ACK command from the slave, that certifies us
that the slave correctly loaded the RDB file and is ready to get more
data.
2014-11-11 17:12:12 +01:00
antirez
f5c6ebbfe3 Disconnect timedout slave: regression introduced with diskless repl. 2014-11-11 15:10:58 +01:00
Matt Stancliff
0014966c1e Networking: add more outbound IP binding fixes
Same as the original bind fixes (we just missed these the
first time around).

This helps Redis not automatically send
connections from the first IP on an interface if we are bound
to a specific IP address (e.g. with multiple IP aliases on one
interface, you want to send from _your_ IP, not from the first IP
on the interface).
2014-10-29 15:09:09 -04:00
antirez
9ec22d9223 Diskless replication: missing listRewind() added.
This caused BGSAVE to be triggered a second time without any need when
we switch from socket to disk target via the command

    CONFIG SET repl-diskless-sync no

and there is already a slave waiting for the BGSAVE to start.
Also comments clarified about what is happening.
2014-10-29 12:48:22 +01:00
antirez
4b8f4b90b9 Log slave ip:port in more log messages. 2014-10-27 12:30:07 +01:00
antirez
8a416ca46e Added a function to get slave name for logs. 2014-10-27 11:58:20 +01:00
antirez
a27befc495 Diskless replication: log BGSAVE delay only when it is non-zero. 2014-10-27 10:48:39 +01:00
antirez
707352439c Diskless sync delay is now configurable. 2014-10-27 10:36:30 +01:00
antirez
c4dbc7cdec Remove duplicated log message about starting BGSAVE. 2014-10-24 10:38:42 +02:00