This commit fixes a serious Lua scripting replication issue, described
by Github issue #1549. The root cause of the problem is that scripts
were put inside the script cache, assuming that slaves and AOF already
contained it, even if the scripts sometimes produced no changes in the
data set, and were not actaully propagated to AOF/slaves.
Example:
eval "if tonumber(KEYS[1]) > 0 then redis.call('incr', 'x') end" 1 0
Then:
evalsha <sha1 step 1 script> 1 0
At this step sha1 of the script is added to the replication script cache
(the script is marked as known to the slaves) and EVALSHA command is
transformed to EVAL. However it is not dirty (there is no changes to db),
so it is not propagated to the slaves. Then the script is called again:
evalsha <sha1 step 1 script> 1 1
At this step master checks that the script already exists in the
replication script cache and doesn't transform it to EVAL command. It is
dirty and propagated to the slaves, but they fail to evaluate the script
as they don't have it in the script cache.
The fix is trivial and just uses the new API to force the propagation of
the executed command regardless of the dirty state of the data set.
Thank you to @minus-infinity on Github for finding the issue,
understanding the root cause, and fixing it.
server.lua_time_start is expressed in milliseconds. Use mstime_t instead
of long long, and populate it with mstime() instead of ustime()/1000.
Functionally identical but more natural.
Multiple missing calls to lua_pop prevented the error handler function
pushed on the stack for lua_pcall() to be popped before returning,
causing a memory leak in almost all the code paths of EVAL (both
successful calls and calls returning errors).
This caused two issues: Lua leaking memory (and this was very visible
from INFO memory output, as the 'used_memory_lua' field reported an
always increasing amount of memory used), and as a result slower and
slower GC cycles resulting in all the CPU being used.
Thanks to Tanguy Le Barzic for noticing something was wrong with his 2.8
slave, and for creating a testing EC2 environment where I was able to
investigate the issue.
The old REDIS_CMD_FORCE_REPLICATION flag was removed from the
implementation of Redis, now there is a new API to force specific
executions of a command to be propagated to AOF / Replication link:
void forceCommandPropagation(int flags);
The new API is also compatible with Lua scripting, so a script that will
execute commands that are forced to be propagated, will also be
propagated itself accordingly even if no change to data is operated.
As a side effect, this new design fixes the issue with scripts not able
to propagate PUBLISH to slaves (issue #873).
1) The event handler was no restored after a timeout condition if the
command was eventually executed with success.
2) The command was not converted to EVAL in case of errors in the middle
of the execution.
3) Terrible duplication of code without any apparent reason.
EVALSHA used to crash if the SHA1 was not lowercase (Issue #783).
Fixed using a case insensitive dictionary type for the sha -> script
map used for replication of scripts.
When calling SCRIPT KILL currently you can get two errors:
* No script in timeout (busy) state.
* The script already performed a write.
It is useful to be able to distinguish the two errors, but right now both
start with "ERR" prefix, so string matching (that is fragile) must be used.
This commit introduces two different prefixes.
-NOTBUSY and -UNKILLABLE respectively to reply with an error when no
script is busy at the moment, and when the script already executed a
write operation and can not be killed.
A previous commit introduced Redis.NIL. This commit adds similar helper
functions to return tables with a single field set to the specified
string so that instead of using 'return {err="My Error"}' it is possible
to use a more idiomatic form:
return redis.error_reply("My Error")
return redis.status_reply("OK")
Lua arrays can't contain nil elements (see
http://www.lua.org/pil/19.1.html for more information), so Lua scripts
were not able to return a multi-bulk reply containing nil bulk
elements inside.
This commit introduces a special conversion: a table with just
a "nilbulk" field set to a boolean value is converted by Redis as a nil
bulk reply, but at the same time for Lua this type is not a "nil" so can
be used inside Lua arrays.
This type is also assigned to redis.NIL, so the following two forms
are equivalent and will be able to return a nil bulk reply as second
element of a three elements array:
EVAL "return {1,redis.NIL,3}" 0
EVAL "return {1,{nilbulk=true},3}" 0
The result in redis-cli will be:
1) (integer) 1
2) (nil)
3) (integer) 3
SORT is able to return (faster than when ordering) unordered output if
the "BY" clause is used with a constant value. However we try to play
well with scripting requirements of determinism providing always sorted
outputs when SORT (and other similar commands) are called by Lua
scripts.
However we used the general mechanism in place in scripting in order to
reorder SORT output, that is, if the command has the "S" flag set, the
Lua scripting engine will take an additional step when converting a
multi bulk reply to Lua value, calling a Lua sorting function.
This is suboptimal as we can do it faster inside SORT itself.
This is also broken as issue #545 shows us: basically when SORT is used
with a constant BY, and additionally also GET is used, the Lua scripting
engine was trying to order the output as a flat array, while it was
actually a list of key-value pairs.
What we do know is to recognized if the caller of SORT is the Lua client
(since we can check this using the REDIS_LUA_CLIENT flag). If so, and if
a "don't sort" condition is triggered by the BY option with a constant
string, we force the lexicographical sorting.
This commit fixes this bug and improves the performance, and at the same
time simplifies the implementation. This does not mean I'm smart today,
it means I was stupid when I committed the original implementation ;)
Lua scripting uses a fake client in order to run commands in the context
of a client, accumulate the reply, and convert it into a Lua object
to return to the caller. This client is reused again and again, and is
referenced by the server.lua_client globally accessible pointer.
However after every call to redis.call() or redis.pcall(), that is
handled by the luaRedisGenericCommand() function, the reply_bytes field
of the client was not set back to zero. This filed is used to estimate
the amount of memory currently used in the reply. Because of the lack of
reset, script after script executed, this value used to get bigger and
bigger, and in the end on 32 bit systems it triggered the following
assert:
redisAssert(c->reply_bytes < ULONG_MAX-(1024*64));
On 64 bit systems this does not happen because it takes too much time to
reach values near to 2^64 for users to see the practical effect of the
bug.
Now in the cleanup stage of luaRedisGenericCommand() we reset the
reply_bytes counter to zero, avoiding the issue. It is not practical to
add a test for this bug, but the fix was manually tested using a
debugger.
This commit fixes issue #656.
Redis used to crash with a call like the following:
EVAL "redis.call()" 0
Now the explicit check for at least one argument prevents the problem.
This commit fixes issue #655.
An user reported a crash with Redis scripting (see issue #480 on
github), inspection of the kindly provided strack trace showed that
server.lua_caller was probably set to NULL. The stack trace also slowed
that the call to the hook was originating from a point where we just
used to set/get a few global variables in the Lua state.
What was happening is that we did not set the timeout hook selectively
only when the user script was called. Now we set it more selectively,
specifically only in the context of the lua_pcall() call, and make sure
to remove the hook when the call returns. Otherwise the hook can get
called in random contexts every time we do something with the Lua
state.
Lua global protection can now be simpified becuase we no longer have the
global() function. It's useless to occupy memory with this table, it is
also not faster because the metamethods we use are only called when a
global object does not exist or we are trying to create it from a
script.
After considering the interaction between ability to delcare globals in
scripts using the 'global' function, and the complexities related to
hanlding replication and AOF in a sane way with globals AND ability to
turn protection On and Off, we reconsidered the design. The new design
makes clear that there is only one good way to write Redis scripts, that
is not using globals. In the rare cases state must be retained across
calls a Redis key can be used.
This commit introduces support for read only slaves via redis.conf and CONFIG GET/SET commands. Also various semantical fixes are implemented here:
1) MULTI/EXEC with only read commands now work where the server is into a state where writes (or commands increasing memory usage) are not allowed. Before this patch everything inside a transaction would fail in this conditions.
2) Scripts just calling read-only commands will work against read only
slaves, when the server is out of memory, or when persistence is into an
error condition. Before the patch EVAL always failed in this condition.
