Commit Graph

11385 Commits

Author SHA1 Message Date
Binbin
c2b0c13d5c
Fix Invalid node address specified in redis-cli --cluster create/add-node (#11151)
This bug was introduced in #10344 (7.0.3), and it breaks the
redis-cli --cluster create usage in #10436 (7.0 RC3).

At the same time, the cluster-port support introduced in #10344
cannot use the DNS lookup brought by #10436.
2022-09-19 13:59:36 +03:00
sundb
13d25dd95e
Fix crash due to delete entry from compress quicklistNode and wrongly split quicklistNode (#11242)
This PR mainly deals with 2 crashes introduced in #9357,
and fix the QUICKLIST-PACKED-THRESHOLD mess in external test mode.

1. Fix crash due to deleting an entry from a compress quicklistNode
   When inserting a large element, we need to create a new quicklistNode first,
   and then delete its previous element, if the node where the deleted element is
   located is compressed, it will cause a crash.
   Now add `dont_compress` to quicklistNode, if we want to use a quicklistNode
   after some operation, we can use this flag like following:

    ```c
    node->dont_compress = 1; /* Prevent to be compressed */
    some_operation(node); /* This operation might try to compress this node */
    some_other_operation(node); /* We can use this node without decompress it */
    node->dont_compress = 0; /* Re-able compression */
    quicklistCompressNode(node);
    ```

   Perhaps in the future, we could just disable the current entry from being
   compressed during the iterator loop, but that would require more work.

2. Fix crash due to wrongly split quicklist
   before #9357, the offset param of _quicklistSplitNode() will not negative.
   For now, when offset is negative, the split extent will be wrong.
   following example:
    ```c
    int orig_start = after ? offset + 1 : 0;
    int orig_extent = after ? -1 : offset;
    int new_start = after ? 0 : offset;
    int new_extent = after ? offset + 1 : -1;
    # offset: -2, after: 1, node->count: 2
    # current wrong range: [-1,-1] [0,-1]
    # correct range: [1,-1] [0, 1]
    ```

   Because only `_quicklistInsert()` splits the quicklistNode and only
   `quicklistInsertAfter()`, `quicklistInsertBefore()` call _quicklistInsert(), 
   so `quicklistReplaceEntry()` and `listTypeInsert()` might occur this crash.
   But the iterator of `listTypeInsert()` is alway from head to tail(iter->offset is
   always positive), so it is not affected.
   The final conclusion is this crash only occur when we insert a large element
   with negative index into a list, that affects `LSET` command and `RM_ListSet`
   module api.
     
3. In external test mode, we need to restore quicklist packed threshold after
   when the end of test.
4. Show `node->count` in quicklistRepr().
5. Add new tcl proc `config_get_set` to support restoring config in tests.
2022-09-19 09:47:52 +03:00
zhaozhao.zz
464aa04188
fix infinite sleep in performEvictions when have lazyfree jobs (#11237)
This bug is introduced in #7653. (Redis 6.2.0)

When `server.maxmemory_eviction_tenacity` is 100, `eviction_time_limit_us` is
`ULONG_MAX`, and if we cannot find the best key to delete (e.g. maxmemory-policy
is `volatile-lru` and all keys with ttl have been evicted), in `cant_free` redis will sleep
forever if some items are being freed in the lazyfree thread.
2022-09-18 17:46:24 +03:00
Adi Pinsky
d144dc927a
Adds listnode to client struct for clients_pending_write list (#11220) 2022-09-14 22:39:47 -05:00
Viktor Söderqvist
42e4241ece
Avoid crash when a cluster node is a replica of a replica of itself (#11263) 2022-09-13 17:48:48 -07:00
Madelyn Olson
6c03786b66
Prevent use after free for inbound cluster link (#11255) 2022-09-13 16:19:29 -05:00
Eduardo Semprebon
36abc0fa8f
Improve redis.conf documentation on repl-diskless-load (#11213)
Just noticed that there are some inaccurate, or at least confusing information about `repl-diskless-load` in `redis.conf`
It shouldn't scare away users willing to spend the extra memory.

`may mean that we have to flush the contents of the current database before the full rdb was received.`: this is likely related to the time when there was an option `always`, where content on replica was flushed before loading from master.
2022-09-11 11:22:59 +03:00
Wen Hui
5389fa62a6
Update group and consumer description in json file for Unifying Stream command format (#11190)
For the stream data type, some commands, such as **XGROUP CREATE, XGROUP DESTROY, XGROUP CREATECONSUMER, 
XGROUP DELCONSUMER and XINFO CONSUMERS** use groupname and consumername in the command description;

However, for the commands **XREADGROUP GROUP, XPENDING, XACK , XCLAIM  and XAUTOCLAIM**  use term "group and consumer", clients could be confused.

This PR goal is to unify all the commands to groupname and consumername.
2022-09-08 09:37:49 +03:00
Mingyi Kang
e67d06ee6b
Optimize setGenericCommand(): no need to remove the expiration entry when 'expire' is not NULL (#11244) 2022-09-07 22:35:25 -05:00
*caco
89018ff6a7
fix semantic uncorresponding in comment of ziplist example (#11141)
fix semantic uncorresponding in comment of ziplist example
2022-09-07 21:48:11 -05:00
tmoshaiov
fb1d56bc2a
Added API to initialize dictionary iterators without memory allocation (#11245)
* Added api to use dictionary iterators without calling malloc.
2022-09-07 20:57:43 -05:00
ranshid
c0ce97facc
fix test Migrate the last slot away from a node using redis-cli (#11221)
When using cli to add node, there can potentially be a race condition in
which all nodes presenting cluster state o.k even though the added node
did not yet meet all cluster nodes.
this adds another utility function to wait until all cluster nodes see the same cluster size
2022-09-06 16:54:24 -07:00
Ozan Tezcan
3761fdb048
Use cached value correctly inside connectionTypeTls() (#11236)
When Redis is built without TLS support, connectionTypeTls() function 
keeps searching connection type as cached connection type is NULL. 

Added another variable to track if we cached the connection type to 
prevent search after the first time. 

Noticed a log warning message is printed repeatedly by connectionTypeTls.

Co-authored-by: zhenwei pi <pizhenwei@bytedance.com>
Co-authored-by: Oran Agra <oran@redislabs.com>
2022-09-06 09:04:33 +03:00
Shogo Hayashi
e764e2a627
Fix typo in DEBUG REPLYBUFFER RESIZING comment
This command is related with reply buffer, not replay buffer

Co-authored-by: Shogo Hayashi <hayshogo@amazon.co.jp>
2022-09-06 08:40:37 +03:00
chendianqiang
e42d98ed27
Correctly handle scripts with shebang (not read-only) on a cluster replica (#11223)
EVAL scripts are by default not considered `write` commands, so they were allowed on a replica.
But when adding a shebang, they become `write` command (unless the `no-writes` flag is added).
With this change we'll handle them as write commands, and reply with MOVED instead of
READONLY when executed on a redis cluster replica.

Co-authored-by: chendianqiang <chendianqiang@meituan.com>
2022-09-05 16:59:14 +03:00
Shaya Potter
87e7973c7e
Add a dry run flag to RM_Call execution (#11158)
Add a new "D" flag to RM_Call which runs whatever verification the user requests,
but returns before the actual execution of the command.

It automatically enables returning error messages as CallReply objects to distinguish
success (NULL) from failure (CallReply returned).
2022-09-05 16:19:32 +03:00
David CARLIER
22f763aa10
zmalloc api set malloc attributes for api giving non aliased pointers. (#11196)
micro optimizations, giving the hints that the returned addresses
 are guaranteed to be unique. The alloc_size attribute gives an extra hint
 about the source of the size, useful mostly for calloc-like calls or when there
 are extra arguments.
2022-09-05 16:09:28 +03:00
dependabot[bot]
c66eaf4e4a
Bump vmactions/freebsd-vm from 0.2.3 to 0.2.4 (#11203)
Bumps [vmactions/freebsd-vm](https://github.com/vmactions/freebsd-vm) from 0.2.3 to 0.2.4.
- [Release notes](https://github.com/vmactions/freebsd-vm/releases)
- [Commits](https://github.com/vmactions/freebsd-vm/compare/v0.2.3...v0.2.4)

---
updated-dependencies:
- dependency-name: vmactions/freebsd-vm
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-08-31 10:13:48 +03:00
Oran Agra
c3b7bde914
fix false valgrind error on new hash test (#11200)
New test fails on valgrind because strtold("+inf") with valgrind returns a non-inf result
same thing is done in incr.tcl.
2022-08-29 10:25:24 +03:00
Shaya Potter
bed6d759bc
Improve cmd_flags for script/functions in RM_Call (#11159)
When RM_Call was used with `M` (reject OOM), `W` (reject writes),
as well as `S` (rejecting stale or write commands in "Script mode"),
it would have only checked the command flags, but not the declared
script flag in case it's a command that runs a script.

Refactoring: extracts out similar code in server.c's processCommand
to be usable in RM_Call as well.
2022-08-28 13:10:10 +03:00
weimeng
8945067544
bugfix:del keys in slot replicate to replica, and trigger other invalidations (#11084)
Bugfix:
with the scenario if we force assigned a slot to other master,
old master will lose the slot ownership, then old master will
call the function delKeysInSlot() to delete all keys which in
the slot. These delete operations should replicate to replicas,
avoid the data divergence issue in master and replicas.

Additionally, in this case, we now call:
* signalModifiedKey (to invalidate WATCH)
* moduleNotifyKeyspaceEvent (key space notification for modules)
* dirty++ (to signal that the persistence file may be outdated)

Co-authored-by: weimeng <weimeng@didiglobal.com>
Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
2022-08-28 11:37:26 +03:00
chendianqiang
bc7fe41e58
fix hincrbyfloat not to create a key if the new value is invalid (#11149)
Check the validity of the value before performing the create operation,
prevents new data from being generated even if the request fails to execute.

Co-authored-by: Oran Agra <oran@redislabs.com>
Co-authored-by: chendianqiang <chendianqiang@meituan.com>
Co-authored-by: Binbin <binloveplay1314@qq.com>
2022-08-28 11:33:41 +03:00
Huang Zhw
a7da7473cb
Remove the NONDETERMINISTIC_OUTPUT flag from most CLUSTER sub-commands. (#11157)
TLDR: the CLUSTER command originally had the `random` flag,
so all the sub-commands initially got that new flag, but in fact many
of them don't need it.
The only effect of this change is on the output of COMMAND INFO.
2022-08-28 11:24:47 +03:00
Moti Cohen
246f44d723
Removing old redundant code from bio.c (#11136)
* Remove redundant array bio_pending[]. Value at index i identically reflects the
length of list bio_jobs[i]. Better use listLength() instead and discard this array.
(no critical section issues to concern about).

changed returned value of bioPendingJobsOfType() from "long long" to "long".

Remove unused API. Maybe we will use this API later.
2022-08-26 09:09:23 -07:00
Paul Menzel
14e026e685
Correct grammatical error in for DENIED error message (#11192)
networking: Spell verb *set up* with space in error message
2022-08-26 09:06:55 -07:00
Oran Agra
c789fb0aa7
Fix assertion when a key is lazy expired during cluster key migration (#11176)
Redis 7.0 has #9890 which added an assertion when the propagation queue
was not flushed and we got to beforeSleep.
But it turns out that when processCommands calls getNodeByQuery and
decides to reject the command, it can lead to a key that was lazy
expired and is deleted without later flushing the propagation queue.

This change prevents lazy expiry from deleting the key at this stage
(not as part of a command being processed in `call`)
2022-08-24 19:39:15 +03:00
Binbin
78259826cd
Bump codespell from 2.1.0 to 2.2.1 in /.codespell (#11184)
add a few terms to the white list, and fix a few newly detected typos
2022-08-24 15:07:43 +03:00
Brad Dunbar
c07212372c
Cleanup in GETDEL: Strings are never freed lazily (#11175)
The GETDEL command only operates on strings, and strings are never freed
lazily, so there's no need to use `dbAsyncDelete` or `shared.unlink`.
2022-08-24 14:11:04 +03:00
Meir Shpilraien (Spielrein)
3603f19496
fix test timeout wait command (#11181)
Fix `Test replication with lazy expire` test to not timeout the wait command.
This fix will allow the test to pass on slow environments and when running with valgrind.
2022-08-24 13:38:55 +03:00
Meir Shpilraien (Spielrein)
c1bd61a4a5
Reverts most of the changes of #10969 (#11178)
The PR reverts the changes made on #10969.
The reason for revert was trigger because of occasional test failure
that started after the PR was merged.

The issue is that if there is a lazy expire during the command invocation,
the `del` command is added to the replication stream after the command
placeholder. So the logical order on the primary is:

* Delete the key (lazy expiration)
* Command invocation

But the replication stream gets it the other way around:

* Command invocation (because the command is written into the placeholder)
* Delete the key (lazy expiration)

So if the command write to the key that was just lazy expired we will get
inconsistency between primary and replica.

One solution we considered is to add another lazy expire replication stream
and write all the lazy expire there. Then when replicating, we will replicate the
lazy expire replication stream first. This will solve this specific test failure but
we realize that the issues does not ends here and the more we dig the more
problems we find.One of the example we thought about (that can actually
crashes Redis) is as follow:

* User perform SINTERSTORE
* When Redis tries to fetch the second input key it triggers lazy expire
* The lazy expire trigger a module logic that deletes the first input key
* Now Redis hold the robj of the first input key that was actually freed

We believe we took the wrong approach and we will come up with another
PR that solve the problem differently, for now we revert the changes so we
will not have the tests failure.

Notice that not the entire code was revert, some parts of the PR are changes
that we would like to keep. The changes that **was** reverted are:

* Saving a placeholder for replication at the beginning of the command (`call` function)
* Order of the replication stream on active expire and eviction (we will decide how
  to handle it correctly on follow up PR)
* `Spop` changes are no longer needed (because we reverted the placeholder code)

Changes that **was not** reverted:

* On expire/eviction, wrap the `del` and the notification effect in a multi exec.
* `PropagateNow` function can still accept a special dbid, -1, indicating not to replicate select.
* Keep optimisation for reusing the `alsoPropagate` array instead of allocating it each time.

Tests:

* All tests was kept and only few tests was modify to work correctly with the changes
* Test was added to verify that the revert fixes the issues.
2022-08-24 12:51:36 +03:00
Oran Agra
41d9eb0291
Merge: Fully abstract connection and make TLS dynamically loadable (#9320)
There are many commits in this PR, the detailed changes is described
in each commit message.

### Main changes in this PR

* Fully abstract connection type, and hide connection type specified methods.
  Ex, currently TLS class looks like:
```
static ConnectionType CT_TLS = {
    /* connection type */
    .get_type = connTLSGetType,

    /* connection type initialize & finalize & configure */
    .init = tlsInit,
    .cleanup = tlsCleanup,
    .configure = tlsConfigure,

    /* ae & accept & listen & error & address handler */
    .ae_handler = tlsEventHandler,
    .accept_handler = tlsAcceptHandler,
    .addr = connTLSAddr,
    .listen = connTLSListen,

    /* create/close connection */
    .conn_create = connCreateTLS,
    .conn_create_accepted = connCreateAcceptedTLS,
    .close = connTLSClose,

    /* connect & accept */
    .connect = connTLSConnect,
    .blocking_connect = connTLSBlockingConnect,
    .accept = connTLSAccept,

    /* IO */
    .read = connTLSRead,
    .write = connTLSWrite,
    .writev = connTLSWritev,
    .set_write_handler = connTLSSetWriteHandler,
    .set_read_handler = connTLSSetReadHandler,
    .get_last_error = connTLSGetLastError,
    .sync_write = connTLSSyncWrite,
    .sync_read = connTLSSyncRead,
    .sync_readline = connTLSSyncReadLine,

    /* pending data */
    .has_pending_data = tlsHasPendingData,
    .process_pending_data = tlsProcessPendingData,

    /* TLS specified methods */
    .get_peer_cert = connTLSGetPeerCert,
};

int RedisRegisterConnectionTypeTLS()
{
    return connTypeRegister(&CT_TLS);
}
```

* Also abstract Unix socket class. Currently, the connection framework becomes like:
```
                       uplayer
                          |
                   connection layer
                     /    |     \
                   TCP   Unix   TLS
    
```

* It's possible to build TLS as a shared library (`make BUILD_TLS=module`).
  Loading the shared library(redis-tls.so) into Redis by Redis module subsystem,
  and Redis starts to listen TLS port. Ex:
```
    ./src/redis-server --tls-port 6379 --port 0 \
        --tls-cert-file ./tests/tls/redis.crt \
        --tls-key-file ./tests/tls/redis.key \
        --tls-ca-cert-file ./tests/tls/ca.crt \
        --loadmodule src/redis-tls.so
```

### Interface changes
* RM_GetContextFlags supports a new flag: REDISMODULE_CTX_FLAGS_SERVER_STARTUP
* INFO SERVER includes a list of listeners:
```
listener0:name=tcp,bind=127.0.0.1,port=6380
listener1:name=unix,bind=/run/redis.sock
listener2:name=tls,bind=127.0.0.1,port=6379
```

### Other notes

* Fix wrong signature of RedisModuleDefragFunc, this could break
  compilation of a module, but not the ABI
* Some reordering of initialization order in server.c:
  * Move initialization of listeners to be after loading the modules
  * Config TLS after initialization of listeners
  * Init cluster after initialization of listeners
* Sentinel does not support the TLS module or any connection module
  since it uses hiredis for outbound connections, so when TLS is built as
  a module, sentinel lacks TLS support.
2022-08-24 08:35:46 +03:00
Oran Agra
4faddf18ca Build TLS as a loadable module
* Support BUILD_TLS=module to be loaded as a module via config file or
  command line. e.g. redis-server --loadmodule redis-tls.so
* Updates to redismodule.h to allow it to be used side by side with
  server.h by defining REDISMODULE_CORE_MODULE
* Changes to server.h, redismodule.h and module.c to avoid repeated
  type declarations (gcc 4.8 doesn't like these)
* Add a mechanism for non-ABI neutral modules (ones who include
  server.h) to refuse loading if they detect not being built together with
  redis (release.c)
* Fix wrong signature of RedisModuleDefragFunc, this could break
  compilation of a module, but not the ABI
* Move initialization of listeners in server.c to be after loading
  the modules
* Config TLS after initialization of listeners
* Init cluster after initialization of listeners
* Add TLS module to CI
* Fix a test suite race conditions:
  Now that the listeners are initialized later, it's not sufficient to
  wait for the PID message in the log, we need to wait for the "Server
  Initialized" message.
* Fix issues with moduleconfigs test as a result from start_server
  waiting for "Server Initialized"
* Fix issues with modules/infra test as a result of an additional module
  present

Notes about Sentinel:
Sentinel can't really rely on the tls module, since it uses hiredis to
initiate connections and depends on OpenSSL (won't be able to use any
other connection modules for that), so it was decided that when TLS is
built as a module, sentinel does not support TLS at all.
This means that it keeps using redis_tls_ctx and redis_tls_client_ctx directly.

Example code of config in redis-tls.so(may be use in the future):
RedisModuleString *tls_cfg = NULL;

void tlsInfo(RedisModuleInfoCtx *ctx, int for_crash_report) {
    UNUSED(for_crash_report);
    RedisModule_InfoAddSection(ctx, "");
    RedisModule_InfoAddFieldLongLong(ctx, "var", 42);
}

int tlsCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc)
{
    if (argc != 2) return RedisModule_WrongArity(ctx);
    return RedisModule_ReplyWithString(ctx, argv[1]);
}

RedisModuleString *getStringConfigCommand(const char *name, void *privdata) {
    REDISMODULE_NOT_USED(name);
    REDISMODULE_NOT_USED(privdata);
    return tls_cfg;
}

int setStringConfigCommand(const char *name, RedisModuleString *new, void *privdata, RedisModuleString **err) {
    REDISMODULE_NOT_USED(name);
    REDISMODULE_NOT_USED(err);
    REDISMODULE_NOT_USED(privdata);
    if (tls_cfg) RedisModule_FreeString(NULL, tls_cfg);
    RedisModule_RetainString(NULL, new);
    tls_cfg = new;
    return REDISMODULE_OK;
}

int RedisModule_OnLoad(void *ctx, RedisModuleString **argv, int argc)
{
    ....
    if (RedisModule_CreateCommand(ctx,"tls",tlsCommand,"",0,0,0) == REDISMODULE_ERR)
        return REDISMODULE_ERR;

    if (RedisModule_RegisterStringConfig(ctx, "cfg", "", REDISMODULE_CONFIG_DEFAULT, getStringConfigCommand, setStringConfigCommand, NULL, NULL) == REDISMODULE_ERR)
        return REDISMODULE_ERR;

    if (RedisModule_LoadConfigs(ctx) == REDISMODULE_ERR) {
        if (tls_cfg) {
            RedisModule_FreeString(ctx, tls_cfg);
            tls_cfg = NULL;
        }
        return REDISMODULE_ERR;
    }
    ...
}

Co-authored-by: zhenwei pi <pizhenwei@bytedance.com>
Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
2022-08-23 12:37:56 +03:00
Ariel Shtul
90223759a3
[PERF] use snprintf once in addReplyDouble (#11093)
The previous implementation calls `snprintf` twice, the second time used to
'memcpy' the output of the first, which could be a very large string.
The new implementation reserves space for the protocol header ahead
of the formatted double, and then prepends the string length ahead of it.

Measured improvement of simple ZADD of some 25%.
2022-08-23 09:37:59 +03:00
Itamar Haber
407b5c912f
Replaces a made-up term with a real one (#11169) 2022-08-22 15:05:01 +03:00
zhenwei pi
89e1148688 Introduce redis module ctx flag 'server startup'
A module may be loaded only during initial stage, a typical case is
connection type shared library.

Introduce REDISMODULE_CTX_FLAGS_SERVER_STARTUP context flag
to tell the module the stage of Redis. Then the module gets the flag
by RedisModule_GetContextFlags(ctx), tests flags and returns error in
onload handler.

Suggested-by: Oran Agra <oran@redislabs.com>
Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
2022-08-22 19:44:55 +08:00
zhenwei pi
8a59c19310 Introduce redis module ctx flag 'server startup' & 'sentinel'
A module may be loaded only during initial stage, a typical case is
connection type shared library.

Introduce REDISMODULE_CTX_FLAGS_SERVER_STARTUP context flag
to tell the module the stage of Redis. Then the module gets the flag
by RedisModule_GetContextFlags(ctx), tests flags and returns error in
onload handler.

Also introduce 'REDISMODULE_CTX_FLAGS_SENTINEL' context flag to tell
the module the sentinel mode or not.

Suggested-by: Oran Agra <oran@redislabs.com>
Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
2022-08-22 15:16:42 +08:00
zhenwei pi
0c4d2fcc8e Add listeners info string for 'INFO' command
Suggested by Oran, add necessary listeners information in 'INFO'
command. It would be helpful for debug.

Example of this:
127.0.0.1:6379> INFO SERVER
redis_version:255.255.255
...
listener0:name=tcp,bind=127.0.0.1,port=6380
listener1:name=unix,bind=/run/redis.sock
listener2:name=tls,bind=127.0.0.1,port=6379
...

Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
2022-08-22 15:16:26 +08:00
zhenwei pi
0b27cfe37d Introduce .listen into connection type
Introduce listen method into connection type, this allows no hard code
of listen logic. Originally, we initialize server during startup like
this:
    if (server.port)
        listenToPort(server.port,&server.ipfd);
    if (server.tls_port)
        listenToPort(server.port,&server.tlsfd);
    if (server.unixsocket)
        anetUnixServer(...server.unixsocket...);

    ...
    if (createSocketAcceptHandler(&server.ipfd, acceptTcpHandler) != C_OK)
    if (createSocketAcceptHandler(&server.tlsfd, acceptTcpHandler) != C_OK)
    if (createSocketAcceptHandler(&server.sofd, acceptTcpHandler) != C_OK)
    ...

If a new connection type gets supported, we have to add more hard code
to setup listener.

Introduce .listen and refactor listener, and Unix socket supports this.
this allows to setup listener arguments and create listener in a loop.

What's more, '.listen' is defined in connection.h, so we should include
server.h to import 'struct socketFds', but server.h has already include
'connection.h'. To avoid including loop(also to make code reasonable),
define 'struct connListener' in connection.h instead of 'struct socketFds'
in server.h. This leads this commit to get more changes.

There are more fields in 'struct connListener', hence it's possible to
simplify changeBindAddr & applyTLSPort() & updatePort() into a single
logic: update the listener config from the server.xxx, and re-create
the listener.

Because of the new field 'priv' in struct connListener, we expect to pass
this to the accept handler(even it's not used currently), this may be used
in the future.

Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
2022-08-22 15:16:08 +08:00
zhenwei pi
45617385e7 Use connection name of string
Suggested by Oran, use an array to store all the connection types
instead of a linked list, and use connection name of string. The index
of a connection is dynamically allocated.

Currently we support max 8 connection types, include:
- tcp
- unix socket
- tls

and RDMA is in the plan, then we have another 4 types to support, it
should be enough in a long time.

Introduce 3 functions to get connection type by a fast path:
- connectionTypeTcp()
- connectionTypeTls()
- connectionTypeUnix()

Note that connectionByType() is designed to use only in unlikely code path.

Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
2022-08-22 15:15:37 +08:00
zhenwei pi
eb94d6d36d Introduce unix socket connection type
Unix socket uses different accept handler/create listener from TCP,
to hide these difference to avoid hard code, use a new unix socket
connection type. Also move 'acceptUnixHandler' into unix.c.

Currently, the connection framework becomes like following:

                   uplayer
                      |
               connection layer
                 /    |     \
               TCP   Unix   TLS

It's possible to build Unix socket support as a shared library, and
load it dynamically. Because TCP and Unix socket don't require any
heavy dependencies or overheads, we build them into Redis statically.

Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
2022-08-22 15:12:31 +08:00
zhenwei pi
0ae02ce95b Abstract accept handler
Abstract accept handler for socket&TLS, and add helper function
'connAcceptHandler' to get accept handler by specified type.

Also move acceptTcpHandler into socket.c, and move
acceptTLSHandler into tls.c.

Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
2022-08-22 15:12:18 +08:00
zhenwei pi
41fff55d52 Use socketFds for unix
socketFds is also suitable for Unix socket, then we can use
'createSocketAcceptHandler' to create accept handler.
And then, we can abstract accept handler in the future.

Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
2022-08-22 15:12:04 +08:00
zhenwei pi
1234e3a562 Fully abstract connection type
Abstract common interface of connection type, so Redis can hide the
implementation and uplayer only calls connection API without macro.

               uplayer
                  |
           connection layer
             /          \
          socket        TLS

Currently, for both socket and TLS, all the methods of connection type
are declared as static functions.

It's possible to build TLS(even socket) as a shared library, and Redis
loads it dynamically in the next step.

Also add helper function connTypeOfCluster() and
connTypeOfReplication() to simplify the code:
link->conn = server.tls_cluster ? connCreateTLS() : connCreateSocket();
-> link->conn = connCreate(connTypeOfCluster());

Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
2022-08-22 15:11:44 +08:00
zhenwei pi
c4c02f8036 Introduce TLS specified APIs
Introduce .get_peer_cert, .get_ctx and .get_client_ctx for TLS, also
hide redis_tls_ctx & redis_tls_client_ctx.

Then outside could access the variables by connection API only:
- redis_tls_ctx -> connTypeGetCtx(CONN_TYPE_TLS)
- redis_tls_client_ctx -> connTypeGetClientCtx(CONN_TYPE_TLS)

Also remove connTLSGetPeerCert(), use connGetPeerCert() instead.

Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
2022-08-22 15:11:25 +08:00
zhenwei pi
709b55b09d Introduce pending data for connection type
Introduce .has_pending_data and .process_pending_data for connection
type, and hide tlsHasPendingData() and tlsProcessPendingData(). Also
set .has_pending_data and .process_pending_data as NULL explicitly in
socket.c.

Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
2022-08-22 15:11:06 +08:00
zhenwei pi
8234a5123d Introduce connection layer framework
Use connTypeRegister() to register a connection type into redis, and
query connection by connectionByType() via type.

With this change, we can hide TLS specified methods into connection
type:
- void tlsInit(void);
- void tlsCleanup(void);
- int tlsConfigure(redisTLSContextConfig *ctx_config);
- int isTlsConfigured(void);

Merge isTlsConfigured & tlsConfigure, use an argument *reconfigure*
to distinguish:
   tlsConfigure(&server.tls_ctx_config)
-> onnTypeConfigure(CONN_TYPE_TLS, &server.tls_ctx_config, 1)

   isTlsConfigured() && tlsConfigure(&server.tls_ctx_config)
-> connTypeConfigure(CONN_TYPE_TLS, &server.tls_ctx_config, 0)

Finally, we can remove USE_OPENSSL from config.c. If redis is built
without TLS, and still run redis with TLS, then redis reports:
 # Missing implement of connection type 1
 # Failed to configure TLS. Check logs for more info.

The log can be optimised, let's leave it in the future. Maybe we can
use connection type as a string.

Although uninitialized fields of a static struct are zero, we still
set them as NULL explicitly in socket.c, let them clear to read & maintain:
    .init = NULL,
    .cleanup = NULL,
    .configure = NULL,

Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
2022-08-22 15:09:59 +08:00
zhenwei pi
bff7ecc786 Introduce connAddr
Originally, connPeerToString is designed to get the address info from
socket only(for both TCP & TLS), and the API 'connPeerToString' is
oriented to operate a FD like:
int connPeerToString(connection *conn, char *ip, size_t ip_len, int *port) {
    return anetFdToString(conn ? conn->fd : -1, ip, ip_len, port, FD_TO_PEER_NAME);
}

Introduce connAddr and implement .addr method for socket and TLS,
thus the API 'connAddr' and 'connFormatAddr' become oriented to a
connection like:
static inline int connAddr(connection *conn, char *ip, size_t ip_len, int *port, int remote) {
    if (conn && conn->type->addr) {
        return conn->type->addr(conn, ip, ip_len, port, remote);
    }

    return -1;
}

Also remove 'FD_TO_PEER_NAME' & 'FD_TO_SOCK_NAME', use a boolean type
'remote' to get local/remote address of a connection.

With these changes, it's possible to support the other connection
types which does not use socket(Ex, RDMA).

Thanks to Oran for suggestions!

Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
2022-08-22 15:01:40 +08:00
zhenwei pi
b9d7728824 Reorder methods for ConnectionType
Reorder methods for CT_Socket & CT_TLS, also add comments to make the
methods clear.

Also move the CT_TLS to the end of file, other methods can be static
in the next step.

Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
2022-08-22 15:01:32 +08:00
zhenwei pi
8045e26efa Move 'connGetSocketError' to 'anetGetError'
getsockopt is part of TCP, rename 'connGetSocketError' to
'anetGetError', and move it into anet.c.

Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
2022-08-22 15:01:16 +08:00
zhenwei pi
dca5c6ff11 Move several conn functions to connection.h
These functions are really short enough and they are the connection
functions, separate them from the socket source.

Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
2022-08-22 15:01:01 +08:00