Commit Graph

12191 Commits

Author SHA1 Message Date
Binbin
4cae66f5e8
Use shard-id of the master if the replica does not support shard-id (#12805)
If there are nodes in the cluster that do not support shard-id, they
will gossip shard-id. From the perspective of nodes that support shard-id,
their shard-id is meaningless (since shard-id is randomly generated when
we create a node.)

Nodes that support shard-id will save the shard-id information in nodes.conf.
If the node is restarted according to nodes.conf, the server will report a
corrupted cluster config file error. Because auxShardIdSetter will reject
configurations with inconsistent master-replica shard-ids.

A cluster-wide consensus for the node's shard_id is not necessary. The key
is maintaining consistency of the shard_id on each individual 7.2 node.
As the cluster progressively upgrades to version 7.2, we can expect the
shard_ids across all nodes to naturally converge and align.

In this PR, when processing the gossip, if sender is a replica and does not
support shard-id, set the shard_id to the shard_id of its master.
2024-01-06 20:24:41 -08:00
dependabot[bot]
38f0234946
Bump cross-platform-actions/action from 0.21.1 to 0.22.0 (#12904)
Bumps
[cross-platform-actions/action](https://github.com/cross-platform-actions/action)
from 0.21.1 to 0.22.0.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/cross-platform-actions/action/releases">cross-platform-actions/action's
releases</a>.</em></p>
<blockquote>
<h2>Cross Platform Action 0.22.0</h2>
<h3>Added</h3>
<ul>
<li>
<p>Added support for using the action in multiple steps in the same job
(<a
href="https://redirect.github.com/cross-platform-actions/action/issues/26">#26</a>).
All the inputs need to be the same for all steps, except for the
following
inputs: <code>sync_files</code>, <code>shutdown_vm</code> and
<code>run</code>.</p>
</li>
<li>
<p>Added support for specifying that the VM should not shutdown after
the action
has run. This adds a new input parameter: <code>shutdown_vm</code>. When
set to <code>false</code>,
this will hopefully mitigate very frequent freezing of VM during
teardown (<a
href="https://redirect.github.com/cross-platform-actions/action/issues/61">#61</a>,
<a
href="https://redirect.github.com/cross-platform-actions/action/issues/72">#72</a>).</p>
</li>
</ul>
<h3>Changed</h3>
<ul>
<li>
<p>Always terminate VM instead of shutting down. This is more efficient
and this
will hopefully mitigate very frequent freezing of VM during teardown
(<a
href="https://redirect.github.com/cross-platform-actions/action/issues/61">#61</a>,
<a
href="https://redirect.github.com/cross-platform-actions/action/issues/72">#72</a>).</p>
</li>
<li>
<p>Use <code>unsafe</code> as the cache mode for QEMU disks. This should
improve performance (<a
href="https://redirect.github.com/cross-platform-actions/action/issues/67">#67</a>).</p>
</li>
</ul>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/cross-platform-actions/action/blob/master/changelog.md">cross-platform-actions/action's
changelog</a>.</em></p>
<blockquote>
<h2>[0.22.0] - 2023-12-27</h2>
<h3>Added</h3>
<ul>
<li>
<p>Added support for using the action in multiple steps in the same job
(<a
href="https://redirect.github.com/cross-platform-actions/action/issues/26">#26</a>).
All the inputs need to be the same for all steps, except for the
following
inputs: <code>sync_files</code>, <code>shutdown_vm</code> and
<code>run</code>.</p>
</li>
<li>
<p>Added support for specifying that the VM should not shutdown after
the action
has run. This adds a new input parameter: <code>shutdown_vm</code>. When
set to <code>false</code>,
this will hopefully mitigate very frequent freezing of VM during
teardown (<a
href="https://redirect.github.com/cross-platform-actions/action/issues/61">#61</a>,
<a
href="https://redirect.github.com/cross-platform-actions/action/issues/72">#72</a>).</p>
</li>
</ul>
<h3>Changed</h3>
<ul>
<li>
<p>Always terminate VM instead of shutting down. This is more efficient
and this
will hopefully mitigate very frequent freezing of VM during teardown
(<a
href="https://redirect.github.com/cross-platform-actions/action/issues/61">#61</a>,
<a
href="https://redirect.github.com/cross-platform-actions/action/issues/72">#72</a>).</p>
</li>
<li>
<p>Use <code>unsafe</code> as the cache mode for QEMU disks. This should
improve performance (<a
href="https://redirect.github.com/cross-platform-actions/action/issues/67">#67</a>).</p>
</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="5800fa0060"><code>5800fa0</code></a>
Release 0.22.0</li>
<li><a
href="20ad4b2ceb"><code>20ad4b2</code></a>
Fix <a
href="https://redirect.github.com/cross-platform-actions/action/issues/67">#67</a>:
Use <code>unsafe</code> as the cache mode disks</li>
<li><a
href="d9184930c3"><code>d918493</code></a>
Always terminate VM instead of shutting down.</li>
<li><a
href="626f1d6c95"><code>626f1d6</code></a>
Fix error when terminating the VM</li>
<li><a
href="d59f08dc5c"><code>d59f08d</code></a>
Print stack trace for uncaught exceptions</li>
<li><a
href="7f2fab9c56"><code>7f2fab9</code></a>
Revert &quot;Run SSH in verbose mode when debug mode is
enabled&quot;</li>
<li><a
href="0f566c356e"><code>0f566c3</code></a>
[no ci] Update the changelog</li>
<li><a
href="b7f77446bb"><code>b7f7744</code></a>
[no ci] Fix spelling</li>
<li><a
href="9894a9b118"><code>9894a9b</code></a>
Wrap <code>host</code> module in namespace</li>
<li><a
href="87fdd346a2"><code>87fdd34</code></a>
Fix broken test-vm-shutdown tests</li>
<li>Additional commits viewable in <a
href="https://github.com/cross-platform-actions/action/compare/v0.21.1...v0.22.0">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=cross-platform-actions/action&package-manager=github_actions&previous-version=0.21.1&new-version=0.22.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-01-04 22:38:33 +02:00
Lior Kogan
5189838350
Update CONTRIBUTING.md (#12907)
- Referring to Redis Discord channel instead of the mailing list.
- Referring to the licensing instead of repeating it.
2024-01-03 17:21:19 +02:00
Madelyn Olson
068051e378
Handle recursive serverAsserts and provide more information for recursive segfaults (#12857)
This change is trying to make two failure modes a bit easier to deep dive:
1. If a serverPanic or serverAssert occurs during the info (or module)
printing, it will recursively panic, which is a lot of fun as it will
just keep recursively printing. It will eventually stack overflow, but
will generate a lot of text in the process.
2. When a segfault happens during the segfault handler, no information
is communicated other than it happened. This can be problematic because
`info` may help diagnose the real issue, but without fixing the
recursive crash it might be hard to get at that info.
2024-01-02 18:20:22 -08:00
AshMosh
c3f8b542ee
Manage number of new connections per cycle (#12178)
There are situations (especially in TLS) in which the engine gets too occupied managing a large number of new connections. Existing connections may time-out while the server is processing the new connections initial TLS handshakes, which may cause cause new connections to be established, perpetuating the problem. To better manage the tradeoff between new connection rate and other workloads, this change adds a new config to manage maximum number of new connections per event loop cycle, instead of using a predetermined number (currently 1000).

This change introduces two new configurations, max-new-connections-per-cycle and max-new-tls-connections-per-cycle. The default value of the tcp connections is 10 per cycle and the default value of tls connections per cycle is 1.
---------

Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
2024-01-02 15:15:03 -08:00
Chen Tianjie
9d0158bf89
Reorder signalModifiedKey in xaddCommand. (#12895)
This PR is a supplement to #11144, moving `signalModifiedKey` in
`xaddCommand` after the trimming, to ensure the state of key eventual
consistency. Currently there is no problem with Redis, but it is better
to avoid issues in future development on Redis.
2023-12-28 13:29:27 +02:00
dependabot[bot]
2c5b51ad26
Bump github/codeql-action from 2 to 3 (#12869)
Bumps [github/codeql-action](https://github.com/github/codeql-action)
from 2 to 3.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/github/codeql-action/releases">github/codeql-action's
releases</a>.</em></p>
<blockquote>
<h2>CodeQL Bundle v2.15.4</h2>
<p>Bundles CodeQL CLI v2.15.4</p>
<ul>
<li>(<a
href="https://github.com/github/codeql-cli-binaries/blob/HEAD/CHANGELOG.md">changelog</a>,
<a
href="https://github.com/github/codeql-cli-binaries/releases/tag/v2.15.4">release</a>)</li>
</ul>
<p>Includes the following CodeQL language packs from <a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.4"><code>github/codeql@codeql-cli/v2.15.4</code></a>:</p>
<ul>
<li><code>codeql/cpp-queries</code> (<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.4/cpp/ql/src/CHANGELOG.md">changelog</a>,
<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.4/cpp/ql/src">source</a>)</li>
<li><code>codeql/cpp-all</code> (<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.4/cpp/ql/lib/CHANGELOG.md">changelog</a>,
<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.4/cpp/ql/lib">source</a>)</li>
<li><code>codeql/csharp-queries</code> (<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.4/csharp/ql/src/CHANGELOG.md">changelog</a>,
<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.4/csharp/ql/src">source</a>)</li>
<li><code>codeql/csharp-all</code> (<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.4/csharp/ql/lib/CHANGELOG.md">changelog</a>,
<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.4/csharp/ql/lib">source</a>)</li>
<li><code>codeql/go-queries</code> (<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.4/go/ql/src/CHANGELOG.md">changelog</a>,
<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.4/go/ql/src">source</a>)</li>
<li><code>codeql/go-all</code> (<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.4/go/ql/lib/CHANGELOG.md">changelog</a>,
<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.4/go/ql/lib">source</a>)</li>
<li><code>codeql/java-queries</code> (<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.4/java/ql/src/CHANGELOG.md">changelog</a>,
<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.4/java/ql/src">source</a>)</li>
<li><code>codeql/java-all</code> (<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.4/java/ql/lib/CHANGELOG.md">changelog</a>,
<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.4/java/ql/lib">source</a>)</li>
<li><code>codeql/javascript-queries</code> (<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.4/javascript/ql/src/CHANGELOG.md">changelog</a>,
<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.4/javascript/ql/src">source</a>)</li>
<li><code>codeql/javascript-all</code> (<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.4/javascript/ql/lib/CHANGELOG.md">changelog</a>,
<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.4/javascript/ql/lib">source</a>)</li>
<li><code>codeql/python-queries</code> (<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.4/python/ql/src/CHANGELOG.md">changelog</a>,
<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.4/python/ql/src">source</a>)</li>
<li><code>codeql/python-all</code> (<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.4/python/ql/lib/CHANGELOG.md">changelog</a>,
<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.4/python/ql/lib">source</a>)</li>
<li><code>codeql/ruby-queries</code> (<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.4/ruby/ql/src/CHANGELOG.md">changelog</a>,
<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.4/ruby/ql/src">source</a>)</li>
<li><code>codeql/ruby-all</code> (<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.4/ruby/ql/lib/CHANGELOG.md">changelog</a>,
<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.4/ruby/ql/lib">source</a>)</li>
<li><code>codeql/swift-queries</code> (<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.4/swift/ql/src/CHANGELOG.md">changelog</a>,
<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.4/swift/ql/src">source</a>)</li>
<li><code>codeql/swift-all</code> (<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.4/swift/ql/lib/CHANGELOG.md">changelog</a>,
<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.4/swift/ql/lib">source</a>)</li>
</ul>
<h2>CodeQL Bundle</h2>
<p>Bundles CodeQL CLI v2.15.3</p>
<ul>
<li>(<a
href="https://github.com/github/codeql-cli-binaries/blob/HEAD/CHANGELOG.md">changelog</a>,
<a
href="https://github.com/github/codeql-cli-binaries/releases/tag/v2.15.3">release</a>)</li>
</ul>
<p>Includes the following CodeQL language packs from <a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.3"><code>github/codeql@codeql-cli/v2.15.3</code></a>:</p>
<ul>
<li><code>codeql/cpp-queries</code> (<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.3/cpp/ql/src/CHANGELOG.md">changelog</a>,
<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.3/cpp/ql/src">source</a>)</li>
<li><code>codeql/cpp-all</code> (<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.3/cpp/ql/lib/CHANGELOG.md">changelog</a>,
<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.3/cpp/ql/lib">source</a>)</li>
<li><code>codeql/csharp-queries</code> (<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.3/csharp/ql/src/CHANGELOG.md">changelog</a>,
<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.3/csharp/ql/src">source</a>)</li>
<li><code>codeql/csharp-all</code> (<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.3/csharp/ql/lib/CHANGELOG.md">changelog</a>,
<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.3/csharp/ql/lib">source</a>)</li>
<li><code>codeql/go-queries</code> (<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.3/go/ql/src/CHANGELOG.md">changelog</a>,
<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.3/go/ql/src">source</a>)</li>
<li><code>codeql/go-all</code> (<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.3/go/ql/lib/CHANGELOG.md">changelog</a>,
<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.3/go/ql/lib">source</a>)</li>
<li><code>codeql/java-queries</code> (<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.3/java/ql/src/CHANGELOG.md">changelog</a>,
<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.3/java/ql/src">source</a>)</li>
<li><code>codeql/java-all</code> (<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.3/java/ql/lib/CHANGELOG.md">changelog</a>,
<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.3/java/ql/lib">source</a>)</li>
<li><code>codeql/javascript-queries</code> (<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.3/javascript/ql/src/CHANGELOG.md">changelog</a>,
<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.3/javascript/ql/src">source</a>)</li>
<li><code>codeql/javascript-all</code> (<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.3/javascript/ql/lib/CHANGELOG.md">changelog</a>,
<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.3/javascript/ql/lib">source</a>)</li>
<li><code>codeql/python-queries</code> (<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.3/python/ql/src/CHANGELOG.md">changelog</a>,
<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.3/python/ql/src">source</a>)</li>
<li><code>codeql/python-all</code> (<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.3/python/ql/lib/CHANGELOG.md">changelog</a>,
<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.3/python/ql/lib">source</a>)</li>
<li><code>codeql/ruby-queries</code> (<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.3/ruby/ql/src/CHANGELOG.md">changelog</a>,
<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.3/ruby/ql/src">source</a>)</li>
<li><code>codeql/ruby-all</code> (<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.3/ruby/ql/lib/CHANGELOG.md">changelog</a>,
<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.3/ruby/ql/lib">source</a>)</li>
<li><code>codeql/swift-queries</code> (<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.3/swift/ql/src/CHANGELOG.md">changelog</a>,
<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.3/swift/ql/src">source</a>)</li>
<li><code>codeql/swift-all</code> (<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.3/swift/ql/lib/CHANGELOG.md">changelog</a>,
<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.3/swift/ql/lib">source</a>)</li>
</ul>
<h2>CodeQL Bundle</h2>
<p>Bundles CodeQL CLI v2.15.2</p>
<ul>
<li>(<a
href="https://github.com/github/codeql-cli-binaries/blob/HEAD/CHANGELOG.md">changelog</a>,
<a
href="https://github.com/github/codeql-cli-binaries/releases/tag/v2.15.2">release</a>)</li>
</ul>
<p>Includes the following CodeQL language packs from <a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.2"><code>github/codeql@codeql-cli/v2.15.2</code></a>:</p>
<ul>
<li><code>codeql/cpp-queries</code> (<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.2/cpp/ql/src/CHANGELOG.md">changelog</a>,
<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.2/cpp/ql/src">source</a>)</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/github/codeql-action/blob/main/CHANGELOG.md">github/codeql-action's
changelog</a>.</em></p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="3a9f6a89e0"><code>3a9f6a8</code></a>
update javascript files</li>
<li><a
href="cc4fead714"><code>cc4fead</code></a>
update version in various hardcoded locations</li>
<li><a
href="183559cea8"><code>183559c</code></a>
Merge branch 'main' into update-bundle/codeql-bundle-v2.15.4</li>
<li><a
href="5b52b36d41"><code>5b52b36</code></a>
reintroduce PR check that confirm action can be still be compiled on
node16</li>
<li><a
href="5b19bef41e"><code>5b19bef</code></a>
change to node20 for all actions</li>
<li><a
href="f2d0c2e7ae"><code>f2d0c2e</code></a>
upgrade node type definitions</li>
<li><a
href="d651fbc494"><code>d651fbc</code></a>
change to node20 for all actions</li>
<li><a
href="382a50a028"><code>382a50a</code></a>
Merge pull request <a
href="https://redirect.github.com/github/codeql-action/issues/2021">#2021</a>
from github/mergeback/v2.22.9-to-main-c0d1daa7</li>
<li><a
href="458b4226ad"><code>458b422</code></a>
Update checked-in dependencies</li>
<li><a
href="5e0f9dbc48"><code>5e0f9db</code></a>
Update changelog and version after v2.22.9</li>
<li>See full diff in <a
href="https://github.com/github/codeql-action/compare/v2...v3">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=github/codeql-action&package-manager=github_actions&previous-version=2&new-version=3)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-12-28 11:32:23 +02:00
guybe7
12b611b374
WAITAOF: Try to wake blocked clients ASAP in the next beforeSleep (#12627)
In case server.fsynced_reploff changed (e.g. flushAppendOnly set it to
server.master_repl_offset in case there was nothing to fsync) we want to
avoid sleeping before the next beforeSleep so we can call
blockedBeforeSleep ASAP.
without that, in case there's no incoming traffic, we could be waiting
for the next cron timer event to wake us up.
2023-12-28 11:27:58 +02:00
Binbin
99c468c38c
Fix crash caused by pubsubShardUnsubscribeAllChannelsInSlot not deleting the client (#12896)
The code does not delete the corresponding node when traversing clients,
resulting in a loop, causing the dictDelete() == DICT_OK assertion to
fail.

In addition, did a cleanup, in the dictCreate scenario, we can avoid a
dictFind call since the dict is empty.

Issue was introduced in #12804.
2023-12-28 08:32:51 +02:00
Binbin
5b1fe925f2
Adjust redis-cli --cluster create arity from -2 to -1 (#12892)
When arity is -2, it allows us to input two nodes, but returns:
```
*** ERROR: Invalid configuration for cluster creation.
*** Redis Cluster requires at least 3 master nodes.
*** This is not possible with 2 nodes and 0 replicas per node.
*** At least 3 nodes are required.
```

When we input one node, it return:
```
[ERR] Wrong number of arguments for specified --cluster sub command
```

Strictly speaking -2 should also be rejected, because redis-cli
requires at least three nodes. However, the error message was not
very friendly, so decided to change it arity -1.

This closes #12891.
2023-12-28 08:26:23 +02:00
Chen Tianjie
8527959598
Replace slots_to_channels radix tree with slot specific dictionaries for shard channels. (#12804)
We have achieved replacing `slots_to_keys` radix tree with key->slot
linked list (#9356), and then replacing the list with slot specific
dictionaries for keys (#11695).

Shard channels behave just like keys in many ways, and we also need a
slots->channels mapping. Currently this is still done by using a radix
tree. So we should split `server.pubsubshard_channels` into 16384 dicts
and drop the radix tree, just like what we did to DBs.

Some benefits (basically the benefits of what we've done to DBs):
1. Optimize counting channels in a slot. This is currently used only in
removing channels in a slot. But this is potentially more useful:
sometimes we need to know how many channels there are in a specific slot
when doing slot migration. Counting is now implemented by traversing the
radix tree, and with this PR it will be as simple as calling `dictSize`,
from O(n) to O(1).
2. The radix tree in the cluster has been removed. The shard channel
names no longer require additional storage, which can save memory.
3. Potentially useful in slot migration, as shard channels are logically
split by slots, thus making it easier to migrate, remove or add as a
whole.
4. Avoid rehashing a big dict when there is a large number of channels.

Drawbacks:
1. Takes more memory than using radix tree when there are relatively few
shard channels.

What this PR does:
1. in cluster mode, split `server.pubsubshard_channels` into 16384
dicts, in standalone mode, still use only one dict.
2. drop the `slots_to_channels` radix tree.
3. to save memory (to solve the drawback above), all 16384 dicts are
created lazily, which means only when a channel is about to be inserted
to the dict will the dict be initialized, and when all channels are
deleted, the dict would delete itself.
5. use `server.shard_channel_count` to keep track of the number of all
shard channels.

---------

Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2023-12-27 17:40:45 +08:00
Moshe Kaplan
fa751f9bef
config.c: Avoid leaking file handle if file is 0 bytes (#12828)
If fopen() is successful and redis_fstat determines that the file is 0
bytes, the file handle stored in fp will leak. This change closes the
filehandle stored in fp if the file is 0 bytes.

Second attempt at fixing Coverity 390029

This is a follow-up to #12796
2023-12-27 08:53:56 +02:00
sundb
bef5715374
Fix oom-score-adj test due to no permission (#12887)
Fix #12792

On ubuntu 23(lunar), non-root users will not be allowed to change the
oom_score_adj of a process to a value that is too low.
Since terminal's default oom_score_adj is 200, if we run the test on
terminal, we won't be able to set the oom_score_adj of the redis process
to 9 or 22, which is too low.

Reproduction on ubuntu 23(lunar) terminal:
```sh
$ cat /proc/`pgrep redis-server`/oom_score_adj
200
$ echo 100 > /proc/`pgrep redis-server`/oom_score_adj
# success without error
$ echo 99 > /proc/`pgrep redis-server`/oom_score_adj
echo: write error: Permission denied
```

As from the output above, we can only set the minimum oom score of redis
processes to 100.
By modifying the test, make oom_score_adj only increase upwards and not
decrease.

---------

Co-authored-by: debing.sun <debing.sun@redis.com>
2023-12-27 08:42:46 +02:00
Andy Pan
1aa633d61b
Implement TCP Keep-Alives across most Unix-like systems (#12782)
## TCP Keep-Alives

[TCP
Keep-Alives](https://datatracker.ietf.org/doc/html/rfc9293#name-tcp-keep-alives)
provides a way to detect whether a TCP connection is alive or dead,
which can be useful for reducing system resources by cleaning up dead
connections.

There is full support of TCP Keep-Alives on Linux and partial support on
macOS in `redis` at present.

This PR intends to complete the rest.

## Unix-like OS's support

`TCP_KEEPIDLE`, `TCP_KEEPINTVL`, and `TCP_KEEPCNT` are not included in
the POSIX standard for `setsockopts`, while these three socket options
are widely available on most Unix-like systems and Windows.

### References

- [AIX](https://www.ibm.com/support/pages/ibm-aix-tcp-keepalive-probes)
- [DragonflyBSD](https://man.dragonflybsd.org/?command=tcp&section=4)
- [FreeBSD](https://www.freebsd.org/cgi/man.cgi?query=tcp)
-
[HP-UX](https://docstore.mik.ua/manuals/hp-ux/en/B2355-60130/TCP.7P.html)
- [illumos](https://illumos.org/man/4P/tcp)
- [Linux](https://man7.org/linux/man-pages/man7/tcp.7.html)
- [NetBSD](https://man.netbsd.org/NetBSD-8.0/tcp.4)
-
[Windows](https://learn.microsoft.com/en-us/windows/win32/winsock/ipproto-tcp-socket-options)

### Mac OS

In earlier versions, macOS only supported setting `TCP_KEEPALIVE` (the
equivalent of `TCP_KEEPIDLE` on other platforms), but since macOS 10.8
it has supported `TCP_KEEPINTVL` and `TCP_KEEPCNT`.

Check out [this mailing
list](https://lists.apple.com/archives/macnetworkprog/2012/Jul/msg00005.html)
and [the source
code](https://github.com/apple/darwin-xnu/blob/main/bsd/netinet/tcp.h#L215-L230)
for more details.

### Solaris

Solaris claimed it supported the TCP-Alives mechanism, but
`TCP_KEEPIDLE`, `TCP_KEEPINTVL`, and `TCP_KEEPCNT` were not available on
Solaris until the latest version 11.4. Therefore, we need to simulate
the TCP-Alives mechanism on other platforms via
`TCP_KEEPALIVE_THRESHOLD` + `TCP_KEEPALIVE_ABORT_THRESHOLD`.

- [Solaris
11.3](https://docs.oracle.com/cd/E86824_01/html/E54777/tcp-7p.html)
- [Solaris
11.4](https://docs.oracle.com/cd/E88353_01/html/E37851/tcp-4p.html)

---------

Co-authored-by: Oran Agra <oran@redislabs.com>
2023-12-26 18:44:18 +02:00
Jeff Liu
27a8e3b04e
fix missing comments (#12878)
add a missing comment for `dont_compress` and fix the bits calculation
2023-12-25 19:30:05 -08:00
zalj
baf5699d77
fix comment of aeProcessEvents (#12884)
The implementation of aeProcessEvents seems have different behavior from
the top comment.
The implementation process file events first, then process time events.
2023-12-25 18:58:14 -08:00
Binbin
71f31da66f
Add restart option to create-cluster script (#12885)
When testing and debugging the cluster code before, you need
to stop the cluster after making changes, and then start the
cluster again. Add a restart option for ease of use.
2023-12-25 18:36:44 -08:00
Slava Koyfman
20214b26a4
Don't disconnect all clients in ACL LOAD (#12171)
Previous implementation would disconnect _all_ clients when running `ACL
LOAD`, which wasn't very useful.

This change brings the behavior in line with that of `ACL SETUSER`, `ACL
DELUSER`, in that only clients whose user is deleted or clients
subscribed to channels which they no longer have access to will be
disconnected.

---------

Co-authored-by: Oran Agra <oran@redislabs.com>
Co-authored-by: Madelyn Olson <34459052+madolson@users.noreply.github.com>
2023-12-24 11:56:44 +02:00
Binbin
09e0d338f5
redis-cli adds -4 / -6 options to determine IPV4 / IPV6 priority in DNS lookup (#11315)
This PR, we added -4 and -6 options to redis-cli to determine
IPV4 / IPV6 priority in DNS lookup.
This was mentioned in
https://github.com/redis/redis/pull/11151#issuecomment-1231570651

For now it's only used in CLUSTER MEET.

The options also made it possible to reliably test dns lookup in CI,
using this option, we can add some localhost tests for #11151.

The commit was cherry-picked from #11151, back then we decided to split
the PR.

Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2023-12-24 10:40:34 +02:00
Binbin
23e980e77a
Move cliVersion to cli_common and add --version support for redis-check-aof (#10856)
Let us see which version of redis this tool is part of.
Similarly to redis-cli, redis-benchmark and redis-check-rdb

redis-rdb-check and redis-aof-check are actually symlinks to redis,
so they will directly use getVersion in server, the format became:
```
{title} v={redis_version} sha={sha}:{dirty} malloc={malloc} bits={bits} build={build}
```

Move cliVersion into cli_common, redis-cli and redis-benchmark will
use it, and the format is not change:
```
{title} {redis_version} (git:{sha})
```
2023-12-21 13:51:46 +02:00
Wen Hui
5dc631d880
Add missing test cases for hash commands (#12851)
We dont have test for hgetall against key doesnot exist so added the
test in test suite and along with this, added wrong type cases for other
missing commands.
2023-12-17 14:02:53 +02:00
Binbin
adbb534f03
Always keep an in-memory history of all commands in redis-cli (#12862)
redis-cli avoids saving sensitive commands in it's history (doesn't
persist them to the history file).
this means that if you had a typo and you wanna re-run the command, you
can't easily do that.
This PR changes that to keep an in-memory history of all the redacted
commands, and just
not persist them to disk. This way we would be able to press the up
arrow and
re-try the command freely, and it'll just not survive a redis-cli
restart.
2023-12-15 17:22:02 +02:00
zhaozhao.zz
d8a21c5767
Unified db rehash method for both standalone and cluster (#12848)
After #11695, we added two functions `rehashingStarted` and
`rehashingCompleted` to the dict structure. We also registered two
handlers for the main database's dict and expire structures. This allows
the main database to record the dict in `rehashing` list when rehashing
starts. Later, in `serverCron`, the `incrementallyRehash` function is
continuously called to perform the rehashing operation. However,
currently, when rehashing is completed, `rehashingCompleted` does not
remove the dict from the `rehashing` list. This results in the
`rehashing` list containing many invalid dicts. Although subsequent cron
checks and removes dicts that don't require rehashing, it is still
inefficient.

This PR implements the functionality to remove the dict from the
`rehashing` list in `rehashingCompleted`. This is achieved by adding
`metadata` to the dict structure, which keeps track of its position in
the `rehashing` list, allowing for quick removal. This approach avoids
storing duplicate dicts in the `rehashing` list.

Additionally, there are other modifications:

1. Whether in standalone or cluster mode, the dict in database is
inserted into the rehashing linked list when rehashing starts. This
eliminates the need to distinguish between standalone and cluster mode
in `incrementallyRehash`. The function only needs to focus on the dicts
in the `rehashing` list that require rehashing.
2. `rehashing` list is moved from per-database to Redis server level.
This decouples `incrementallyRehash` from the database ID, and in
standalone mode, there is no need to iterate over all databases,
avoiding unnecessary access to databases that do not require rehashing.
In the future, even if unsharded-cluster mode supports multiple
databases, there will be no risk involved.
3. The insertion and removal operations of dict structures in the
`rehashing` list are decoupled from `activerehashing` config.
`activerehashing` only controls whether `incrementallyRehash` is
executed in serverCron. There is no need for additional steps when
modifying the `activerehashing` switch, as in #12705.
2023-12-15 10:42:53 +08:00
Guillaume Koenig
967fb3c6e8
Extend rax usage by allowing any long long value (#12837)
The raxFind implementation uses a special pointer value (the address of
a static string) as the "not found" value. It works as long as actual
pointers were used. However we've seen usages where long long,
non-pointer values have been used. It creates a risk that one of the
long long value precisely is the address of the special "not found"
value. This commit changes raxFind to return 1 or 0 to indicate
elementhood, and take in a new void **value to optionally return the
associated value.

By extension, this also allow the RedisModule_DictSet/Replace operations
to also safely insert integers instead of just pointers.
2023-12-14 14:50:18 -08:00
Chen Tianjie
e95a5d4831
Support by/get options for sort(_ro) in cluster mode when pattern implies slot. (#12728)
The by/get options of sort/sort_ro command used to be forbidden in
cluster mode, since we are not sure which slot the pattern may be in.

As the optimization done in #12536, patterns now can be mapped to slots,
we should allow by/get options in cluster mode when the pattern maps to
the same slot as the key.
2023-12-13 21:16:36 +02:00
Binbin
3c0fd25201
Redact ACL username information and mark *-key-file-pass configs as sensitive (#12860)
In #11489, we consider acl username to be sensitive information,
and consider the ACL GETUSER a sensitive command and remove it
from redis-cli historyfile.

This PR redact username information in ACL GETUSER and ACL DELUSER
from SLOWLOG, and also remove ACL DELUSER from redis-cli historyfile.

This PR also mark tls-key-file-pass and tls-client-key-file-pass
as sensitive config, will redact it from SLOWLOG and also
remove them from redis-cli historyfile.
2023-12-13 15:28:13 +02:00
Chen Tianjie
f9cc25c1dd
Add metric to INFO CLIENTS: pubsub_clients. (#12849)
In INFO CLIENTS section, we already have blocked_clients and
tracking_clients. We should add a new metric showing the number of
pubsub connections, which helps performance monitoring and trouble
shooting.
2023-12-13 13:44:13 +08:00
Binbin
c85a9b7896
Fix delKeysInSlot server events are not executed inside an execution unit (#12745)
This is a follow-up fix to #12733. We need to apply the same changes to
delKeysInSlot. Refer to #12733 for more details.

This PR contains some other minor cleanups / improvements to the test
suite and docs.
It uses the postnotifications test module in a cluster mode test which
revealed a leak in the test module (fixed).
2023-12-11 20:15:19 +02:00
Binbin
62419c01db
Handle missing fields in dbSwapDatabases and swapMainDbWithTempDb (#12763)
The change in dbSwapDatabases seems harmless. Because in non-clustered
mode, dbBuckets calculations are strictly accurate and in cluster mode,
we only have one DB. Modify it for uniformity (just like resize_cursor).

The change in swapMainDbWithTempDb is needed in case we swap with the
temp db, otherwise the overhead memory usage of db can be miscalculated.

In addition we will swap all fields (including rehashing list), just for
completeness (and reduce the chance of surprises in the future).

Introduced in #12697.
2023-12-10 17:30:20 +02:00
Binbin
e6423b7a7e
Fix rehashingStarted miscalculating bucket_count in dict initialization (#12846)
In the old dictRehashingInfo implementation, for the initialization
scenario,
it mistakenly directly set to_size to DICTHT_SIZE(DICT_HT_INITIAL_EXP),
which
is 4 in our code by default.

In scenarios where dictExpand directly passes the target size as
initialization,
the code will calculate bucket_count incorrectly. For example, in DEBUG
POPULATE
or RDB load scenarios, it will cause the final bucket_count to be
initialized to
65536 (16384 * 4), see:
```
before:
DB 0: 10000000 keys (0 volatile) in 65536 slots HT.

it should be:
DB 0: 10000000 keys (0 volatile) in 16777216 slots HT.
```

In PR, new ht will also be initialized before calling rehashingStarted
in
_dictExpand, so that the calls in dictRehashingInfo can be unified.

Bug was introduced in #12697.
2023-12-10 10:55:30 +02:00
Binbin
a3ae2ed37b
Remove dead code around should_expand_db (#12767)
when dbExpand is called from rdb.c with try_expand set to 0, it will
either panic panic on OOM, or be non-fatal (should not fail RDB loading)

At the same time, the log text has been slightly adjusted to make it
more unified.
2023-12-10 10:40:15 +02:00
Binbin
7410d985bc
Remove overhead.hashtable.slot-to-keys from memory-stats reply_schema (#12784)
overhead.hashtable.slot-to-keys was added in 7.0 in #10017, then removed
in #11695. Now remove it from reply_schema.
2023-12-10 09:46:21 +02:00
Yanqi Lv
15b993f1ef
Optimize dictExpand of empty dict (#12847)
If a dict is empty before `dictExpand`, we don't need to do rehashing
actually. We can just create a new dict and set it as ht_table[0] to
skip incremental rehashing and free the memory quickly.
2023-12-08 17:48:52 +08:00
bentotten
826b39e016
Align server.lastsave and server.rdb_save_time_last by removing multiple calls to time(NULL) (#12823)
This makes sure the various times (server.lastsave and server.rdb_save_time_last) are aligned by using the result of the same time call.
2023-12-07 17:03:51 -08:00
Chen Tianjie
f2d59c4f91
Avoid unnecessary slot computing in KEYS command. (#12843)
If not in cluster mode, there is no need to compute slot.

A bit optimization for #12754
2023-12-07 19:48:15 +08:00
zhaozhao.zz
8e11f84ded
Fix replica node cannot expand dicts when loading legacy RDB (#12839)
When loading RDB on cluster nodes, it is necessary to consider the
scenario where a node is a replica.

For example, during a rolling upgrade, new version instances are often
mounted as replicas on old version instances. In this case, the full
synchronization legacy RDB does not contain slot information, and the
new version instance, acting as a replica, should be able to handle the
legacy RDB correctly for `dbExpand`.

Additionally, renaming `getMyClusterSlotCount` to `getMyShardSlotCount`
would be appropriate.

Introduced in #11695
2023-12-07 14:30:48 +08:00
Moshe Kaplan
e2a3f3091f
coverity.yml: Upload should go to project redis-unstable (#12841)
Coverity project name was changed from redis to redis-unstable. Fix the
upload destination to also go to redis-unstable.

Continuation of #12807
2023-12-06 20:51:58 +02:00
Binbin
2f6d4dabaa
Fix outdated LFU comments to eliminate confusion (#12244)
The decrement time was replaced by access time in
583c314725.

The halved and doubled LFU_INIT_VAL logic has been changed in
06ca9d6839.
Now we just decrement the counter by num_periods. This has been
previously fixed in redis.conf, #11108.
2023-12-06 20:46:57 +02:00
Moshe Kaplan
77e69d8884
GH Workflows: Create CI job for Coverity scan (#12807)
I've noticed that https://scan.coverity.com/projects/redis already
exists, but appears to be only updated on an ad-hoc basis. creating
[redis-unstable](https://scan.coverity.com/projects/redis-unstable?tab=project_settings)
project in coverity for this CI.

This PR adds a GitHub Action-based CI job to create a new Coverity build
once daily, so that there is always a recent scan available.

This is within the limit, as Redis is ~150K LOC and per
https://scan.coverity.com/faq#frequency :

> Up to 21 builds per week, with a maximum of 3 builds per day, for
projects with 100K to 500K lines of code

Before this is merged in, two new secrets will need to be created:

COVERITY_SCAN_EMAIL with the email address used for accessing Coverity
COVERITY_SCAN_TOKEN with the Project token from
https://scan.coverity.com/projects/redis-unstable?tab=project_settings

---------

Co-authored-by: Oran Agra <oran@redislabs.com>
2023-12-06 14:50:00 +02:00
zhaozhao.zz
b730404c2f
Fix multi dbs donot dbExpand when loading RDB (#12840)
Currently, during RDB loading, once a `dbExpand` is performed, the
`should_expand_db` flag is set to 0. This causes the remaining DBs
unable to do `dbExpand` when there are multiple DBs.

To fix this issue, we need to set `should_expand_db` back to 1 whenever
we encounter `RDB_OPCODE_RESIZEDB`. This ensures that each DB can
perform `dbExpand` correctly.

Additionally, the initial value of `should_expand_db` should also be set
to 0 to prevent invalid `dbExpand` in older versions of RDB where
`RDB_OPCODE_RESIZEDB` is not present.

problem introduced in #11695
2023-12-06 10:59:56 +02:00
zhaozhao.zz
9ee1cc33a3
Make the sampling logic in eviction clearer (#12781)
Additional optimizations for the eviction logic in #11695:

To make the eviction logic clearer and decouple the number of sampled
keys from the running mode (cluster or standalone).
* When sampling in each database, we only care about the number of keys
in the current database (not the dicts we sampled from).
* If there are a insufficient number of keys in the current database
(e.g. 10 times the value of `maxmemory_samples`), we can break out
sooner (to avoid looping on a sparse database).
* We'll never try to sample the db dicts more times than the number of
non-empty dicts in the db (max 1 in non-cluster mode).

And it also ensures that each database has a sufficient amount of
sampled keys, so even if unsharded-cluster supports multiple databases,
there won't be any issues.

other changes:
1. keep track of the number of non-empty dicts in each database.
2. move key_count tracking into cumulativeKeyCountAdd rather than all
it's callers

---------

Co-authored-by: Oran Agra <oran@redislabs.com>
2023-12-06 09:37:24 +08:00
Chen Tianjie
991aff1c0f
Optimize KEYS when pattern includes hashtag and implies a single slot. (#12754)
in #12536 we made a similar optimization for SCAN, now that hashtags in
patterns. When we can make sure all keys matching the pettern will be in
the same slot, we can limit the iteration to run only one one.
2023-12-05 16:21:50 +02:00
Madelyn Olson
35c8d616cf
Only rebuild server when src or deps are modified (#12799)
`make test` will unnecessary rebuild `redis-server` if tests are modified because mkheader will touch releae.c.
This changes scopes down when we re-generate release.c to only when actual source files are modified.
2023-12-04 19:24:24 -08:00
Binbin
764838d66f
Check whether the client is NULL in luaCreateFunction (#12829)
It was first added to load lua from RDB, see 28dfdca7. After #9812,
we no longer save lua in RDB. luaCreateFunction will only be called
in script load and eval*, both of which are available in the client.

It could be that that some day we'll still want to load scripts from
somewhere that's not a client. This fix is in dead code.
2023-12-04 20:12:48 +02:00
Binbin
8a4ccb01b3
Fix clusterLoadConfig aux_argv minor memory leak (#12726)
We forgot to call sdsfreesplitres. This is just a cleanup since it will
only be leaked in the error paths, and we will exit on the error paths.
2023-12-03 11:00:53 +02:00
sundb
91309f7981
Fix compilation warning in KeySpace_ServerEventCallback and add CFLAGS=-Werror flag for module CI (#12786)
Warning:
```
postnotifications.c:216:77: warning: format specifies type 'long' but the argument has type 'uint64_t' (aka 'unsigned long long') [-Wformat]
        RedisModule_Log(ctx, "warning", "Got an unexpected subevent '%ld'", subevent);
                                                                     ~~~    ^~~~~~~~
                                                                     %llu
```

CI:
https://github.com/redis/redis/actions/runs/6937308713/job/18871124342#step:6:115

## Other
Add `CFLAGS=-Werror` flag for module CI.

---------

Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2023-11-30 17:41:00 +02:00
Binbin
e216c83909
Change addReplyErrorFormat to addReplyError when there is no format (#12641)
This is just a cleanup, although they are both correct, the change
is normatively better, and addReplyError is also much faster.
Although not important, speed is not important for these error cases.
2023-11-30 12:36:17 +02:00
lyq2333
423565f784
Optimize CPU cache efficiency on dict while rehashing in dictTwoPhaseUnlinkFind (#12818)
In #5692, we optimize CPU cache efficiency on dict while rehashing but
missed the modification in dictTwoPhaseUnlinkFind.

This PR supplements it.
2023-11-30 13:50:09 +08:00
Binbin
22cc9b5122
Use CLZ in _dictNextExp to get the next power of two (#12815)
In the past, we did not call _dictNextExp frequently. It was only
called when the dictionary was expanded.

Later, dictTypeExpandAllowed was introduced in #7954, which is 6.2.
For the data dict and the expire dict, we can check maxmemory before
actually expanding the dict. This is a good optimization to avoid
maxmemory being exceeded due to the dict expansion.

And in #11692, we moved the dictTypeExpandAllowed check before the
threshold check, this caused a bit of performance degradation, every
time a key is added to the dict, dictTypeExpandAllowed is called to
check.

The main reason for degradation is that in a large dict, we need to
call _dictNextExp frequently, that is, every time we add a key, we
need to call _dictNextExp once. Then the threshold is checked to see
if the dict needs to be expanded. We can see that the order of checks
here can be optimized.

So we moved the dictTypeExpandAllowed check back to after the threshold
check in #12789. In this way, before the dict is actually expanded (that
is, before the threshold is reached), we will not do anything extra
compared to before, that is, we will not call _dictNextExp frequently.

But note we'll still hit the degradation when we over the thresholds.
When the threshold is reached, because #7954, we may delay the dict
expansion due to maxmemory limitations. In this case, we will call
_dictNextExp every time we add a key during this period.

This PR use CLZ in _dictNextExp to get the next power of two. CLZ (count
leading zeros) can easily give you the next power of two. It should be
noted that we have actually introduced the use of __builtin_clzl in
#8687,
which is 7.0. So i suppose all the platforms we use have it (even if the
CPU doesn't have an instruction).

We build 67108864 (2**26) keys through DEBUG POPULTE, which will use
approximately 5.49G memory (used_memory:5898522936). If expansion is
triggered, the additional hash table will consume approximately 1G
memory (2 ** 27 * 8). So we set maxmemory to 6871947673 (that is, 6.4G),
which will be less than 5.49G + 1G, so we will delay the dict rehash
while addint the keys.

After that, each time an element is added to the dict, an allow check
will be performed, that is, we can frequently call _dictNextExp to test
the comparison before and after the optimization. Using DEBUG HTSTATS 0
to
check and make sure that our dict expansion is dealyed.

Using `./src/redis-server redis.conf --save "" --maxmemory 6871947673`.
Using `./src/redis-benchmark -P 100 -r 1000000000 -t set -n 5000000`.
After ten rounds of testing:
```
unstable:           this PR:
769585.94           816860.00
771724.00           818196.69
775674.81           822368.44
781983.12           822503.69
783576.25           828088.75
784190.75           828637.75
791389.69           829875.50
794659.94           835660.69
798212.00           830013.25
801153.62           833934.56
```

We can see there is about 4-5% performance improvement in this case.
2023-11-29 14:42:22 +02:00
Binbin
bdceaf50e4
Fix the check for new RDB_OPCODE_SLOT_INFO in redis-check-rdb (#12768)
We did not read expires_slot_size, causing its check to fail.
An overlook in #11695.
2023-11-29 14:25:17 +02:00