Commit Graph

53 Commits

Author SHA1 Message Date
Oran Agra
1a57af629c
Split daily CI into smaller chunks (#10469)
this should aid find the CI issues with freebsd and macos runs, and also
get faster results from valgrind and tls
2022-03-22 17:38:01 +02:00
Oran Agra
9478d5a134
enable daily CI on release branches (#10357) 2022-02-28 13:17:56 +02:00
Oran Agra
1193e96d02
Add workflow_dispatch filters for daily CI. (#10289)
sometimes you just wanna run one test on one system (e.g. memefficiency
on macos), so you want all other tests to be skipped
2022-02-13 17:43:19 +02:00
dependabot[bot]
edc050cc57
Bump vmactions/freebsd-vm from 0.1.5 to 0.1.6 (#10219) 2022-02-02 10:39:34 +02:00
Matthieu MOREL
d5a3b3f5ec
Setup dependabot for github-actions and codespell (#9857)
This sets up  dependabot to check weekly updates for pip and github-actions dependencies.
If it finds an update it will create a PR to update the dependency. More information can be found here

It includes the update of:

* vmactions/freebsd-vm from 0.1.4 to 0.1.5
* codespell from 2.0.0 to 2.1.0

Also includes spelling fixes found by the latest version of codespell.
Includes a dedicated .codespell folder so dependabot can read a requirements.txt file and every files dedicated to codespell can be grouped in the same place

Co-Authored-By: Matthieu MOREL <mmorel-35@users.noreply.github.com>
Co-Authored-By: MOREL Matthieu <matthieu.morel@cnp.fr>
2022-01-04 16:19:28 +02:00
Binbin
980bb3ae19
Add REDIS_CFLAGS='-Werror' to CI tests (#9828)
Update CI so that warnings cause build failures.

Also fix a warning in `test-sanitizer-address`:
```
In function ‘strncpy’,
   inlined from ‘clusterUpdateMyselfIp’ at cluster.c:545:13:

/usr/include/x86_64-linux-gnu/bits/string_fortified.h:106:10:
error: ‘__builtin_strncpy’ specified bound 46 equals destination size [-Werror=stringop-truncation]

  106 |   return __builtin___strncpy_chk (__dest, __src, __len, __bos (__dest));
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
cc1: all warnings being treated as errors
```
2021-11-29 10:30:35 +02:00
Meir Shpilraien (Spielrein)
a8c1253b6f
Fix Lua C API violation on lua msgpack lib. (#9832)
msgpack lib missed using lua_checkstack and so on rare
cases overflow the stack by at most 2 elements. This is a
violation of the Lua C API. Notice that Lua allocates
additional 5 more elements on top of lua->stack_last
so Redis does not access an invalid memory. But it is an
API violation and we should avoid it.

This PR also added a new Lua compilation option. The new
option can be enable using environment variable called
LUA_DEBUG. If set to `yes` (by default `no`), Lua will be
compiled without optimizations and with debug symbols (`-O0 -g`).
In addition, in this new mode, Lua will be compiled with the
`-DLUA_USE_APICHECK` flag that enables extended Lua C API
validations.

In addition, set LUA_DEBUG=yes on daily valgrind flow so we
will be able to catch Lua C API violations in the future.
2021-11-28 11:33:09 +02:00
Ozan Tezcan
a68b71ac02
Don't use accurate option with ASAN unit tests (#9818)
specifically the ziplist and listpack unit tests and benchmarks run for too long with address sanitizer and --accurate
2021-11-21 18:27:54 +02:00
Binbin
020092e08f
Add --dump-logs to ci tests and daily tests (#9803)
So that we can print server log on test failure.
2021-11-18 16:04:01 +02:00
Oran Agra
45144fc79f
Daily tests flags inconsistency (#9792)
Add --accurate to unit tests (new feature recently added)
Add --no-latency to valgrind run (was present only for modules)
add --no-latency to macos and freebsd runs (was not present for modules)
add --timeout to freebsd (same one we have for valgrind)
2021-11-16 21:00:13 +02:00
Ozan Tezcan
bcb7961f12
Enable running daily CI from forks (#9771)
Was impossible to run the daily CI from private forks due to "redis/redis" repo check.
Let's disable that check for manual triggers.
2021-11-11 14:39:20 +02:00
Ozan Tezcan
b91d8b289b
Add sanitizer support and clean up sanitizer findings (#9601)
- Added sanitizer support. `address`, `undefined` and `thread` sanitizers are available.  
- To build Redis with desired sanitizer : `make SANITIZER=undefined`
- There were some sanitizer findings, cleaned up codebase
- Added tests with address and undefined behavior sanitizers to daily CI.
- Added tests with address sanitizer to the per-PR CI (smoke out mem leaks sooner).

Basically, there are three types of issues : 

**1- Unaligned load/store** : Most probably, this issue may cause a crash on a platform that
does not support unaligned access. Redis does unaligned access only on supported platforms.

**2- Signed integer overflow.** Although, signed overflow issue can be problematic time to time
and change how compiler generates code, current findings mostly about signed shift or simple
addition overflow. For most platforms Redis can be compiled for, this wouldn't cause any issue
as far as I can tell (checked generated code on godbolt.org).

 **3 -Minor leak** (redis-cli), **use-after-free**(just before calling exit());

UB means nothing guaranteed and risky to reason about program behavior but I don't think any
of the fixes here worth backporting. As sanitizers are now part of the CI, preventing new issues
will be the real benefit.
2021-11-11 13:51:33 +02:00
Oran Agra
0927a0dd24
Try solving test timeout on freebsd CI (#9768)
First, avoid using --accurate on the freebsd CI, we only care about
systematic issues there due to being different platform, but not
accuracy

Secondly, when looking at the test which timed out it seems silly and
outdated:
- it used KEYS to attempt to trigger lazy expiry, but KEYS doesn't do
  that anymore.
- it used some hard coded sleeps rather than waiting for things to
  happen and exiting ASAP
2021-11-10 19:39:26 +02:00
Oran Agra
978eadbad4
Increase test timeout in valgrind runs (#9767)
We saw some tests sporadically time out on valgrind (namely the ones
from #9323).

Increasing valgrind timeout from 20 mins to 40 mins in CI.
And fixing an outdated help message.
2021-11-10 19:38:58 +02:00
YaacovHazan
03406fcb6c
fix short timeout in replication short read tests (#9763)
In both tests, "diskless loading short read" and "diskless loading short read with module",
the timeout of waiting for the replica to respond to a short read and log it, is too short.

Also, add --dump-logs in runtest-moduleapi for valgrind runs.
2021-11-09 22:37:18 +02:00
perryitay
f27083a4a8
Add support for list type to store elements larger than 4GB (#9357)
Redis lists are stored in quicklist, which is currently a linked list of ziplists.
Ziplists are limited to storing elements no larger than 4GB, so when bigger
items are added they're getting truncated.
This PR changes quicklists so that they're capable of storing large items
in quicklist nodes that are plain string buffers rather than ziplist.

As part of the PR there were few other changes in redis: 
1. new DEBUG sub-commands: 
   - QUICKLIST-PACKED-THRESHOLD - set the threshold of for the node type to
     be plan or ziplist. default (1GB)
   - QUICKLIST <key> - Shows low level info about the quicklist encoding of <key>
2. rdb format change:
   - A new type was added - RDB_TYPE_LIST_QUICKLIST_2 . 
   - container type (packed / plain) was added to the beginning of the rdb object
     (before the actual node list).
3. testing:
   - Tests that requires over 100MB will be by default skipped. a new flag was
     added to 'runtest' to run the large memory tests (not used by default)

Co-authored-by: sundb <sundbcn@gmail.com>
Co-authored-by: Oran Agra <oran@redislabs.com>
2021-11-03 20:47:18 +02:00
Yossi Gottlieb
6d5a911707
Fix daily failures due to macos-latest change. (#9637)
* Fix test modules linking on macOS 11.x.
* Use macOS 10.x for FreeBSD VM as VirtualBox is not yet supported on
  11.
2021-10-17 00:07:27 +03:00
Huang Zhw
71d452876e
On 32 bit platform, the bit position of GETBIT/SETBIT/BITFIELD/BITCOUNT,BITPOS may overflow (see CVE-2021-32761) (#9191)
GETBIT, SETBIT may access wrong address because of wrap.
BITCOUNT and BITPOS may return wrapped results.
BITFIELD may access the wrong address but also allocate insufficient memory and segfault (see CVE-2021-32761).

This commit uses `uint64_t` or `long long` instead of `size_t`.
related https://github.com/redis/redis/pull/8096

At 32bit platform:
> setbit bit 4294967295 1
(integer) 0
> config set proto-max-bulk-len 536870913
OK
> append bit "\xFF"
(integer) 536870913
> getbit bit 4294967296
(integer) 0

When the bit index is larger than 4294967295, size_t can't hold bit index. In the past,  `proto-max-bulk-len` is limit to 536870912, so there is no problem.

After this commit, bit position is stored in `uint64_t` or `long long`. So when `proto-max-bulk-len > 536870912`, 32bit platforms can still be correct.

For 64bit platform, this problem still exists. The major reason is bit pos 8 times of byte pos. When proto-max-bulk-len is very larger, bit pos may overflow.
But at 64bit platform, we don't have so long string. So this bug may never happen.

Additionally this commit add a test cost `512MB` memory which is tag as `large-memory`. Make freebsd ci and valgrind ci ignore this test.
2021-07-21 16:25:19 +03:00
Oran Agra
9894ce5fc4
add missing skips to daily cycle (#9196)
minor cleanups.
2021-07-05 08:46:19 +03:00
Oran Agra
a5bc54f01e
Re-fix daily CI (#9141)
The freebsd and macos jobs were still broken.
also add a few more skip jobs options.
2021-06-24 12:50:10 +03:00
Oran Agra
eb09fbce1e
daily CI, fix odd yaml parsing error (#9137) 2021-06-23 13:56:40 -07:00
Oran Agra
0676a9fc6b
Fix broken daily due to manual action triggers. and attempt to improve them (#9134)
The daily CI was broken by #9119 seems that for cron scheduled tasks, these ifs aren't evaluated to false.
But also it turns out that workflow_dispatch is only able to run CI on branches in the main repo (not on PRs).
this is an attempt to overcome that by being able to checkout from any repo we want.
2021-06-23 22:13:24 +03:00
Oran Agra
b466471a52
Add manual triggers for Daily CI (#9119)
Hopefully now we'll be able to manually trigger extensive tests to PR branches without modifying them.
2021-06-22 17:23:59 +03:00
yoav-steinberg
2e88b06396
Bump freebsd-vm version to fix CI failures (#8876)
Specifically we had issues with NTP sync failure which was resolved here: 457af73456
2021-04-27 16:22:22 +03:00
Oran Agra
611959eee5
fuzz tester, try to print hung command (#8837) 2021-04-25 13:08:46 +03:00
sundb
95d6297db8
Add run all test support with define REDIS_TEST (#8570)
1. Add `redis-server test all` support to run all tests.
2. Add redis test to daily ci.
3. Add `--accurate` option to run slow tests for more iterations (so that
   by default we run less cycles (shorter time, and less prints).
4. Move dict benchmark to REDIS_TEST.
5. fix some leaks in tests
6. make quicklist tests run on a specific fill set of options rather than huge ranges
7. move some prints in quicklist test outside their loops to reduce prints
8. removing sds.h from dict.c since it is now used in both redis-server and
   redis-cli (uses hiredis sds)
2021-03-10 09:13:11 +02:00
Harkrishn Patro
25f8d4fb41
Run daily workflows only on redis/redis repo. (#8625)
Co-authored-by: Harkrishn Patro <harkrisp@amazon.com>
2021-03-09 12:35:41 +02:00
Yossi Gottlieb
3ea4c43add
Cleanup usage of malloc_usable_size. (#8554)
* Add better control of malloc_usable_size() usage.
* Use malloc_usable_size on alpine libc daily job.
* Add no-malloc-usable-size daily jobs.
* Fix zmalloc(0) when HAVE_MALLOC_SIZE is undefined.

In order to align with the jemalloc behavior, this should never return
NULL or OOM panic.
2021-02-25 09:24:41 +02:00
Yossi Gottlieb
dd885780d6
Fix compile errors with no HAVE_MALLOC_SIZE. (#8533)
Also adds a new daily CI test, relying on the fact that we don't use malloc_size() on alpine libmusl.

Fixes #8531
2021-02-23 17:08:49 +02:00
Yossi Gottlieb
95ea74549c
Fix failed tests on Linux Alpine and add a CI job. (#8532)
* Remove linux/version.h dependency.

This introduces unnecessary dependencies, and generally not a good idea
as the platform we build on may be different than the platform we run
on.

To determine if sync_file_range exists we can simply rely on header file
hints.

* Fix setproctitle() on libmusl.

The previous ifdef checks were a bit too strict for no apparent
reason.

* Fix tests failure on Linux with no backtrace.

* Add alpine daily CI job.
2021-02-23 12:57:45 +02:00
filipe oliveira
b5ca1e9e53
Removed time sensitive checks from block on background tests. Fixed uninitialized variable (#8479)
- removes time sensitive checks from block on background tests during leak checks.
- fix uninitialized variable on RedisModuleBlockedClient() when calling
  RM_BlockedClientMeasureTimeEnd() without RM_BlockedClientMeasureTimeStart()
2021-02-10 08:59:07 +02:00
Oran Agra
018f7b7378
Update CI on Ubuntu to tcl8.6 (since 20.04 is now used) (#8460)
Github started shifting some repositoreis to use ubuntu 20.04 by default
tcl8.5 is missing in these, but 8.6 exists in both 20.04 and 18.04
2021-02-07 15:41:49 +02:00
Yossi Gottlieb
5b8350aaaa
Add --dump-logs tests option. (#8459)
Dump the entire server log if a test failed, to easy troubleshooting
with no access to log files.
2021-02-07 12:37:24 +02:00
Yossi Gottlieb
de6f3ad017
Fix FreeBSD tests and CI Daily issues. (#8438)
* Add bash temporarily to allow sentinel fd leaks test to run.
* Use vmactions-freebsd rdist sync to work around bind permission denied
  and slow execution issues.
* Upgrade to tcl8.6 to be aligned with latest Ubuntu envs.
* Concat all command executions to avoid ignoring failures.
* Skip intensive fuzzer on FreeBSD. For some yet unknown reason, generate_fuzzy_traffic_on_key causes TCL to significantly bloat on FreeBSD resulting with out of memory.
2021-02-03 17:35:28 +02:00
Guy Korland
ac5f21d613
Add CI for FreeBSD (#8292)
Co-authored-by: Oran Agra <oran@redislabs.com>
2021-01-20 14:07:09 +02:00
Yossi Gottlieb
522d93607a
Add io-thread daily CI tests. (#8232)
This adds basic coverage to IO threads by running the cluster and few selected Redis test suite tests with the IO threads enabled.

Also provides some necessary additional improvements to the test suite:

* Add --config to sentinel/cluster tests for arbitrary configuration.
* Fix --tags whitelisting which was broken.
* Add a `network` tag to some tests that are more network intensive. This is work in progress and more tests should be properly tagged in the future.
2021-01-17 15:48:48 +02:00
Yossi Gottlieb
84e0489562
Remove end-of-life CentOS 6 workflows. (#8121) 2020-12-02 13:47:25 +02:00
Yossi Gottlieb
843a13e88f
Add a --no-latency tests flag. (#7939)
Useful for running tests on systems which may be way slower than usual.
2020-10-22 11:10:53 +03:00
Wang Yuan
445a4b669a
Implement redisAtomic to replace _Atomic C11 builtin (#7707)
Redis 6.0 introduces I/O threads, it is so cool and efficient, we use C11
_Atomic to establish inter-thread synchronization without mutex. But the
compiler that must supports C11 _Atomic can compile redis code, that brings a
lot of inconvenience since some common platforms can't support by default such
as CentOS7, so we want to implement redis atomic type to make it more portable.

We have implemented our atomic variable for redis that only has 'relaxed'
operations in src/atomicvar.h, so we implement some operations with
'sequentially-consistent', just like the default behavior of C11 _Atomic that
can establish inter-thread synchronization. And we replace all uses of C11
_Atomic with redis atomic variable.

Our implementation of redis atomic variable uses C11 _Atomic, __atomic or
__sync macros if available, it supports most common platforms, and we will
detect automatically which feature we use. In Makefile we use a dummy file to
detect if the compiler supports C11 _Atomic. Now for gcc, we can compile redis
code theoretically if your gcc version is not less than 4.1.2(starts to support
__sync_xxx operations). Otherwise, we remove use mutex fallback to implement
redis atomic variable for performance and test. You will get compiling errors
if your compiler doesn't support all features of above.

For cover redis atomic variable tests, we add other CI jobs that build redis on
CentOS6 and CentOS7 and workflow daily jobs that run the tests on them.
For them, we just install gcc by default in order to cover different compiler
versions, gcc is 4.4.7 by default installation on CentOS6 and 4.8.5 on CentOS7.

We restore the feature that we can test redis with Helgrind to find data race
errors. But you need install Valgrind in the default path configuration firstly
before running your tests, since we use macros in helgrind.h to tell Helgrind
inter-thread happens-before relationship explicitly for avoiding false positives.
Please open an issue on github if you find data race errors relate to this commit.

Unrelated:
- Fix redefinition of typedef 'RedisModuleUserChangedFunc'
  For some old version compilers, they will report errors or warnings, if we
  re-define function type.
2020-09-17 16:01:45 +03:00
Oran Agra
78a6e5eb2b
Fix failing valgrind installation in github actions (#7792)
These tests started failing every day on http 404 (not being able to
install valgrind)
2020-09-13 13:51:21 +03:00
Oran Agra
5496b4a7cd
Add daily CI for MacOS (#7759) 2020-09-08 10:59:25 +03:00
Yossi Gottlieb
fb2a94af3f
TLS: relax verification on CONFIG SET. (#7665)
Avoid re-configuring (and validating) SSL/TLS configuration on `CONFIG
SET` when TLS is not actively enabled for incoming connections, cluster
bus or replication.

This fixes failures when tests run without `--tls` on binaries that were
built with TLS support.

An additional benefit is that it's now possible to perform a multi-step
configuration process while TLS is disabled. The new configuration will
be verified and applied only when TLS is effectively enabled.
2020-08-17 17:36:50 +03:00
Yossi Gottlieb
944cca98c8
Run daily workflow on main repo only (no forks). (#7646) 2020-08-11 10:59:47 +03:00
Yossi Gottlieb
f2da3efc78
Fix test-centos7-tls daily job. (#7598) 2020-07-31 13:55:57 +03:00
Yossi Gottlieb
4ac1f9ac55
CI: Add daily CentOS 7.x jobs. (#7582) 2020-07-30 13:25:10 +03:00
Oran Agra
6d92eee69b
Daily github action: run cluster and sentinel tests with tls (#7575) 2020-07-27 15:30:36 +03:00
Oran Agra
19f192c3a5
Run daily CI on PRs to release a branch (#7535) 2020-07-20 13:57:54 +03:00
Scott Brenner
2f4e9c3f9f
GitHub Actions workflows - use latest version of actions/checkout (#7534) 2020-07-20 09:22:24 +03:00
Oran Agra
7f19a04f0f
update release scripts for new hosts, and CI to run more tests (#7480)
* update daily CI to include cluster and sentinel tests
* update daily CI to run when creating a new release
* update release scripts to work on the new redis.io hosts
2020-07-12 13:55:26 +03:00
Oran Agra
15bcb813d4 32bit CI needs to build modules correctly 2020-05-27 18:19:30 +03:00