Commit Graph

113 Commits

Author SHA1 Message Date
Tino Reichardt
e0bf43d64e ZTS: Make use of optimal CPU pinning
With CPU pinning, we should get some speedup because of better
cpu cache re-use.

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Tino Reichardt <milky-zfs@mcmilk.de>
Closes #16641
2024-10-13 19:20:49 -07:00
Tino Reichardt
e7b64159f8 ZTS: Optimize Kernel Same-page Merging (KSM)
Kernel same-page Merging (KSM) allows KVM guests to share identical
memory pages. These shared pages are usually common libraries or other
identical, high-use data.

The current configuration was a bit to lazy - so KSM didn't work very
well. With the new configuration I could run 3 Linux VMs in parralel.

FreeBSD can't benefit from it. But FreeBSD is not so memory hungry in
general, so there is no need for it ;)

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Tino Reichardt <milky-zfs@mcmilk.de>
Closes #16641
2024-10-13 19:19:34 -07:00
Brian Behlendorf
34efa8e2d8
CI: Stick with ubuntu-22.04 for CodeQL analysis
The ubuntu-latest alias now refers to ubuntu-24.04 instead of
ubuntu-22.04 which causes CodeQL's autobuild to fail with:

    cpp/autobuilder: deptrace not supported in ubuntu 24.04

Until deptrace is supported by ubuntu-24.04 hosted runners request
ubuntu-22.04 which is supported.

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Tino Reichardt <milky-zfs@mcmilk.de>
Reviewed-by: George Melikov <mail@gmelikov.ru>
Closes #16639
2024-10-11 14:16:00 -07:00
Tino Reichardt
995a3a61fd
ZTS: Fix summary page creation again - second try
In PR #16599 I used 'return' like in C - which is wrong :/
This fix generates the summary as needed.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Tino Reichardt <milky-zfs@mcmilk.de>
Closes #16611
2024-10-06 14:32:08 -07:00
Tino Reichardt
87ca6ba9a8
ZTS: Remove FreeBSD 13.4-STABLE
Current CI is failing on FreeBSD 13.4-STABLE, because samba4 can't be
installed there. Lets remove it for now.

Update also the FreeBSD version definitions a bit.

The naming is like this now:

FreeBSD variants:
- freebsd13-3r, freebsd13-4r, freebsd14-0r, freebsd14-1r (RELEASE)
- freebsd13-4s, freebsd14-1s (STABLE)
- freebsd15-0c (CURRENT)

RHL based distros:
- almalinux8, almalinux9, centos-stream9, fedora39, fedora40

Debian based:
- debian11, debian12, ubuntu20, ubuntu22, ubuntu24

Misc Linux distros:
- archlinux, tumbleweed

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Tino Reichardt <milky-zfs@mcmilk.de>
Closes #16610
2024-10-06 14:29:20 -07:00
Tino Reichardt
3d0175d10e
ZTS: Fix summary page creation
There are cases, where some needed files for the summary page aren't
created. Currently the whole Summary Page creation will fail then.
Sample run: https://github.com/openzfs/zfs/actions/runs/11148248072/job/30999748588

Fix this, by properly checking for existence of the needed files.

Reviewed-by: Tony Hutter <hutter2@llnl.gov>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Rob Norris <robn@despairlabs.com>
Signed-off-by: Tino Reichardt <milky-zfs@mcmilk.de>
Closes #16599
2024-10-03 11:42:25 -07:00
Tino Reichardt
b1958b531b
ZTS: Replace MD5 and SHA256 wit XXH128
For data integrity checks as done in ZTS, the verification for
unintended data corruption with xxhash128 should be a lot faster
and perfectly usable.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Tino Reichardt <milky-zfs@mcmilk.de>
Closes #16577
2024-09-28 09:24:05 -07:00
Brian Behlendorf
d7ab2816ee
CI: Add logs to zloop workflow
On failure attempt to include the most relevant portions of the
ztest logs in the CI output.  This full logs are still available
for download but often a backtrace and the last output is enough.

Install libunwind to improve the odds of a useful backtrace.

Reviewed-by: Tino Reichardt <milky-zfs@mcmilk.de>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #16573
2024-09-27 09:05:49 -07:00
Shengqi Chen
05a7a9594e
CI: run only sanity check on limited OSes for nonbehavioral changes
The commit uses heuristics to determine whether a PR is behavioral:

It runs "quick" CI (i.e., only use sanity.run on fewer OSes)
if (explicitly requested by user):
- the *last* commit message contains a line 'ZFS-CI-Type: quick',
or if (by heuristics):
- the files changed are not in the list of specified directory, and
- all commit messages does not contain 'ZFS-CI-Type: full'.

It runs "full" CI otherwise.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Tino Reichardt <milky-zfs@mcmilk.de>
Signed-off-by: Shengqi Chen <harry-chen@outlook.com>
Closes #16564
2024-09-25 07:46:52 -07:00
Brian Behlendorf
4b5fbf70f2
CI: cancel workflows when PRs are updated (#16562)
For checkstyle, zloop, zfs-qemu, and codeql workflows cancel
in-progress jobs when the PR is updated.

Relevant GitHub Actions documentation:

  The following concurrency group cancels in-progress jobs or run
  on pull_request events only; if github.head_ref is undefined, the
  concurrency group will fallback to the run ID, which is guaranteed
  to be both unique and defined for the run.

https://docs.github.com/en/actions/writing-workflows/workflow-syntax-for-github-actions#example-using-a-fallback-value

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #16562
2024-09-24 09:02:01 -07:00
Brian Behlendorf
9c03b22f8f
ZTS: CI Documentation Updates
Update the CONTRIBUTING.md documentation to refer to the GitHub Actions
workflows which have replaced the buildbot infrastructure.

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #16561
2024-09-23 22:04:27 -07:00
Brian Behlendorf
56ab541521
ZTS: CodeQL Action v3 update
Switch from v2 to v3 CodeQL Actions.  The v2 actions will no longer
be supported as of Dec '24 so we need to move to v3.  According to
the release notes they should be functionally equivalent.

    Note that the only difference between v2 and v3 of the CodeQL
    Action is the node version they support, ... For example 3.22.11
    was the first v3 release and is functionally identical to 2.22.11.

https://github.com/github/codeql-action/blob/main/CHANGELOG.md

Reviewed-by: George Melikov <mail@gmelikov.ru>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #16560
2024-09-23 17:07:29 -07:00
Tino Reichardt
53b77c32af
ZTS: Fix Summary Page
The qemu-9-summary-page.sh script reads the file env.txt in the
first lines. When the module didn't build, this file was not copied
into the tarfile - causing the scipt to abort.

Fix: copy needed files into the tarfile in case of module build
failures. The fix ignores also empty tarfiles in future.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Tino Reichardt <milky-zfs@mcmilk.de>
Closes #16555
2024-09-22 09:21:42 -07:00
Tino Reichardt
45a4a94b64 ZTS: Remove functional tests via matrix
This commit changes the workflow of the github actions.

- Ubuntu 20.04, 22.04, 24.04 will be tested via QEMU now
- remove unused scripts of this commit: b7bc334d1
- re-add the zloop standalone testings via zloop.yml

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Tony Hutter <hutter2@llnl.gov>
Signed-off-by: Tino Reichardt <milky-zfs@mcmilk.de>
Closes #16549
2024-09-20 08:01:05 -07:00
Tino Reichardt
7a965b9bc8 ZTS: Fix Test Summary page generation
Fix that error: "cat /tmp/failed.txt: No such file or directory"

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Tony Hutter <hutter2@llnl.gov>
Signed-off-by: Tino Reichardt <milky-zfs@mcmilk.de>
Closes #16549
2024-09-20 08:00:40 -07:00
Tino Reichardt
bca9b64e7b ZTS: Use QEMU for tests on Linux and FreeBSD
This commit adds functional tests for these systems:
- AlmaLinux 8, AlmaLinux 9, ArchLinux
- CentOS Stream 9, Fedora 39, Fedora 40
- Debian 11, Debian 12
- FreeBSD 13, FreeBSD 14, FreeBSD 15
- Ubuntu 20.04, Ubuntu 22.04, Ubuntu 24.04

- enabled by default:
 - AlmaLinux 8, AlmaLinux 9
 - Debian 11, Debian 12
 - Fedora 39, Fedora 40
 - FreeBSD 13, FreeBSD 14

Workflow for each operating system:
- install qemu on the github runner
- download current cloud image of operating system
- start and init that image via cloud-init
- install dependencies and poweroff system
- start system and build openzfs and then poweroff again
- clone build system and start 2 instances of it
- run functional testings and complete in around 3h
- when tests are done, do some logfile preparing
- show detailed results for each system
- in the end, generate the job summary

Real-world benefits from this PR:

1. The github runner scripts are in the zfs repo itself. That means
   you can just open a PR against zfs, like "Add Fedora 41 tester", and
   see the results directly in the PR. ZFS admins no longer need
   manually to login to the buildbot server to update the buildbot config
   with new version of Fedora/Almalinux.

2. Github runners allow you to run the entire test suite against your
   private branch before submitting a formal PR to openzfs. Just open a
   PR against your private zfs repo, and the exact same
   Fedora/Alma/FreeBSD runners will fire up and run ZTS. This can be
   useful if you want to iterate on a ZTS change before submitting a
   formal PR.

3. buildbot is incredibly cumbersome. Our buildbot config files alone
   are ~1500 lines (not including any build/setup scripts)!
   It's a huge pain to setup.

4. We're running the super ancient buildbot 0.8.12. It's so ancient
   it requires python2. We actually have to build python2 from source
   for almalinux9 just to get it to run. Ugrading to a more modern
   buildbot is a huge undertaking, and the UI on the newer versions is
   worse.

5. Buildbot uses EC2 instances. EC2 is a pain because:
   * It costs money
   * They throttle IOPS and CPU usage, leading to mysterious,
   * hard-to-diagnose, failures and timeouts in ZTS.
   * EC2 is high maintenance. We have to setup security groups, SSH
   * keys, networking, users, etc, in AWS and it's a pain. We also
   * have to periodically go in an kill zombie EC2 instances that
   * buildbot is unable to kill off.

6. Buildbot doesn't always handle failures well. One of the things we
   saw in the past was the FreeBSD builders would often die, and each
   builder death would take up a "slot" in buildbot. So we would
   periodically have to restart buildbot via a cron job to get the slots
   back.

7. This PR divides up the ZTS test list into two parts, launches two
   VMs, and on each VM runs half the test suite. The test results are
   then merged and shown in the sumary page. So we're basically
   parallelizing ZTS on the same github runner. This leads to lower
   overall ZTS runtimes (2.5-3 hours vs 4+ hours on buildbot), and one
   unified set of results per runner, which is nice.

8. Since the tests are running on a VM, we have much more control over
   what happens. We can capture the serial console output even if the
   test completely brings down the VM. In the future, we could also
   restart the test on the VM where it left off, so that if a single test
   panics the VM, we can just restart it and run the remaining ZTS tests
   (this functionaly is not yet implemented though, just an idea).

9. Using the runners, users can manually kill or restart a test run
   via the github IU. That really isn't possible with buildbot unless
   you're an admin.

10. Anecdotally, the tests seem to be more stable and constant under
    the QEMU runners.

Reviewed by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Tino Reichardt <milky-zfs@mcmilk.de>
Signed-off-by: Tony Hutter <hutter2@llnl.gov>
Closes #16537
2024-09-17 12:03:27 -07:00
Ameer Hamza
9c56b8ec78
Github workflow: fix typo in zloop artifact
Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Ameer Hamza <ahamza@ixsystems.com>
Closes #16432
2024-08-09 16:49:19 -07:00
Andrew Innes
3965c9ba38
Move nodes into correct subgraphs
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Tino Reichardt <milky-zfs@mcmilk.de>
Signed-off-by: Andrew Innes <andrew.c12@gmail.com>
Closes #15828
2024-01-29 09:16:02 -08:00
Tino Reichardt
fb27698cfc
ZTS: Update deprecated Github Action version numbers
GitHub Actions is transitioning from Node 16 to Node 20.

So we need to update these:
- actions/checkout@v3 -> v4
- actions/download-artifact@v3 -> v4
- actions/upload-artifact@v3 -> v4 and some minor changes

Update also the documentation of the testings workflow.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Andrew Innes <andrew.c12@gmail.com>
Signed-off-by: Tino Reichardt <milky-zfs@mcmilk.de>
Closes #15820
2024-01-26 14:22:26 -08:00
Richard Yao
e7af89d972
Switch to CodeQL to detect prohibited function use
The LLVM/Clang developers pointed out that using the CPP to detect use
of functions that our QA policies prohibit risks invoking undefined
behavior. To resolve this, we configure CodeQL to detect forbidden
function usage.

Note that cpp in the context of CodeQL refers to C/C++, rather than the
C PreProcessor, which C++ also uses. It really should have been written
cxx, but that ship sailed a long time ago. This misuse of the term cpp
is retained in the CodeQL configuration for consistency with upstream
CodeQL.

As a side benefit, verbose make no longer is a wall of text showing a
bunch of CPP macros, which can make debugging slightly easier.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Richard Yao <richard.yao@alumni.stonybrook.edu>
Closes #15819 
Closes #14134
2024-01-26 14:11:33 -08:00
Tino Reichardt
dac0bae561
ZTS: Apply small changes for speeding up the tests
The Github Action Runner got some new hardware metrics.  We should use
the provided and empty disk which is pre-mounted at /mnt now.

Disk1: 89GiB -> rootfs + bootfs with ~80MB/s -> don't care
Disk2: 64GiB -> /mnt with 420MB/s -> new testing ssd

This commit will mount the new disk to /var/tmp and provide hopefully
some speedups within our testings.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Andrew Innes <andrew.c12@gmail.com>
Signed-off-by: Tino Reichardt <milky-zfs@mcmilk.de>
Closes #15811
2024-01-26 13:36:59 -08:00
Tino Reichardt
480d809703
Refine some details for the github actions update
Set the retention-days variable to 14 days for these artifacts:
- the zloop error logs
- the zloop vdev files
- the compiled modules

Add the abality to re-run some part of the functional testings.
Fix some comments and remove the deleting of the modules artifact.

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Tino Reichardt <milky-zfs@mcmilk.de>
Closes #14637
2023-03-16 10:00:14 -07:00
Tino Reichardt
c9c463a508
Add git repo checkout to testing workflow
Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Tino Reichardt <milky-zfs@mcmilk.de>
Closes #14634
2023-03-15 15:37:56 -07:00
Tino Reichardt
b7bc334d13
Split functional testings via github action matrix
This commit changes the workflow of the github actions.

We split the workflow into different parts:

1) build zfs modules for Ubuntu 20.04 and 22.04 (~25m)
2) 2x zloop test (~10m) + 2x sanity test (~25m)
3) functional testings in parts 1..5 (each ~1h)
   - these could be triggered, when sanity tests are ok
   - currently I just start them all in the same time
4) cleanup and create summary

When everything is fine, the full run with all testings
should be done in around 2 hours.

The codeql.yml and checkstyle.yml are not part in this circle.

The testings are also modified a bit:
- report info about CPU and checksum benchmarks
- reset the debugging logs for each test
  - when some error occurred, we call dmesg with -c to get
    only the log output for the last failed test
  - we empty also the dbgsys

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Tino Reichardt <milky-zfs@mcmilk.de>
Closes #14078
2023-03-15 10:41:05 -07:00
Tino Reichardt
0f9ed58f43
Update reclaim disk space script
The script uses systemd-run, which does the job in background.
We should take the the time and wait for the job to finish.

Maybe some functional tests suffer from not really freed disk
space and fail because of this.

We also add some trimming in the end of the script.

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Tino Reichardt <milky-zfs@mcmilk.de>
Closes #14554
2023-03-01 12:25:09 -08:00
Brian Behlendorf
000985fc15
Workaround GitHub Action failure
Ubuntu 20.04 and 22.04 workflows are failing due to an error
which is hit when running `apt-get update`.  Until the
problematic package is fixed apply the suggested workaround
described here:

  https://github.com/orgs/community/discussions/47863

Reviewed-by: Matthew Ahrens <mahrens@delphix.com>
Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Richard Yao <richard.yao@alumni.stonybrook.edu>
Reviewed-by: Tino Reichardt <milky-zfs@mcmilk.de>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #14530
2023-02-27 09:19:25 -08:00
Brian Behlendorf
07281f7d13
CI: Reclaim space after package operations
Rather than reclaiming space before updating the packages do
it afterwards.  This avoids issues with apt returning an
error due to missing files on the system.

This commit includes a revert for 6320b9e6.

Reviewed-by: George Melikov <mail@gmelikov.ru>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #14387
2023-01-17 09:52:27 -08:00
Brian Behlendorf
6320b9e68e
CI: remove unused packages/snaps
Removing portions of packages/snaps directly with rm can result in
unexpected errors when running `apt update`.  Free up the additional
space by removing (some) packages with the proper tools.

This change frees up slightly less space than before, but it is
expected to still be sufficient.

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Richard Yao <richard.yao@alumni.stonybrook.edu>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #14374
2023-01-11 15:18:51 -08:00
Umer Saleem
e6e31dd540
Add native-deb* targets to build native Debian packages
In continuation of previous #13451, this commits adds native-deb*
targets for make to build native debian packages. Github workflows
are updated to build and test native Debian packages.

Native packages only build with pre-configured paths (see the
dh_auto_configure section in contrib/debian/rules.in). While
building native packages, paths should not be configured. Initial
config flags e.g. '--enable-debug' are replaced in
contrib/debian/rules.in.

Additional packages on top of existing zfs packages required to
build native packages include debhelper-compat, dh-python, dkms,
po-debconf, python3-all-dev, python3-sphinx.

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Ryan Moeller <ryan@iXsystems.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Umer Saleem <usaleem@ixsystems.com>
Closes #14265
2022-12-13 17:33:05 -08:00
Brian Behlendorf
3a6d89ae52
Retire Ubuntu 18.04 CI builder
The GitHub-hosted Ubuntu 18.04 has been deprecated and will be
entirely unsupported as of April 2023.  Leading up to this there
will be scheduled "brownouts" to encourage users to update their
workflows.

This commit retires our use of the GitHub-hosted Ubuntu 18.04
runners in advance of their removal.

Reviewed-by: Richard Yao <richard.yao@alumni.stonybrook.edu>
Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Damian Szuberski <szuberskidamian@gmail.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #14238
2022-11-30 10:25:57 -08:00
Richard Yao
022bdb5b0b
Set multiple make jobs on CodeQL github workflows
github supports 2 processors right now, although we use nproc instead of
hard coding -j2, so if that ever increases, we will take advantage of it
right away.

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Damian Szuberski <szuberskidamian@gmail.com>
Reviewed-by: Tino Reichardt <milky-zfs@mcmilk.de>
Signed-off-by: Richard Yao <richard.yao@alumni.stonybrook.edu>
Closes #14217
2022-11-28 13:29:55 -08:00
szubersk
9e7fc5da38 Ubuntu 22.04 integration: GitHub workflows
- GitHub workflows are run on Ubuntu 22.04

- Extract the `checkstyle` workflow dependencies to a separate file.

- Refresh the `build-dependencies.txt` list.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Richard Yao <richard.yao@alumni.stonybrook.edu>
Reviewed-by: George Melikov <mail@gmelikov.ru>
Signed-off-by: szubersk <szuberskidamian@gmail.com>
Closes #14148
2022-11-18 11:27:03 -08:00
Richard Yao
c8ae0ca11a
Add CodeQL workflow
CodeQL is a static analyzer from github with a very low false positive
rate. We have long wanted to have static analysis runs done on every
pull request and using CodeQL, we can.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Andrew Innes <andrew.c12@gmail.com>
Signed-off-by: Richard Yao <richard.yao@alumni.stonybrook.edu>
Closes #14087
2022-10-27 09:36:17 -07:00
George Melikov
87d5ff8ecd CI: bump actions/upload-artifact to v3
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: George Melikov <mail@gmelikov.ru>
Closes #14018
2022-10-12 15:18:39 -07:00
George Melikov
4bbd7a0fbe CI: bump actions/checkout to v3
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: George Melikov <mail@gmelikov.ru>
Closes #14018
2022-10-12 15:18:19 -07:00
George Melikov
6f8602a5ed
CI: revert --with-config=dist to hotfix Ubuntu 20.04
Recently Github action runners started to fail on kmod build.  
Revert --with-config=dist from ./configure section of github 
runners to stabilize CI for now.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: George Melikov <mail@gmelikov.ru>
Closes #13894
2022-09-14 16:26:57 -07:00
Brian Behlendorf
d98a67a53a
Replace EXTRA_DIST with dist_noinst_DATA
The EXTRA_DIST variable is ignored when used in the FALSE conditional
of a Makefile.am.  This results in the `make dist` target omitting
these files from the generated tarball unless CONFIG_USER is defined.
This issue can be avoided by switching to use the dist_noinst_DATA
variable which is handled as expected by autoconf.

This change also adds support for --with-config=dist as an alias
for --with-config=srpm and updates the GitHub workflows to use it.

Reviewed-by: Ahelenia Ziemiańska <nabijaczleweli@nabijaczleweli.xyz>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #13459
Closes #13505
2022-05-26 09:24:50 -07:00
Brian Behlendorf
196e7c7617
ztest: reduce runtile of zloop.sh in CI
The zloop.sh script is primarily designed to randomly stress
the DMU and SPA layers.  This can result in some unrealistic
(or even impossible) scenarios being tested which then fail.

Since the longer we run zloop.sh the more likely this is to occur
this commit reduces the runtime.  The intention being that normally
this will result in a clean CI run unless the PR does introduce
serious breaking change.

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Tony Hutter <hutter2@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #13453
2022-05-12 09:11:29 -07:00
наб
c8970f52ed autoconf: use include directives instead of recursing down lib
As a bonus, this also adds zfs-mount-generator (previously undescended
down) and libzstd (not included) to CppCheck

As a bonus bonus, abigail rules work out-of-tree, too

Against current trunk:
  $ diff -U0 ./destdir.listing ~/store/code/zfs/destdir.listing
  -destdir/usr/local/include/libspl/sscanf.h

  $ diff --color -U0 ./zfs-2.1.99.tar.gz.listing ../oot/zfs-2.1.99.tar.gz.listing | grep -v @@ | grep -v /Makefile
  -zfs-2.1.99/config/Abigail.am
  -zfs-2.1.99/lib/libspl/include/util/
  -zfs-2.1.99/lib/libspl/include/util/sscanf.h

  $ diff --color -U0 ./zfs-2.1.99.tar.gz.listing ../oot/zfs-2.1.99.tar.gz.listing | grep -v @@ | grep /Makefile
  -zfs-2.1.99/lib/libavl/Makefile.in
  -zfs-2.1.99/lib/libefi/Makefile.in
  -zfs-2.1.99/lib/libicp/Makefile.in
  -zfs-2.1.99/lib/libnvpair/Makefile.in
  -zfs-2.1.99/lib/libshare/Makefile.in
  -zfs-2.1.99/lib/libspl/include/Makefile.in
  -zfs-2.1.99/lib/libspl/include/os/freebsd/Makefile.am
  -zfs-2.1.99/lib/libspl/include/os/freebsd/Makefile.in
  -zfs-2.1.99/lib/libspl/include/os/freebsd/sys/Makefile.am
  -zfs-2.1.99/lib/libspl/include/os/freebsd/sys/Makefile.in
  -zfs-2.1.99/lib/libspl/include/os/linux/Makefile.am
  -zfs-2.1.99/lib/libspl/include/os/linux/Makefile.in
  -zfs-2.1.99/lib/libspl/include/os/linux/sys/Makefile.am
  -zfs-2.1.99/lib/libspl/include/os/linux/sys/Makefile.in
  -zfs-2.1.99/lib/libspl/include/os/Makefile.am
  -zfs-2.1.99/lib/libspl/include/os/Makefile.in
  -zfs-2.1.99/lib/libspl/include/rpc/Makefile.am
  -zfs-2.1.99/lib/libspl/include/rpc/Makefile.in
  -zfs-2.1.99/lib/libspl/include/sys/dktp/Makefile.am
  -zfs-2.1.99/lib/libspl/include/sys/dktp/Makefile.in
  -zfs-2.1.99/lib/libspl/include/sys/Makefile.am
  -zfs-2.1.99/lib/libspl/include/sys/Makefile.in
  -zfs-2.1.99/lib/libspl/include/util/Makefile.am
  -zfs-2.1.99/lib/libspl/include/util/Makefile.in
  -zfs-2.1.99/lib/libspl/Makefile.in
  -zfs-2.1.99/lib/libtpool/Makefile.in
  -zfs-2.1.99/lib/libunicode/Makefile.in
  -zfs-2.1.99/lib/libuutil/Makefile.in
  -zfs-2.1.99/lib/libzfsbootenv/Makefile.in
  -zfs-2.1.99/lib/libzfs_core/Makefile.in
  -zfs-2.1.99/lib/libzfs/Makefile.in
  -zfs-2.1.99/lib/libzpool/Makefile.in
  -zfs-2.1.99/lib/libzstd/Makefile.in
  -zfs-2.1.99/lib/libzutil/Makefile.in
  -zfs-2.1.99/lib/Makefile.in

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Ahelenia Ziemiańska <nabijaczleweli@nabijaczleweli.xyz>
Closes #13316
2022-05-10 10:18:11 -07:00
George Melikov
427fad7f24
CI: Log test name to /dev/kmsg in ZTS
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: John Kennedy <john.kennedy@delphix.com>
Signed-off-by: George Melikov <mail@gmelikov.ru>
Closes  #13249
2022-03-24 08:01:26 -06:00
Aleksa Sarai
669683c4cb
ZTS: switch to rsync for directory diffs
While "diff -r" is the most straightforward way of comparing directory
trees for differences, it has two major issues:

 * File metadata is not compared, which means that subtle bugs may be
   missed even if a test is written that exercises the buggy behaviour.
 * diff(1) doesn't know how to compare special files -- it assumes they
   are always different, which means that a test using diff(1) on
   special files will always fail (resulting in such tests not being
   added).

rsync can be used in a very similar manner to diff (with the -ni flags),
but has the additional benefit of being able to detect and resolve many
more differences between directory trees. In addition, rsync has a
standard set of features and flags while diffs feature set depends on
whether you're using GNU or BSD binutils.

Note that for several of the test cases we expect that file timestamps
will not match. For example, the ctime for a file creation or modify
event is stored in the intent log but not the mtime. Thus when replaying
the log the correct ctime is set but the current mtime is used. This is
the expected behavior, so to prevent these tests from failing, there's a
replay_directory_diff function which ignores those kinds of changes.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Aleksa Sarai <cyphar@cyphar.com>
Closes #12588
2022-03-01 10:05:32 -08:00
Rich Ercolani
0df22afea2
Colorize the Github test output
Let's color our workflow test output for better readability.

Reviewed by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Ahelenia Ziemiańska <nabijaczleweli@nabijaczleweli.xyz>
Signed-off-by: Rich Ercolani <rincebrain@gmail.com>
Closes #13000
2022-02-16 11:40:25 -08:00
Damian Szuberski
e2909fae8f
Remove unused files from GitHub runners
Majority of the software installed by default in GitHub runners is
irrelevant to OpenZFS. Reclaimed space could be used for more/bigger
vdev files. File deletion happens in the background, leveraging
`systemd-run` - the workflow is not significantly slowed down.

Before
```
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        84G   53G   31G  63% /
```

After
```
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        84G   15G   70G  18% /
```

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Ahelenia Ziemiańska <nabijaczleweli@nabijaczleweli.xyz>
Signed-off-by: szubersk <szuberskidamian@gmail.com>
Closes #13066
2022-02-09 11:50:13 -08:00
Damian Szuberski
add15e9539
Extract workflows dependencies
- Move build dependencies moved to
  `.github/workflows/build-dependencies.txt` shared among workflows.

- Change `ubuntu-latest` -> `ubuntu-20.04` to avoid unexpected
  runner environment updates in `zloop` workflow.

- Change `ubuntu-20.04` -> `ubuntu-latest` to track changes in
  runner environment in `checkstyle` workflow.

- Kernel buffer is flushed before ZTS invocation to avoid storing
  the same data after each test case run.

- `make` is invoked with consistent set of options to reduce
  clutter in logs.

Reviewed-by: George Melikov <mail@gmelikov.ru>
Signed-off-by: szubersk <szuberskidamian@gmail.com>
Closes #13037
2022-02-07 11:44:17 -08:00
Damian Szuberski
63652e1546
Add --enable-asan and --enable-ubsan switches
`configure` now accepts `--enable-asan` and `--enable-ubsan` switches
which results in passing `-fsanitize=address`
and `-fsanitize=undefined`, respectively, to the compiler. Those
flags are enabled in GitHub workflows for ZTS and zloop. Errors
reported by both instrumentations are corrected, except for:

- Memory leak reporting is (temporarily) suppressed. The cost of
  fixing them is relatively high compared to the gains.

- Checksum computing functions in `module/zcommon/zfs_fletcher*`
  have UBSan errors suppressed. It is completely impractical
  to enforce 64-byte payload alignment there due to performance
  impact.

- There's no ASan heap poisoning in `module/zstd/lib/zstd.c`. A custom
  memory allocator is used there rendering that measure
  unfeasible.

- Memory leaks detection has to be suppressed for `cmd/zvol_id`.
  `zvol_id` is run by udev with the help of `ptrace(2)`. Tracing is
  incompatible with memory leaks detection.

Reviewed-by: Ahelenia Ziemiańska <nabijaczleweli@nabijaczleweli.xyz>
Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: szubersk <szuberskidamian@gmail.com>
Closes #12928
2022-02-03 14:35:38 -08:00
Rich Ercolani
cd26b217dc
Add explicit timeout to test step
If we die from timeout of the whole GH action run, we don't run the
collect step afterward, which can make it hard to investigate the
timeout.

If we timeout first in the test action, though, it qualifies as
failure, and collects appropriately.

(330 minutes seems like an acceptable tradeoff between the 6h
timeout by default on the action and the 4h and change "functional"
usually takes.)

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Rich Ercolani <rincebrain@gmail.com>
Closes #12999
2022-01-24 15:54:52 -08:00
Damian Szuberski
8a7c4efd3c
Removed Python 2 and Python 3.5- support
Deprecation of Python versions below 3.6 gives opportunity to unify the
build and install requirements for OpenZFS packages. The minimal
supported Python version is 3.6 as this is the most recent Python
package CentOS/RHEL 7 users can get.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Rich Ercolani <rincebrain@gmail.com>
Reviewed-by: John Kennedy <john.kennedy@delphix.com>
Signed-off-by: szubersk <szuberskidamian@gmail.com>
Closes #12925
2022-01-13 09:51:12 -07:00
John Wren Kennedy
ddc026f59b
Strip colons from all test result filenames
The upload artifact functionality in github can't handle colons in
filenames. The current code handles this for files under the most
recent set of results. With the ability to rerun failed tests, now
there can be multiple sets of results, and they all need to be
processed in the same way.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Tony Nguyen <tony.nguyen@delphix.com>
Signed-off-by: John Kennedy <john.kennedy@delphix.com>
Closes #12815
2021-12-01 16:18:45 -08:00
Paul Dagnelie
2320e6eb43
Add zfs-test facility to automatically rerun failing tests
This was a project proposed as part of the Quality theme for the
hackthon for the 2021 OpenZFS Developer Summit. The idea is to improve
the usability of the automated tests that get run when a PR is created
by having failing tests automatically rerun in order to make flaky
tests less impactful.

Reviewed-by: John Kennedy <john.kennedy@delphix.com>
Reviewed-by: Tony Nguyen <tony.nguyen@delphix.com>
Signed-off-by: Paul Dagnelie <pcd@delphix.com>
Closes #12740
2021-12-01 10:38:53 -07:00
Damian Szuberski
6d680e61ef
Update checkstyle workflow env to ubuntu-20.04
- `checkstyle` workflow uses ubuntu-20.04 environment
- improved `mancheck.sh` readability

Reviewed-by: Matthew Ahrens <mahrens@delphix.com>
Reviewed-by: John Kennedy <john.kennedy@delphix.com>
Signed-off-by: szubersk <szuberskidamian@gmail.com>
Closes #12713
2021-11-02 14:02:57 -06:00