beenull/crowdsec
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity
crowdsec/tests/ansible/README.md
mmetc bd3722f075
Ansible testing improvements (#1700)
2022-08-04 11:25:34 +02:00

159 lines
9.1 KiB
Markdown
Raw Blame History

# Ansible playbooks for functional testing
These playbooks allow you to test crowdsec in a real environment, where the
application is running as root, deployed with the OS package manager, and
uses the standard init system for the distribution (systemd or other).
This way, you can test not only the application's feature but also the packaging
boilerplate, integration scripts, and compatibility with new distribution releases,
operating systems, or architectures.
The ansible hosts should be expendable machines with at least 1GB RAM, do not
expect them to be stable if you use them for anything else after the tests.
Install the requirements with `ansible-galaxy install -r requiements.yml`.
There are several Ansible playbooks. You can use `run-all.yml` to configure the
installation and run the tests, or run the playbooks separately to iterate while developing.
- run-all.yml: run the other playbooks in the correct order.
- provision-dependencies.yml: install the bats requirements (bash, netcat, cfssl, etc.), compilers, and database.
- provision-test-suite.yml: install the tests scripts and bats environment.
- prepare-tests.yml: install the package under test, and create the test fixture data.
- run-tests.yml: run the functional tests. This is not idempotent and can be run multiple times.
The tasks use the following environment variables. They must be exported or
ansible won't be able to see them.
- `TEST_SUITE_GIT` (default "https://github.com/crowdsecurity/crowdsec"),
`TEST_SUITE_VERSION` (default "master"): repo URL and branch/tag/commit of
the crowdsec sources containing the test fixture and scripts.
- `TEST_SUITE_ZIP`: optional, archive of a `crowdsecurity/crowdsec` repository
containing the test fixture and scripts. Overrides `TEST_SUITE_GIT` and
`TEST_SUITE_VERSION`. It can be created with `zip -r crowdsec.zip .` from
the root directory of the repository.
- `DB_BACKEND`: Required. Set to "sqlite", "pgx", "mysql", "postgres".
Postgres is automatically provisioned when required. There is no
provisioning code for mysql/mariadb yet, but it can be added.
- `PACKAGE_TESTING`: when set to false or not defined, the crowdsec binaries
to be tested are built from the sources that come from `TEST_SUITE_GIT` or
`TEST_SUITE_ZIP`. Crowdsec is then run as non-root, in a local directory.
This is basically a fancy wrapper to run `make bats-test` in a vm.
When `PACKAGE_TESTING` is set to true, however, crowdsec is installed from
a binary package (see variables below), it is run as root from systemd (or
equivalent) and uses the system-wide `/etc/crowdsec` and `/var/lib`
directories to store the test data.
- `TEST_PACKAGE_VERSION_DEB`, `TEST_PACKAGE_VERSION_RPM`: Optional, the
version of the package under test (ex. "1.4.0-rc5"), can be in the
packagecloud "stable" or "testing" repository. Requires
`PACKAGE_TESTING=true`. You must set both variables to reuse the same set of
variables for Debian and RedHat-based distributions, because stable releases
require a package version suffix in the RPM file names.
- `TEST_PACKAGE_FILE`: optional, file pointing to the package under test (.deb,
.rpm, .pkg...). It can be a glob expression but it must match a single file,
and the pattern works only on the filename. If both `TEST_PACKAGE_VERSION_*`
and `TEST_PACKAGE_FILE` are provided, both are be installed (to test upgrades
for example). Requires `PACKAGE_TESTING=true`
- `TEST_PACKAGE_DIR`: optional (but conflicts with `TEST_PACKAGE_FILE`), the path
to a directory containing packages with the following layout:
For DEB: `{{ package_dir }}/{{ ansible_distribution_release }}/{{ ansible_architecture.replace('x86_64', 'amd64' }}/crowdsec_*.deb`
For RPM: `{{ package_dir }}/{{ releasever }}/RPMS/{{ ansible_architecture }}/crowdsec-*.{{ releasever }}.{{ ansible_architecture }}.rpm`
- `TEST_SKIP`: optional, comma-separated list of scripts that won't be executed.
Example: `TEST_SKIP=02_nolapi.bats,03_noagent.bats`
## Running tests with Vagrant + Ansible
You don't need Vagrant to run the ansible tests, if you can manage your own
vm creation and inventory.
However, to avoid relying on (and paying for..) a public cloud, we wrote vagrant
configuration files for the most common distributions we support.
To test with Vagrant, you need to:
- have a working libvirt environment (if you can use virt-manager to create VMs, you're golden)
- install the vagrant-libvirt plugin (`vagrant plugin install vagrant-libvirt`.
If it complains about gem versions, blame Ruby and see if you can remove some
other conflicting plugin).
- copy one of the `./env/*.sh` scripts to `environment.sh`, edit to your
needs, and execute it with "source environment.sh"
- `cd vagrant/<distro-of-your-choice>`
- `vagrant up --no-provision; vagrant provision`. The first command creates
the VM, the second installs all the dependencies, test suite and package
under test, then runs the tests. If you run a plain `vagrant up`, it does
everything with a single command, but also destroys the VM in case of test
failure so you are left with nothing to debug.
- `vagrant destroy` when you want to remove the VM. If you want to free up the
space taken by the base VM images, they are in
`/var/lib/libvirt/images/*VAGRANT*`
The above steps are automated in the script `./prepare-run` (requires bash
>=4.4). It takes an enviroment file, and optionally a list of directories with
vagrant configurations. With a single parameter, it loops over all the
directories in alphabetical order, excluding those in the `experimental`
directory. Watch out for running VMs if you break the loop by hand.
After this, you will find up to 30GB of base images in `/var/lib/libvirt/images`,
which you need to remove by hand when you have finished testing or leave them
around for the next time.
You can give more memory or CPU juice to the VMs by editing [Vagrantfile.common](vagrant/Vagrantfile.common).
## Test Matrix
Tests fail with unsupported configurations or when the environment is not prepared correctly
due to missing setup/teardown parts in Ansible or functional tests. False positives
are also possible due to timing issues or flaky network connections.
If you have a result that deviates from the following matrix, that's probably a genuine bug or regression.
The data was created with crowdsec v1.4.1.
| | source/sqlite | pkg/sqlite | source/postgres | source/pgx | source/mysql (0) |
| ------------------------- | ------------- | ---------- | --------------- | ---------- | ---------------- |
| AmazonLinux 2 | ✓ (1) | ✓ (1) | old-db | old-db | wip |
| CentOS 7 | ✓ | ✓ | old-db | old-db | ✓ |
| CentOS 8 | ✓ | ✓ | ✓ | ✓ | ✓ |
| Debian 9 (stretch) | ✓ | ✓ | old-db | old-db | wip |
| Debian 10 (buster) | ✓ | ✓ | ✓ | ✓ | ✓ |
| Debian 11 (bullseye) | ✓ | ✓ | ✓ | ✓ | ✓ |
| Debian (testing/bookworm) | ✓ | ✓ | ✓ | ✓ | wip |
| Fedora 33 | ✓ | ✓ | wip | wip | wip |
| Fedora 34 | ✓ | ✓ | ✓ | ✓ | wip |
| Fedora 35 | ✓ | ✓ | ✓ | ✓ | wip |
| Fedora 36 | ✓ | ✓ | ✓ | ✓ | wip |
| FreeBSD 12 | ✓ | wip | wip | wip | wip |
| FreeBSD 13 | ✓ | wip | wip | wip | wip |
| Oracle 7 | ✓ | ✓ | old-db | old-db | ✓ |
| Oracle 8 | ✓ | ✓ | ✓ | ✓ | ✓ |
| Ubuntu 16.04 (xenial) | ✓ | ✓ | old-db | old-db | ✓ |
| Ubuntu 18.04 (bionic) | ✓ | ✓ | ✓ | ✓ | ✓ |
| Ubuntu 20.04 (focal) | ✓ | ✓ | ✓ | ✓ | ✓ |
| Ubuntu 22.04 (jammy) | ✓ | ✓ | ✓ | ✓ | ✓ |
| | | | | | |
Note: all tests with `local/<database>` are expected to pass for `pkg/<database>` as well.
wip - missing ansible or bats parts, working on it
old-db - the database that ships with the distribution is not supported (Postgres < 10)
0 - MySQL or MariaDB, depending on distribution defaults
1 - ansible may hang, passes all tests if run by hand
Reference in a new issue View git blame Copy permalink