beenull/crowdsec
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity
crowdsec/pkg/setup/README.md
mmetc b6be18ca65
cscli setup (#1923) 
Detect running services and generate acquisition configuration
2023-02-06 07:33:04 +01:00

340 lines
10 KiB
Markdown
Raw Blame History

> **_NOTE_**: The following document describes an experimental, work-in-progress feature. To enable the `cscli setup` command, set the environment variable `CROWDSEC_FEATURE_CSCLI_SETUP=true` or add the line " - cscli_setup" to `/etc/crowdsec/feature.yaml`. Any feedback is welcome.
---
# cscli setup
The "cscli setup" command can configure a crowdsec instance based on the services that are installed or running on the server.
There are three main subcommands:
- `cscli setup detect`: *detect* the services, the OS family, version or the Linux distribution
- `cscli setup install-hub`: *install* the recommended collections, parsers, etc. based on the detection result
- `cscli setup datasources`: *generate* the appropriate acquisition rules
The setup command is used in the `wizard.sh` script, but can also be invoked by hand or customized via a configuration file
by adding new services, log locations and detection rules.
Detection and installation are performed as separate steps, as you can see in the following diagram:
```
+-------------+
| |
| detect.yaml |
| |
+-------------+
|
v
setup detect
|
v
+--------------+
| +---> setup install-hub +-----------------------+
| setup.yaml | | |
| +---> setup datasources --->| etc/crowdsec/acquis.d |
+--------------+ | |
+-----------------------+
```
You can inspect and customize the intermediary file (`setup.yaml`), which is useful
in case of many instances, deployment automation or unusual setups.
A subcommand can be used to check your changes in this case:
- `cscli setup validate`: *validate* or report errors on a setup file
## Basic usage
Identify the existing services and write out what was detected:
```console
# cscli setup detect > setup.yaml
```
See what was found.
```console
# cscli setup install-hub setup.yaml --dry-run
dry-run: would install collection crowdsecurity/apache2
dry-run: would install collection crowdsecurity/linux
dry-run: would install collection crowdsecurity/pgsql
dry-run: would install parser crowdsecurity/whitelists
```
Install the objects (parsers, scenarios...) required to support the detected services:
```console
# cscli setup install-hub setup.yaml
INFO[29-06-2022 03:16:14 PM] crowdsecurity/apache2-logs : OK
INFO[29-06-2022 03:16:14 PM] Enabled parsers : crowdsecurity/apache2-logs
INFO[29-06-2022 03:16:14 PM] crowdsecurity/http-logs : OK
[...]
INFO[29-06-2022 03:16:18 PM] Enabled crowdsecurity/linux
```
Generate the datasource configuration:
```console
# cscli setup datasources setup.yaml --to-dir /etc/crowdsec/acquis.d
```
With the above command, each detected service gets a corresponding file in the
`acquis.d` directory. Running `cscli setup` again may add more services as they
are detected, but datasource files or hub items are never removed
automatically.
## The detect.yaml file
A detect.yaml file is downloaded when you first install crowdsec, and is updated by the `cscli hub update`
command.
> **_NOTE_**: XXX XXX - this is currently not the case, the file is distributed in the crowdsec repository, but it should change.
You can see the default location with `cscli setup detect --help | grep detect-config`
The YAML file contains a version number (always 1.0) and a list of sections, one per supported service.
Each service defines its detection rules, the recommended hub items and
recommended datasources. The same software can be defined in multiple service
sections: for example, apache on debian and fedora have different detection
rules and different datasources so it requires two sections to support both platforms.
The following are minimal `detect.yaml` examples just to show a few concepts.
```yaml
version: 1.0
services:
apache2:
when:
- ProcessRunning("apache2")
install:
collections:
- crowdsecurity/apache2
datasources:
source: file
labels:
type: apache2
filenames:
- /var/log/apache2/*.log
- /var/log/httpd/*.log
```
- `ProcessRunning()` matches the process name of a running application. The
`when:` clause can contain any number of expressions, they are all evaluated
and must all return true for a service to be detected (implied *and* clause, no
short-circuit). A missing or empty `when:` section is evaluated as true.
The [expression
engine](https://github.com/antonmedv/expr/blob/master/docs/Language-Definition.md)
is the same one used by CrowdSec parser filters. You can force the detection of
a process by using the `cscli setup detect... --force-process <processname>`
flag. It will always behave as if `<processname>` was running.
The `install:` section can contain any number of collections, parsers, scenarios
and postoverflows. In practices, it's most often a single collection.
The `datasource:` section is copied as-is in the acquisition file.
> **_NOTE_**: XXX TODO - the current version does not validate the `datasource:` mapping. Bad content is written to acquis.d until crowdsec chokes on it.
Detecting a running process may seem a good idea, but if a process manager like
systemd is available it's better to ask it for the information we want.
```yaml
version: 1.0
services:
apache2-systemd:
when:
- UnitFound("apache2.service")
- OS.ID != "centos"
install:
collections:
- crowdsecurity/apache2
datasource:
source: file
labels:
type: syslog
filenames:
- /var/log/apache2/*.log
apache2-systemd-centos:
when:
- UnitFound("httpd.service")
- OS.ID == "centos"
install:
collections:
- crowdsecurity/apache2
datasource:
source: file
labels:
type: syslog
filenames:
- /var/log/httpd/*.log
```
Here we see two more detection methods:
- `UnitFound()` matches the name of systemd units, if the are in state enabled,
generated or static. You can see here that CentOS is using a different unit
name for Apache so it must have its own service section. You can force the
detection of a unit by using the `cscli setup detect... --force-unit <unitname>` flag.
- OS.Family, OS.ID and OS.RawVersion are read from /etc/os-release in case of
Linux, and detected by other methods for FreeBSD and Windows. Under FreeBSD
and Windows, the value of OS.ID is the same as OS.Family. If OS detection
fails, it can be overridden with the flags `--force-os-family`, `--force-os-id`
and `--force-os-version`.
If you want to ignore one or more services (i.e. not install anything and not
generate acquisition rules) you can specify it with `cscli setup detect...
--skip-service <servicename>`. For example, `--skip-service apache2-systemd`.
If you want to disable systemd unit detection, use `cscli setup detect... --snub-systemd`.
If you used the `--force-process` or `--force-unit` flags, but none of the
defined services is looking for them, you'll have an error like "detecting
services: process(es) forced but not supported".
> **_NOTE_**: XXX XXX - having an error for this is maybe too much, but can tell that a configuration is outdated. Could this be a warning with optional flag to make it an error?
We used the `OS.ID` value to check for the linux distribution, but since the same configuration
is required for CentOS and the other RedHat derivatives, it's better to check for the existence
of a file that is known to exist in all of them:
```yaml
version: 1.0
services:
apache2-systemd-deb:
when:
- UnitFound("apache2.service")
- PathExists("/etc/debian_version")
install:
# [...]
apache2-systemd-rpm:
when:
- UnitFound("httpd.service")
- PathExists("/etc/redhat-release")
install:
# [...]
```
- `PathExists()` evaluates to true if a file, directory or link exists at the
given path. It does not check for broken links.
Rules can be used to detect operating systems and environments:
```yaml
version: 1.0
services:
linux:
when:
- OS.Family == "linux"
install:
collections:
- crowdsecurity/linux
datasource:
type: file
labels:
type: syslog
log_files:
- /var/log/syslog
- /var/log/kern.log
- /var/log/messages
freebsd:
when:
- OS.Family == "freebsd"
install:
collections:
- crowdsecurity/freebsd
windows:
when:
- OS.Family == "windows"
install:
collections:
- crowdsecurity/windows
```
The OS object contains a methods to check for version numbers:
`OS.VersionCheck("<constraint>")`. It uses the
[Masterminds/semver](https://github.com/Masterminds/semver) package and accepts
a variety of operators.
Instead of: OS.RawVersion == "1.2.3" you should use `OS.VersionCheck("~1")`,
`OS.VersionCheck("~1.2")` depending if you want to match the major or the minor
version. It's unlikely that you need to match the exact patch level.
Leading zeroes are permitted, to allow comparison of Ubuntu versions: strict semver rules would treat "22.04" as invalid.
# The `setup.yaml` file
This file does not actually have a specific name, as it's usually written to standard output.
For example, on a Debian system running Apache under systemd you can execute:
```console
$ cscli setup detect --yaml
setup:
- detected_service: apache2-systemd-deb
install:
collections:
- crowdsecurity/apache2
datasource:
filenames:
- /var/log/apache2/*.log
labels:
type: apache2
- detected_service: linux
install:
collections:
- crowdsecurity/linux
datasource:
filenames:
- /var/log/syslog
- /var/log/kern.log
- /var/log/messages
labels:
type: syslog
- detected_service: whitelists
install:
parsers:
- crowdsecurity/whitelists
```
The default output format is JSON, which is compatible with YAML but less readable to humans.
- `detected_service`: used to generate a name for the files written to `acquis.d`
- `install`: can contain collections, parsers, scenarios, postoverflows
- `datasource`: copied to `acquis.d`
```console
$ cscli setup datasources --help
generate datasource (acquisition) configuration from a setup file
Usage:
cscli setup datasources [setup_file] [flags]
Flags:
-h, --help help for datasources
--to-dir string write the configuration to a directory, in multiple files
[...]
```
If the `--to-dir` option is not specified, a single monolithic `acquis.yaml` is printed to the standard output.
Reference in a new issue View git blame Copy permalink