doc : add debugging in cheatsheets (#179)
This commit is contained in:
parent
747065229e
commit
a3b638890d
3 changed files with 112 additions and 4 deletions
108
docs/cheat_sheets/debugging_configs.md
Normal file
108
docs/cheat_sheets/debugging_configs.md
Normal file
|
@ -0,0 +1,108 @@
|
|||
|
||||
|
||||
|
||||
# Debugging Scenarios and Parsers
|
||||
|
||||
## General Advice
|
||||
|
||||
When trying to debug a parser or a scenario :
|
||||
|
||||
- Work on "cold logs" (with the `-file` and `-type` options) rather than live ones
|
||||
- Use the `/etc/crowdsec/config/user.yaml` configuration files to have logs on stdout
|
||||
|
||||
## Example
|
||||
|
||||
```bash
|
||||
crowdsec -c /etc/crowdsec/config/user.yaml -file mylogs.log.gz -type syslog
|
||||
INFO[05-08-2020 16:15:47] Crowdsec v0.3.0-rc3-7525f11975a0107746213862dc41c69e00122ac7
|
||||
INFO[05-08-2020 16:15:47] Loading grok library
|
||||
...
|
||||
WARN[05-08-2020 16:16:12] 182.x.x.x triggered a 4h0m0s ip ban remediation for [crowdsecurity/http-probing] bucket_id=misty-moon event_time="2019-01-01 22:58:32 +0100 CET" scenario=crowdsecurity/http-probing source_ip=182.x.x.x
|
||||
...
|
||||
```
|
||||
|
||||
- `/etc/crowdsec/config/user.yaml` disables demonization and push logs to stdout/stderr
|
||||
- `-type` must respect expected log type (ie. `nginx` `syslog` etc.)
|
||||
- `-file` must point to a flat file or a gzip file
|
||||
|
||||
When processing logs like this, {{crowdsec.name}} runs in "time machine" mode, and relies on the timestamps *in* the logs to evaluate scenarios. You will most likely need the `crowdsecurity/dateparse-enrich` parser for this.
|
||||
|
||||
|
||||
## Using debug
|
||||
|
||||
Both scenarios and parsers support a `debug: true|false` option which produce useful debug.
|
||||
|
||||
<details>
|
||||
<summary>Debug parsing output (expand)</summary>
|
||||
```bash
|
||||
DEBU[05-08-2020 15:25:36] eval(evt.Parsed.program == 'nginx') = TRUE id=lively-smoke name=crowdsecurity/nginx-logs stage=s01-parse
|
||||
DEBU[05-08-2020 15:25:36] eval variables: id=lively-smoke name=crowdsecurity/nginx-logs stage=s01-parse
|
||||
DEBU[05-08-2020 15:25:36] evt.Parsed.program = 'nginx' id=lively-smoke name=crowdsecurity/nginx-logs stage=s01-parse
|
||||
DEBU[05-08-2020 15:25:36] Event entering node id=icy-dew name=child-crowdsecurity/nginx-logs stage=s01-parse
|
||||
DEBU[05-08-2020 15:25:36] + Grok 'NGINXACCESS' returned 10 entries to merge in Parsed id=icy-dew name=child-crowdsecurity/nginx-logs stage=s01-parse
|
||||
DEBU[05-08-2020 15:25:36] .Parsed['request'] = '/data.php' id=icy-dew name=child-crowdsecurity/nginx-logs stage=s01-parse
|
||||
DEBU[05-08-2020 15:25:36] .Parsed['http_user_agent'] = 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0' id=icy-dew name=child-crowdsecurity/nginx-logs stage=s01-parse
|
||||
DEBU[05-08-2020 15:25:36] .Parsed['http_referer'] = '-' id=icy-dew name=child-crowdsecurity/nginx-logs stage=s01-parse
|
||||
DEBU[05-08-2020 15:25:36] .Parsed['remote_addr'] = '123.x.x.x' id=icy-dew name=child-crowdsecurity/nginx-logs stage=s01-parse
|
||||
DEBU[05-08-2020 15:25:36] .Parsed['remote_user'] = '-' id=icy-dew name=child-crowdsecurity/nginx-logs stage=s01-parse
|
||||
DEBU[05-08-2020 15:25:36] .Parsed['time_local'] = '01/Jan/2019:01:39:06 +0100' id=icy-dew name=child-crowdsecurity/nginx-logs stage=s01-parse
|
||||
DEBU[05-08-2020 15:25:36] .Parsed['method'] = 'POST' id=icy-dew name=child-crowdsecurity/nginx-logs stage=s01-parse
|
||||
DEBU[05-08-2020 15:25:36] .Parsed['body_bytes_sent'] = '162' id=icy-dew name=child-crowdsecurity/nginx-logs stage=s01-parse
|
||||
DEBU[05-08-2020 15:25:36] .Parsed['http_version'] = '1.1' id=icy-dew name=child-crowdsecurity/nginx-logs stage=s01-parse
|
||||
DEBU[05-08-2020 15:25:36] .Parsed['status'] = '404' id=icy-dew name=child-crowdsecurity/nginx-logs stage=s01-parse
|
||||
DEBU[05-08-2020 15:25:36] .Meta[log_type] = 'http_access-log' id=icy-dew name=child-crowdsecurity/nginx-logs stage=s01-parse
|
||||
DEBU[05-08-2020 15:25:36] evt.StrTime = '01/Jan/2019:01:39:06 +0100' id=icy-dew name=child-crowdsecurity/nginx-logs stage=s01-parse
|
||||
DEBU[05-08-2020 15:25:36] Event leaving node : ok id=icy-dew name=child-crowdsecurity/nginx-logs stage=s01-parse
|
||||
DEBU[05-08-2020 15:25:36] child is success, OnSuccess=next_stage, skip id=lively-smoke name=crowdsecurity/nginx-logs stage=s01-parse
|
||||
```
|
||||
</details>
|
||||
|
||||
|
||||
<details>
|
||||
<summary>Debug scenario output (expand)</summary>
|
||||
```bash
|
||||
DEBU[05-08-2020 16:02:26] eval(evt.Meta.service == 'http' && evt.Meta.http_status in ['404', '403', '400'] && evt.Parsed.static_ressource == 'false') = TRUE cfg=black-wave file=config/scenarios/http-probing.yaml name=crowdsecurity/http-probing
|
||||
DEBU[05-08-2020 16:02:26] eval variables: cfg=black-wave file=config/scenarios/http-probing.yaml name=crowdsecurity/http-probing
|
||||
DEBU[05-08-2020 16:02:26] evt.Meta.service = 'http' cfg=black-wave file=config/scenarios/http-probing.yaml name=crowdsecurity/http-probing
|
||||
DEBU[05-08-2020 16:02:26] evt.Meta.http_status = '404' cfg=black-wave file=config/scenarios/http-probing.yaml name=crowdsecurity/http-probing
|
||||
DEBU[05-08-2020 16:02:26] evt.Parsed.static_ressource = 'false' cfg=black-wave file=config/scenarios/http-probing.yaml name=crowdsecurity/http-probing
|
||||
```
|
||||
</details>
|
||||
|
||||
|
||||
# Test environments
|
||||
|
||||
From a [{{crowdsec.name}} release archive]({{crowdsec.download_url}}), you can deploy a test (non-root) environment that is very suitable to write/debug/test parsers and scenarios. Environment is deployed using `./test_env.sh` script from tgz directory, and creates a test environment in `./tests` :
|
||||
|
||||
```bash
|
||||
$ cd crowdsec-v0.3.0/
|
||||
$ ./test_env.sh
|
||||
...
|
||||
[08/05/2020:04:19:18 PM][INFO] Setting up configurations
|
||||
INFO[0000] Wrote new 75065 bytes index to config/crowdsec-cli/.index.json
|
||||
INFO[0000] crowdsecurity/syslog-logs : OK
|
||||
INFO[0000] crowdsecurity/geoip-enrich : OK
|
||||
...
|
||||
INFO[0007] Enabled collections : crowdsecurity/linux
|
||||
INFO[0007] Enabled crowdsecurity/linux
|
||||
[08/05/2020:04:19:26 PM][INFO] Environment is ready in /home/bui/github/crowdsec/crowdsec/crowdsec-v0.3.0/tests
|
||||
$ cd tests
|
||||
$ ./cscli -c dev.yaml list
|
||||
...
|
||||
INFO[0000] PARSERS:
|
||||
-------------------------------------------------------------------------------------------------------
|
||||
NAME 📦 STATUS VERSION LOCAL PATH
|
||||
-------------------------------------------------------------------------------------------------------
|
||||
crowdsecurity/geoip-enrich ✔️ enabled 0.2 config/parsers/s02-enrich/geoip-enrich.yaml
|
||||
crowdsecurity/syslog-logs ✔️ enabled 0.3 config/parsers/s00-raw/syslog-logs.yaml
|
||||
crowdsecurity/sshd-logs ✔️ enabled 0.2 config/parsers/s01-parse/sshd-logs.yaml
|
||||
crowdsecurity/dateparse-enrich ✔️ enabled 0.1 config/parsers/s02-enrich/dateparse-enrich.yaml
|
||||
-------------------------------------------------------------------------------------------------------
|
||||
...
|
||||
$ ./crowdsec -c dev.yaml -file sshd.log -type syslog
|
||||
INFO[05-08-2020 16:23:32] Crowdsec v0.3.0-rc3-7525f11975a0107746213862dc41c69e00122ac7
|
||||
INFO[05-08-2020 16:23:32] Loading grok library
|
||||
...
|
||||
```
|
||||
|
||||
|
|
@ -4,10 +4,9 @@
|
|||
Please ensure that you have working env or setup test environment before writing your parser.
|
||||
|
||||
!!! warning "Parser dependency"
|
||||
|
||||
The crowdsecurity/syslog-logs parsers is needed by the core parsing
|
||||
engine. Deletion or modification of this could result of {{crowdsec.name}}
|
||||
being unable to parse logs, so this should be done very carefully.
|
||||
The crowdsecurity/syslog-logs parsers is needed by the core parsing
|
||||
engine. Deletion or modification of this could result of {{crowdsec.name}}
|
||||
being unable to parse logs, so this should be done very carefully.
|
||||
|
||||
> In the current example, we'll write a parser for the logs produced by `iptables` (netfilter) with the `-j LOG` target.
|
||||
> This document aims at detailing the process of writing and testing new parsers.
|
||||
|
|
|
@ -19,6 +19,7 @@ nav:
|
|||
- Ban Management: cheat_sheets/ban-mgmt.md
|
||||
- Configuration Management: cheat_sheets/config-mgmt.md
|
||||
- Hub's taxonomy: https://hub.crowdsec.net/fields
|
||||
- Debugging Parsers & Scenarios: cheat_sheets/debugging_configs.md
|
||||
- Observability:
|
||||
- Overview: observability/overview.md
|
||||
- Logs: observability/logs.md
|
||||
|
|
Loading…
Add table
Reference in a new issue