beenull/crowdsec
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

docker: pre-download all hub items and data, opt-in hub update/upgrade (#2933)

Browse source 
* docker: pre-download all hub items and data, opt-in hub update/upgrade

* docker/bars: don't purge anything before pre-downloading hub

* Docker: README update
This commit is contained in:
mmetc 2024-04-08 14:53:12 +02:00 committed by GitHub
parent 990dd5e08e
commit 0e8a1c681b
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
5 changed files with 36 additions and 22 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

1
Dockerfile
View file

@ -25,6 +25,7 @@ RUN make clean release DOCKER_BUILD=1 BUILD_STATIC=1 && \
./wizard.sh --docker-mode && \
cd - >/dev/null && \
cscli hub update && \
./docker/preload-hub-items && \
cscli collections install crowdsecurity/linux && \
cscli parsers install crowdsecurity/whitelists

11
docker/README.md
View file

@ -134,7 +134,6 @@ labels:
type: apache2
```
## Recommended configuration
### Volumes
@ -146,6 +145,14 @@ to avoid losing credentials and decision data in case of container destruction a
* Acquisition: `/etc/crowdsec/acquis.d` and/or `/etc/crowdsec.acquis.yaml` (yes, they can be nested in `/etc/crowdsec`)
* Database when using SQLite (default): `/var/lib/crowdsec/data`
### Hub updates
To ensure you have the latest version of the collections, scenarios, parsers, etc., you can set the variable `DO_HUB_UPGRADE` to true.
This will perform an update/upgrade of the hub every time the container is started.
Be aware that if your container is misbehaving and caught in a restart loop, the CrowdSec hub may ban your IP for some time and your containers
will run with the version of the hub that is cached in the container's image. If you enable `DO_HUB_UPGRADE`, do it when your infrastructure is running
correctly and make sure you have some monitoring in place.
## Start a Crowdsec instance
@ -316,7 +323,7 @@ config.yaml) each time the container is run.
| `BOUNCERS_ALLOWED_OU` | bouncer-ou | OU values allowed for bouncers, separated by comma |
| | | |
| __Hub management__ | | |
| `NO_HUB_UPGRADE` | false | Skip hub update / upgrade when the container starts |
| `DO_HUB_UPGRADE` | false | Force hub update / upgrade when the container starts. If for some reason the container restarts too often, it may lead to a temporary ban from hub updates. |
| `COLLECTIONS` | | Collections to install, separated by space: `-e COLLECTIONS="crowdsecurity/linux crowdsecurity/apache2"` |
| `PARSERS` | | Parsers to install, separated by space |
| `SCENARIOS` | | Scenarios to install, separated by space |

5
docker/docker_start.sh
View file

@ -304,9 +304,8 @@ conf_set_if "$PLUGIN_DIR" '.config_paths.plugin_dir = strenv(PLUGIN_DIR)'
## Install hub items
cscli hub update || true
if isfalse "$NO_HUB_UPGRADE"; then
if istrue "$DO_HUB_UPGRADE"; then
cscli hub update || true
cscli hub upgrade || true
fi

22
docker/preload-hub-items Executable file
View file

@ -0,0 +1,22 @@
#!/usr/bin/env bash
set -eu
# pre-download everything but don't install anything
echo "Pre-downloading Hub content..."
types=$(cscli hub types -o raw)
for itemtype in $types; do
ALL_ITEMS=$(cscli "$itemtype" list -a -o json | itemtype="$itemtype" yq '.[env(itemtype)][] | .name')
if [[ -n "${ALL_ITEMS}" ]]; then
#shellcheck disable=SC2086
cscli "$itemtype" install \
$ALL_ITEMS \
--download-only \
--error
fi
done
echo " done."

19
test/bin/preload-hub-items
View file

@ -9,20 +9,12 @@ THIS_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
# pre-download everything but don't install anything
echo -n "Purging existing hub..."
echo "Pre-downloading Hub content..."
types=$("$CSCLI" hub types -o raw)
for itemtype in $types; do
"$CSCLI" "${itemtype}" delete --all --error --purge --force
done
echo " done."
echo -n "Pre-downloading Hub content..."
for itemtype in $types; do
ALL_ITEMS=$("$CSCLI" "$itemtype" list -a -o json | jq --arg itemtype "$itemtype" -r '.[$itemtype][].name')
ALL_ITEMS=$("$CSCLI" "$itemtype" list -a -o json | itemtype="$itemtype" yq '.[env(itemtype)][] | .name')
if [[ -n "${ALL_ITEMS}" ]]; then
#shellcheck disable=SC2086
"$CSCLI" "$itemtype" install \
@ -32,11 +24,4 @@ for itemtype in $types; do
fi
done
# XXX: download-only works only for collections, not for parsers, scenarios, postoverflows.
# so we have to delete the links manually, and leave the downloaded files in place
for itemtype in $types; do
"$CSCLI" "$itemtype" delete --all --error
done
echo " done."