Add a new datasource that:
- Receives HTTP requests from remediation components
- Apply rules on them to determine whether they are malicious or not
- Rules can be evaluated in-band (the remediation component will block the request directly) or out-band (the RC will let the request through, but crowdsec can still process the rule matches with scenarios)
The PR also adds support for 2 new hub items:
- appsec-configs: Configure the Application Security Engine (which rules to load, in which phase)
- appsec-rules: a rule that is added in the Application Security Engine (can use either our own format, or seclang)
---------
Co-authored-by: alteredCoder <kevin@crowdsec.net>
Co-authored-by: Sebastien Blot <sebastien@crowdsec.net>
Co-authored-by: mmetc <92726601+mmetc@users.noreply.github.com>
Co-authored-by: Marco Mariani <marco@crowdsec.net>
* Add detection sqlie wal for dashboard chown
* Lean it down a little
* Change to for loop with extensions
* Keep existing uid on files incase user is running as a unpriviledge user
* I have no idea 🤷
* Exclude dash.go and update windows
* Update
* Renam
* Remove the os check since we no longer get to this stage for those os's
---------
Co-authored-by: Manuel Sabban <github@sabban.eu>
* bump gopkg.in/yaml.v3
* test: cannot remove local items with cscli
* test dangling links
* test: cannot install local item with cscli
* pkg/cwhub: reorg (move) functions in files
* allow hub upgrade with local items
* data download: honor Last-Modified header
* fatal -> warning when attempting to remove a local item (allows remove --all)
* cscli...inspect -o yaml|human: rename remote_path -> path
* Correct count of removed items
Still no separate counter for the --purge option, but should be clear enough
* Merge main and apply stash
* Rework some of cscli notif stuff and add a generic test which works with non active profiles
* Update wording
* Fix merge
* Final version
* Cleanup
* Improve whitelist parsing
* Split whitelist check into a function tied to whitelist, also since we check node debug we can make a pointer to node containing whitelist
* No point passing clog as an argument since it is just a pointer to node we already know about
* We should break instead of returning false, false as it may have been whitelisted by ips/cidrs
* reimplement early return if expr errors
* Fix lint and dont need to parse ip back to string just loop over sources
* Log error with node logger as it provides context
* Move getsource to a function cleanup some code
* Change func name
* Split out compile to a function so we can use in tests. Add a bunch of tests
* spell correction
* Use node logger so it has context
* alternative solution
* quick fixes
* Use containswls
* Change whitelist test to use parseipsource and only events
* Make it simpler
* Postoverflow tests, some basic ones to make sure it works
* Use official pkg
* Add @mmetc reco
* Add @mmetc reco
* Change if if to a switch to only evaluate once
* simplify assertions
---------
Co-authored-by: bui <thibault@crowdsec.net>
Co-authored-by: Marco Mariani <marco@crowdsec.net>
* refact jwt:Authenticator
* include scenarios in first login request for machines with tlsAuth
* log.Printf -> log.Infof
* errors.Wrap -> fmt.Errorf
* don't override validation error
* fix test
* cleanup in argument call
* update test as well
* cwhub_tests: reduce verbosity and use helpers
---------
Co-authored-by: Marco Mariani <marco@crowdsec.net>
* use go 1.21.1, require 1.21
* import "slices" from stdlib
* allow codeql to set version number from tags
* codeql: custom WASM build - the automated one can silently fail
* removed unused error; comment
* rename loop variables
* happy path
* rename loop variables
* extract function, method
* log.Printf -> log.Infof
* tests -> testdata
from "go help test":
The go tool will ignore a directory named "testdata", making it available
to hold ancillary data needed by the tests.
* align tags
* extract function toEmoji
* Add parsers length check as it can panic is enrich is empty
* Lets get smarter and loop backwards to find last successful stage
* Shorten code
---------
Co-authored-by: Thibault "bui" Koechlin <thibault@crowdsec.net>
* move function GetLineCountForFile from pkg/types to cscli
* move ParseDuration from pkg/types to pkg/database
* remove unused types.Profile, types.RemediationProfile
now it does not attempt any retry, instead of attempting all retries
immediately
example: cannot reach LAPI
Before:
$ CROWDSEC_FEATURE_DISABLE_HTTP_RETRY_BACKOFF=true cscli decisions list
ERRO[27-07-2023 10:44:44] error while performing request: dial tcp [::1]:8080: connect: connection refused; 4 retries left
INFO[27-07-2023 10:44:44] retrying in 0 seconds (attempt 2 of 5)
[...]
ERRO[27-07-2023 10:44:44] error while performing request: dial tcp [::1]:8080: connect: connection refused; 1 retries left
INFO[27-07-2023 10:44:44] retrying in 0 seconds (attempt 5 of 5)
ERRO[27-07-2023 10:44:44] error while performing request: dial tcp [::1]:8080: connect: connection refused; 0 retries left
FATA[27-07-2023 10:44:44] Unable to list decisions : performing request: Get "http://localhost:8080/v1/alerts?has_active_decision=true&include_capi=false&limit=100": could not get jwt token: Post "http://localhost:8080/v1/watchers/login": dial tcp [::1]:8080: connect: connection refused
After:
$ CROWDSEC_FEATURE_DISABLE_HTTP_RETRY_BACKOFF=true ./test/local/bin/cscli decisions list
FATA[11-08-2023 16:49:58] unable to retrieve decisions: performing request: Get "http://127.0.0.1:8080/v1/alerts?has_active_decision=true&include_capi=false&limit=100": could not get jwt token: Post "http://127.0.0.1:8080/v1/watchers/login": dial tcp 127.0.0.1:8080: connect: connection refused
* apic: minor refactoring
* Add whitelist length check
If user configures the file but fails to define and actual whitelist we should check length to save allocs
* Init with length from file
* extract loop method from ApplyApicWhitelists
* pass pointer
* extract loop method updateBlocklist
---------
Co-authored-by: Laurence Jones <laurence.jones@live.co.uk>
* Add bouncers prune command
* No point overloading functions
* Add prune to list of commands
* change all short desc to be similar, and made it really really clear when pruning it is not recoverable
* Dont use log. and dont return error on user input to abort
* Add machines prune command
* Fix scope variable for naming scheme
* Add some freshness and add new features
* Fix force and fix duration if less than 60
* Allow duration to be more readable
* Fix description
* Improve func wording and make int machines length
* No point overloading functions
* Add prune to list of commands
* Check if GID is already the group if so no need to chown
* Revert "Check if GID is already the group if so no need to chown"
This reverts commit c7cef1773e.
* change all short desc to be similar, and made it really really clear when pruning it is not recoverable
* Better examples
* Match bouncer like for like
* Fix merge error
* Dont use log. and dont return error on user input to abort
* Add version override and update
* Ooppsie
* Quick fix
* fgs copilot
* Allow user to overwrite image, add warning for exposing metabase and general cleanup
* One ix
* Default image if not found in config, and add a warning to remove and update
* Reorder check system memory checks so it inline with @mmetc best pratices
* No need for err
* Clean up some group code
* Change ipv6 as [] seems to wildcard
* Split loopback warn and disclaimer. Add force yes to start to allow user to accept disclaimer by default
* All cmd commands are RunE clean up
* Update flag name and dont allow a shorthand
This on the other hand, gives a new fatal error when there are no valid datasources.
In the previous version, crowdsec kept running with just a warning if no
acquisition yaml or dir were specified.
The lumberjack package fixed the issue in natefinch/lumberjack#83 (tested with umask 002) and this code is now redundant since we updated the dependency to v2.2.1.
* do not send more than group_threshold alerts at once to a notification plugin
* Use generic Chunks function, updated tests
---------
Co-authored-by: Sebastien Blot <sebastien@crowdsec.net>
* Update KV ignore whitespace before and after `=`
* Update helpers.go
Don't need whitespace infront of KEY
* Add some tests to ensure edge cases
* Ensure quoted and unquoted values act the same
* add infof command if err was nil
* Fix golint
* Make message more readable and log individual stats
* Missed a d
* Remove '
* simplify if/else logic
---------
Co-authored-by: Marco Mariani <marco@crowdsec.net>
