Commit graph

639 commits

Author SHA1 Message Date
Laurence Jones
0334a9afe8
Add method name to child logger so we can see which function is erroring when in enrichers (#2411) 2023-08-08 13:38:11 +01:00
Laurence Jones
a18df9c3bb
Add bouncers prune command (#2379)
* Add bouncers prune command

* No point overloading functions

* Add prune to list of commands

* change all short desc to be similar, and made it really really clear when pruning it is not recoverable

* Dont use log. and dont return error on user input to abort
2023-07-28 15:37:39 +01:00
mmetc
ffadd42779
update dependency on go-cs-lib; drop the pkg/ part (#2393) 2023-07-28 16:35:08 +02:00
Laurence Jones
55247cd46a
Add machines prune command (#2011)
* Add machines prune command

* Fix scope variable for naming scheme

* Add some freshness and add new features

* Fix force and fix duration if less than 60

* Allow duration to be more readable

* Fix description

* Improve func wording and make int machines length

* No point overloading functions

* Add prune to list of commands

* Check if GID is already the group if so no need to chown

* Revert "Check if GID is already the group if so no need to chown"

This reverts commit c7cef1773e.

* change all short desc to be similar, and made it really really clear when pruning it is not recoverable

* Better examples

* Match bouncer like for like

* Fix merge error

* Dont use log. and dont return error on user input to abort
2023-07-28 15:23:47 +01:00
mmetc
ae53c0f1cc
fix "crowdsec-cli/require" log verbosity (#2390) 2023-07-28 09:56:20 +02:00
Thibault "bui" Koechlin
718721b341
fix a confusing debug message (#2386)
* fix a confusing debug message

* make CTIHelper simply log the error to avoid failing template rendering
2023-07-28 09:52:21 +02:00
mmetc
5cb7013575
Check cscli preconditions with crowdsec-cli/require package (#2388) 2023-07-27 17:02:20 +02:00
mmetc
a01ce18b98
replace imports of path with path/filepath (#2330) 2023-07-26 10:29:58 +02:00
Laurence Jones
389ea4293f
Add metabase version override and update (#2370)
* Add version override and update

* Ooppsie

* Quick fix

* fgs copilot

* Allow user to overwrite image, add warning for exposing metabase and general cleanup

* One ix

* Default image if not found in config, and add a warning to remove and update

* Reorder check system memory checks so it inline with @mmetc best pratices

* No need for err

* Clean up some group code

* Change ipv6 as [] seems to wildcard

* Split loopback warn and disclaimer. Add force yes to start to allow user to accept disclaimer by default

* All cmd commands are RunE clean up

* Update flag name and dont allow a shorthand
2023-07-25 14:21:25 +01:00
mmetc
395cace69f
fix double push of metrics by properly handling tickers (#2374) 2023-07-25 12:19:26 +02:00
blotus
7106d396dc
expose the FormatAlert function to other packages (#2248) 2023-07-25 09:55:39 +02:00
AlteredCoder
b52b4252c1
scenario labels to map string interface (#2201)
* labels are now map string interface

* restore api url

---------

Co-authored-by: Laurence Jones <laurence.jones@live.co.uk>
2023-07-24 15:19:28 +02:00
mmetc
46fff0b544
Update dependency: docker/docker (#2360) 2023-07-24 11:53:33 +02:00
mmetc
b6b6fd026b
typo fix, uppercase 'API', adjusted log level (#2361) 2023-07-21 23:23:24 +02:00
Manuel Sabban
9ac5aeda79
fix the ci by adding the ability to enforce event ordering (#2347)
* fix the ci by adding the ability to enforce event ordering
2023-07-20 11:41:30 +02:00
blotus
f9ca14f010
add object key in src for S3 acquis (#2342) 2023-07-07 10:09:18 +02:00
blotus
1295de928a
Properly match new files on windows when doing file acquisition (#2329) 2023-07-06 14:45:38 +02:00
mmetc
c10bca93df update dependencies on go-plugin and go-hclog (#2341)
* update dependencies on go-plugin and go-hclog
* bump logrus (panic fix)
* implement HCLogAdapter.Getleve() to satisfy the new interface
2023-07-06 12:01:07 +02:00
mmetc
9967d60987
errors.Wrap -> fmt.Errorf (#2333) 2023-07-06 10:14:45 +02:00
mmetc
17cd792826
CI: update ansible tests for re2 (#2318) 2023-06-29 16:35:19 +02:00
mmetc
bd41f855cf
errors.Wrap -> fmt.Errorf (#2317) 2023-06-29 11:34:59 +02:00
blotus
e61d5a3034
rename status to state in fire response (#2313) 2023-06-29 11:06:49 +02:00
mmetc
893394ef5f
rename metabase APIClient to avoid confusion (#2305) 2023-06-27 15:07:16 +02:00
mmetc
e404e0b608
raise error with invalid 'on_success', 'on_failure' in profile (#2303) 2023-06-27 15:03:07 +02:00
mmetc
85839b0199
support for stdin with "cscli decision import" and raw values (#2291)
and remove Origin from the struct, which was ignored anyway
2023-06-27 14:29:42 +02:00
mmetc
a910b7beca
non-fatal error if some datasource can't be run (i.e. journalctl but systemd is missing) (#2309)
This on the other hand, gives a new fatal error when there are no valid datasources.
In the previous version, crowdsec kept running with just a warning if no
acquisition yaml or dir were specified.
2023-06-27 10:13:13 +02:00
mmetc
507da49b5a
send metrics immediately if agents are added or removed (#2296) 2023-06-23 14:06:04 +02:00
mmetc
9beb5388cb
errors.Wrap -> fmt.Errorf; clean up imports (#2301) 2023-06-23 14:04:58 +02:00
mmetc
e42841cd00
Change api_key encoding to base64 to comply with bcrypt max size (#2302) 2023-06-23 13:54:36 +02:00
mmetc
62caffb102
update leakybucket readme (#2298) 2023-06-22 15:35:01 +02:00
mmetc
fddf597040
errors.Wrap -> fmt.Errorf; clean up imports (#2297) 2023-06-22 15:01:34 +02:00
mmetc
8bfeb7d90d
Update go dependencies (#2293)
- update fatih/color (fix windows issue)
- update mongo-driver (fix build issue)
- go.mod: merge two "require" blocks
- update semver dependency (same version as indirect dep), fix test checks in cscli setup
- remove gotest.tools dependency (use testify, cstest)
- update x/ exp, mod, sys dependencies
2023-06-22 11:31:41 +02:00
Emanuel Seemann
40e6b205bc
Add bayesian bucket type (#2290) 2023-06-21 15:08:27 +02:00
mmetc
da6106bd23
spellcheck/style leakybucket readme (#2294) 2023-06-21 11:47:07 +02:00
mmetc
f7409d47be
fix error message when failing to parse ip address (#2292)
Co-authored-by: Thibault "bui" Koechlin <thibault@crowdsec.net>
2023-06-21 09:22:25 +02:00
Laurence Jones
2c8769adf6
Update jsonextract.go (#2287)
Return nil instead of empty string as ParseKV does the same
2023-06-16 18:34:55 +01:00
mmetc
b9a3acb03f
light pkg/parser cleanup (#2279)
* pkg/parser: clean up imports
* remove duplicate import
* simplify boolean expression
* don't check length before range
* if..else if.. -> switch/case
* errors.Wrap -> fmt.Errorf
* typo, lint
* redundant break
2023-06-13 13:16:13 +02:00
mmetc
76429f033a
trim pkg/types: move DataSet/GetData to pkg/cwhub, removed unused Clone function (#2271) 2023-06-08 16:49:51 +02:00
mmetc
cf747d65e0
fix missing import (#2275) 2023-06-08 15:49:37 +02:00
mmetc
25bb23d8b7
minor refactor to pkg/types, cscli machines (#2270)
* cleanup: separate ui and logic
* trim some code from pkg/types
2023-06-08 15:08:51 +02:00
mmetc
6096cb3c9b
Move grok_pattern.go away from pkg/types to trim bouncer dependencies (#2269) 2023-06-08 15:07:30 +02:00
mmetc
8da9d5eefd
don't log notification error if not running under systemd (#2274) 2023-06-08 15:04:48 +02:00
mmetc
5b3200173e
don't pre-create log files (not required anymore) (#2267)
The lumberjack package fixed the issue in natefinch/lumberjack#83 (tested with umask 002) and this code is now redundant since we updated the dependency to v2.2.1.
2023-06-07 12:58:35 +02:00
mmetc
edd062522d
build against libre2-dev if found (#2255) 2023-06-06 15:46:25 +02:00
mmetc
3cc6b2c0d0
CI: add tests for metrics configuration (#2251) 2023-06-05 23:17:30 +02:00
mmetc
0191faf3a8
update notif threshold test on windows (#2265) 2023-06-05 22:58:13 +02:00
mmetc
e3cb4ab2c4
do not send more than group_threshold alerts at once to a notification plugin (#2264)
* do not send more than group_threshold alerts at once to a notification plugin
* Use generic Chunks function, updated tests

---------

Co-authored-by: Sebastien Blot <sebastien@crowdsec.net>
2023-06-05 12:55:03 +02:00
mmetc
a4eee41fd7
log.Warning if a notification is configured twice (#2240) 2023-06-02 14:41:50 +02:00
mmetc
396dcf8e6e
dependencies: replaced function calls to pkg/types, errors.Wrap (#2235)
we now use a generic pointer function, and slowly remove the deprecated pkg/errors
2023-06-01 16:31:56 +02:00
mmetc
12c32d507c
CI: refactoring pkg/csplugin tests (#2247) 2023-06-01 10:33:08 +02:00
mmetc
92a9d6c321
types.InSlice() -> slices.Contains() (#2246) 2023-05-31 12:39:22 +02:00
Laurence Jones
4fbc3402fb
Update KV ignore whitespace before and after = (#2236)
* Update KV ignore whitespace before and after `=`

* Update helpers.go

Don't need whitespace infront of KEY

* Add some tests to ensure edge cases

* Ensure quoted and unquoted values act the same
2023-05-26 15:35:46 +01:00
blotus
6720d89845
fix lock when dumping the parsing state in explain mode (#2234) 2023-05-26 15:23:50 +01:00
blotus
f6924f8c57
generate asserts for evt.Unmarshaled in hubtest (#2214) 2023-05-26 11:44:58 +02:00
mmetc
9167bd107d
decouple bouncer dependencies: use go-cs-lib/pkg/ptr (#2228) 2023-05-25 15:43:39 +02:00
mmetc
b2d3520519
decouple bouncer dependencies: use go-cs-lib in test code (#2229) 2023-05-25 15:37:44 +02:00
mmetc
364b833d67
test cleanup: remove /tmp/crowdsec_tests* directories (#2232) 2023-05-25 15:32:32 +02:00
Laurence Jones
0416a41d58
Log info capi whitelists (#2220)
* add infof command if err was nil

* Fix golint

* Make message more readable and log individual stats

* Missed a d

* Remove '

* simplify if/else logic

---------

Co-authored-by: Marco Mariani <marco@crowdsec.net>
2023-05-25 10:28:08 +01:00
mmetc
025f14f879
merge system cert pool with own certs (#2226) 2023-05-25 10:10:58 +02:00
mmetc
e5fe74ce77
decouple bouncer dependencies: use go-cs-lib/pkg/ptr in apiclient (#2227) 2023-05-25 10:08:52 +02:00
mmetc
534328ca30
decouple bouncer dependencies: use go-cs-lib/pkg/* (#2216)
* decouple bouncer dependencies: use go-cs-lib/pkg/trace
* decouple bouncer dependencies: use go-cs-lib/pkg/version
* decouple bouncer dependencies: use go-cs-lib/pkg/yamlpatch
* decouple bouncer dependencies: use go-cs-lib/pkg/csstring
* unused import
2023-05-23 10:52:47 +02:00
blotus
6e3ca35941
fallback to master for hub index download if it does not exist (#2210) 2023-05-17 11:20:53 +02:00
blotus
412b4c4b0b
fix incorrect version strip (#2206) 2023-05-17 01:13:55 +02:00
Thibault "bui" Koechlin
77f2968267
fix the behavior of json unmarshal to not return the full map (#2199) 2023-05-16 09:10:38 +02:00
Laurence Jones
424215f228
Add ParseKV helper and rework UnmarshalJSON as a proper helper (#2184) 2023-05-12 09:43:01 +02:00
mmetc
e1f5ed41df
Implement "cscli config show-yaml" (#2191) 2023-05-11 21:01:13 +02:00
blotus
4ae41a363d
add Hostname helper in expr and templating (#2193) 2023-05-11 14:25:04 +02:00
blotus
71b7a594bd
add indexes on the FK between alerts and {decisions,metas,events} (#2188) 2023-05-11 13:49:01 +02:00
blotus
2701454f23
defaults to inotify to detect changes in file datasource to avoid too many call to stat() (#2181) 2023-05-09 10:03:55 +02:00
blotus
e1f4a71357
readd KeyExists expr helper (#2180) 2023-05-04 16:55:34 +02:00
blotus
a753ea6981
Add B64decode expr helper (#2183) 2023-05-04 14:15:20 +02:00
Thibault "bui" Koechlin
8f71edaadd
do not error on this filter (#2182) 2023-05-04 13:06:15 +02:00
Thibault "bui" Koechlin
4ff8f498ce
add a LogInfo expr helper (#2179) 2023-05-03 10:07:11 +02:00
AlteredCoder
6bb20fa951
fix issue #2172 (#2177) 2023-04-28 16:32:46 +02:00
AlteredCoder
c0e6c1ac78
Fix chooseHubBranch when latest() doesn't work (#2178)
* Fix chooseHubBranch when latest() doesn't works
2023-04-28 11:24:04 +02:00
Thibault "bui" Koechlin
3041023ed8
add an optional flag to disable the fetch (#2169) 2023-04-14 11:39:16 +02:00
Thibault "bui" Koechlin
66dfded0cf
significantly increase the max number of scenarios to be sent (#2170) 2023-04-14 11:39:07 +02:00
mmetc
0c5d233563
Minor cleanup and dead code removal (#2166) 2023-04-12 16:57:38 +02:00
Laurence Jones
9a5a937695
Make it more obvious that parser succeeded but was whitelisted (#2167)
* Make it more obvious that parser succeeded but was whitelisted

* Add more verbose by placing whitelist reason next to why it is ignored
2023-04-12 10:48:42 +01:00
blotus
0279e549bd
check if the acquis tomb is dying while processing logs in replay mode for file/s3/docker (#2152) 2023-04-04 13:57:06 +02:00
mmetc
3132aa54b7
Properly load k8s audit configuration (#2158) 2023-04-03 21:55:31 +02:00
mmetc
38ab6be7c2
Allow feature.yml to change available subcommands (#2156) 2023-04-03 10:11:56 +02:00
mmetc
3fa555fb25
Rename k8s_audit to k8s-audit (easier to type, consistent with labels) (#2153) 2023-04-03 09:53:38 +02:00
blotus
61bea26486
Add transform configuration option for acquisition (#2144) 2023-03-29 16:04:17 +02:00
blotus
772d5b5c32
Add experimental support for re2 (#2138) 2023-03-28 16:26:47 +02:00
blotus
1095f6c875
use expr.Function for custom functions instead of passing them in the env (#2133) 2023-03-28 10:49:01 +02:00
Thibault "bui" Koechlin
169b844212
fix awkward stacktrace in conditional filter (#2145) 2023-03-27 16:01:42 +02:00
mmetc
d769fff1e8
File acquisition: log "file reopen" events instead of writing to stderr (#2139) 2023-03-24 11:24:36 +01:00
mmetc
3884c5f47d
Unit tests: remove leftover files (#2134) 2023-03-22 13:51:37 +01:00
Thibault "bui" Koechlin
a3e5f0a3a0
fix dateparse (#2135) 2023-03-22 08:20:21 +01:00
blotus
91eb39cff6
New PAPI commands: reauth + force_pull (#2129) 2023-03-21 14:06:19 +01:00
blotus
dc38e5ac00
S3 acquisition datasource (#2130) 2023-03-21 13:54:52 +01:00
Thibault "bui" Koechlin
a74e424d53
support ip and cidr based whitelists for capi and 3rd party blocklists (#2132)
* support ip and cidr based whitelists for capi and 3rd party blocklist
2023-03-21 11:50:10 +01:00
Thibault "bui" Koechlin
d87f088b8f
match expr helper (#2126)
* match expr helper
2023-03-21 10:39:17 +01:00
Thibault "bui" Koechlin
618be9ff68
properly update the time structure within event (#2122)
* properly update the time structure within event to ensure it works in time-machine

* move LIVE and TIMEMACHINE to pkg/types : less code needs to import leakybucket package, and we avoid duplicating constants
2023-03-16 16:25:50 +01:00
blotus
c77fe16943
actually fix expr-debugger to work with the new version (#2124) 2023-03-16 15:20:48 +01:00
blotus
94c7efdb5b
add ToString() helper (#2100) 2023-03-16 15:20:31 +01:00
blotus
b1f2063a9a
Only support pgx driver for postgresql (#2118) 2023-03-16 11:02:31 +01:00
Thibault "bui" Koechlin
855f9e6f8d
protect map w/ mutex to avoid concurrent map writes with cscli explain when having many concurrent parser routines (#2113) 2023-03-16 11:01:25 +01:00
Manuel Sabban
b451d190b7
try to make reproducible build work (#2119)
Co-authored-by: sabban <15465465+sabban@users.noreply.github.com>
2023-03-13 17:26:33 +01:00