Commit graph

470 commits

Author SHA1 Message Date
mmetc
d5c7870826
Refact cwhub: remove global hub instance (#2587)
* csConfig.Cscli is always loaded now, configuration paths too
* Remove global/singleton hub instance
2023-11-10 17:32:12 +01:00
mmetc
9d7ed12950
Refact cwhub (#2586)
* Inspect item: always show tainted, installed, etc. when false
* cleanup, comments, unused stuff
* download collection content after downloading dependencies, avoid duplicate call
* Return instances from Item.SubItems()
* shorter i/o code
* inline / simplify getData()
* Handle timeout connections when downloading from hub or data
2023-11-10 10:25:29 +01:00
mmetc
ab8de19506
Refact cwhub: move methods from hub to item (#2585)
* Add back pointer Item.hub
* Hub.enableItem() -> Item.enable()
* Rename variable i -> idx (i is used for item instances)
* Move Hub.purgeItem() -> Item.purge()
* Move Hub.disableItem() -> Item.disable()
* Move Hub.downloadItem() -> Item.download()
* Move Hub.downloadLatest() -> Item.downloadLatest()
* Move Hub.DownloadDataIfNeeded() -> Item.DownloadDataIfNeeded()
* Move Hub.InstallItem() -> Item.Install()
* Move Hub.RemoveItem() -> Item.Remove()
* Move Hub.UpgradeItem() -> Item.Upgrade()
* store hub items as pointers
* No need to re-add items to the hub if we use pointers
* Fix parameter calling order + regression test
2023-11-09 15:19:38 +01:00
mmetc
f80d841188
Refact cwhub: make some methods private (#2584)
* make hub.enableItem() private
* make hub.downloadLatest() private
* make getData() private
* make hub.disableItem() private
* make hub.downloadItem() private
* make hub.syncDir() private
* make hub.localSync() private; keep warnings in Hub struct (no need to call LocalSync to get them)
2023-11-09 12:07:09 +01:00
mmetc
ec4b5bdc86
Refact cwhub (#2583)
* no need to use NewRequest()
* download error messages
* cscli hub list: fix item stats
* Method item.HasSubItems() - avoid explicit type check
* cscli config restore: drop silent install, just call InstallItem
* no backpointer yet
2023-11-09 11:34:14 +01:00
Sebastien Blot
1154ada2df
up 2023-11-08 20:32:58 +01:00
Sebastien Blot
694028f769
merge hub branch 2023-11-08 20:25:42 +01:00
Sebastien Blot
152c940774
wip 2023-11-08 20:24:44 +01:00
mmetc
f4b5bcb865
Refact cwhub: version comparison and branch selection (#2581)
* simplify GetItemByPath
* hub: sort version numbers by semver
* replace golang.org/x/mod/semver with github.com/Masterminds/semver/v3 (would not compare correctly)
* fix nil dereference with tainted items
* update tests for collections, postoverflows
* fix nil deref
* don't fallback to master if hub is not found, improve message
* explicit message for unknown version / tainted collections
2023-11-08 13:21:59 +01:00
mmetc
5cd4406f5e
typos/grammar (#2561) 2023-11-07 15:07:36 +01:00
mmetc
ad54b99bf9
Refact pkg/hubtest (#2580)
* pkg/hubtest: lint (whitespace, empty lines)
* use existing function to sort keys
* lint
* cscli hubtest: set TZ=UTC
* dedup Coverage struct
* pre-compile regexps
* remove redundant type declarations or global vars
2023-11-07 14:02:02 +01:00
mmetc
41d19de092
Refact cwhub (#2578)
* Fix suggest functional tests
* comments
* non-empty SubItems() implies collections type
* use "slices" from stdlib
* No need to repeat author field in the index -- take it from the item key
2023-11-06 17:35:33 +01:00
Sebastien Blot
26c876dc38
merge hub-1.6 branch 2023-11-06 15:02:11 +01:00
mmetc
450c263826
Refact cwhub: minor cleanups and comments (#2574)
* check response status before body; close file
* err check one-liners, lint, comments
* simplify function logic, reduce code
* comments, xxx, whitespace
2023-10-31 16:32:29 +01:00
mmetc
590a19b768
Refact pkg/cwhub: constructor, cscli output
* Single constructor: NewHub() to replace InitHub(), InitHubUpdate()
* sort cscli hub list output
* log.Fatal -> fmt.Errorf
2023-10-31 12:47:39 +01:00
mmetc
17662e59a9
Refact pkg/cwhub, cscli: hub upgrades (#2568)
* fix bats test for "upgrade all items"
* refact UpgradeConfig() -> UpgradeItem(): one item only
* refact RemoveMany() -> RemoveItem()
* Computed value: Item.Local -> Item.IsLocal()
* refact url/branch configuration with LocalHubCfg/RemoteHubCfg
2023-10-30 17:23:50 +01:00
mmetc
6b8ed0c9d0
Refactor hub URL/branch configuration (#2559)
* Refactor hub URL/branch configuration
* docker: using --force to implement $DISABLE (required for items in collections)
* use pointer receiver for consistency
2023-10-27 10:25:29 +02:00
bui
dd49620922 our shortcut for waap events 2023-10-24 17:23:29 +02:00
mmetc
ac98256602
Refact pkg/cwhub, cmd/crowdsec-cli (#2557)
- pkg/cwhub: change file layout, rename functions
 - method Item.SubItems
 - cmd/crowdsec-cli: generic code for hub items
 - cscli: removing any type of items in a collection now requires --force
 - tests
2023-10-20 14:32:35 +02:00
Sebastien Blot
ef118a49ff
add waap-configs hub item 2023-10-19 16:53:00 +02:00
Sebastien Blot
15120a6d8f
merge hub-1.5.6 2023-10-19 14:19:37 +02:00
Sebastien Blot
350e8979b1
merge hub-1.5.6 branch 2023-10-19 12:18:16 +02:00
mmetc
88e4f7c157
Refact pkg/csconfig, pkg/cwhub (#2555)
* csconfig: drop redundant hub information on *Cfg structs
* rename validItemFileName() -> item.validPath()
* Methods on hub object
* updated tests to reduce need of csconfig.Config or global state
2023-10-19 12:04:29 +02:00
Sebastien Blot
ecbdf2f0e1
merge master branch 2023-10-19 10:51:54 +02:00
bui
c89b42939e naming 2023-10-18 17:17:57 +02:00
bui
98fb84d3e7 be consistent : waap-rules 2023-10-18 17:11:43 +02:00
mmetc
be6555e46c
Refact pkg/csconfig, HubCfg (#2552)
- rename csconfig.Hub -> HubCfg
 - move some Load*() functions to NewConfig()
 - config.yaml: optional common section
 - remove unused working_dir
2023-10-18 09:38:33 +02:00
mmetc
4eae40865e
HubIndex struct, comments, name changes (#2549)
* pkg/cwhub: rename PARSERS_OVFLW -> POSTOVERFLOWS
* mostly comments, some light cleanup
* move type hubtest.HubIndex -> cwhub.HubIndex
* move and rename LoadPkgIndex -> ParseIndex
* move displaySummary(), skippedLocal, skippedTainted to HubIndex struct
2023-10-17 16:17:37 +02:00
mmetc
810a8adcf0 fix build (#2548) 2023-10-17 16:12:41 +02:00
mmetc
325003bb69 Refact cscli item listing, tests (#2547)
* hub diet; taint tests
* cmd/crowdsec-cli: split utils.go, moved cwhub.GetHubStatusForItemType()
* cscli: refactor hub list commands, fix edge cases
2023-10-17 16:12:41 +02:00
mmetc
f496bd1692 bats: more cscli hub tests (#2541)
- updated logs and user messages
- added func tests for all the items: install, remove, upgrade, list
- rewritten taint tests for collections
- removed redundant csconfig.LoadPrometheus()
2023-10-17 16:12:41 +02:00
mmetc
a00bae6039 cmd/crowdsec-cli: remove global prometheusURL (#2542)
* cmd/crowdsec-cli: remove global prometheusURL
* PrometheusUrl now includes the path (/metrics)
2023-10-17 16:12:41 +02:00
mmetc
734ba46e6a Refact cscli hub/item commands (#2536)
* log.Fatal -> fmt.Errorf
* lint cmd/crowdsec-cli hub items and split collection commands
* cscli collections: add examples
* cscli parsers: avoid globals
* cscli scenarios: avoid globals
* cscli collections, postoverflows: avoid globals
* cscli hub: avoid globals
* remove unused globals
2023-10-17 16:12:41 +02:00
mmetc
7db5bf8979 pkg/csconfig: set prometheus address:port defaults (#2533)
We set these default in one place (after loading the configuration)
instead of leaving that to both metric server and consumer.
2023-10-17 16:12:41 +02:00
mmetc
a6b55f2b5e
cscli config feeature-flags: point user to the right location of feature.yaml (#2539) 2023-10-13 09:52:51 +02:00
mmetc
3b1563a538
Refact cscli hub / pkg/cwhub (part 6) (#2524)
* hub.ConfigDir -> hub.InstallDir; hub.DataDir -> hub.InstallDataDir
* cleanup GetInstalledItemsAsString()
* lint: ReferenceMissingError -> ErrMissingReference
* lint: parent_dir -> parentDir
* link: export Walker type
* lint: return error last
* lint: shadow
* move around and group variable definitions
2023-10-09 21:33:35 +02:00
Manuel Sabban
6e228f3f3f
pkg/cwhub: cleanup in argument call (#2527)
* cleanup in argument call
* update test as well
* cwhub_tests: reduce verbosity and use helpers

---------

Co-authored-by: Marco Mariani <marco@crowdsec.net>
2023-10-09 13:26:34 +02:00
mmetc
9ae8bd79c5
Refact pkg/csconfig tests (#2526)
* remove unused method
* whitespace, redundant comments
* use test helpers
* move DumpConsoleConfig() from pkg/csconfig to cscli
* package doc header
* var -> const
* rename ./tests -> ./testdata
* shorter tests with more error checks
* lint/formatting
* use helpers; fix tests that didn't actually test
* lint; rename expectedResult -> expected
2023-10-09 11:10:51 +02:00
mmetc
338141f067
Refact cscli hub / pkg/cwhub (part 5) (#2521)
* remove unused yaml tags
* cscli/cwhub: deduplicate, remove dead code
* log.Fatal -> fmt.Errorf
* deflate utils.go by moving functions to respective files
* indexOf() -> slices.Index()
* ItemStatus() + toEmoji() -> Item.status()
* Item.versionStatus()
* move getSHA256() to loader.go
2023-10-06 13:59:51 +02:00
mmetc
9235f55c47
Refact pkg/cwhub (part 4) (#2518)
* generalize function: GetInstalledItems, GetInstalledItemsAsString
* extracted function itemKey, happy path
* review comments / remove redundant; rename file to remove build tags
* remove unused fields in Item struct
* unix build tag
2023-10-05 09:35:03 +02:00
mmetc
61d4ccbfdd
use go 1.21.1 (#2418)
* use go 1.21.1, require 1.21
* import "slices" from stdlib
* allow codeql to set version number from tags
* codeql: custom WASM build - the automated one can silently fail
2023-10-04 13:01:57 +02:00
mmetc
5618ba9f46
cscli: refactor hub commands (#2500) 2023-10-04 10:42:47 +02:00
mmetc
cba6de024f
cscli: restore config correctly if acquis.d already exists (#2504) 2023-10-02 13:31:04 +02:00
mmetc
3cb9dbdb21
notification-email: configurable timeouts (#2465)
* configurable timeouts
* parse email timeouts as duration string
* add helo_host to email.yaml
* move html and body tags outside of the loops
* added quotes to href=.., and formatting test
2023-09-29 16:59:06 +02:00
mmetc
95ed308207
cscli setup: accept stdin; fix proftpd detection test and service unmask (#2496) 2023-09-29 12:58:35 +02:00
Laurence Jones
9dba6db676
add alert alias (#2485) 2023-09-23 19:35:02 +01:00
mmetc
d45bec4047
minor log message improvements (#2455) 2023-09-12 11:04:56 +02:00
bui
5a0b1b72d3 up 2023-09-12 10:42:28 +02:00
Laurence Jones
702da0f59a
[enhancement] cscli explain --labels (#2461)
* Add label support for explain and allow user to provide multiple labels

* Change my mind about empty string

* Add debug and im an idiot 😄
2023-09-11 14:18:04 +01:00
mmetc
fd94e2c056
refactor alert/decisions insert/update to avoid database locking in bulk operations (#2446) 2023-09-04 14:21:45 +02:00
mmetc
22146eb3e4
fix "cscli console disable --all"; cleanup "cscli console" command (#2444) 2023-08-29 11:44:23 +02:00
mmetc
b562103024
Make: build with debug symbols in func tests or if DEBUG=1; drop BUILD_VENDOR_FLAGS (#2443) 2023-08-28 15:58:26 +02:00
mmetc
2aa55e9444
move plugins/notifications/* to cmd/notification-* (#2429)
This ensures keeping all dependencies in sync, and simplifies
packaging under freebsd/gentoo/etc because there is a single
vendor directory.
2023-08-24 09:46:25 +02:00
mmetc
e36df40ba7
pkg/types cleanup (#2398)
* move function GetLineCountForFile from pkg/types to cscli
* move ParseDuration from pkg/types to pkg/database
* remove unused types.Profile, types.RemediationProfile
2023-08-24 09:44:46 +02:00
Rasmus
b4d9223625
Update main.go (#2431) 2023-08-22 14:09:53 +02:00
Efren
1ec52431b6
Remove duplicate line (#2432)
Remove duplicate line (60) which is outputting `Configuration Folder` twice in `cscli config show`
2023-08-22 14:09:19 +02:00
mmetc
e8e2ade8f0
remove calls to log.Fatal (#2399)
* remove log.Fatal from scenarios.go
* remove log.Fatal from collections.go
* remove log.Fatal from parsers.go and postoverflows.go
2023-08-16 21:04:46 +02:00
mmetc
caaed7c515
Timeout on shutdown while waiting for events to be flushed (#2423) 2023-08-16 21:03:15 +02:00
AlteredCoder
31c5727a90
Simplify context add (#2408) 2023-08-04 16:50:35 +02:00
mmetc
644c767019
cscli decisions list -o json => [] instead of null; same for alerts (#2397) 2023-08-03 12:51:50 +02:00
Laurence Jones
6ba682a32f
Update bouncers.go (#2404)
Fix wrong short
2023-08-03 11:26:08 +01:00
Sebastien Blot
711f0474d9
merge from master 2023-07-31 17:05:25 +02:00
Sebastien Blot
dd83bdea6b
revert previous bad merge 2023-07-31 17:00:06 +02:00
Laurence Jones
a18df9c3bb
Add bouncers prune command (#2379)
* Add bouncers prune command

* No point overloading functions

* Add prune to list of commands

* change all short desc to be similar, and made it really really clear when pruning it is not recoverable

* Dont use log. and dont return error on user input to abort
2023-07-28 15:37:39 +01:00
mmetc
ffadd42779
update dependency on go-cs-lib; drop the pkg/ part (#2393) 2023-07-28 16:35:08 +02:00
Laurence Jones
55247cd46a
Add machines prune command (#2011)
* Add machines prune command

* Fix scope variable for naming scheme

* Add some freshness and add new features

* Fix force and fix duration if less than 60

* Allow duration to be more readable

* Fix description

* Improve func wording and make int machines length

* No point overloading functions

* Add prune to list of commands

* Check if GID is already the group if so no need to chown

* Revert "Check if GID is already the group if so no need to chown"

This reverts commit c7cef1773e.

* change all short desc to be similar, and made it really really clear when pruning it is not recoverable

* Better examples

* Match bouncer like for like

* Fix merge error

* Dont use log. and dont return error on user input to abort
2023-07-28 15:23:47 +01:00
mmetc
ae53c0f1cc
fix "crowdsec-cli/require" log verbosity (#2390) 2023-07-28 09:56:20 +02:00
mmetc
5cb7013575
Check cscli preconditions with crowdsec-cli/require package (#2388) 2023-07-27 17:02:20 +02:00
mmetc
a01ce18b98
replace imports of path with path/filepath (#2330) 2023-07-26 10:29:58 +02:00
mmetc
1a6f12c88e
Build target for "make tidy" (#2378)
The make tidy target runs "go mod tidy" in the root directory and all plugins.
2023-07-26 10:24:37 +02:00
Laurence Jones
389ea4293f
Add metabase version override and update (#2370)
* Add version override and update

* Ooppsie

* Quick fix

* fgs copilot

* Allow user to overwrite image, add warning for exposing metabase and general cleanup

* One ix

* Default image if not found in config, and add a warning to remove and update

* Reorder check system memory checks so it inline with @mmetc best pratices

* No need for err

* Clean up some group code

* Change ipv6 as [] seems to wildcard

* Split loopback warn and disclaimer. Add force yes to start to allow user to accept disclaimer by default

* All cmd commands are RunE clean up

* Update flag name and dont allow a shorthand
2023-07-25 14:21:25 +01:00
mmetc
4bc225f26b
change output of "cscli metrics -o [json|raw]" from list of objects to map with table names (#2375) 2023-07-25 13:33:50 +02:00
mmetc
b6b6fd026b
typo fix, uppercase 'API', adjusted log level (#2361) 2023-07-21 23:23:24 +02:00
Manuel Sabban
9ac5aeda79
fix the ci by adding the ability to enforce event ordering (#2347)
* fix the ci by adding the ability to enforce event ordering
2023-07-20 11:41:30 +02:00
mmetc
3c16139c44
Reduce log verbosity at startup (#2363)
A configuration syntax test is performed every time the service is
started from systemd. The resulting error, if any, is shown on
journalctl logs.
This PR removes the unnecessary output in crowdsec.log generated by the
configuration test.
2023-07-19 13:28:52 +02:00
mmetc
bb16552aca
Use same levenshtein package for cscli, ent, hcl (#2359)
remove one dependency, slightly smaller binary
2023-07-18 11:30:14 +02:00
bui
8baeb70998 add metrics 2023-07-10 18:00:19 +02:00
mmetc
9967d60987
errors.Wrap -> fmt.Errorf (#2333) 2023-07-06 10:14:45 +02:00
alteredCoder
84b6570554 Revert "Merge remote-tracking branch 'origin' into coraza_poc_acquis"
This reverts commit 7098e971c7, reversing
changes made to 13512891e4.
2023-07-04 18:46:20 +02:00
alteredCoder
7098e971c7 Merge remote-tracking branch 'origin' into coraza_poc_acquis 2023-07-04 17:42:39 +02:00
mmetc
85839b0199
support for stdin with "cscli decision import" and raw values (#2291)
and remove Origin from the struct, which was ignored anyway
2023-06-27 14:29:42 +02:00
mmetc
a910b7beca
non-fatal error if some datasource can't be run (i.e. journalctl but systemd is missing) (#2309)
This on the other hand, gives a new fatal error when there are no valid datasources.
In the previous version, crowdsec kept running with just a warning if no
acquisition yaml or dir were specified.
2023-06-27 10:13:13 +02:00
mmetc
62caffb102
update leakybucket readme (#2298) 2023-06-22 15:35:01 +02:00
mmetc
fddf597040
errors.Wrap -> fmt.Errorf; clean up imports (#2297) 2023-06-22 15:01:34 +02:00
mmetc
25bb23d8b7
minor refactor to pkg/types, cscli machines (#2270)
* cleanup: separate ui and logic
* trim some code from pkg/types
2023-06-08 15:08:51 +02:00
mmetc
6096cb3c9b
Move grok_pattern.go away from pkg/types to trim bouncer dependencies (#2269) 2023-06-08 15:07:30 +02:00
mmetc
4e2c9c185b
Implement "crowdsec -fatal" flag; change help message (#2266)
The -trace...-fatal flags do not change the log destination but only the
verbosity. This change reflects that, and implements "-fatal" which was missing.
2023-06-08 15:06:06 +02:00
mmetc
8da9d5eefd
don't log notification error if not running under systemd (#2274) 2023-06-08 15:04:48 +02:00
Sebastien Blot
415e2dc68d
merge 2023-06-08 11:22:16 +02:00
mmetc
3cc6b2c0d0
CI: add tests for metrics configuration (#2251) 2023-06-05 23:17:30 +02:00
mmetc
9ccdddaab1
CI: refactor makefile for plugins and vendor target (#2256) 2023-06-05 23:15:18 +02:00
Sebastien Blot
7078d79ce4
merge 2023-06-05 14:30:14 +02:00
mmetc
2a8e97d558
show option -winsvc only under windows (#2258) 2023-06-05 13:49:31 +02:00
mmetc
228e4f9acc
cscli: add G (1e9) suffix to metric units (#2254) 2023-06-02 14:38:11 +02:00
mmetc
396dcf8e6e
dependencies: replaced function calls to pkg/types, errors.Wrap (#2235)
we now use a generic pointer function, and slowly remove the deprecated pkg/errors
2023-06-01 16:31:56 +02:00
mmetc
92a9d6c321
types.InSlice() -> slices.Contains() (#2246) 2023-05-31 12:39:22 +02:00
mmetc
9167bd107d
decouple bouncer dependencies: use go-cs-lib/pkg/ptr (#2228) 2023-05-25 15:43:39 +02:00
mmetc
1f9f81da70
makefiles: de-duplicate, simplify and remove unused code (#2222) 2023-05-25 10:32:05 +02:00
mmetc
534328ca30
decouple bouncer dependencies: use go-cs-lib/pkg/* (#2216)
* decouple bouncer dependencies: use go-cs-lib/pkg/trace
* decouple bouncer dependencies: use go-cs-lib/pkg/version
* decouple bouncer dependencies: use go-cs-lib/pkg/yamlpatch
* decouple bouncer dependencies: use go-cs-lib/pkg/csstring
* unused import
2023-05-23 10:52:47 +02:00
blotus
6e3ca35941
fallback to master for hub index download if it does not exist (#2210) 2023-05-17 11:20:53 +02:00