Commit graph

1491 commits

Author SHA1 Message Date
mmetc
4a6fd338e0
replace 'timeout' helper with async python script; allow hub preload in func tests; improve item removal (#2591)
* replace 'timeout' helper with async python script; allow hub preload in func tests; improve item removal
* func tests: cscli hub update/upgrade
* docker test update
* Update docker entrypoint to disable items with --force

The --force flag was not transmitted to cscli, but is required after the hub refact
to disable items inside installed collections
2023-11-14 17:36:07 +01:00
mmetc
f8c91d20b0
enable CI tests for hub-1.5.6 (#2592) 2023-11-14 15:20:28 +01:00
Marco Mariani
120f7cf578 Merge branch 'master' into hub-1.5.6 2023-11-14 15:04:50 +01:00
mmetc
042d316fab
Refact cwhub: remove global hub, func test improvements (#2588)
* csConfig.Cscli is always loaded now, configuration paths too
* Remove global/singleton hub instance
* read {index_path} from config instead of assuming {hub_dir}/.index.json
* fix segfault with cscli explain when no parser is installed
* cscli: help text
* hub download timeout 20 sec
* reduce log verbosity
* allow func tests with empty hub or pre-download
* cscli <itemtype> remove --all --purge
2023-11-14 14:58:36 +01:00
mmetc
d5c7870826
Refact cwhub: remove global hub instance (#2587)
* csConfig.Cscli is always loaded now, configuration paths too
* Remove global/singleton hub instance
2023-11-10 17:32:12 +01:00
mmetc
9d7ed12950
Refact cwhub (#2586)
* Inspect item: always show tainted, installed, etc. when false
* cleanup, comments, unused stuff
* download collection content after downloading dependencies, avoid duplicate call
* Return instances from Item.SubItems()
* shorter i/o code
* inline / simplify getData()
* Handle timeout connections when downloading from hub or data
2023-11-10 10:25:29 +01:00
mmetc
ab8de19506
Refact cwhub: move methods from hub to item (#2585)
* Add back pointer Item.hub
* Hub.enableItem() -> Item.enable()
* Rename variable i -> idx (i is used for item instances)
* Move Hub.purgeItem() -> Item.purge()
* Move Hub.disableItem() -> Item.disable()
* Move Hub.downloadItem() -> Item.download()
* Move Hub.downloadLatest() -> Item.downloadLatest()
* Move Hub.DownloadDataIfNeeded() -> Item.DownloadDataIfNeeded()
* Move Hub.InstallItem() -> Item.Install()
* Move Hub.RemoveItem() -> Item.Remove()
* Move Hub.UpgradeItem() -> Item.Upgrade()
* store hub items as pointers
* No need to re-add items to the hub if we use pointers
* Fix parameter calling order + regression test
2023-11-09 15:19:38 +01:00
mmetc
f80d841188
Refact cwhub: make some methods private (#2584)
* make hub.enableItem() private
* make hub.downloadLatest() private
* make getData() private
* make hub.disableItem() private
* make hub.downloadItem() private
* make hub.syncDir() private
* make hub.localSync() private; keep warnings in Hub struct (no need to call LocalSync to get them)
2023-11-09 12:07:09 +01:00
mmetc
ec4b5bdc86
Refact cwhub (#2583)
* no need to use NewRequest()
* download error messages
* cscli hub list: fix item stats
* Method item.HasSubItems() - avoid explicit type check
* cscli config restore: drop silent install, just call InstallItem
* no backpointer yet
2023-11-09 11:34:14 +01:00
mmetc
f4b5bcb865
Refact cwhub: version comparison and branch selection (#2581)
* simplify GetItemByPath
* hub: sort version numbers by semver
* replace golang.org/x/mod/semver with github.com/Masterminds/semver/v3 (would not compare correctly)
* fix nil dereference with tainted items
* update tests for collections, postoverflows
* fix nil deref
* don't fallback to master if hub is not found, improve message
* explicit message for unknown version / tainted collections
2023-11-08 13:21:59 +01:00
guangwu
ddd6ee8e42
fix: typo (#2582)
Signed-off-by: guoguangwu <guoguangwu@magic-shield.com>
2023-11-08 09:26:34 +01:00
mmetc
5cd4406f5e
typos/grammar (#2561) 2023-11-07 15:07:36 +01:00
Manuel Sabban
4934fce769
update gantsign.golang name (#2558) 2023-11-07 14:53:14 +01:00
mmetc
ad54b99bf9
Refact pkg/hubtest (#2580)
* pkg/hubtest: lint (whitespace, empty lines)
* use existing function to sort keys
* lint
* cscli hubtest: set TZ=UTC
* dedup Coverage struct
* pre-compile regexps
* remove redundant type declarations or global vars
2023-11-07 14:02:02 +01:00
Marco Mariani
84be2b8c97 Merge branch 'master' into hub-1.5.6 2023-11-07 13:25:18 +01:00
mmetc
bfd94ceda7
make ParserIndex(), DownloadIndex() private methods (#2579)
* unnecessary pointer type
* ParseIndex() as hub method, don't collect missing items since they are never used
* don't export hub.parseIndex(), hub.downloadIndex()
2023-11-07 10:27:33 +01:00
mmetc
41d19de092
Refact cwhub (#2578)
* Fix suggest functional tests
* comments
* non-empty SubItems() implies collections type
* use "slices" from stdlib
* No need to repeat author field in the index -- take it from the item key
2023-11-06 17:35:33 +01:00
mmetc
450c263826
Refact cwhub: minor cleanups and comments (#2574)
* check response status before body; close file
* err check one-liners, lint, comments
* simplify function logic, reduce code
* comments, xxx, whitespace
2023-10-31 16:32:29 +01:00
Marco Mariani
fcd6c468c4 fix lint 2023-10-31 13:12:28 +01:00
mmetc
590a19b768
Refact pkg/cwhub: constructor, cscli output
* Single constructor: NewHub() to replace InitHub(), InitHubUpdate()
* sort cscli hub list output
* log.Fatal -> fmt.Errorf
2023-10-31 12:47:39 +01:00
mmetc
17662e59a9
Refact pkg/cwhub, cscli: hub upgrades (#2568)
* fix bats test for "upgrade all items"
* refact UpgradeConfig() -> UpgradeItem(): one item only
* refact RemoveMany() -> RemoveItem()
* Computed value: Item.Local -> Item.IsLocal()
* refact url/branch configuration with LocalHubCfg/RemoteHubCfg
2023-10-30 17:23:50 +01:00
mmetc
6b8ed0c9d0
Refactor hub URL/branch configuration (#2559)
* Refactor hub URL/branch configuration
* docker: using --force to implement $DISABLE (required for items in collections)
* use pointer receiver for consistency
2023-10-27 10:25:29 +02:00
mmetc
272cf543b3
Release action: fix asset upload (#2565) 2023-10-25 14:51:36 +02:00
mmetc
ac98256602
Refact pkg/cwhub, cmd/crowdsec-cli (#2557)
- pkg/cwhub: change file layout, rename functions
 - method Item.SubItems
 - cmd/crowdsec-cli: generic code for hub items
 - cscli: removing any type of items in a collection now requires --force
 - tests
2023-10-20 14:32:35 +02:00
Marco Mariani
b89c5652ca Merge branch 'master' into hub-1.5.6 2023-10-19 12:05:19 +02:00
mmetc
88e4f7c157
Refact pkg/csconfig, pkg/cwhub (#2555)
* csconfig: drop redundant hub information on *Cfg structs
* rename validItemFileName() -> item.validPath()
* Methods on hub object
* updated tests to reduce need of csconfig.Config or global state
2023-10-19 12:04:29 +02:00
mmetc
57d3ebba12
typo (#2556) 2023-10-18 10:03:02 +02:00
mmetc
be6555e46c
Refact pkg/csconfig, HubCfg (#2552)
- rename csconfig.Hub -> HubCfg
 - move some Load*() functions to NewConfig()
 - config.yaml: optional common section
 - remove unused working_dir
2023-10-18 09:38:33 +02:00
Laurence Jones
d2d788c5dc
[hubtest] escpae scenario asssert meta keys (#2551) 2023-10-17 15:29:21 +01:00
mmetc
4eae40865e
HubIndex struct, comments, name changes (#2549)
* pkg/cwhub: rename PARSERS_OVFLW -> POSTOVERFLOWS
* mostly comments, some light cleanup
* move type hubtest.HubIndex -> cwhub.HubIndex
* move and rename LoadPkgIndex -> ParseIndex
* move displaySummary(), skippedLocal, skippedTainted to HubIndex struct
2023-10-17 16:17:37 +02:00
mmetc
810a8adcf0 fix build (#2548) 2023-10-17 16:12:41 +02:00
mmetc
325003bb69 Refact cscli item listing, tests (#2547)
* hub diet; taint tests
* cmd/crowdsec-cli: split utils.go, moved cwhub.GetHubStatusForItemType()
* cscli: refactor hub list commands, fix edge cases
2023-10-17 16:12:41 +02:00
mmetc
f496bd1692 bats: more cscli hub tests (#2541)
- updated logs and user messages
- added func tests for all the items: install, remove, upgrade, list
- rewritten taint tests for collections
- removed redundant csconfig.LoadPrometheus()
2023-10-17 16:12:41 +02:00
mmetc
a00bae6039 cmd/crowdsec-cli: remove global prometheusURL (#2542)
* cmd/crowdsec-cli: remove global prometheusURL
* PrometheusUrl now includes the path (/metrics)
2023-10-17 16:12:41 +02:00
mmetc
734ba46e6a Refact cscli hub/item commands (#2536)
* log.Fatal -> fmt.Errorf
* lint cmd/crowdsec-cli hub items and split collection commands
* cscli collections: add examples
* cscli parsers: avoid globals
* cscli scenarios: avoid globals
* cscli collections, postoverflows: avoid globals
* cscli hub: avoid globals
* remove unused globals
2023-10-17 16:12:41 +02:00
mmetc
7db5bf8979 pkg/csconfig: set prometheus address:port defaults (#2533)
We set these default in one place (after loading the configuration)
instead of leaving that to both metric server and consumer.
2023-10-17 16:12:41 +02:00
Thibault "bui" Koechlin
a4dc5053d2
fix null deref in cti calls if key is empty (#2540)
* fix null deref in cti calls if key is empty

* avoid hardcoded error check
2023-10-17 09:34:53 +01:00
Laurence Jones
19de3a8a77
Runtime whitelist parsing improvement (#2422)
* Improve whitelist parsing

* Split whitelist check into a function tied to whitelist, also since we check node debug we can make a pointer to node containing whitelist

* No point passing clog as an argument since it is just a pointer to node we already know about

* We should break instead of returning false, false as it may have been whitelisted by ips/cidrs

* reimplement early return if expr errors

* Fix lint and dont need to parse ip back to string just loop over sources

* Log error with node logger as it provides context

* Move getsource to a function cleanup some code

* Change func name

* Split out compile to a function so we can use in tests. Add a bunch of tests

* spell correction

* Use node logger so it has context

* alternative solution

* quick fixes

* Use containswls

* Change whitelist test to use parseipsource and only events

* Make it simpler

* Postoverflow tests, some basic ones to make sure it works

* Use official pkg

* Add @mmetc reco

* Add @mmetc reco

* Change if if to a switch to only evaluate once

* simplify assertions

---------

Co-authored-by: bui <thibault@crowdsec.net>
Co-authored-by: Marco Mariani <marco@crowdsec.net>
2023-10-16 10:08:57 +01:00
Laurence Jones
e7ad3d88ae
Clear up some community confusion (#2543) 2023-10-16 10:08:41 +01:00
Thibault "bui" Koechlin
3cd4847093
sort map keys when generating asserts (#2494)
* sort map keys when generating asserts
2023-10-16 09:54:19 +02:00
Laurence Jones
b2a6eb92bf
Dont create 3 maps just pass the same one to expr (#2421) 2023-10-13 22:35:30 +01:00
Laurence Jones
f0cda0406b
Load file only once if specified twice, and bail earlier if type is unknown (#2419) 2023-10-13 22:34:57 +01:00
Laurence Jones
ff7acd3347
Reset grokky once all patterns are compiled as we do not need to hold them in memoory (#2420) 2023-10-13 12:53:42 +01:00
mmetc
a6b55f2b5e
cscli config feeature-flags: point user to the right location of feature.yaml (#2539) 2023-10-13 09:52:51 +02:00
mmetc
a254b436c7
use go 1.12.3 (#2535) 2023-10-12 16:28:24 +02:00
mmetc
3b1563a538
Refact cscli hub / pkg/cwhub (part 6) (#2524)
* hub.ConfigDir -> hub.InstallDir; hub.DataDir -> hub.InstallDataDir
* cleanup GetInstalledItemsAsString()
* lint: ReferenceMissingError -> ErrMissingReference
* lint: parent_dir -> parentDir
* link: export Walker type
* lint: return error last
* lint: shadow
* move around and group variable definitions
2023-10-09 21:33:35 +02:00
mmetc
0ecb6eefee
add missing scenarios in first login when authenticating with TLS (#2454)
* refact jwt:Authenticator
* include scenarios in first login request for machines with tlsAuth
* log.Printf -> log.Infof
* errors.Wrap -> fmt.Errorf
* don't override validation error
* fix test
2023-10-09 15:26:38 +02:00
Manuel Sabban
6e228f3f3f
pkg/cwhub: cleanup in argument call (#2527)
* cleanup in argument call
* update test as well
* cwhub_tests: reduce verbosity and use helpers

---------

Co-authored-by: Marco Mariani <marco@crowdsec.net>
2023-10-09 13:26:34 +02:00
Laurence Jones
28238cb01f
reverse nil statement instead of else (#2530) 2023-10-09 11:36:05 +01:00
Laurence Jones
0dd22e8b93
convert ifelseif to switch (#2529) 2023-10-09 11:23:19 +01:00