Commit graph

674 commits

Author SHA1 Message Date
mmetc
ca784b147b
test and log fixes (#2690)
* cscli inspect: suggest --diff if an item is tainted
* appropriate warning, or error if context configuration file is empty
* fix user/group lookup unit test
* fix: allow hub upgrade --force with local items
* fix pkg/parser lookup for 8.8.8.8
* fix func test
* fix hubtests: machines add --force
2024-01-03 09:33:52 +01:00
blotus
b6f272d09a
always set the transaction in the current request (#2682) 2023-12-22 11:44:06 +01:00
blotus
a62e28fdfb
always set inband transaction even if we have no rules (#2681) 2023-12-22 10:18:35 +01:00
blotus
33e3fdabe4
Appsec additional fixes (#2676) 2023-12-21 11:51:04 +01:00
mmetc
6e34d609b7
cscli: silence cwhub logger for non-hub related commands (#2675) 2023-12-19 17:20:09 +01:00
mmetc
822fcdacbb
fflags: don't print deprecation warning if there is no message (papi) (#2666) 2023-12-18 09:35:57 +01:00
mmetc
08694adf1b
lint (errorlint) (#2644) 2023-12-18 09:35:28 +01:00
mmetc
a79fcaf378
Add "taintedBy" and "--diff" flag to cscli... inspect (#2665)
* "cscli inspect" reports tainted sub-items
* cscli... inspect --diff
* unified diff
* option --diff --rev
* tainted message
* correctly report multiple taint reasons
2023-12-15 15:27:22 +01:00
blotus
9b07e1f7ce
update scenarios and parsers constraints (#2663) 2023-12-14 16:34:51 +01:00
AlteredCoder
a941576acc
Improvement to run hubtest for appsec in docker (#2660) 2023-12-14 16:05:16 +01:00
mmetc
89f704ef18
light pkg/api{client,server} refact (#2659)
* tests: don't run crowdsec if not necessary
* make listen_uri report the random port number when 0 is requested
* move apiserver.getTLSAuthType() -> csconfig.TLSCfg.GetAuthType()
* move apiserver.isEnrolled() -> apiclient.ApiClient.IsEnrolled()
* extract function apiserver.recoverFromPanic()
* simplify and move APIServer.GetTLSConfig() -> TLSCfg.GetTLSConfig()
* moved TLSCfg type to csconfig/tls.go
* APIServer.InitController(): early return / happy path
* extract function apiserver.newGinLogger()
* lapi tests
* update unit test
* lint (testify)
* lint (whitespace, variable names)
* update docker tests
2023-12-14 14:54:11 +01:00
mmetc
67cdf91f94
Short build tag in version number (#2658)
* use short commit hash in version number
* var -> const
* cscli: extract version.go, doc.go
* don't repeat commit hash in version number
2023-12-14 09:16:38 +01:00
Thibault "bui" Koechlin
51f70e47e3
Minor improvements to hubtest and appsec component (#2656) 2023-12-13 17:45:56 +01:00
blotus
04f3dc09f9
remove PAPI feature flag (#2601) 2023-12-08 14:55:45 +01:00
AlteredCoder
b1f85693c2
Appsec improvement and fixes after merge (#2645) 2023-12-08 10:25:00 +01:00
mmetc
4acb4f8df3
cwhub: context type (#2631)
* add hub type "context"
* cscli lapi: log.Fatal -> fmt.Errorf; lint
* tests for context.yaml
* load console context from hub
* original & compiled context
* deprecate "cscli lapi context delete"
$ cscli lapi context delete
Command "delete" is deprecated, please manually edit the context file.
* cscli completion: add appsec-rules, appsec-configs, explain, hubtest
2023-12-07 16:20:13 +01:00
Thibault "bui" Koechlin
8cca4346a5
Application Security Engine Support (#2273)
Add a new datasource that:
- Receives HTTP requests from remediation components
- Apply rules on them to determine whether they are malicious or not
- Rules can be evaluated in-band (the remediation component will block the request directly) or out-band (the RC will let the request through, but crowdsec can still process the rule matches with scenarios)

The PR also adds support for 2 new hub items:
- appsec-configs: Configure the Application Security Engine (which rules to load, in which phase)
- appsec-rules: a rule that is added in the Application Security Engine (can use either our own format, or seclang)

---------

Co-authored-by: alteredCoder <kevin@crowdsec.net>
Co-authored-by: Sebastien Blot <sebastien@crowdsec.net>
Co-authored-by: mmetc <92726601+mmetc@users.noreply.github.com>
Co-authored-by: Marco Mariani <marco@crowdsec.net>
2023-12-07 12:21:04 +01:00
mmetc
90d3a21853
CI: use go 1.21.5 (#2640)
* use go 1.21.5
* Simpler go:build directives
2023-12-06 12:38:36 +01:00
mmetc
1ab4487b65
cscli hub list: show only non-empty tables with -o human
* agent config: remove unused LintOnly bool
* Item.IsLocal() -> Item.State.IsLocal(); split method InstallStatus()
* cscli hub list: show only non-empty tables with -o human
2023-12-05 13:38:52 +01:00
mmetc
23968e472d
Refact bouncer auth (#2456)
Co-authored-by: blotus <sebastien@crowdsec.net>
2023-12-04 23:06:01 +01:00
Laurence Jones
d1bfaddb69
[Plugin] Pass down ctx and use it (#2626)
* Pass down cancellable context and update http plugin

* Use context where we can
2023-12-04 12:05:26 +00:00
Laurence Jones
ed3d501081
[Metabase] QOL Changes and chown wal files (#2627)
* Add detection sqlie wal for dashboard chown

* Lean it down a little

* Change to for loop with extensions

* Keep existing uid on files incase user is running as a unpriviledge user

* I have no idea 🤷

* Exclude dash.go and update windows

* Update

* Renam

* Remove the os check since we no longer get to this stage for those os's

---------

Co-authored-by: Manuel Sabban <github@sabban.eu>
2023-12-04 10:06:41 +00:00
mmetc
7e5ab344a2
command "cscli hub types" (#2632)
* Command "cscli hub types"; de-duplicate test/bin/preload-hub-items
* don't export Hub.Items -> hub.items
2023-12-01 09:36:38 +01:00
Cristian Nitescu
7c5cbef51a
manage force_pull message for one blocklist (#2615)
* manage force_pull message for one blocklist

* fix info message on force pull blocklist
2023-11-29 11:37:46 +01:00
mmetc
6b0bdc5eeb
Refact pkg/cwhub: fix some known issues and reorganize files (#2616)
* bump gopkg.in/yaml.v3
* test: cannot remove local items with cscli
* test dangling links
* test: cannot install local item with cscli
* pkg/cwhub: reorg (move) functions in files
* allow hub upgrade with local items
* data download: honor Last-Modified header
* fatal -> warning when attempting to remove a local item (allows remove --all)
* cscli...inspect -o yaml|human: rename remote_path -> path
* Correct count of removed items
Still no separate counter for the --purge option, but should be clear enough
2023-11-28 23:51:51 +01:00
blotus
380cbf70a9
force rfc 3339 date format in metrics push (#2402) 2023-11-28 16:30:20 +01:00
Laurence Jones
6a61b919e7
[cscli] notifications test command and slight re write (#2391)
* Merge main and apply stash

* Rework some of cscli notif stuff and add a generic test which works with non active profiles

* Update wording

* Fix merge

* Final version

* Cleanup
2023-11-28 13:17:54 +00:00
mmetc
15542b78fb
refact BulkDeleteDecisions (#2308)
Code cleanup and de-duplication.
2023-11-26 22:30:03 +01:00
mmetc
ffcab0b2bc
Refactor hub management and cscli commands (#2545) 2023-11-24 15:57:32 +01:00
mmetc
ec199162dc
iso8601: use yyyy-mm-dd in log timestamps instead of dd-mm-yyyy (#2564)
Co-authored-by: Thibault "bui" Koechlin <thibault@crowdsec.net>
2023-11-24 14:59:28 +01:00
Thibault "bui" Koechlin
1dcf9d1ae1
Improved expr debugger (#2495)
* new expr debugger

---------

Co-authored-by: mmetc <92726601+mmetc@users.noreply.github.com>
2023-11-24 11:10:54 +01:00
blotus
ec53c672dc
Kafka acquisition: warn if no consumer group id and allow to read from a specific partition (#2612) 2023-11-23 10:02:53 +01:00
lperdereau
92f923cfa8
Loki integration #2 (#2306)
* Add support for loki datasource

---------

Co-authored-by: Mathieu Lecarme <mathieu@garambrogne.net>
Co-authored-by: Sebastien Blot <sebastien@crowdsec.net>
Co-authored-by: Thibault "bui" Koechlin <thibault@crowdsec.net>
2023-11-22 13:31:39 +01:00
he2ss
947b247a40
kafkaAcquisition: add more debug (#2609)
* kafkaAcquisition: add more debug
2023-11-22 09:35:58 +01:00
blotus
d7ef51e6ba
properly update the cs_syslogsource_parsed_total metric (#2608) 2023-11-22 09:04:23 +01:00
mmetc
5cd4406f5e
typos/grammar (#2561) 2023-11-07 15:07:36 +01:00
Laurence Jones
d2d788c5dc
[hubtest] escpae scenario asssert meta keys (#2551) 2023-10-17 15:29:21 +01:00
Thibault "bui" Koechlin
a4dc5053d2
fix null deref in cti calls if key is empty (#2540)
* fix null deref in cti calls if key is empty

* avoid hardcoded error check
2023-10-17 09:34:53 +01:00
Laurence Jones
19de3a8a77
Runtime whitelist parsing improvement (#2422)
* Improve whitelist parsing

* Split whitelist check into a function tied to whitelist, also since we check node debug we can make a pointer to node containing whitelist

* No point passing clog as an argument since it is just a pointer to node we already know about

* We should break instead of returning false, false as it may have been whitelisted by ips/cidrs

* reimplement early return if expr errors

* Fix lint and dont need to parse ip back to string just loop over sources

* Log error with node logger as it provides context

* Move getsource to a function cleanup some code

* Change func name

* Split out compile to a function so we can use in tests. Add a bunch of tests

* spell correction

* Use node logger so it has context

* alternative solution

* quick fixes

* Use containswls

* Change whitelist test to use parseipsource and only events

* Make it simpler

* Postoverflow tests, some basic ones to make sure it works

* Use official pkg

* Add @mmetc reco

* Add @mmetc reco

* Change if if to a switch to only evaluate once

* simplify assertions

---------

Co-authored-by: bui <thibault@crowdsec.net>
Co-authored-by: Marco Mariani <marco@crowdsec.net>
2023-10-16 10:08:57 +01:00
Thibault "bui" Koechlin
3cd4847093
sort map keys when generating asserts (#2494)
* sort map keys when generating asserts
2023-10-16 09:54:19 +02:00
Laurence Jones
b2a6eb92bf
Dont create 3 maps just pass the same one to expr (#2421) 2023-10-13 22:35:30 +01:00
Laurence Jones
f0cda0406b
Load file only once if specified twice, and bail earlier if type is unknown (#2419) 2023-10-13 22:34:57 +01:00
Laurence Jones
ff7acd3347
Reset grokky once all patterns are compiled as we do not need to hold them in memoory (#2420) 2023-10-13 12:53:42 +01:00
mmetc
a6b55f2b5e
cscli config feeature-flags: point user to the right location of feature.yaml (#2539) 2023-10-13 09:52:51 +02:00
mmetc
3b1563a538
Refact cscli hub / pkg/cwhub (part 6) (#2524)
* hub.ConfigDir -> hub.InstallDir; hub.DataDir -> hub.InstallDataDir
* cleanup GetInstalledItemsAsString()
* lint: ReferenceMissingError -> ErrMissingReference
* lint: parent_dir -> parentDir
* link: export Walker type
* lint: return error last
* lint: shadow
* move around and group variable definitions
2023-10-09 21:33:35 +02:00
mmetc
0ecb6eefee
add missing scenarios in first login when authenticating with TLS (#2454)
* refact jwt:Authenticator
* include scenarios in first login request for machines with tlsAuth
* log.Printf -> log.Infof
* errors.Wrap -> fmt.Errorf
* don't override validation error
* fix test
2023-10-09 15:26:38 +02:00
Manuel Sabban
6e228f3f3f
pkg/cwhub: cleanup in argument call (#2527)
* cleanup in argument call
* update test as well
* cwhub_tests: reduce verbosity and use helpers

---------

Co-authored-by: Marco Mariani <marco@crowdsec.net>
2023-10-09 13:26:34 +02:00
Laurence Jones
28238cb01f
reverse nil statement instead of else (#2530) 2023-10-09 11:36:05 +01:00
Laurence Jones
0dd22e8b93
convert ifelseif to switch (#2529) 2023-10-09 11:23:19 +01:00
mmetc
9ae8bd79c5
Refact pkg/csconfig tests (#2526)
* remove unused method
* whitespace, redundant comments
* use test helpers
* move DumpConsoleConfig() from pkg/csconfig to cscli
* package doc header
* var -> const
* rename ./tests -> ./testdata
* shorter tests with more error checks
* lint/formatting
* use helpers; fix tests that didn't actually test
* lint; rename expectedResult -> expected
2023-10-09 11:10:51 +02:00