Commit graph

452 commits

Author SHA1 Message Date
Marco Mariani
91b0f8fee1 load custom configuration paths when agent is disabled 2023-01-18 15:15:18 +01:00
Marco Mariani
2e91a82aa7 load feature.yaml as soon as possible 2023-01-18 15:15:18 +01:00
Thibault "bui" Koechlin
f25fdecc3f
normalize scopes for alerts and decisions (#2001)
* normalize scopes for alerts and decisions
2023-01-18 14:50:03 +01:00
mmetc
51800132cd
improve feature flag logging (#1986)
For cscli: it should provide a terse output, not nag users with configuration details. Although it's usually important that cscli and crowdsec have the same enabled features, having it list them every time the command is invoked can be too much.

For crowdsec: when features are set from the environment, it's too early to log where we should. So we can use log.Debug at activation time, and list them again once logging is configured.

 - wrap some functions in csconfig for convenience and DRY
 - for each enabled feature, log.Debug
 - log all enabled features once as Info (crowdsec) or Debug (cscli)
 - file does not exist -> log.Trace
2023-01-13 13:42:42 +01:00
Cristian Nitescu
73663ff9e7
log the request error even in case of retry (#1988) 2023-01-13 12:58:12 +01:00
mmetc
ba4396e52c
fix flaky parser unit test (#1985) 2023-01-12 17:03:25 +01:00
Thibault "bui" Koechlin
6fb962a941
Allow parsers to capture data for future enrichment (#1969)
* Allow parsers to capture data in a cache, that can be later accessed via expr helpers (fake multi-line support)
2023-01-11 15:01:02 +01:00
mmetc
cd4dabde0e
silence yaml.local explicitly in cscli, keep in crowdsec/bouncer logs (#1981) 2023-01-11 09:50:46 +01:00
Laurence Jones
ca12432a2a
Change patch to debug, if user has a local overide they will get informed every cscli call (#1980) 2023-01-10 10:05:18 +00:00
Cristian Nitescu
7284c0a47a
retry with backoff requests to CAPI (#1957)
* backoff on refresh token error

* fix tls communication with lapi and user/pw auth (#1956)

allow self-signed TLS encryption with user/pw auth

docker:
 - remove defaults for certificate file locations
 - new envvar INSECURE_SKIP_VERIFY
 - register agent before TLS settings (cscli machine add removes them
   from the credentials file)

* separate cscli cobra constructors:  lapi, machines, bouncers, postoverflows (#1945)

* use feature toggling to improve testability with http retry backoff

* Add parse unix to dateparse enricher (#1958)

Add parse unix is we do have a strTime but wasnt parsed using convential golang time

* func tests: redirect stderr to filter extra logs (#1961)

* backoff on refresh token error

* use feature toggling to improve testability with http retry backoff

* refactor feature backoff toggle for tests

Co-authored-by: mmetc <92726601+mmetc@users.noreply.github.com>
Co-authored-by: Laurence Jones <laurence.jones@live.co.uk>
2023-01-09 14:49:21 +01:00
blotus
a84e4b6b15
Add conditional bucket (#1962) 2023-01-06 09:26:16 +01:00
AlteredCoder
185f9ad541
Alert context (#1895)
Co-authored-by: bui <thibault@crowdsec.net>
2023-01-04 16:50:02 +01:00
mmetc
033082a31e
ParseUnix() test fix: force UTC (#1970) 2023-01-04 16:22:17 +01:00
mmetc
2d81e751a1
fix parser test 2k23 (#1971) 2023-01-04 15:46:16 +01:00
Laurence Jones
fd1c38811e
Add parse unix to dateparse enricher (#1958)
Add parse unix is we do have a strTime but wasnt parsed using convential golang time
2022-12-30 12:47:14 +00:00
mmetc
72c1753fb7
fix tls communication with lapi and user/pw auth (#1956)
allow self-signed TLS encryption with user/pw auth

docker:
 - remove defaults for certificate file locations
 - new envvar INSECURE_SKIP_VERIFY
 - register agent before TLS settings (cscli machine add removes them
   from the credentials file)
2022-12-29 22:00:11 +01:00
Laurence Jones
401739b036
Add unix expr helper (#1952)
* Add unix expr helper

* Add original value not parsed error

* return early if cannot parse

* Add tests

* Fix negative value
2022-12-29 14:53:06 +00:00
Thibault "bui" Koechlin
e4463c412b
Improve warnings around lack of evt.StrTime field (#1954)
* fix #1951 : improve error messages

* make hubtest warn you if you're missing evt.StrTime in your logs
2022-12-29 15:03:32 +01:00
mmetc
6efc2688b1
simplify feature flags (#1947)
Now checking for a feature flag is a one liner,
with no need to control errors.

if fflag.Crowdsec.CscliSetup.IsEnabled() {
   ...
}
2022-12-26 14:23:41 +01:00
mmetc
5d2c99bb17
runtime feature flag initialization 2022-12-21 17:19:20 +01:00
mmetc
ff88faf402
updated localstack dependencies, added build cache 2022-12-21 12:20:01 +01:00
mmetc
a32aa96752
feature flags (#1933)
Package fflag provides a simple feature flag system.

 Feature names are lowercase and can only contain letters, numbers, undercores
 and dots.

 good: "foo", "foo_bar", "foo.bar"
 bad: "Foo", "foo-bar"

 A feature flag can be enabled by the user with an environment variable
 or by adding it to {ConfigDir}/feature.yaml

 I.e. CROWDSEC_FEATURE_FOO_BAR=true
 or in feature.yaml:
```
 ---
 - foo_bar
```

 If the variable is set to false, the feature can still be enabled
 in feature.yaml. Features cannot be disabled in the file.

 A feature flag can be deprecated or retired. A deprecated feature flag is
 still accepted but a warning is logged. A retired feature flag is ignored
 and an error is logged.

 A specific deprecation message is used to inform the user of the behavior
 that has been decided when the flag is/was finally retired.
2022-12-20 16:11:51 +01:00
he2ss
579cecde04
apiclient: fix http roundtrip (clone body also) (#1758)
* apiclient: fix http roundtrip (clone body also)
2022-12-14 16:42:46 +01:00
Laurence Jones
fe23da6e0c
Add postgres socket support, clean some code (#1926) 2022-12-12 16:08:19 +00:00
Laurence Jones
11965f08db
Add socket support to mysql (#1911) 2022-12-08 09:33:08 +00:00
mmetc
cc228f1868
Typos, grammar (#1905) 2022-12-06 15:55:27 +01:00
blotus
fdda940ac0
Add Kubernetes audit acquisition (#1767) 2022-12-06 13:47:29 +01:00
mmetc
fd3e668fe1
add -error flag to crowdsec binary (#1903) 2022-12-03 08:56:11 +01:00
mmetc
fa0e590778
removed pid_dir (#1906) 2022-12-02 13:42:43 +01:00
mmetc
4a6a9c4355
acquisition: validate datasources before configuration (static checks) (#1841)
* acquisition: validate datasources before configuration (allow static configuration checks)

* remove comment

* import reviser, format

* error wrap
2022-11-30 17:36:56 +01:00
blotus
60f1228030
use a copy of bucket processors in LeakRoutine (#1902) 2022-11-30 10:59:47 +01:00
mmetc
104f5d1fe6
lint: error handling cleanup (#1877) 2022-11-29 09:16:07 +01:00
mmetc
66543493b5
fix nil dereference: check that httpServer is set before shutting down (#1893) 2022-11-28 11:55:08 +01:00
mmetc
fde9640364
Docker refactoring, tls setup (#1869) 2022-11-28 10:35:12 +01:00
blotus
c5079ac15e
invalidate agent token on 403 as well (#1888) 2022-11-25 14:35:50 +01:00
mmetc
5bdd3bbfcb
require at least go 1.18 to build (#1884) 2022-11-24 11:29:54 +01:00
Laurence Jones
4ac01ed880
Update perms for group read (#1876) 2022-11-21 09:49:56 +00:00
mmetc
3beb84bcfe
print missing "AS" values as empty strings instead of "0 " (#1867) 2022-11-14 09:55:53 +01:00
Thibault "bui" Koechlin
523343b174
notify when community-blocklist starts pull (#1845)
* minor change to notify blocklist pull update, will make eventual troubleshooting easier
2022-11-08 10:44:25 +01:00
Thibault "bui" Koechlin
3b4da7e637
fix #1860 : Only repeat the WAL warning once (#1863)
* fix #1860
2022-11-07 16:36:39 +01:00
mmetc
895691dad1
enabled linters: gocritic, nilerr (#1853) 2022-11-07 10:36:50 +01:00
Manuel Sabban
8aca00326d
fix ticker (#1858)
Co-authored-by: sabban <15465465+sabban@users.noreply.github.com>
2022-11-04 13:56:43 +01:00
Laurence Jones
668627f890
Add error checking to lookup host (#1847) 2022-10-31 18:38:01 +00:00
mmetc
344b1dc559
fixed package tests w/wal, gitignore/typos (#1849) 2022-10-31 10:02:51 +01:00
mmetc
df88f4e1e9
randomize pull, push and metric intervals; reload crowdsec only when hub changed (#1846) 2022-10-28 13:55:59 +02:00
mmetc
02d2eab18c
update golangci-lint to 1.50 and fixes (#1828) 2022-10-26 15:11:37 +02:00
ThinkChaos
22479a289d
Add LookupHost expr lib func (#1775) 2022-10-26 10:17:48 +01:00
mmetc
2088bb1f91
fix for #1839 (#1840) 2022-10-26 11:02:12 +02:00
blotus
b7c4bfd4e3
Use explicit transaction when inserting community blocklist (#1835) 2022-10-26 10:48:17 +02:00
mmetc
e545933923
fix(cscli): correct and test the behavior of "cscli collections delete" (#1824) 2022-10-25 14:10:51 +02:00