beenull/crowdsec
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity
703 commits 83 branches 177 tags 151 MiB
Commit graph

271 commits

Author SHA1 Message Date
AlteredCoder
cf57c89177
add name and alias in cscli console enroll (#950) 
* add name and alias in cscli console enroll
 2021-10-26 15:33:17 +02:00
blotus
25a2d528b0
Alerts flush: Optimization of the flush mechanism (batch and limit to one job) + add cscli alerts flush command (#1024) 
- Don't allow running more than one alert flush job at a time to prevent runaway CPU usage in some case. (fix High CPU after Upgrade to 1.2.0 #1022)
 - Add a cscli alerts flush command to manually flush the alerts in the database (fixes Improvement/Manual flush mechanism #1023 ).
 - Enable cascading deletion on alerts as we upgraded ent: Deleting an alert in the database will automatically delete all related decisions, events and meta
 - Add an index on alerts.id to try to improve flush performance with very big sqlite database.
- Flush alert now operates in batch
 2021-10-26 13:33:45 +02:00
Thibault "bui" Koechlin
3f99330b3d
Entgo 0.9 (#1018) 
* update entgo & sqlite to latest version

* schema update
 2021-10-22 16:15:57 +02:00
Shivam Sandbhor
a7b1c02bd5
Fix bugs in cloudwatch acq (#991) 
* Fix bugs in cloudwatch acq

- Fix concurrent writes to map streamIndexes
- Fix multiple cases of modifying while iterating on slice.
- Fix order of fetching cloudwatch events.
- Remove `startup` hack.

Signed-off-by: Shivam Sandbhor <shivam.sandbhor@gmail.com>

* Fix cloudwatch tests

Signed-off-by: Shivam Sandbhor <shivam.sandbhor@gmail.com>
 2021-10-22 10:35:05 +02:00
Thibault "bui" Koechlin
3bb2128bf4
fix sort :/ (#1007) 2021-10-12 19:16:24 +02:00
Thibault "bui" Koechlin
1bd6b8f7b9
Multiple fixes (#1006) 
* fix #1005 : timestamp in trigger timemachine buckets

* attempt at consistent bucket order for hubtest
 2021-10-12 14:09:17 +02:00
Thibault "bui" Koechlin
2961a0ed02
ensure machineID is included early enough into the alert (#1004) 2021-10-11 15:02:16 +02:00
blotus
2bc9f33e12
add ParseUri() expr helper (#994) 2021-10-08 16:50:31 +02:00
AlteredCoder
0ccc69696b
Break on success when alert already has decision (#997) (#999) 
* Break on success when alert already has decision (#997)

Signed-off-by: Shivam Sandbhor <shivam.sandbhor@gmail.com>
 2021-10-05 11:30:34 +02:00
Thibault "bui" Koechlin
af4bb350c0
hubtests revamp + cscli explain (#988) 
* New hubtest CI for scenarios/parsers from the hub
 * New `cscli explain` command to visualize parsers/scenarios pipeline

Co-authored-by: alteredCoder <kevin@crowdsec.net>
Co-authored-by: Sebastien Blot <sebastien@crowdsec.net>
Co-authored-by: he2ss <hamza.essahely@gmail.com>
Co-authored-by: Cristian Nitescu <cristian@crowdsec.net>
 2021-10-04 17:14:52 +02:00
Thibault "bui" Koechlin
c2fd173d1e
fix node success logic (#993) 
* fix node success logic : only fail node on child failure if mother node has no successfull grok
 2021-09-28 17:58:07 +02:00
he2ss
fb308d5596
fix plugins logging in right level (#990) 2021-09-28 14:44:21 +02:00
he2ss
db5ffb0040
Update test env (#987) 
* update test_env
 2021-09-24 18:06:30 +02:00
blotus
f0db3742de
fix usage of regex.Match in cloudwatch module (#986) 2021-09-23 13:52:05 +02:00
Shivam Sandbhor
cca76da2d6
Fix crash if plugin config is broken (#964) 
Signed-off-by: Shivam Sandbhor <shivam.sandbhor@gmail.com>
 2021-09-10 14:25:34 +02:00
he2ss
e651379964
add jsonExtractUnescape Helper (#962) 
* add jsonExtractUnescape Helper
 2021-09-10 12:43:11 +02:00
AlteredCoder
5ae69aa293
fix stacktrace when mmdb file are not present (#935) 
* fix stacktrace when mmdb file are not present
 2021-09-09 16:27:30 +02:00
blotus
7a1b955ad1
use our fork of grokky (#953) 2021-09-09 14:46:16 +02:00
Shivam Sandbhor
b8e24a1e0b
Make plugin runner configurable and run only registered plugins (#944) 
* Make plugin runner configurable and run only registered plugins
 2021-09-08 11:36:42 +02:00
Thibault "bui" Koechlin
0ad6165ed2
fix release drafter + readme + remove dead readme for acquis (#933) 2021-09-03 09:07:24 +02:00
Manuel Sabban
d7d591ff84
update to use cdn for hub (#920) 
* update to use cdn for hub
* add cdn for version
* fix unit tests accodingly with new cdn

Co-authored-by: sabban <15465465+sabban@users.noreply.github.com>
 2021-09-02 15:17:37 +02:00
Thibault "bui" Koechlin
bed90a832e
fix #919 : display error message (#929) 
* fix #919

* fix tests
 2021-09-02 12:46:32 +02:00
Thibault "bui" Koechlin
589cb72d41
enforce a bit more parsing for resillience (#928) 2021-09-02 12:34:20 +02:00
Shivam Sandbhor
b40fd36607
Add plugin interface code in protobufs package (#921) 
Signed-off-by: Shivam Sandbhor <shivam.sandbhor@gmail.com>
 2021-08-31 14:40:17 +02:00
Thibault "bui" Koechlin
68c11dd827
don't try to send/don't notify if plugin chan is nil (#923) 2021-08-31 14:39:32 +02:00
blotus
b5d0d56a11
add support for --since in journalctl DSN (#917) 2021-08-31 12:40:22 +02:00
ThinkChaos
448a227079
Minor changes to specific logs (#900) 
- Minor changes to specific logs
- Fix LAPI to not push signals to CAPI when disabled #907
 2021-08-25 18:30:05 +02:00
Thibault "bui" Koechlin
c188d401a3
Improve CAPI pull management (#871) 
* prepare for new consensus : thousands of ips

Co-authored-by: Sebastien Blot <sebastien@crowdsec.net>
 2021-08-25 11:45:29 +02:00
Thibault "bui" Koechlin
950759f6d6
Output plugins (#878) 
* Add plugin system for notifications (#857)
 2021-08-25 11:43:29 +02:00
Manuel Sabban
4dbbd4b3c4
Download datafile (#895) 
* add the ability to download datafile on cscli hub upgrade on files are missing
* fix stuff + lint
* fix error management

Co-authored-by: sabban <15465465+sabban@users.noreply.github.com>
 2021-08-19 09:08:20 +02:00
Shivam Sandbhor
f64f20fd53
Document scope parameter for stream API (#897) 
Signed-off-by: Shivam Sandbhor <shivam@crowdsec.net>
 2021-08-18 16:05:56 +02:00
Nanik
b0746fbc4d
fix: add /health endpoint (#881) 
* fix: add /health endpoint
 2021-08-18 09:06:01 +02:00
Thibault "bui" Koechlin
05ac3ca402
if profile is in debug, log debug even if it matched the profile (#894) 2021-08-17 16:50:16 +02:00
Thibault "bui" Koechlin
25ed1c265d
fix #885 : remove dead dependencies for plugin (#891) 2021-08-17 10:32:15 +02:00
Thibault "bui" Koechlin
fc7369c4ea
Fix big serialized entries (#877) 
* bump serialized to 8k

* handle oversized serialized entry : progressively strip its size down
 2021-08-03 15:46:10 +02:00
Thibault "bui" Koechlin
01028d0a09
Goroutine leak hunt (#874) 
* close the writers of gin loggers + kill the tomb of httpServer

* body close defer
 2021-07-30 11:41:17 +02:00
blotus
cedfca07c2
don't wait for acquis tomb if we have no sources (#868) 2021-07-28 08:58:44 +02:00
Thibault "bui" Koechlin
b6ee006078
ensure decisions from CAPI have proper case (#848) 2021-07-02 11:23:46 +02:00
Thibault "bui" Koechlin
033c8e17e8
fix #842 #837 (#845) 
* fix #842 and move preflight checks tgth

* handle new container name

Co-authored-by: AlteredCoder <AlteredCoder>
 2021-07-01 18:15:22 +02:00
blotus
3994aec7fe
add console enroll command to cscli (#828) 2021-06-28 17:34:19 +02:00
Thibault "bui" Koechlin
7f0cac8ee6
add support for 'expression' (fix #822) in grok patterns (#830) 
* add support for 'expression' (fix #822) in grok patterns

* add tests
 2021-06-21 09:07:33 +02:00
Thibault "bui" Koechlin
ce6a61df1c
Refactor Acquisition Interface (#773) 
* Add new acquisition interface + new modules (cloudwatch, syslog)

Co-authored-by: Sebastien Blot <sebastien@crowdsec.net>
 2021-06-11 09:53:53 +02:00
Thibault "bui" Koechlin
71c1d9431f
fix #823 : lower JsonExtract debug (#824) 
* lower key not found log level, fix #823
 2021-06-02 14:27:34 +02:00
Shivam Sandbhor
f25d02a7c8
Allow bouncers to filter decisions by scope (#817) 
Signed-off-by: Shivam Sandbhor <shivam@crowdsec.net>
 2021-05-31 15:07:09 +02:00
Thibault "bui" Koechlin
bf6b791420
fix #781 - avoid unconsistent body : do not send NbDeleted on error (#812) 2021-05-28 11:17:30 +02:00
blotus
c1c76645a7
improve emoji for local configuration when listing (#811) 2021-05-28 11:11:53 +02:00
svesve
6693bff2f5
Add postgres sslmode option (#772) 
Co-authored-by: aleksandr.drozdin <aleksandr.drozdin@karuna.group>
 2021-05-19 17:03:23 +02:00
he2ss
eb0bd70046
fix #787 : load simulation config at startup (#793) 
* fix #787 : load simulation config at startup
 2021-05-17 11:54:28 +02:00
Thibault "bui" Koechlin
f881510f79
delete orphan nodes (fix #778) (#794) 
* delete orphan nodes (for #778 and partially #781)

* and do it as well for decisions
 2021-05-17 11:45:01 +02:00
AlteredCoder
fd830b4293
Fix some bugs (#788) 
* fix config restore

* fix panic on middleware

Co-authored-by: AlteredCoder <AlteredCoder>
 2021-05-07 18:40:01 +02:00
AlteredCoder
a19f13ab45
fix cscli alerts delete -all (#769) 
Co-authored-by: AlteredCoder <AlteredCoder>
 2021-04-27 11:59:18 +02:00
Thibault "bui" Koechlin
b0d4744b15
add System to cwversion to know platform, add it in UA (#763) 2021-04-23 15:23:46 +02:00
registergoofy
7e9ce901a4
add TimeNow in the exprlib helpers (#756) 
* add TimeNow in the exprlib helpers
* add a default date when none is detected: when no date is recognised by ParseDate, then use time.Now()
 2021-04-16 19:13:48 +02:00
Lars Lehtonen
d86ba98cff
pkg/apiserver: fix dropped error (#700) 
* pkg/apiserver: fix dropped error

* pkg/apiserver: remove unused Context from APIServer{}
 2021-04-07 14:51:00 +02:00
Thibault "bui" Koechlin
4bb34d8e77
fix #723 : intercept http2 stream closed errors (#724) 
* fix #723 : intercept http2 stream closed errors

* factorize the 'dump stacktrace' code
 2021-04-07 14:31:03 +02:00
Thibault "bui" Koechlin
cd06929e75
honor log levels for api : don't log access logs if level is warn/err (#732) 
* honor log levels for api : don't log access logs if level is warn/err

* add basic test for logging of api server
 2021-04-07 11:39:24 +02:00
Thibault "bui" Koechlin
20ef67a699
cscli hub mgmt improvements (#710) 
* avoid this confusing behaviour where 'cscli parsers/scenarios/... upgrade' won't tell a thing if no arguments are given (and won't do anything neither)

* avoid repeating warnings about available update to the user
 2021-03-29 10:33:23 +02:00
AlteredCoder
1e899c2211
Refactor configuration management (#698) 2021-03-24 18:16:17 +01:00
Thibault "bui" Koechlin
6d28599efa
Ensure LAPI logs respect log_media (#707) 
* if log_media is set to file, don't try to log to stdout

* use the log media no matter what
 2021-03-22 17:46:55 +01:00
AlteredCoder
4166d9ff48
fix pattern registration (#715) 2021-03-22 17:17:24 +01:00
Thibault "bui" Koechlin
1938e1a62d
clarify doc on onsuccess in parsers + add new date formats for dateparse (#703) 2021-03-19 16:33:10 +01:00
Lars Lehtonen
7f8faa7565
pkg/apiclient: pick up dropped errors (#676) 2021-03-17 12:36:47 +01:00
Thibault "bui" Koechlin
28446b6d29
Ent update : 0.7.0 (#692) 
* up regenerate new schema

* new ent

* update documentation for min required versions

* update documentation
 2021-03-15 18:46:52 +01:00
AlteredCoder
c1abf69979
fix #677 (#684) 2021-03-12 15:10:56 +01:00
AlteredCoder
f2d14c8ca2
update the config.yaml file (#674) 2021-03-11 11:18:09 +01:00
Thibault "bui" Koechlin
0981aa98d8
Pattern syntax consistence (#675) 
* fix #667

* improved error message

* mark the compability, ordered pattern_syntax will be tagged as 'version 2'

* fix tests + add tests to check grok subpattern dependencies
 2021-03-10 18:27:21 +01:00
Lars Lehtonen
7863bad596
pkg/metabase: fix dropped error (#652) 2021-03-10 15:11:56 +01:00
registergoofy
a8b16a66b1
truely don't try to send anything with empty online credentials configuration file (#657) 
* truely don't try to send anything with empty online credentials config file

Co-authored-by: AlteredCoder <AlteredCoder>
 2021-03-02 09:25:12 +01:00
Thibault "bui" Koechlin
70055b3fd6
Doc api + minor api fixes (#654) 
* add doc for API

* link users guide on metabase without docker

* rename doc and swagger
 2021-02-26 17:42:45 +01:00
registergoofy
5b7ac4a473
[Rebased] fix races (#633) 
* get rid of dead code
* have LeakRoutined started in a tomb
* fix race and multiple small issues in the way we handle tombs
* yet another race fix
* another race
* get rid of leaky.KillSwitch for proper tomb use
* fix deadlock
* empty overflow before exiting
* fix an obvious typo
* proper use of waitgroup
* have a smart signalisation for allowing LeakRoutine being killed
* ugly workaround
* fix lint error
* fix compilation
* fix panic
* shorten lock
* up lock both copy
* wait for crowdsec to die
* fix coding style and lint issue
* go mod tidy

Co-authored-by: bui <thibault@crowdsec.net>
 2021-02-25 11:26:46 +01:00
AlteredCoder
8b504e9f67
improve logging in cscli and wizard (#643) 2021-02-25 11:20:36 +01:00
Thibault "bui" Koechlin
a3d00fe130
skip empty lines to avoid issue of #630 (#631) 
* skip empty lines to avoid issue of #630

* add tests on empty lines and comms
 2021-02-25 09:57:24 +01:00
Thibault "bui" Koechlin
22ada59393
Allow for acquisition files to be specified from a directory as well (#619) 
* allow a acquisition_dir in crowdsec's config + change the behaviour of config loading so that it's working with a list instead. keep backward compat with acquisition_path

* remove the default behaviour of 'guessing' acquis path if param isn't present, and error
 2021-02-17 13:55:36 +01:00
Thibault "bui" Koechlin
7d93302e05
add a prometheus_uri option for cscli's config (#625) 
* add a prometheus_uri option for cscli's config, and update documentation

* specify min version
 2021-02-17 13:53:57 +01:00
Thibault "bui" Koechlin
7f40160f6e
only set logfile dir if media is file (#615) 2021-02-11 18:28:01 +01:00
AlteredCoder
dae4458a6f
create crowdsec group for metabase and crowdsec.db (#606) 2021-02-10 09:23:33 +01:00
blotus
260332c726
Add use_forwarded_for_headers configuration option for LAPI (#610) 
* Add use_forwarded_for_headers configuration option for LAPI

* update documentation
 2021-02-09 19:10:14 +01:00
AlteredCoder
22c4962768
don't load lapi creds when running only api (#608) 
Co-authored-by: AlteredCoder <AlteredCoder>
 2021-02-09 17:59:35 +01:00
AlteredCoder
50ee846e87
enable item when they have been added to a collection since previous release (#599) 
Co-authored-by: AlteredCoder <AlteredCoder>
 2021-02-04 17:17:51 +01:00
AlteredCoder
359a9cb8ce
allow environment variable in configuration file (#601) 2021-02-04 17:17:01 +01:00
Thibault "bui" Koechlin
e74f221044
Fix default configurations (#597) 
* fix default perms on SQLite file

* seed the prng securely

* fix defaults to enforce certificates verification

* ensure file is within path

* ensure the directory doesn't exist beforehand

* verify certificate by default

* disable http ip forward headers
 2021-02-02 14:15:13 +01:00
Shivam Sandbhor
36844e50b3
Fix typo in apic.go logs (#592) 2021-01-31 11:42:17 +01:00
Thibault "bui" Koechlin
25562e9575
drop the platform argument to avoid being compatible ONLY with API 1.41 (#582) 2021-01-18 15:25:07 +01:00
AlteredCoder
81e7db71ed
Fix bugs in wizard and cscli (#577) 
* fix id generation bug

* fix api client response

Co-authored-by: AlteredCoder <AlteredCoder>
 2021-01-15 18:14:50 +01:00
AlteredCoder
5544000d38
lapi: fix ipv6 operations (#567) 2021-01-14 16:27:45 +01:00
Thibault "bui" Koechlin
9ec0ea08bb
fix jwt token desynchronization between crowdsec and lapi (#572) 2021-01-14 16:04:10 +01:00
AlteredCoder
c2517e8eb4
fix docker container creation for metabase (#563) 2021-01-08 14:32:29 +01:00
registergoofy
eda9c03c82
jwt token generation improvement (#557) 
* add some warning comment for those who want to choose their secret
* strictly follow the golang doc for using crypto/rand
* fatal if not enough entropy
* add a check when using pre-choosen secret
 2021-01-07 14:24:53 +01:00
Thibault "bui" Koechlin
ad4521f2cc
gin: broken pipe (#538) 
* broken pipe

* don't fail if release isn't here
 2020-12-14 17:48:32 +01:00
registergoofy
13881edbaa
export node logger (#537) 2020-12-14 14:12:22 +01:00
Thibault "bui" Koechlin
f2b30db684
ensure decisions from local or tainted scenarios aren't push, neither are manual decisions (#536) 2020-12-14 12:46:07 +01:00
Thibault "bui" Koechlin
bb679310c7
deal with LAPI down : ensure client will reauthenticate (#527) 
* to avoid keeping apiclient in broken state, reset the token on error
 2020-12-14 11:54:16 +01:00
erenJag
b6d73f48cd
Fix some bugs : update doc, codename and fix wizard (#522) 
* change localhost to 127.0.0.1 + fix uninstall in wizard
* remove beta from repo
 2020-12-08 12:45:36 +01:00
erenJag
339cb6cce7
update prometheus doc (#509) 2020-12-04 11:24:12 +01:00
registergoofy
f411ab4fcd
Fix a crash (#503) 
* fix a crash
 2020-12-03 17:34:57 +01:00
erenJag
fd744408c3
fix cwhub remove func (#501) 2020-12-03 12:05:27 +01:00
erenJag
9d016f262f
fix & improve cscli remove action + improve cscli args vars (#498) 2020-12-02 18:47:17 +01:00
Thibault "bui" Koechlin
2e76097d35
Fix overflows of overflows requesting for different decision scope (#499) 2020-12-02 17:15:48 +01:00
Thibault "bui" Koechlin
b7190c9ecc
improve error management of cscli bouncers add (#495) 2020-12-01 16:16:01 +01:00
erenJag
71325d9134
Improve create alerts input (#493) 
* check decisions start_ip & end_ip fields
 2020-12-01 14:42:53 +01:00