Commit graph

756 commits

Author SHA1 Message Date
Thibault "bui" Koechlin
0981aa98d8
Pattern syntax consistence (#675)
* fix #667

* improved error message

* mark the compability, ordered pattern_syntax will be tagged as 'version 2'

* fix tests + add tests to check grok subpattern dependencies
2021-03-10 18:27:21 +01:00
Lars Lehtonen
7863bad596
pkg/metabase: fix dropped error (#652) 2021-03-10 15:11:56 +01:00
registergoofy
a8b16a66b1
truely don't try to send anything with empty online credentials configuration file (#657)
* truely don't try to send anything with empty online credentials config file

Co-authored-by: AlteredCoder <AlteredCoder>
2021-03-02 09:25:12 +01:00
Thibault "bui" Koechlin
70055b3fd6
Doc api + minor api fixes (#654)
* add doc for API

* link users guide on metabase without docker

* rename doc and swagger
2021-02-26 17:42:45 +01:00
registergoofy
5b7ac4a473
[Rebased] fix races (#633)
* get rid of dead code
* have LeakRoutined started in a tomb
* fix race and multiple small issues in the way we handle tombs
* yet another race fix
* another race
* get rid of leaky.KillSwitch for proper tomb use
* fix deadlock
* empty overflow before exiting
* fix an obvious typo
* proper use of waitgroup
* have a smart signalisation for allowing LeakRoutine being killed
* ugly workaround
* fix lint error
* fix compilation
* fix panic
* shorten lock
* up lock both copy
* wait for crowdsec to die
* fix coding style and lint issue
* go mod tidy

Co-authored-by: bui <thibault@crowdsec.net>
2021-02-25 11:26:46 +01:00
AlteredCoder
8b504e9f67
improve logging in cscli and wizard (#643) 2021-02-25 11:20:36 +01:00
Thibault "bui" Koechlin
a3d00fe130
skip empty lines to avoid issue of #630 (#631)
* skip empty lines to avoid issue of #630

* add tests on empty lines and comms
2021-02-25 09:57:24 +01:00
Thibault "bui" Koechlin
22ada59393
Allow for acquisition files to be specified from a directory as well (#619)
* allow a acquisition_dir in crowdsec's config + change the behaviour of config loading so that it's working with a list instead. keep backward compat with acquisition_path

* remove the default behaviour of 'guessing' acquis path if param isn't present, and error
2021-02-17 13:55:36 +01:00
Thibault "bui" Koechlin
7d93302e05
add a prometheus_uri option for cscli's config (#625)
* add a prometheus_uri option for cscli's config, and update documentation

* specify min version
2021-02-17 13:53:57 +01:00
Thibault "bui" Koechlin
7f40160f6e
only set logfile dir if media is file (#615) 2021-02-11 18:28:01 +01:00
AlteredCoder
dae4458a6f
create crowdsec group for metabase and crowdsec.db (#606) 2021-02-10 09:23:33 +01:00
blotus
260332c726
Add use_forwarded_for_headers configuration option for LAPI (#610)
* Add use_forwarded_for_headers configuration option for LAPI

* update documentation
2021-02-09 19:10:14 +01:00
AlteredCoder
22c4962768
don't load lapi creds when running only api (#608)
Co-authored-by: AlteredCoder <AlteredCoder>
2021-02-09 17:59:35 +01:00
AlteredCoder
50ee846e87
enable item when they have been added to a collection since previous release (#599)
Co-authored-by: AlteredCoder <AlteredCoder>
2021-02-04 17:17:51 +01:00
AlteredCoder
359a9cb8ce
allow environment variable in configuration file (#601) 2021-02-04 17:17:01 +01:00
Thibault "bui" Koechlin
e74f221044
Fix default configurations (#597)
* fix default perms on SQLite file

* seed the prng securely

* fix defaults to enforce certificates verification

* ensure file is within path

* ensure the directory doesn't exist beforehand

* verify certificate by default

* disable http ip forward headers
2021-02-02 14:15:13 +01:00
Shivam Sandbhor
36844e50b3
Fix typo in apic.go logs (#592) 2021-01-31 11:42:17 +01:00
Thibault "bui" Koechlin
25562e9575
drop the platform argument to avoid being compatible ONLY with API 1.41 (#582) 2021-01-18 15:25:07 +01:00
AlteredCoder
81e7db71ed
Fix bugs in wizard and cscli (#577)
* fix id generation bug

* fix api client response

Co-authored-by: AlteredCoder <AlteredCoder>
2021-01-15 18:14:50 +01:00
AlteredCoder
5544000d38
lapi: fix ipv6 operations (#567) 2021-01-14 16:27:45 +01:00
Thibault "bui" Koechlin
9ec0ea08bb
fix jwt token desynchronization between crowdsec and lapi (#572) 2021-01-14 16:04:10 +01:00
AlteredCoder
c2517e8eb4
fix docker container creation for metabase (#563) 2021-01-08 14:32:29 +01:00
registergoofy
eda9c03c82
jwt token generation improvement (#557)
* add some warning comment for those who want to choose their secret
* strictly follow the golang doc for using crypto/rand
* fatal if not enough entropy
* add a check when using pre-choosen secret
2021-01-07 14:24:53 +01:00
Thibault "bui" Koechlin
ad4521f2cc
gin: broken pipe (#538)
* broken pipe

* don't fail if release isn't here
2020-12-14 17:48:32 +01:00
registergoofy
13881edbaa
export node logger (#537) 2020-12-14 14:12:22 +01:00
Thibault "bui" Koechlin
f2b30db684
ensure decisions from local or tainted scenarios aren't push, neither are manual decisions (#536) 2020-12-14 12:46:07 +01:00
Thibault "bui" Koechlin
bb679310c7
deal with LAPI down : ensure client will reauthenticate (#527)
* to avoid keeping apiclient in broken state, reset the token on error
2020-12-14 11:54:16 +01:00
erenJag
b6d73f48cd
Fix some bugs : update doc, codename and fix wizard (#522)
* change localhost to 127.0.0.1 + fix uninstall in wizard
* remove beta from repo
2020-12-08 12:45:36 +01:00
erenJag
339cb6cce7
update prometheus doc (#509) 2020-12-04 11:24:12 +01:00
registergoofy
f411ab4fcd
Fix a crash (#503)
* fix a crash
2020-12-03 17:34:57 +01:00
erenJag
fd744408c3
fix cwhub remove func (#501) 2020-12-03 12:05:27 +01:00
erenJag
9d016f262f
fix & improve cscli remove action + improve cscli args vars (#498) 2020-12-02 18:47:17 +01:00
Thibault "bui" Koechlin
2e76097d35
Fix overflows of overflows requesting for different decision scope (#499) 2020-12-02 17:15:48 +01:00
Thibault "bui" Koechlin
b7190c9ecc
improve error management of cscli bouncers add (#495) 2020-12-01 16:16:01 +01:00
erenJag
71325d9134
Improve create alerts input (#493)
* check decisions start_ip & end_ip fields
2020-12-01 14:42:53 +01:00
erenJag
a16fb1475d
add info message when there is no hub index (#492) 2020-12-01 12:33:14 +01:00
AlteredCoder
c6eb2afa20
push to CAPI in go routine (#489)
Co-authored-by: AlteredCoder <AlteredCoder>
2020-11-30 17:46:02 +01:00
Thibault "bui" Koechlin
71ac0d2fce
Apiclient tests (#484)
Co-authored-by: AlteredCoder
Co-authored-by: erenJag
2020-11-30 16:15:07 +01:00
Thibault "bui" Koechlin
dbb420f79e
local api (#482)
Co-authored-by: AlteredCoder
Co-authored-by: erenJag
2020-11-30 10:37:17 +01:00
registergoofy
f0ea8312db
set default hub branch to master in cscli (#279) 2020-10-01 15:02:53 +02:00
registergoofy
9b97633043
fix ban flush (#277)
* fix ban flush by soft-deleting entries in database
* fix unit tests accordingly
2020-10-01 08:26:59 +02:00
registergoofy
c6aab9893a
add randomness to machine-id when registering. (#261)
* add randomness to machine-id when registering.

* add some regexp check for machine_id

* typo fix

* fix cwapi unit tests
2020-09-29 13:17:33 +02:00
AlteredCoder
b7286d6a85
make cscli use crowdsec version for hub (#194) 2020-09-01 14:32:45 +02:00
AlteredCoder
b81c735d81 remove debug while fixing test 2020-08-30 16:04:51 +02:00
AlteredCoder
7f45c43eb1 change github workflow to get latest release 2020-08-28 16:37:35 +02:00
Thibault "bui" Koechlin
b2ef6a555c
add support for 'prometheus_mode' configuration directive that can be set to 'aggregation' to limit the cardinality of prometheus metrics (#192) 2020-08-24 11:51:50 +02:00
erenJag
6624fce66a
fix tests (#191)
* fix leakybucket test
2020-08-24 10:25:52 +02:00
AlteredCoder
a6fabcf481 fix CI in leakybuckets 2020-08-23 23:42:24 +02:00
AlteredCoder
3801dcc277 fix parser tests 2020-08-23 23:34:12 +02:00
erenJag
25dfcebf4c
improve fileInit func by not loading unspecified data (#189)
* improve fileInit func by not loading unsepcified data
2020-08-21 14:20:44 +02:00
Thibault "bui" Koechlin
5595070e67
handle multiple plugins for now (append results rather than taking the 'last one' (#185) 2020-08-21 12:28:59 +02:00
Thibault "bui" Koechlin
1956f52be5
add a warning when a grok pattern ends with \n (#183) 2020-08-20 15:07:50 +02:00
Thibault "bui" Koechlin
742435f178
Acquisition extra tests (#188)
* acquisition testing
2020-08-20 13:55:52 +02:00
Thibault "bui" Koechlin
1398a74c6d
add extra tests for exprlib visitors (#187)
* add extra tests for exprlib visitors
2020-08-20 11:53:47 +02:00
Thibault "bui" Koechlin
ceb69f0cef
documentation improvment (#182) 2020-08-07 09:40:43 +02:00
AlteredCoder
747065229e
fix expr debugger (#178)
Co-authored-by: AlteredCoder <AlteredCoder>
2020-08-05 16:27:16 +02:00
Thibault "bui" Koechlin
7525f11975
improve tests in pkg/leakybuckets (#171) 2020-08-05 11:39:54 +02:00
Thibault "bui" Koechlin
8128dcf61b
add more tests for pkg/parser (config loading) (#172) 2020-08-05 11:20:03 +02:00
AlteredCoder
b10c7e9bef
fix expr debugger when no variable is present (#174)
Co-authored-by: AlteredCoder <AlteredCoder>
2020-08-04 16:44:03 +02:00
AlteredCoder
d8f0f5a3a9
Add debug expr (#168)
* debug expr in node.go
* update documentation

Co-authored-by: AlteredCoder <AlteredCoder>
Co-authored-by: Thibault bui Koechlin <thibault@crowdsec.net>
2020-08-03 12:21:15 +02:00
Thibault "bui" Koechlin
085dcc5eb6
add a basic functional tests in the CI (#169) 2020-07-31 16:16:23 +02:00
AlteredCoder
b7096be6e6
fix message to verbose in expr helper (#166)
Co-authored-by: AlteredCoder <AlteredCoder>
2020-07-30 17:12:47 +02:00
Thibault "bui" Koechlin
0356f8404b
add tests for pkg/database (#151) 2020-07-30 15:58:06 +02:00
AlteredCoder
d23512e9c6
improve logging (#164)
Co-authored-by: AlteredCoder <AlteredCoder>
2020-07-30 15:30:02 +02:00
Thibault "bui" Koechlin
b19046939c
sqlite set busy timeout to 10s rather than 1s (#162) 2020-07-30 11:39:06 +02:00
Thibault "bui" Koechlin
d2ffb190f9
add a default configuration file if none is specified (#159) 2020-07-29 16:58:25 +02:00
AlteredCoder
5e561e30bd
fix cwapi bug with new sling usage (#157)
* fix sling usage

Co-authored-by: AlteredCoder <AlteredCoder>
2020-07-29 15:15:33 +02:00
erenJag
89c8d1a527
rename metrics and update metrics helps (#152)
* rename metrics and update metrics helps

* add meta info about crowdsec

Co-authored-by: erenJag <erenJag>
2020-07-29 15:03:15 +02:00
Thibault "bui" Koechlin
acc0960c17
match the hashes version in reverse order (#156) 2020-07-29 15:02:52 +02:00
AlteredCoder
794d3221d0
add tests in pkg/csconfig and improve pkg/exprhelpers tests (#150)
* add tests for csconfig & improve exprhelpers tests
2020-07-28 15:38:48 +02:00
AlteredCoder
e6cb7f3a79 fix 2020-07-27 18:52:00 +02:00
AlteredCoder
55d5b6842c fix 2020-07-27 18:35:31 +02:00
Thibault "bui" Koechlin
056c7801c6
add unitest in cwhub package (#144) 2020-07-27 13:47:32 +02:00
Thibault "bui" Koechlin
151af2d0d8
No sql transaction + proper time-machine wait (#148) 2020-07-27 13:42:30 +02:00
AlteredCoder
851ad300cb
Add unitest in pkg/acquisition and pkg/cwapi (#145)
* ci for acquisition and cwapi

* update README


Co-authored-by: AlteredCoder <AlteredCoder>
2020-07-27 12:18:55 +02:00
Thibault "bui" Koechlin
a104e6d053
fix ban deduplication (#143) 2020-07-21 10:48:06 +02:00
AlteredCoder
40b7bfaf69
fix (#142)
Co-authored-by: AlteredCoder <AlteredCoder>
2020-07-20 17:44:45 +02:00
Thibault "bui" Koechlin
66161bc8ae
fix auto-delete (#140) 2020-07-20 11:52:00 +02:00
AlteredCoder
0864f13cb8
fix post merge (#138)
* fix post merge (#138)
2020-07-17 15:26:46 +02:00
Thibault "bui" Koechlin
177480cff7
updated mysql plugin support (#135)
* add support for plugin, support mysql & so on

* fix queries

Co-authored-by: erenJag <erenJag>
Co-authored-by: AlteredCoder <AlteredCoder>
2020-07-16 16:05:03 +02:00
Thibault "bui" Koechlin
7fe6741df3
Simulation support (#136)
* support simulation mode
2020-07-16 15:59:09 +02:00
AlteredCoder
87a90583fe
Fix#80 (#133)
* fix #80 

Co-authored-by: AlteredCoder <AlteredCoder>
Co-authored-by: Thibault bui Koechlin <thibault@crowdsec.net>
2020-07-16 15:48:46 +02:00
Thibault "bui" Koechlin
bc2566f3e5
support multiple args for all cscli upgrade/install/remove commands (#132) 2020-07-10 10:43:22 +02:00
erenJag
44304a30e7
fix #124 (#127)
* fix #124
2020-07-09 12:41:18 +02:00
Thibault "bui" Koechlin
a0c1ca49d0
Doc : fix whitelists documentation + document data for parsers/scenarios + document expr helpers + link taxonomy (#126) 2020-07-08 10:58:20 +02:00
AlteredCoder
d0ac43b00f
Allow comments with # in expr wordlists
Co-authored-by: AlteredCoder <AlteredCoder>
2020-07-07 16:26:00 +02:00
Thibault "bui" Koechlin
a62bac0ca0
verbosity (#121) 2020-07-03 18:26:23 +02:00
Thibault "bui" Koechlin
7691e5b663
re-enable postoverflows (#117)
* re-enable postoverflows

* debug

* yoloooo

* remove debug

* remove error print

* fix test

* fix leakybucket test

* fix

Co-authored-by: AlteredCoder <AlteredCoder>
2020-07-02 17:56:39 +02:00
AlteredCoder
eef1847873
add whitelisted flag in signal occurence (#114) 2020-07-02 11:44:27 +02:00
erenJag
f6826c7e47
add expr helper to check if IP is in ipRange (#113)
* add expr helper to check if IP is in ipRange

* update helper name

Co-authored-by: erenJag <erenJag>
2020-07-02 11:09:40 +02:00
Thibault "bui" Koechlin
b9ae94b874
Sqlite : Support automatic db flushing (#91)
* add support for sqlite retention : max_records, max_records_age

* reduce verbosity of cwhub
2020-07-01 17:04:29 +02:00
Thibault "bui" Koechlin
e4993996a5
exclude fields so that they are not serialized when we're rewritting api.yaml file (#103) 2020-06-29 17:47:57 +02:00
Thibault "bui" Koechlin
02b2193d64
fix collection dependencies : install subparts of collection, even if the collec file itself exists (#101) 2020-06-29 17:31:37 +02:00
Thibault "bui" Koechlin
652b54ee81
SignalOccurence can't lead to BanApplication if there is no source in the Event (#96) 2020-06-29 13:22:56 +02:00
AlteredCoder
2e30793188
Allow CrowdSec to start if geoip data are not downloaded (#92)
* Allow CrowdSec to start if `geoip` data are not downloaded
2020-06-25 12:36:01 +02:00
Thibault "bui" Koechlin
5446857377
Add crowdsec reload + cscli metrics minor improvements (#79) 2020-06-19 13:57:44 +02:00
Thibault "bui" Koechlin
8651a1aefc
skip ~ files #69 (#71)
* skip ~ files

* only keep .yaml et .yml files
2020-06-12 17:55:35 +02:00
Thibault "bui" Koechlin
26f77bed88
CI: provide makefile for CI and fixe types.Event (#70) 2020-06-11 15:03:43 +02:00
Thibault "bui" Koechlin
64c5fa7360
CI: add a CI to test parsers (#67) 2020-06-10 12:14:27 +02:00
AlteredCoder
fc05a49cc3
Fix API push (#62)
* add debug

* debug

* remove debug

* remove typo

* fix linter

Co-authored-by: AlteredCoder <AlteredCoder>
2020-06-03 14:24:07 +02:00
Thibault "bui" Koechlin
f9a4ae2b3f
fix #60 - sqlite lock + badges (#61)
* don't fatal is DB is locked + Fix #60
2020-06-03 14:18:00 +02:00
AlteredCoder
08c0167f15
fix debug (#58)
Co-authored-by: AlteredCoder <AlteredCoder>
2020-06-02 16:17:30 +02:00
AlteredCoder
72d1fe4c3b
fix (#51)
Co-authored-by: AlteredCoder <AlteredCoder>
2020-06-01 18:32:01 +02:00
Thibault "bui" Koechlin
430b7cd90d
add a 'Upper' expr helper (#50)
* add a 'Upper' expr helper

* remove redundant startsWith and endsWith
2020-06-01 16:12:48 +02:00
Thibault "bui" Koechlin
f2e38b0d28
fix #47 #35 (#49) 2020-06-01 12:56:32 +02:00
AlteredCoder
82d6e6938a
Fix expr helpers "StartsWith" and "EndsWith" (#48)
* fix

* fix typo

Co-authored-by: AlteredCoder <AlteredCoder>
2020-05-29 17:25:09 +02:00
Thibault "bui" Koechlin
4e8aa19c09
bump version 0.0.4 + typo (#46)
* fix typo in history rewrite
* and bump version
2020-05-28 12:03:29 +02:00
AlteredCoder
ea4e4153af fix review 2020-05-28 11:32:00 +02:00
AlteredCoder
937bd20c18 fix 2020-05-28 11:32:00 +02:00
AlteredCoder
083b9897d8 fix 2020-05-28 11:32:00 +02:00
Thibault "bui" Koechlin
cda7beddbf add support in cscli to switch branches of hub (#43) 2020-05-28 11:32:00 +02:00
Thibault bui Koechlin
b51d666dcb type 2020-05-28 11:32:00 +02:00
Thibault bui Koechlin
7563975eef only set if non-nil, avoid crash on unconfigured logger 2020-05-28 11:32:00 +02:00
Thibault bui Koechlin
80387fe66e simplify the code 2020-05-28 11:32:00 +02:00
Thibault bui Koechlin
87ff0883cf lower verbosity for this, give context to that 2020-05-28 11:32:00 +02:00
Thibault bui Koechlin
372d8680c3 unify loggers and improve the log message about groks when they're not called by name 2020-05-28 11:31:55 +02:00
Thibault bui Koechlin
a06f8373ae move the setLogger config and ConfigureLogger to be part of types for reuse accross modules 2020-05-28 11:28:49 +02:00
Thibault bui Koechlin
3fa5122db7 only test directories 2020-05-28 11:28:49 +02:00
AlteredCoder
8d43abbf4c add context in log 2020-05-28 11:28:49 +02:00
AlteredCoder
ed919a55be fiw review 2020-05-28 11:28:49 +02:00
AlteredCoder
fd6ecd25df fix review 2020-05-28 11:28:49 +02:00
Thibault bui Koechlin
7657a0cc37 fix non unique grok name 2020-05-28 11:28:49 +02:00
Thibault bui Koechlin
a199cd8b36 unify paths 2020-05-28 11:28:49 +02:00
Thibault bui Koechlin
7557ce8156 add 'in File(...)' tests for parsers and leakybuckets 2020-05-28 11:28:49 +02:00
Thibault bui Koechlin
6cb5ec0460 add a parser unit test using the 'in File(...)' construct 2020-05-28 11:28:49 +02:00
AlteredCoder
160478b419 add test for exprhelpers 2020-05-28 11:28:49 +02:00
Thibault "bui" Koechlin
1abdfc9b10 make message better 2020-05-28 11:28:49 +02:00
erenJag
c020ff8c64
Merge pull request #22 from crowdsecurity/add_expr_helpers
add new expr helpers
2020-05-25 12:12:49 +02:00
Thibault "bui" Koechlin
d9a37683e7
Merge pull request #27 from crowdsecurity/add_custom_error
Adding custom error.
2020-05-25 12:12:03 +02:00
FaricaUnknown
060a595244
Merge pull request #25 from crowdsecurity/json_extractor
add json support via expr helpers
2020-05-25 11:47:47 +02:00
Thibault bui Koechlin
ed24638200 fix remark 2020-05-25 11:37:52 +02:00
Thibault bui Koechlin
aad2e1421e fix printf 2020-05-25 11:35:32 +02:00
erenJag
34a6156097 add custom error to be able to ignore or fatalF when reference is missing in collection 2020-05-25 10:55:05 +02:00
AlteredCoder
1b54bbb909 add data folder parameter to hubMgmt 2020-05-24 19:53:05 +02:00
AlteredCoder
1f91bd8af0 make var public 2020-05-24 19:17:03 +02:00
AlteredCoder
6757fa3cee merge 2020-05-24 18:15:59 +02:00
Thibault bui Koechlin
1970a90813 debug is false in benchmark 2020-05-24 12:59:26 +02:00
Thibault bui Koechlin
88c72340e3 unique pattern names 2020-05-24 12:44:33 +02:00
Thibault bui Koechlin
aa9badc70c reset stages 2020-05-24 12:41:45 +02:00
Thibault bui Koechlin
d601e21afb working tests for json 2020-05-23 13:22:43 +02:00
AlteredCoder
74dd723ebf fix 2020-05-22 18:24:54 +02:00
AlteredCoder
f8821b8982 fix dev config 2020-05-22 18:22:43 +02:00
Thibault bui Koechlin
32f32b41c7 add json support via expr helpers 2020-05-22 18:12:33 +02:00
AlteredCoder
019c8d3e18 fix package config name 2020-05-22 17:45:08 +02:00
AlteredCoder
62daa98bf3 add new helpers 2020-05-22 13:55:48 +02:00
Thibault bui Koechlin
ada5d3fcfe explain linter bypasses 2020-05-22 10:12:35 +02:00
Thibault bui Koechlin
e86b163ba5 linter fixes 2020-05-20 18:05:05 +02:00
Thibault bui Koechlin
e643bb5b31 linter fixes, inefficient assignments 2020-05-20 17:50:56 +02:00
Thibault bui Koechlin
e6cad40ac4 more linting fixes 2020-05-20 11:26:21 +02:00
Thibault bui Koechlin
fe68914628 more linter fixes (simplicity mostly) 2020-05-20 11:00:25 +02:00
Thibault bui Koechlin
db9e1e280d fix linter warnings : dead code, simplification 2020-05-20 10:49:17 +02:00
Thibault bui Koechlin
604b97a519 LGMT fixes + avoid capitalizing fmt.Errorf strings 2020-05-19 21:31:06 +02:00
Thibault bui Koechlin
3d2d39e77d up 2020-05-15 12:13:57 +02:00
Thibault bui Koechlin
eb0b49cbbc up 2020-05-15 11:49:17 +02:00
Thibault bui Koechlin
58a9bb0bc0 rename 2020-05-15 11:42:56 +02:00
Thibault bui Koechlin
2016167654 initial import 2020-05-15 11:39:16 +02:00