Commit graph

386 commits

Author SHA1 Message Date
Mgrdich
c2be45fd06
[FE] Fix invalid SeekType value for the offset and timestamp fields (#3057)
* SeekType fix the value during the offset and timestamp fields

* BE: thread interruption logging added to Emitters

Co-authored-by: iliax <ikuramshin@provectus.com>
2022-12-19 21:44:14 +04:00
Ilya Kuramshin
f9906b5d30
Brokers API improvements (#2743)
* Brokers API improvements:
1. broker io rates stats added
2. active controller property set to node id
3. minor refactoring
4. FE: Add an indicator for an active broker controller

Co-authored-by: Mgrdich <mgotm13@gmail.com>
Co-authored-by: iliax <ikuramshin@provectus.com>
Co-authored-by: Hrant Abrahamyan <113341474+habrahamyanpro@users.noreply.github.com>
Co-authored-by: Mgrdich <mgotm13@gmail.com>
2022-12-12 14:15:55 +04:00
Narekmat
b81c20d36d
Serde-api release adjustments (#3050)
* kafka-api-serde-api pom fixes

* change pom and add settings

* fix xml

Co-authored-by: iliax <ikuramshin@provectus.com>
Co-authored-by: Roman Zabaluev <rzabaluev@provectus.com>
2022-12-09 17:59:25 +04:00
Ilya Kuramshin
1108c760e5
[BE] Fix a case when controller is not found (#3030)
* ISSUE-3024: kui not working when controller not found
1. ReactiveAdminClient::getClusterVersion now takes into account that controller can not be visible
2. Uniq clientId generation added for consumers and adminClients
3. ReactiveAdminClient::toMonoWithExceptionFilter implementation simplified

* uniq adminclient clientId generation

Co-authored-by: iliax <ikuramshin@provectus.com>
2022-12-07 08:51:22 +04:00
Ilya Kuramshin
ed61852e40
Implement per-cluster polling rates limits (#2981)
* Polling throttling:
1. PollingThrottler class for per-cluster throttle managing
2. UI-publishing throttle for TailingEmitter
3. Elastic scheduler assigned for places where Serde interfaces are used

* int cast fix

* compilation fix

* test threshold fix

Co-authored-by: iliax <ikuramshin@provectus.com>
Co-authored-by: Roman Zabaluev <rzabaluev@provectus.com>
2022-12-07 08:51:11 +04:00
Boris Djurdjevic
b87665a12c
[KSQL] Implement mTLS configuration (#2984)
* feat(ksqldb): Add mTLS support

* doc(ksqldb): Add SSL configuration

* chore(compose): Add ksqldb0 to key material

* feat(ksqldb): Add SSL-enabled demo ksqlDB instance

Co-authored-by: Roman Zabaluev <rzabaluev@provectus.com>
2022-12-07 07:38:13 +04:00
Ilya Kuramshin
43ccca43c2
Fix random topic creation fails (#3004)
ReactiveAdminClient::listOffsetsUnsafe now skipping non-initialized partitions. This prevents UnknownTopicOrPartitionException to be leaked to high-level logic levels (preventing TopicService::loadTopicAfterCreation to do retries)
2022-12-05 17:02:45 +04:00
Ilya Kuramshin
052d8af456
ISSUE-2968: Deleting a config parameter of topic doesn't take effect (#3001)
ReactiveAdminClient::incrementalAlterConfig now replaces all topic configs, not only altering changed
2022-12-01 21:09:23 +04:00
Ilya Kuramshin
c804a6d00b
Serde autoconfiguration fix (#2997)
Co-authored-by: iliax <ikuramshin@provectus.com>
2022-12-01 13:20:48 +04:00
Ilya Kuramshin
216b4acd3d
Serdes improvements and fixes (#2989)
Serdes improvements and fixes:
1. Built-in serdel's auto-configuration separated from explicit configuration (when properties set in serde configuration)
2. Serdes initialization logic extracted into SerdesInitializer.java
3. docker-compose example with serdes configuration added
4. Serialization.md document fixed
2022-11-30 18:43:48 +04:00
Ilya Kuramshin
20ecc74dd9
Data masking (#2850)
Data masking:
1. properties & mapping added to ClustersProperties
2. DataMasking provides function that doing masking for specified topic & target
3. Masking policies implemented: MASK, REMOVE, REPLACE
2022-11-28 12:54:07 +04:00
Ilya Kuramshin
e77b913164
Documentation for pluggable serdes (#2702)
* 1. Minor serde's code improvements
2. Javadoc added to serde-related code

* Serdes configuration docs WIP

* wip

* DeserializeResult.toString impl added

* SchemeRegistry.canDeserialize checks subject existence

* Update Serialization.md

* Update README.md

Co-authored-by: iliax <ikuramshin@provectus.com>
Co-authored-by: Roman Zabaluev <rzabaluev@provectus.com>
2022-11-25 14:35:06 +04:00
Ilya Kuramshin
5900f8e957
Snappy native impl fix (#2940)
* wip

* wip

* wip

* wip

Co-authored-by: iliax <ikuramshin@provectus.com>
2022-11-15 16:05:06 +04:00
Jochen Schalanda
0d0a244915
Centralize dependency versions and bump where necessary (#2929) 2022-11-15 14:22:16 +04:00
Boris Djurdjevic
e26f9787d8
Implement mTLS for KC (#2833)
* feat(kafka-connect): Implement kafka-connect mTLS configuration

* chore(kafka-connect): Use SecuredWebClient

* feat(ssl-demo): Configure SANs and recreate certificates

* feat(ssl-demo): Add docker-compose demo with TLS enabled components
2022-11-15 14:11:57 +04:00
Jochen Schalanda
d6571ec7ae
CI: Scan Java source files with CodeQL (#2898)
* Scan Java code with CodeQL

* REMOVE ME: Trigger workflow run
2022-11-07 19:16:26 +04:00
Jochen Schalanda
3277186fc0
Add support for multiple Protobuf files (#2874)
* Add support for multiple Protobuf files

* Ensure `protobufFiles` list is not empty

* Address review comments
2022-11-07 15:15:40 +04:00
Jochen Schalanda
62f35e3e29
Upgrade to Java 17 (Azul Zulu) (#2879)
* Feature: Upgrade to Java 17 (Azul Zulu)

Java/OpenJDK 13 has reached end of life (and end of security support)
more than 2.5 years ago: https://endoflife.date/java

This change set updates the build, CI pipeline, and Docker image of Kafka UI to Java 17 (LTS).

* Add back maven-compile-plugin to kafka-ui-e2e-checks
2022-11-07 14:57:59 +04:00
Boris Djurdjevic
99e50f8814
Support SSL for schema registry (#2832)
* feat(schema-registry): Implement schema-registry mTLS configuration

* chore(linter): Make checkstyle happy

* feat(schema-registry): Implement schema-registry mTLS configuration

* chore(linter): Make checkstyle happy

* chore(schema-registry): Refactor WebClient creation for PR #2832

Co-authored-by: Ilya Kuramshin <iliax@proton.me>
2022-10-31 18:05:28 +04:00
Ilya Kuramshin
2c74b4e1c2
ISSUE-2680: Smart filters stats propagation to UI (#2771)
ISSUE-2680: Smart filters stats propagation to UI : 
1. DONE event type implemented to flag polling end
2. MessageFilterStats added to emitters context to count filter apply errors
2022-10-31 17:41:11 +04:00
Ilya Kuramshin
aa6c3083c2
ISSUE-2103: Fixing AdminClient timeout on listOffsets (#2781)
ISSUE-2103: Fixing AdminClient timeout on listOffsets when partition's leader in not available:
1. ReactiveAdminClient.listOffsets methods now check if partitions have leader before executing
2. In all places where ReactiveAdminClient.listOffsets methods are used, we specify what to do if partition leader is not available - fail or skip such partitions
3. AdminClient.listOffsets calls splitted with partitionCalls
2022-10-31 16:46:19 +04:00
Roman Zabaluev
cb8049e769
Add a separate endpoint for frontend time format (#2847) 2022-10-31 13:17:23 +04:00
Roman Zabaluev
2f2e19d1bc
Fix topic internal indication (#2846) 2022-10-31 12:46:27 +04:00
Ilya Kuramshin
9a3cbfa14b
ISSUE-2787: Setting default config values properly (#2788)
ISSUE-2787:
1. Getting topic config defaults from server response
2. rm KafkaConstants
3. doc field added to TopicConfigDTO
4. ReactiveAdminClient.SupportedFeature refactor
2022-10-30 21:59:49 +04:00
Egorka Voronyansky
7044e57095
Drop deprecated methods (#2837)
Co-authored-by: vrnsky <vrnsky@protonmail.com>
2022-10-28 12:50:04 +04:00
Ilya Kuramshin
d60808a2f2
Drop deprecated ksql api (#2796)
Co-authored-by: iliax <ikuramshin@provectus.com>
2022-10-23 19:49:11 +04:00
Ilya Kuramshin
4558466ff6
Emitters logic refactoring (#2729)
* Emitters logic refactoring:
1. consumers seeking moved to SeekOperations class
2. offsets info gathering moved to OffsetsInfo class

* wip

* checkstyle fix

* checkstyle fix

* minor improvements

Co-authored-by: iliax <ikuramshin@provectus.com>
2022-10-23 19:47:21 +04:00
Egorka Voronyansky
b0e0da4c0d
Bump version of mockito up to 4.8.1 (#2793)
* Bump version of mockito up to 4.8.1

* Change scope of byte buddy dependency to test

Co-authored-by: vrnsky <vrnsky@protonmail.com>
2022-10-21 09:40:35 +04:00
Ilya Kuramshin
63b6bb3fa1
ISSUE-2202: smart filters headers test added (#2763) 2022-10-17 16:24:14 +04:00
Roman Zabaluev
2b5dd270e2
Implement Connectors sorting / backend (#2291)
Co-authored-by: Artem Kazlanzhy <akazlanzhy@provectus.com>
2022-10-14 13:16:29 +04:00
Ilya Kuramshin
8b91f50af5 Smart filter offset binding (#2732)
* adding "offset" var to smart filter binding

Co-authored-by: iliax <ikuramshin@provectus.com>
2022-10-13 22:41:43 +04:00
Ilya Kuramshin
19e38fb1bf
Msk serverless support (BE) (#2737)
Msk serverless support:
1. ReactiveAdminClient.loadBrokersConfig returns empty map if configs retrieval not supported by kafka backend
2. ReactiveAdminClient.toMone exception unwrapping added
3. FeatureService delete topics enabled set true by default
4. TopicCreationDTO.replicationFactor made optional


Co-authored-by: iliax <ikuramshin@provectus.com>
2022-10-13 22:30:07 +04:00
Marat Gumarov
6dff8f105e
Support prometheus as a metrics interface (#2190)
* Unified internal metric representation (RawMetric)
* Prometheus retriever implemented
* Jmx metrics formatted to jmx-exporter openmentrics format

Co-authored-by: Mohamad Choukair <mchoukair@provectus.com>
Co-authored-by: Roman Zabaluev <rzabaluev@provectus.com>
Co-authored-by: Daniel Ching <110129035+daching-provectus@users.noreply.github.com>
Co-authored-by: Daniel Ching <daching@provectus.com>
Co-authored-by: German Osin <germanosin@Germans-MacBook-Pro.local>
Co-authored-by: iliax <ikuramshin@provectus.com>
Co-authored-by: Ilya Kuramshin <iliax@proton.me>
2022-10-07 14:36:27 +04:00
Ilya Kuramshin
b19b9d82e9
Minor backend fixes (#2696)
Minor backend fixes:
1. InternalTopic.leader fiels filling fix
2. bytesIn/OutPerSec fields set to TopicDTO
3. "subject" property added to SR SchemaDescription
2022-10-03 17:16:55 +04:00
Ilya Kuramshin
2f786c080b
ISSUE-2080: Consumer groups not found for topic (#2628)
* Fixing consumers visibility when offsets not committed
Co-authored-by: iliax <ikuramshin@provectus.com>
2022-09-30 13:21:54 +04:00
Roman Zabaluev
7db55d5acf
Expose build info for actuator (#2583) 2022-09-28 15:56:03 +04:00
Ilya Kuramshin
97f1c639a3
1. fixing infinite recursion with cyclic references (#2640)
2. using record names in unions to make possible to use several record
3. tests refactored

Co-authored-by: iliax <ikuramshin@provectus.com>
2022-09-28 14:08:17 +04:00
Ilya Kuramshin
049b35fc99
Pluggable serde (#2125)
* checkstyle

* node version reverted

* api comments

* unused properties removed

* serde api fix

* renamings

* PR review fixes

* checkstyle fix

* PR fixes

* UUID serde tests added

* UUID serde tests added

* fixed due to review comments

* 1. kafka-ui-serde-api renamed 2. kafka-clients dependency removed from kafka-ui-serde-api module

* dep fix

* Add serde to send message component (#2620)

* Add serde to send message component

* adds serde selects to messages views

Co-authored-by: iliax <ikuramshin@provectus.com>
Co-authored-by: Roman Zabaluev <rzabaluev@provectus.com>
Co-authored-by: Kristina Kiparoidze <kkiparoidze@provectus.com>
Co-authored-by: Oleg Shur <workshur@gmail.com>
2022-09-27 19:52:59 +04:00
Ilya Kuramshin
d673ea63a1
Fixing infinite polls in ForwardRecordEmitter for topic where no record found for last offset (compact topic) (#2627)
Co-authored-by: iliax <ikuramshin@provectus.com>
2022-09-22 09:19:00 +04:00
Roman Zabaluev
596f4233fc
Disable ldap beans auto configuration (#2605) 2022-09-19 16:32:47 +04:00
Roman Zabaluev
15d09751a6
Fix flaky backend test (#2592) 2022-09-16 05:37:28 +04:00
Daniel Ching
71aa44a3d3
Get rid of dockerfile-maven-plugin (#2594)
* Resolve #1932

Replacing spotify maven docker plugin to use fabric8 docker maven plugin instead

* Remove unused arg

Co-authored-by: Roman Zabaluev <rzabaluev@provectus.com>
2022-09-16 05:30:55 +04:00
Roman Zabaluev
5db2c17994
Cognito logout + revamp (#1063)
* application yml example

* Update SSO guide

* Add cognito logout handler

* Fix annotations

* impl a separate config file for cognito

* cleanup

* Rollback auth.type change

* Add compose example

* Auth revamp

* Review suggestions

* Use configurationProperties, rename urls to uris
2022-09-15 05:36:21 +04:00
Daniel Ching
fc946a1dd1
Handle error message for schemas (#2520)
Co-authored-by: Roman Zabaluev <rzabaluev@provectus.com>
2022-09-15 05:00:46 +04:00
Jibin Philipose
9f32abcd09
New documentation for building and running without docker (#2574)
* New documentation for building and running without docker and some doc fixes and additions.

* Improvements in docs

* spell fix

* improvements in docs

* minor modification

* improvements
2022-09-15 04:38:05 +04:00
Roman Zabaluev
c8306f5970
Fix consumers sorting (#2447)
* Fix consumers sorting

* Fix tests

* Review fix

* Review suggestions
2022-09-02 16:27:33 +04:00
Sergiy-provectus
085dfec389
Make message timestamp format configurable (backend) (#2277)
* [2163] Make message timestamp format configurable

* [2163] Make message timestamp format configurable - PR comments update

Co-authored-by: Roman Zabaluev <rzabaluev@provectus.com>
2022-08-30 16:28:59 +04:00
Roman Zabaluev
9e1f8d773f
Add a failover recovery for empty cluster versions (#2473)
* Add a failover recovery for empty cluster versions

* Review suggestions
2022-08-30 12:06:05 +04:00
Shubham Jain
01127d8f10
Fix SR subject encoding (#2483) (#2471) (#1995)
* Fix subject encoding frontend (#1995)

* Add tests for schemas with non ascii chars #1995

* Backend: Fix URL encoding while getting schemas (#2471)

Add Test for subject name with non ascii chars
Closes #1995

Co-authored-by: Roman Zabaluev <rzabaluev@provectus.com>

Co-authored-by: Roman Zabaluev <rzabaluev@provectus.com>
2022-08-26 20:51:00 +04:00
Roman Zabaluev
5ac52efb7a
Fix transient jre CVEs (#2480) 2022-08-22 21:04:22 +04:00