Commit graph

406 commits

Author SHA1 Message Date
Ilya Kuramshin
f2ec4d76de
Avro (Embedded) serde implementation (#3266)
* Avro (Embedded) serde implementation

---------

Co-authored-by: iliax <ikuramshin@provectus.com>
2023-02-17 16:18:28 +04:00
Michael Kreis
a87b31aca1
BE: Extend testcase with a recursive reference (#3314)
Fixes #2824

Co-authored-by: Michael Kreis <michael.kreis@raiffeisen.ch>
Co-authored-by: Roman Zabaluev <rzabaluev@provectus.com>
2023-02-15 18:13:53 +04:00
Ilya Kuramshin
ebd25c61b1
Using JRE instead of JDK in docker image construction (#3341)
Co-authored-by: iliax <ikuramshin@provectus.com>
2023-02-14 14:13:52 +04:00
Ilya Kuramshin
45a6e73d29
[BE] SR: maxInMemorySize property setting added to SR client (#3310)
Co-authored-by: iliax <ikuramshin@provectus.com>
Co-authored-by: Roman Zabaluev <rzabaluev@provectus.com>
2023-02-09 16:13:04 +04:00
Ilya Kuramshin
d1f04e05bb
[Be] Fix MessagesServiceTest, waiting of kafka write added (#3287)
Co-authored-by: iliax <ikuramshin@provectus.com>
2023-01-31 11:57:21 +04:00
Batuhan Küçükali
1b09251419
[BE] Fix case-sensitive search in broker's topics (#3276)
Co-authored-by: Roman Zabaluev <rzabaluev@provectus.com>
2023-01-31 11:55:21 +04:00
Ilya Kuramshin
43fcf6dce1
[BE] Suppressing failures when describe log dirs not supported (#3262)
* Suppressing failures when describe log dirs not supported
2023-01-26 11:12:59 +04:00
Ilya Kuramshin
ded4e26825
[BE] Chore: ConfigTest stats initialization added (#3260)
Co-authored-by: iliax <ikuramshin@provectus.com>
2023-01-24 12:52:27 +04:00
Ilya Kuramshin
5d31189609
ISSUE-3222: SchemaRegistrySerde checkSchemaExistenceForDeserialize property added (#3235) 2023-01-17 22:38:44 +04:00
Wanis Fahmy
f55376b40b
[BE] Impl basic auth for prometheus (#3201)
* fix(3143): passes prometheus basic auth credentials

Co-authored-by: theurichde <tim.heurich@idealo.de>

* fix(3143): applies review comments

Co-authored-by: theurichde <tim.heurich@idealo.de>

Co-authored-by: theurichde <tim.heurich@idealo.de>
Co-authored-by: Roman Zabaluev <rzabaluev@provectus.com>
2023-01-16 17:53:06 +04:00
Ilya Kuramshin
799c2c455a
Implement SchemaRegistry openapi client (#3123)
Failover clients for SchemaRegistry, Kafka-connect, KSQL:
1. SchemeRegistry openapi client generation
2. WebClient configuration logic moved to WebClientConfigurator class
3. ReactiveFailover utility implemented 
4. Connect, Ksql moved to ReactiveFailover usage
2023-01-16 15:27:55 +04:00
Ilya Kuramshin
7c6d04bca0
Proto to JsonSchema well-known types support (#3088)
1. Add support for predefined (well-known) protobuf types.
2. `preservingProtoFieldNames` mode is enabled for proto messages formatting to fit proto file's message's field names
3. Invalid oneof-processing logic removed
4. Protobuf -> JsonSchema self-reference issue fixed
5. format added to Timestamp protobuf type
2023-01-16 14:49:04 +04:00
Ilya Kuramshin
578468d090
Consumer groups loading performance impr (#3188)
ISSUE-3148, ISSUE-3188: Topic consumers retrieval performance impr
1. batch listConsumerGroupOffsets method implemented
2. describeConsumerGroups parallelized
3. minor improvements in ReactiveAdminClient
4. (not related to perf) ConsumerGroupMapper messagesBehind setting fixed
2023-01-12 23:14:28 +04:00
Mgrdich
9d98927524
[FE] Implement a full support for custom context path (#3106)
* Vite Custom Plugin code initialization , ( not building on js file )

* Fix the lazy loading functionality in vite

* add lazy loading in the Cluster page

* minor comment code cleanup

* Remove un-necessary files for configuration of the vite

* minor naming modifications

* minor code modification to encode a variable in the root
2023-01-03 19:54:42 +04:00
Roman Zabaluev
9b87d3829b
Fix minor RBAC/auth NPEs, init null collections (#3185) 2023-01-03 19:29:33 +04:00
Ilya Kuramshin
eef63466fb
[BE] Update tests for confluent platform 7+ (#3038)
Co-authored-by: iliax <ikuramshin@provectus.com>
2023-01-03 14:52:28 +04:00
Ilya Kuramshin
0cba8240b7
Bug: UnsupportedVersionException: includeClusterAuthorizedOperations (#3157)
* ISSUE-3141: checking kafka version before setting includeAuthorizedOperations for describeCluster api call
2022-12-28 19:49:26 +04:00
Roman Zabaluev
8b126d843e
Fix google oauth with rbac (#3155) 2022-12-28 14:34:35 +04:00
Roman Zabaluev
027d9b4653
Change message timestamp format (#3113)
* Drop experimental endpoints

* [FE] Change message timestamp format (#3119)

* remove dayjs from the code base

* remove dayjs library and clean up the hook codes

* minor obsolete fixtures deletion

* minor provider code modifications

* minor provider code modifications

* minor linter code modifications

* minor date local and language code modifications

Co-authored-by: Roman Zabaluev <rzabaluev@provectus.com>

* removing timestamp api from the generated sources

Co-authored-by: Mgrdich <46796009+Mgrdich@users.noreply.github.com>
Co-authored-by: Mgrdich <mgotm13@gmail.com>
2022-12-23 17:45:21 +04:00
Roman Zabaluev
5c723d9b44
Role based access control (#2790)
* Role based access control

* Fix build + checkstyle

* Refactoring, some bug fixes, review fixes

* Compile permission value patterns

* Make the resource a enum instead of a string

* Refactoring

* Make clusters required

* Fix formatting

* switch the switch case to a smart switch case

* Get rid of topic analysis actions

* Rename endpoints, fix an issue

* Return a flag indicating if rbac is on and a username

* Fix yaml indent in editorconfig

* Fix github & cognito role name fetching

* Fix case matching for actions

* Update readme

* Add an endpoint to determine if a user can create a resource

* Fix tests (I hope so)

* Fix tests

* Use spring configs instead of a separate file, rename endpoints

* Add "ALL" action
Get rid of unnecessary cache, save groups into spring auth
Review fixes

* Make "all" action case-insensitive

* Role based access control / FrontEnd  (#2933)

* Initial modifications and mocking the For the RoleAccess

* fix the Suspense issue in the components , comment the Tests to implement later

* minor test comment

* Roles and configuration and santization of data

* initialize RoleCheck hook

* make the App test file visible + minor modification in the permission hook

* Structure the data so the Burger header toggle does not rerender the whole application

* add tests to the NavBar and the Page container , add tests

* NavBar and PageContainer bug fixes

* Roles Testing code modification

* covering Topics create button Actions, and Schema create button Actions

* minor typescript code modifications for the cluster required parameter in the rolesHelper

* minor typescript code modifications for the cluster required parameter in the rolesHelper

* minor code modification to describe the Permission tests more clearly

* Produce message Permissions with Tests Suites for Topic

* Add Schema Edit Permission with tests

* Minor role changes

* Add ActionButton Component to handle the Button with tooltip

* Add ActionButton Component to handle the Button with tooltip

* Add Action Button to every Button create Action

* ActionButton add test suites

* usePermission code modification to include regular expressions

* Abstract Actions Component for code repetition, add Configs Edit button Permission + add the tests suites to it.

* Schema Remove functionality Permission and Test Suites + creation of the ActionDropdownItem for Actions

* Topic Edit Clear and delete Topic , Permissions with test suites

* ActionsCell For Topic Message Overview for permissions with tests suites

* Connector Delete , Consumer Groups Permission + writing test suites

* Add Permissions to the Topics ActionCell

* Topic Table Permissions Tests Suites

* Headless Logic for the Permission Part

* add documentation for the headless Part of the permission + add modification of the data version 2 for efficient algorithmic lookup

* replace modify data logic and isPermitted function to have faster access to the data

* Add Permission helpers tests suites

* usePermission hook test suites

* BatchActionsBar add Permissions + minor modification in TopicTable tests suites

* Statistics and Metrics code Permission + add test suites

* Recreate Topic Permissions in the Topic page, add tests suites

* Actions for the Connector components

* Messages NavLink View Permission

* Test suites messages code modifications

* Permissions comment code modifications

* Replacing the Mock Data With the actual code

* Add ActionNavLink test suites

* BatchActionsBar code smell modifications

* maximizing the permissions tests suites

* maximizing the permissions tests suites

* maximizing the permissions tests suites

* Tooltip code refactoring and fix the positions issue

* permissions increase the tests coverage

* add user info at the navigation header and tests suites

* Add Global Schema Selector Permissions with test suites

* Roles minor code removal

* Change the Action Component form hook mixin approach to declarative props approach

* add isPermitted function for multiple Actions , adding tests suites for this particular case

* remove redundant Permissions test blocks from the components

* remove redundant Permissions test blocks from the components

* Action Buttons test suites' coverage + generalizing the code of the Actions

* add invalid Permission check in Action Components tests suites

* Modularization of Actions Components

* Modularization of Actions Components by adding DropDownAction to it.

* Reflect the BE Changes to the UI , by changing the default behavior or the testing of roles.

* Reflect the BE Changes to the UI , by changing the default behavior or the testing of roles.

* Get rid of not necessary usePermission mocks

* Modifications in the UserInfo data , to consider the UI without any login functionality

* minor code modifications in the BatchActionBar component

* change the Query key for the user info

* change the default message for the tooltip

* Fix the Create Role Access for Topics and Schemas

* ListPage Connector create permissions

* add Headless logic for Create Permission with test suites. + add react hook render-er

* Create Button ActionButton logic implementation

* Remove Code smells , by removing the duplications

* increase the test suites for isPermittedToCreate logic

* increase the test suites for isPermittedToCreate logic

* Change the UserResourceType Enum with the new value

* Apply New Resource Creation validation, for Topic, Schema, Connector

* Apply New Resource Creation validation, for Topic, Schema, Connector

* minor code refactor modifications

* minor code modification in the topics useCreate hook

* Async Validation for all the Create Pages

* caching test for optimal performance in async validation schemas

* Reverting the Front End Validation

* Reverting the Front End Validation

* Authorization API minor syntax modifications

* fix SmokeTests

Co-authored-by: Roman Zabaluev <rzabaluev@provectus.com>
Co-authored-by: VladSenyuta <vlad.senyuta@gmail.com>

Co-authored-by: Mgrdich <46796009+Mgrdich@users.noreply.github.com>
Co-authored-by: VladSenyuta <vlad.senyuta@gmail.com>
2022-12-22 17:05:53 +04:00
Mgrdich
c2be45fd06
[FE] Fix invalid SeekType value for the offset and timestamp fields (#3057)
* SeekType fix the value during the offset and timestamp fields

* BE: thread interruption logging added to Emitters

Co-authored-by: iliax <ikuramshin@provectus.com>
2022-12-19 21:44:14 +04:00
Ilya Kuramshin
f9906b5d30
Brokers API improvements (#2743)
* Brokers API improvements:
1. broker io rates stats added
2. active controller property set to node id
3. minor refactoring
4. FE: Add an indicator for an active broker controller

Co-authored-by: Mgrdich <mgotm13@gmail.com>
Co-authored-by: iliax <ikuramshin@provectus.com>
Co-authored-by: Hrant Abrahamyan <113341474+habrahamyanpro@users.noreply.github.com>
Co-authored-by: Mgrdich <mgotm13@gmail.com>
2022-12-12 14:15:55 +04:00
Narekmat
b81c20d36d
Serde-api release adjustments (#3050)
* kafka-api-serde-api pom fixes

* change pom and add settings

* fix xml

Co-authored-by: iliax <ikuramshin@provectus.com>
Co-authored-by: Roman Zabaluev <rzabaluev@provectus.com>
2022-12-09 17:59:25 +04:00
Ilya Kuramshin
1108c760e5
[BE] Fix a case when controller is not found (#3030)
* ISSUE-3024: kui not working when controller not found
1. ReactiveAdminClient::getClusterVersion now takes into account that controller can not be visible
2. Uniq clientId generation added for consumers and adminClients
3. ReactiveAdminClient::toMonoWithExceptionFilter implementation simplified

* uniq adminclient clientId generation

Co-authored-by: iliax <ikuramshin@provectus.com>
2022-12-07 08:51:22 +04:00
Ilya Kuramshin
ed61852e40
Implement per-cluster polling rates limits (#2981)
* Polling throttling:
1. PollingThrottler class for per-cluster throttle managing
2. UI-publishing throttle for TailingEmitter
3. Elastic scheduler assigned for places where Serde interfaces are used

* int cast fix

* compilation fix

* test threshold fix

Co-authored-by: iliax <ikuramshin@provectus.com>
Co-authored-by: Roman Zabaluev <rzabaluev@provectus.com>
2022-12-07 08:51:11 +04:00
Boris Djurdjevic
b87665a12c
[KSQL] Implement mTLS configuration (#2984)
* feat(ksqldb): Add mTLS support

* doc(ksqldb): Add SSL configuration

* chore(compose): Add ksqldb0 to key material

* feat(ksqldb): Add SSL-enabled demo ksqlDB instance

Co-authored-by: Roman Zabaluev <rzabaluev@provectus.com>
2022-12-07 07:38:13 +04:00
Ilya Kuramshin
43ccca43c2
Fix random topic creation fails (#3004)
ReactiveAdminClient::listOffsetsUnsafe now skipping non-initialized partitions. This prevents UnknownTopicOrPartitionException to be leaked to high-level logic levels (preventing TopicService::loadTopicAfterCreation to do retries)
2022-12-05 17:02:45 +04:00
Ilya Kuramshin
052d8af456
ISSUE-2968: Deleting a config parameter of topic doesn't take effect (#3001)
ReactiveAdminClient::incrementalAlterConfig now replaces all topic configs, not only altering changed
2022-12-01 21:09:23 +04:00
Ilya Kuramshin
c804a6d00b
Serde autoconfiguration fix (#2997)
Co-authored-by: iliax <ikuramshin@provectus.com>
2022-12-01 13:20:48 +04:00
Ilya Kuramshin
216b4acd3d
Serdes improvements and fixes (#2989)
Serdes improvements and fixes:
1. Built-in serdel's auto-configuration separated from explicit configuration (when properties set in serde configuration)
2. Serdes initialization logic extracted into SerdesInitializer.java
3. docker-compose example with serdes configuration added
4. Serialization.md document fixed
2022-11-30 18:43:48 +04:00
Ilya Kuramshin
20ecc74dd9
Data masking (#2850)
Data masking:
1. properties & mapping added to ClustersProperties
2. DataMasking provides function that doing masking for specified topic & target
3. Masking policies implemented: MASK, REMOVE, REPLACE
2022-11-28 12:54:07 +04:00
Ilya Kuramshin
e77b913164
Documentation for pluggable serdes (#2702)
* 1. Minor serde's code improvements
2. Javadoc added to serde-related code

* Serdes configuration docs WIP

* wip

* DeserializeResult.toString impl added

* SchemeRegistry.canDeserialize checks subject existence

* Update Serialization.md

* Update README.md

Co-authored-by: iliax <ikuramshin@provectus.com>
Co-authored-by: Roman Zabaluev <rzabaluev@provectus.com>
2022-11-25 14:35:06 +04:00
Ilya Kuramshin
5900f8e957
Snappy native impl fix (#2940)
* wip

* wip

* wip

* wip

Co-authored-by: iliax <ikuramshin@provectus.com>
2022-11-15 16:05:06 +04:00
Jochen Schalanda
0d0a244915
Centralize dependency versions and bump where necessary (#2929) 2022-11-15 14:22:16 +04:00
Boris Djurdjevic
e26f9787d8
Implement mTLS for KC (#2833)
* feat(kafka-connect): Implement kafka-connect mTLS configuration

* chore(kafka-connect): Use SecuredWebClient

* feat(ssl-demo): Configure SANs and recreate certificates

* feat(ssl-demo): Add docker-compose demo with TLS enabled components
2022-11-15 14:11:57 +04:00
Jochen Schalanda
d6571ec7ae
CI: Scan Java source files with CodeQL (#2898)
* Scan Java code with CodeQL

* REMOVE ME: Trigger workflow run
2022-11-07 19:16:26 +04:00
Jochen Schalanda
3277186fc0
Add support for multiple Protobuf files (#2874)
* Add support for multiple Protobuf files

* Ensure `protobufFiles` list is not empty

* Address review comments
2022-11-07 15:15:40 +04:00
Jochen Schalanda
62f35e3e29
Upgrade to Java 17 (Azul Zulu) (#2879)
* Feature: Upgrade to Java 17 (Azul Zulu)

Java/OpenJDK 13 has reached end of life (and end of security support)
more than 2.5 years ago: https://endoflife.date/java

This change set updates the build, CI pipeline, and Docker image of Kafka UI to Java 17 (LTS).

* Add back maven-compile-plugin to kafka-ui-e2e-checks
2022-11-07 14:57:59 +04:00
Boris Djurdjevic
99e50f8814
Support SSL for schema registry (#2832)
* feat(schema-registry): Implement schema-registry mTLS configuration

* chore(linter): Make checkstyle happy

* feat(schema-registry): Implement schema-registry mTLS configuration

* chore(linter): Make checkstyle happy

* chore(schema-registry): Refactor WebClient creation for PR #2832

Co-authored-by: Ilya Kuramshin <iliax@proton.me>
2022-10-31 18:05:28 +04:00
Ilya Kuramshin
2c74b4e1c2
ISSUE-2680: Smart filters stats propagation to UI (#2771)
ISSUE-2680: Smart filters stats propagation to UI : 
1. DONE event type implemented to flag polling end
2. MessageFilterStats added to emitters context to count filter apply errors
2022-10-31 17:41:11 +04:00
Ilya Kuramshin
aa6c3083c2
ISSUE-2103: Fixing AdminClient timeout on listOffsets (#2781)
ISSUE-2103: Fixing AdminClient timeout on listOffsets when partition's leader in not available:
1. ReactiveAdminClient.listOffsets methods now check if partitions have leader before executing
2. In all places where ReactiveAdminClient.listOffsets methods are used, we specify what to do if partition leader is not available - fail or skip such partitions
3. AdminClient.listOffsets calls splitted with partitionCalls
2022-10-31 16:46:19 +04:00
Roman Zabaluev
cb8049e769
Add a separate endpoint for frontend time format (#2847) 2022-10-31 13:17:23 +04:00
Roman Zabaluev
2f2e19d1bc
Fix topic internal indication (#2846) 2022-10-31 12:46:27 +04:00
Ilya Kuramshin
9a3cbfa14b
ISSUE-2787: Setting default config values properly (#2788)
ISSUE-2787:
1. Getting topic config defaults from server response
2. rm KafkaConstants
3. doc field added to TopicConfigDTO
4. ReactiveAdminClient.SupportedFeature refactor
2022-10-30 21:59:49 +04:00
Egorka Voronyansky
7044e57095
Drop deprecated methods (#2837)
Co-authored-by: vrnsky <vrnsky@protonmail.com>
2022-10-28 12:50:04 +04:00
Ilya Kuramshin
d60808a2f2
Drop deprecated ksql api (#2796)
Co-authored-by: iliax <ikuramshin@provectus.com>
2022-10-23 19:49:11 +04:00
Ilya Kuramshin
4558466ff6
Emitters logic refactoring (#2729)
* Emitters logic refactoring:
1. consumers seeking moved to SeekOperations class
2. offsets info gathering moved to OffsetsInfo class

* wip

* checkstyle fix

* checkstyle fix

* minor improvements

Co-authored-by: iliax <ikuramshin@provectus.com>
2022-10-23 19:47:21 +04:00
Egorka Voronyansky
b0e0da4c0d
Bump version of mockito up to 4.8.1 (#2793)
* Bump version of mockito up to 4.8.1

* Change scope of byte buddy dependency to test

Co-authored-by: vrnsky <vrnsky@protonmail.com>
2022-10-21 09:40:35 +04:00
Ilya Kuramshin
63b6bb3fa1
ISSUE-2202: smart filters headers test added (#2763) 2022-10-17 16:24:14 +04:00
Roman Zabaluev
2b5dd270e2
Implement Connectors sorting / backend (#2291)
Co-authored-by: Artem Kazlanzhy <akazlanzhy@provectus.com>
2022-10-14 13:16:29 +04:00