Commit graph

452 commits

Author SHA1 Message Date
Ilya Kuramshin
d1f04e05bb
[Be] Fix MessagesServiceTest, waiting of kafka write added (#3287)
Co-authored-by: iliax <ikuramshin@provectus.com>
2023-01-31 11:57:21 +04:00
Batuhan Küçükali
1b09251419
[BE] Fix case-sensitive search in broker's topics (#3276)
Co-authored-by: Roman Zabaluev <rzabaluev@provectus.com>
2023-01-31 11:55:21 +04:00
Ilya Kuramshin
43fcf6dce1
[BE] Suppressing failures when describe log dirs not supported (#3262)
* Suppressing failures when describe log dirs not supported
2023-01-26 11:12:59 +04:00
Ilya Kuramshin
ded4e26825
[BE] Chore: ConfigTest stats initialization added (#3260)
Co-authored-by: iliax <ikuramshin@provectus.com>
2023-01-24 12:52:27 +04:00
Ilya Kuramshin
5d31189609
ISSUE-3222: SchemaRegistrySerde checkSchemaExistenceForDeserialize property added (#3235) 2023-01-17 22:38:44 +04:00
Wanis Fahmy
f55376b40b
[BE] Impl basic auth for prometheus (#3201)
* fix(3143): passes prometheus basic auth credentials

Co-authored-by: theurichde <tim.heurich@idealo.de>

* fix(3143): applies review comments

Co-authored-by: theurichde <tim.heurich@idealo.de>

Co-authored-by: theurichde <tim.heurich@idealo.de>
Co-authored-by: Roman Zabaluev <rzabaluev@provectus.com>
2023-01-16 17:53:06 +04:00
Ilya Kuramshin
799c2c455a
Implement SchemaRegistry openapi client (#3123)
Failover clients for SchemaRegistry, Kafka-connect, KSQL:
1. SchemeRegistry openapi client generation
2. WebClient configuration logic moved to WebClientConfigurator class
3. ReactiveFailover utility implemented 
4. Connect, Ksql moved to ReactiveFailover usage
2023-01-16 15:27:55 +04:00
Ilya Kuramshin
7c6d04bca0
Proto to JsonSchema well-known types support (#3088)
1. Add support for predefined (well-known) protobuf types.
2. `preservingProtoFieldNames` mode is enabled for proto messages formatting to fit proto file's message's field names
3. Invalid oneof-processing logic removed
4. Protobuf -> JsonSchema self-reference issue fixed
5. format added to Timestamp protobuf type
2023-01-16 14:49:04 +04:00
Ilya Kuramshin
578468d090
Consumer groups loading performance impr (#3188)
ISSUE-3148, ISSUE-3188: Topic consumers retrieval performance impr
1. batch listConsumerGroupOffsets method implemented
2. describeConsumerGroups parallelized
3. minor improvements in ReactiveAdminClient
4. (not related to perf) ConsumerGroupMapper messagesBehind setting fixed
2023-01-12 23:14:28 +04:00
Mgrdich
9d98927524
[FE] Implement a full support for custom context path (#3106)
* Vite Custom Plugin code initialization , ( not building on js file )

* Fix the lazy loading functionality in vite

* add lazy loading in the Cluster page

* minor comment code cleanup

* Remove un-necessary files for configuration of the vite

* minor naming modifications

* minor code modification to encode a variable in the root
2023-01-03 19:54:42 +04:00
Roman Zabaluev
9b87d3829b
Fix minor RBAC/auth NPEs, init null collections (#3185) 2023-01-03 19:29:33 +04:00
Ilya Kuramshin
eef63466fb
[BE] Update tests for confluent platform 7+ (#3038)
Co-authored-by: iliax <ikuramshin@provectus.com>
2023-01-03 14:52:28 +04:00
Ilya Kuramshin
0cba8240b7
Bug: UnsupportedVersionException: includeClusterAuthorizedOperations (#3157)
* ISSUE-3141: checking kafka version before setting includeAuthorizedOperations for describeCluster api call
2022-12-28 19:49:26 +04:00
Roman Zabaluev
8b126d843e
Fix google oauth with rbac (#3155) 2022-12-28 14:34:35 +04:00
Roman Zabaluev
027d9b4653
Change message timestamp format (#3113)
* Drop experimental endpoints

* [FE] Change message timestamp format (#3119)

* remove dayjs from the code base

* remove dayjs library and clean up the hook codes

* minor obsolete fixtures deletion

* minor provider code modifications

* minor provider code modifications

* minor linter code modifications

* minor date local and language code modifications

Co-authored-by: Roman Zabaluev <rzabaluev@provectus.com>

* removing timestamp api from the generated sources

Co-authored-by: Mgrdich <46796009+Mgrdich@users.noreply.github.com>
Co-authored-by: Mgrdich <mgotm13@gmail.com>
2022-12-23 17:45:21 +04:00
Roman Zabaluev
5c723d9b44
Role based access control (#2790)
* Role based access control

* Fix build + checkstyle

* Refactoring, some bug fixes, review fixes

* Compile permission value patterns

* Make the resource a enum instead of a string

* Refactoring

* Make clusters required

* Fix formatting

* switch the switch case to a smart switch case

* Get rid of topic analysis actions

* Rename endpoints, fix an issue

* Return a flag indicating if rbac is on and a username

* Fix yaml indent in editorconfig

* Fix github & cognito role name fetching

* Fix case matching for actions

* Update readme

* Add an endpoint to determine if a user can create a resource

* Fix tests (I hope so)

* Fix tests

* Use spring configs instead of a separate file, rename endpoints

* Add "ALL" action
Get rid of unnecessary cache, save groups into spring auth
Review fixes

* Make "all" action case-insensitive

* Role based access control / FrontEnd  (#2933)

* Initial modifications and mocking the For the RoleAccess

* fix the Suspense issue in the components , comment the Tests to implement later

* minor test comment

* Roles and configuration and santization of data

* initialize RoleCheck hook

* make the App test file visible + minor modification in the permission hook

* Structure the data so the Burger header toggle does not rerender the whole application

* add tests to the NavBar and the Page container , add tests

* NavBar and PageContainer bug fixes

* Roles Testing code modification

* covering Topics create button Actions, and Schema create button Actions

* minor typescript code modifications for the cluster required parameter in the rolesHelper

* minor typescript code modifications for the cluster required parameter in the rolesHelper

* minor code modification to describe the Permission tests more clearly

* Produce message Permissions with Tests Suites for Topic

* Add Schema Edit Permission with tests

* Minor role changes

* Add ActionButton Component to handle the Button with tooltip

* Add ActionButton Component to handle the Button with tooltip

* Add Action Button to every Button create Action

* ActionButton add test suites

* usePermission code modification to include regular expressions

* Abstract Actions Component for code repetition, add Configs Edit button Permission + add the tests suites to it.

* Schema Remove functionality Permission and Test Suites + creation of the ActionDropdownItem for Actions

* Topic Edit Clear and delete Topic , Permissions with test suites

* ActionsCell For Topic Message Overview for permissions with tests suites

* Connector Delete , Consumer Groups Permission + writing test suites

* Add Permissions to the Topics ActionCell

* Topic Table Permissions Tests Suites

* Headless Logic for the Permission Part

* add documentation for the headless Part of the permission + add modification of the data version 2 for efficient algorithmic lookup

* replace modify data logic and isPermitted function to have faster access to the data

* Add Permission helpers tests suites

* usePermission hook test suites

* BatchActionsBar add Permissions + minor modification in TopicTable tests suites

* Statistics and Metrics code Permission + add test suites

* Recreate Topic Permissions in the Topic page, add tests suites

* Actions for the Connector components

* Messages NavLink View Permission

* Test suites messages code modifications

* Permissions comment code modifications

* Replacing the Mock Data With the actual code

* Add ActionNavLink test suites

* BatchActionsBar code smell modifications

* maximizing the permissions tests suites

* maximizing the permissions tests suites

* maximizing the permissions tests suites

* Tooltip code refactoring and fix the positions issue

* permissions increase the tests coverage

* add user info at the navigation header and tests suites

* Add Global Schema Selector Permissions with test suites

* Roles minor code removal

* Change the Action Component form hook mixin approach to declarative props approach

* add isPermitted function for multiple Actions , adding tests suites for this particular case

* remove redundant Permissions test blocks from the components

* remove redundant Permissions test blocks from the components

* Action Buttons test suites' coverage + generalizing the code of the Actions

* add invalid Permission check in Action Components tests suites

* Modularization of Actions Components

* Modularization of Actions Components by adding DropDownAction to it.

* Reflect the BE Changes to the UI , by changing the default behavior or the testing of roles.

* Reflect the BE Changes to the UI , by changing the default behavior or the testing of roles.

* Get rid of not necessary usePermission mocks

* Modifications in the UserInfo data , to consider the UI without any login functionality

* minor code modifications in the BatchActionBar component

* change the Query key for the user info

* change the default message for the tooltip

* Fix the Create Role Access for Topics and Schemas

* ListPage Connector create permissions

* add Headless logic for Create Permission with test suites. + add react hook render-er

* Create Button ActionButton logic implementation

* Remove Code smells , by removing the duplications

* increase the test suites for isPermittedToCreate logic

* increase the test suites for isPermittedToCreate logic

* Change the UserResourceType Enum with the new value

* Apply New Resource Creation validation, for Topic, Schema, Connector

* Apply New Resource Creation validation, for Topic, Schema, Connector

* minor code refactor modifications

* minor code modification in the topics useCreate hook

* Async Validation for all the Create Pages

* caching test for optimal performance in async validation schemas

* Reverting the Front End Validation

* Reverting the Front End Validation

* Authorization API minor syntax modifications

* fix SmokeTests

Co-authored-by: Roman Zabaluev <rzabaluev@provectus.com>
Co-authored-by: VladSenyuta <vlad.senyuta@gmail.com>

Co-authored-by: Mgrdich <46796009+Mgrdich@users.noreply.github.com>
Co-authored-by: VladSenyuta <vlad.senyuta@gmail.com>
2022-12-22 17:05:53 +04:00
Mgrdich
c2be45fd06
[FE] Fix invalid SeekType value for the offset and timestamp fields (#3057)
* SeekType fix the value during the offset and timestamp fields

* BE: thread interruption logging added to Emitters

Co-authored-by: iliax <ikuramshin@provectus.com>
2022-12-19 21:44:14 +04:00
Ilya Kuramshin
f9906b5d30
Brokers API improvements (#2743)
* Brokers API improvements:
1. broker io rates stats added
2. active controller property set to node id
3. minor refactoring
4. FE: Add an indicator for an active broker controller

Co-authored-by: Mgrdich <mgotm13@gmail.com>
Co-authored-by: iliax <ikuramshin@provectus.com>
Co-authored-by: Hrant Abrahamyan <113341474+habrahamyanpro@users.noreply.github.com>
Co-authored-by: Mgrdich <mgotm13@gmail.com>
2022-12-12 14:15:55 +04:00
Narekmat
b81c20d36d
Serde-api release adjustments (#3050)
* kafka-api-serde-api pom fixes

* change pom and add settings

* fix xml

Co-authored-by: iliax <ikuramshin@provectus.com>
Co-authored-by: Roman Zabaluev <rzabaluev@provectus.com>
2022-12-09 17:59:25 +04:00
Ilya Kuramshin
1108c760e5
[BE] Fix a case when controller is not found (#3030)
* ISSUE-3024: kui not working when controller not found
1. ReactiveAdminClient::getClusterVersion now takes into account that controller can not be visible
2. Uniq clientId generation added for consumers and adminClients
3. ReactiveAdminClient::toMonoWithExceptionFilter implementation simplified

* uniq adminclient clientId generation

Co-authored-by: iliax <ikuramshin@provectus.com>
2022-12-07 08:51:22 +04:00
Ilya Kuramshin
ed61852e40
Implement per-cluster polling rates limits (#2981)
* Polling throttling:
1. PollingThrottler class for per-cluster throttle managing
2. UI-publishing throttle for TailingEmitter
3. Elastic scheduler assigned for places where Serde interfaces are used

* int cast fix

* compilation fix

* test threshold fix

Co-authored-by: iliax <ikuramshin@provectus.com>
Co-authored-by: Roman Zabaluev <rzabaluev@provectus.com>
2022-12-07 08:51:11 +04:00
Boris Djurdjevic
b87665a12c
[KSQL] Implement mTLS configuration (#2984)
* feat(ksqldb): Add mTLS support

* doc(ksqldb): Add SSL configuration

* chore(compose): Add ksqldb0 to key material

* feat(ksqldb): Add SSL-enabled demo ksqlDB instance

Co-authored-by: Roman Zabaluev <rzabaluev@provectus.com>
2022-12-07 07:38:13 +04:00
Ilya Kuramshin
43ccca43c2
Fix random topic creation fails (#3004)
ReactiveAdminClient::listOffsetsUnsafe now skipping non-initialized partitions. This prevents UnknownTopicOrPartitionException to be leaked to high-level logic levels (preventing TopicService::loadTopicAfterCreation to do retries)
2022-12-05 17:02:45 +04:00
Ilya Kuramshin
052d8af456
ISSUE-2968: Deleting a config parameter of topic doesn't take effect (#3001)
ReactiveAdminClient::incrementalAlterConfig now replaces all topic configs, not only altering changed
2022-12-01 21:09:23 +04:00
Ilya Kuramshin
c804a6d00b
Serde autoconfiguration fix (#2997)
Co-authored-by: iliax <ikuramshin@provectus.com>
2022-12-01 13:20:48 +04:00
Ilya Kuramshin
216b4acd3d
Serdes improvements and fixes (#2989)
Serdes improvements and fixes:
1. Built-in serdel's auto-configuration separated from explicit configuration (when properties set in serde configuration)
2. Serdes initialization logic extracted into SerdesInitializer.java
3. docker-compose example with serdes configuration added
4. Serialization.md document fixed
2022-11-30 18:43:48 +04:00
Ilya Kuramshin
20ecc74dd9
Data masking (#2850)
Data masking:
1. properties & mapping added to ClustersProperties
2. DataMasking provides function that doing masking for specified topic & target
3. Masking policies implemented: MASK, REMOVE, REPLACE
2022-11-28 12:54:07 +04:00
Ilya Kuramshin
e77b913164
Documentation for pluggable serdes (#2702)
* 1. Minor serde's code improvements
2. Javadoc added to serde-related code

* Serdes configuration docs WIP

* wip

* DeserializeResult.toString impl added

* SchemeRegistry.canDeserialize checks subject existence

* Update Serialization.md

* Update README.md

Co-authored-by: iliax <ikuramshin@provectus.com>
Co-authored-by: Roman Zabaluev <rzabaluev@provectus.com>
2022-11-25 14:35:06 +04:00
Ilya Kuramshin
5900f8e957
Snappy native impl fix (#2940)
* wip

* wip

* wip

* wip

Co-authored-by: iliax <ikuramshin@provectus.com>
2022-11-15 16:05:06 +04:00
Jochen Schalanda
0d0a244915
Centralize dependency versions and bump where necessary (#2929) 2022-11-15 14:22:16 +04:00
Boris Djurdjevic
e26f9787d8
Implement mTLS for KC (#2833)
* feat(kafka-connect): Implement kafka-connect mTLS configuration

* chore(kafka-connect): Use SecuredWebClient

* feat(ssl-demo): Configure SANs and recreate certificates

* feat(ssl-demo): Add docker-compose demo with TLS enabled components
2022-11-15 14:11:57 +04:00
Jochen Schalanda
d6571ec7ae
CI: Scan Java source files with CodeQL (#2898)
* Scan Java code with CodeQL

* REMOVE ME: Trigger workflow run
2022-11-07 19:16:26 +04:00
Jochen Schalanda
3277186fc0
Add support for multiple Protobuf files (#2874)
* Add support for multiple Protobuf files

* Ensure `protobufFiles` list is not empty

* Address review comments
2022-11-07 15:15:40 +04:00
Jochen Schalanda
62f35e3e29
Upgrade to Java 17 (Azul Zulu) (#2879)
* Feature: Upgrade to Java 17 (Azul Zulu)

Java/OpenJDK 13 has reached end of life (and end of security support)
more than 2.5 years ago: https://endoflife.date/java

This change set updates the build, CI pipeline, and Docker image of Kafka UI to Java 17 (LTS).

* Add back maven-compile-plugin to kafka-ui-e2e-checks
2022-11-07 14:57:59 +04:00
Boris Djurdjevic
99e50f8814
Support SSL for schema registry (#2832)
* feat(schema-registry): Implement schema-registry mTLS configuration

* chore(linter): Make checkstyle happy

* feat(schema-registry): Implement schema-registry mTLS configuration

* chore(linter): Make checkstyle happy

* chore(schema-registry): Refactor WebClient creation for PR #2832

Co-authored-by: Ilya Kuramshin <iliax@proton.me>
2022-10-31 18:05:28 +04:00
Ilya Kuramshin
2c74b4e1c2
ISSUE-2680: Smart filters stats propagation to UI (#2771)
ISSUE-2680: Smart filters stats propagation to UI : 
1. DONE event type implemented to flag polling end
2. MessageFilterStats added to emitters context to count filter apply errors
2022-10-31 17:41:11 +04:00
Ilya Kuramshin
aa6c3083c2
ISSUE-2103: Fixing AdminClient timeout on listOffsets (#2781)
ISSUE-2103: Fixing AdminClient timeout on listOffsets when partition's leader in not available:
1. ReactiveAdminClient.listOffsets methods now check if partitions have leader before executing
2. In all places where ReactiveAdminClient.listOffsets methods are used, we specify what to do if partition leader is not available - fail or skip such partitions
3. AdminClient.listOffsets calls splitted with partitionCalls
2022-10-31 16:46:19 +04:00
Roman Zabaluev
cb8049e769
Add a separate endpoint for frontend time format (#2847) 2022-10-31 13:17:23 +04:00
Roman Zabaluev
2f2e19d1bc
Fix topic internal indication (#2846) 2022-10-31 12:46:27 +04:00
Ilya Kuramshin
9a3cbfa14b
ISSUE-2787: Setting default config values properly (#2788)
ISSUE-2787:
1. Getting topic config defaults from server response
2. rm KafkaConstants
3. doc field added to TopicConfigDTO
4. ReactiveAdminClient.SupportedFeature refactor
2022-10-30 21:59:49 +04:00
Egorka Voronyansky
7044e57095
Drop deprecated methods (#2837)
Co-authored-by: vrnsky <vrnsky@protonmail.com>
2022-10-28 12:50:04 +04:00
Ilya Kuramshin
d60808a2f2
Drop deprecated ksql api (#2796)
Co-authored-by: iliax <ikuramshin@provectus.com>
2022-10-23 19:49:11 +04:00
Ilya Kuramshin
4558466ff6
Emitters logic refactoring (#2729)
* Emitters logic refactoring:
1. consumers seeking moved to SeekOperations class
2. offsets info gathering moved to OffsetsInfo class

* wip

* checkstyle fix

* checkstyle fix

* minor improvements

Co-authored-by: iliax <ikuramshin@provectus.com>
2022-10-23 19:47:21 +04:00
Egorka Voronyansky
b0e0da4c0d
Bump version of mockito up to 4.8.1 (#2793)
* Bump version of mockito up to 4.8.1

* Change scope of byte buddy dependency to test

Co-authored-by: vrnsky <vrnsky@protonmail.com>
2022-10-21 09:40:35 +04:00
Ilya Kuramshin
63b6bb3fa1
ISSUE-2202: smart filters headers test added (#2763) 2022-10-17 16:24:14 +04:00
Roman Zabaluev
2b5dd270e2
Implement Connectors sorting / backend (#2291)
Co-authored-by: Artem Kazlanzhy <akazlanzhy@provectus.com>
2022-10-14 13:16:29 +04:00
Ilya Kuramshin
8b91f50af5 Smart filter offset binding (#2732)
* adding "offset" var to smart filter binding

Co-authored-by: iliax <ikuramshin@provectus.com>
2022-10-13 22:41:43 +04:00
Ilya Kuramshin
19e38fb1bf
Msk serverless support (BE) (#2737)
Msk serverless support:
1. ReactiveAdminClient.loadBrokersConfig returns empty map if configs retrieval not supported by kafka backend
2. ReactiveAdminClient.toMone exception unwrapping added
3. FeatureService delete topics enabled set true by default
4. TopicCreationDTO.replicationFactor made optional


Co-authored-by: iliax <ikuramshin@provectus.com>
2022-10-13 22:30:07 +04:00
Marat Gumarov
6dff8f105e
Support prometheus as a metrics interface (#2190)
* Unified internal metric representation (RawMetric)
* Prometheus retriever implemented
* Jmx metrics formatted to jmx-exporter openmentrics format

Co-authored-by: Mohamad Choukair <mchoukair@provectus.com>
Co-authored-by: Roman Zabaluev <rzabaluev@provectus.com>
Co-authored-by: Daniel Ching <110129035+daching-provectus@users.noreply.github.com>
Co-authored-by: Daniel Ching <daching@provectus.com>
Co-authored-by: German Osin <germanosin@Germans-MacBook-Pro.local>
Co-authored-by: iliax <ikuramshin@provectus.com>
Co-authored-by: Ilya Kuramshin <iliax@proton.me>
2022-10-07 14:36:27 +04:00
Ilya Kuramshin
b19b9d82e9
Minor backend fixes (#2696)
Minor backend fixes:
1. InternalTopic.leader fiels filling fix
2. bytesIn/OutPerSec fields set to TopicDTO
3. "subject" property added to SR SchemaDescription
2022-10-03 17:16:55 +04:00