Next level remote data workflows for everyone

X-Pipe is a tool for working with remote connections and the data that can be accessed through them. The focus lies on providing an easy remote connection handling workflow plus data intermediation capabilities such that you can work with all kinds of data located on essentially any remote system with ease. So if you are

• a data analyst/scientist
• a system administrator
• a software engineer
• or generally just work a lot with remote connections and data

then X-Pipe is designed for you!

The X-Pipe project is designed to be an open platform, both in terms of development from a developer's perspective and the workflow from a user's perspective. X-Pipe is open source and is designed to utilize and integrate well with other popular tools and workflows, focusing on augmenting them rather than replacing them.

Connection Explorer

The connection explorer allows you to connect to, manage, and interact with all kinds of remote systems. It comes with the following main features:

Flexible Connector

• Can connect to standard servers, database servers, and more
• Supports established protocols (e.g. SSH and more) plus any custom connection methods that work through the command-line
• Is able to integrate any kind of proxies into the connection process, even ones with different protocols

Instant launch for remote shells and commands

• Automatically login into a shell in your favourite terminal with one click (no need to fill password prompts, etc.)
• Works for all kinds of shells. This includes command shells (e.g. bash, PowerShell, cmd, etc.) and database shells (e.g. PSQL Shell)
• Comes with integrations for all commonly used terminals for all operating systems
• Exclusively uses established CLI tools and therefore works out of the box on most systems and doesn't require any additional setup
• Allows you to customize the launched shell's init environment

All your connections in one place

• Easily create and manage all kinds of remote connections at one location
• Securely stores all information exclusively on your computer and encrypts all secret information
• Share connection configurations to any other trusted party through shareable URLs
• Create desktop shortcuts to open your connections

Data Explorer

Building on top of the connection explorer, the data explorer allows you to manage and work with all kinds of data sources:

Work with your data on a higher level

• X-Pipe utilizes structures of data instead of the raw data itself, allowing for a higher level workflow that is independent of the underlying data format
• Save time when adding data sources by making use of the advanced auto detection feature of X-Pipe where you don't have to worry about encodings, format configurations, and more
• Easily convert between different data representations

Integrate X-Pipe with your favorite tools and workflows

• Easily import and export all kinds of data formats and technologies
• Access data sources from the command-line with the X-Pipe CLI or your favorite programming languages using the X-Pipe APIs
• Connect select third party applications directly to X-Pipe through extensions

Repository Structure

The following for modules make up the X-Pipe API and a licensed under the MIT license:

• core - Shared core classes of the X-Pipe Java API, X-Pipe extensions, and the X-Pipe daemon implementation
• API - The API that can be used to interact with X-Pipe from any JVM-based language. For setup instructions, see the X-Pipe Java API Usage section.
• beacon - The X-Pipe beacon component is responsible for handling all communications between the X-Pipe daemon and the client applications, for example the various programming language APIs and the CLI
• extension - An API to create all different kinds of extensions for the X-Pipe platform For setup instructions, see the X-Pipe extension development section.

The other modules make up the X-Pipe implementation and are licensed under GPL:

• app - Contains the X-Pipe daemon implementation and the X-Pipe desktop application code
• cli - The X-Pipe CLI implementation, a GraalVM native image application
• dist - Tools to create a distributable package of X-Pipe
• ext - Available X-Pipe extensions. Essentially every feature is implemented as an extension

Open source model

X-Pipe utilizes an open core model, which essentially means that the main application core is open source while certain other components are not. In this case these non open source components are planned to be future parts of a potential commercialization. Furthermore, some tests and especially test environments and that run on private servers are also not included in this repository (Don't want to leak server information). Finally, scripts and workflows to create signed executables and installers are also not included to prevent attackers from easily impersonating the shipping the X-Pipe application malware.

The license model is chosen in such a way that you are able to use and integrate X-Pipe within your application through the MIT-licensed API. In any other case where you plan to contribute to the X-Pipe platform itself, which is GPL licensed, I would still have to figure out how to exactly handle these kinds of contributions. It is also planned to move many components to a more permissive license in the future.

Development

Any contribution is welcomed! There are no real formal contribution guidelines right now, they will maybe come later.

Modularity

All X-Pipe components target JDK 19 and make full use of the Java Module System (JPMS). All components are modularized, including all their dependencies. As the CLI utilizes the native image capability of GraalVM, it is recommended to use GraalVM with Java 19 support. In case a dependency is (sadly) not modularized yet, module information is manually added using moditect. Further, note that as this is a pretty complicated Java project that fully utilizes modularity, many IDEs still have problems building this project properly. For example, you can't build this project in eclipse or vscode as it will complain about missing modules. The tested and recommended IDE is intellij.

Requirements

To build the CLI, GraalVM is required. Furthermore, you also need the platform specific toolchains to be installed:

• Windows: Visual Studio 2022
• Linux: build-essential libz-dev zlib1g-dev
• MacOS: xcode with xcode-select --install

Building and Running

You can use the gradle wrapper to build and run the project:

• gradlew app:run will run the desktop application. You can set various useful properties in app/build.gradle
• gradlew buildCli will create a native image for the CLI application
• gradlew dist will create a distributable production version in dist/build/dist/base. To include this CLI executable in this build, make sure to run gradlew buildCli first
• You can also run the CLI application in development mode with something like gradlew :cli:clean :cli:run --args="daemon start". Note here that you should always clean the CLI project first, as the native image plugin is a little buggy in that regard.
• gradlew <project>:test will run the tests of the specified project.

You are also able to properly debug the built production application through two different methods:

• The app/scripts/xpiped_debug script will launch the application in debug mode and with a console attached to it
• The app/scripts/xpiped_debug_attach script attaches a debugger with the help of AttachMe. Just make sure that the attachme process is running within IntelliJ, and the debugger should launch automatically once you start up the application.

Note that when any unit test is run using a debugger, the X-Pipe daemon process that is started will also attempt to connect to that debugger through AttachMe as well.

Development FAQ

Why are there no GitHub actions workflows or other continuous integration pipelines set up for this repository?

There are several test workflows run in a private environment as they use private test connections such as remote server connections and database connections. Other private workflows are responsible for packaging, signing, and distributing the releases. So you can assume that the code is tested!