ladybird/Documentation/Kernel/GraphicsSubsystem.md

Introduction to the Kernel Graphics Subsystem

What is the Kernel Graphics Subsystem?

The Kernel Graphics Subsystem is the kernel subsystem that is responsible to manage all graphics devices, framebuffers, hardware 3D acceleration, memory mappings, etc.

Responsibilities

• Provide a convenient interface to all supported video hardware in the Kernel.
• Manage 3D rendering on supported hardware.

Current Limitations and Future features?

• No locking on who can do mmap on DisplayConnector devices currently, which can lead to malicious applications "fighting" with WindowServer on what is shown to the user from the framebuffer.

DisplayConnector Devices

The Display Connector devices are an abstraction layer to what is essentially the management layer of hardware display (commonly known as scanouts) output connectors. The idea of using such type of device was inspired by Linux, which has a struct called drm_connector as a base structure for other derived structures in the various Linux DRM drivers.

A Display connector device is typically connected to a group of other connectors as well, as it's generally common to have video hardware that utilizes multiple hardware connectors to VGA, DisplayPort, HDMI, DVI, etc. However, it can be a stand-alone device too, which is the case for the GenericDisplayConnector class, that can be initialized without being attached to a parent PCI device object at all.

Each display connector is programmatically accessible via a device file, in the /dev/gpu/ directory with a name connectorX (X is replaced with the minor number).

Each display connector could be mmap-ed to gain control to video RAM directly. This works nicely with the kernel TTY subsystem thanks to the role of virtual memory in the subsystem.

Hardware framebuffers

History lesson on ISA, PCI, VGA (and SVGA)

Since the beginning of video hardware with old-school ISA VGA display adapters, there was a window being mapped in the physical address space, being translated by the motherboard chipset as read/write to video RAM. When SuperVGA came along in the 90s, it expanded the usage of that small VGA window (where it was in very low memory) to high resolution framebuffers in very high memory regions. This tradition continues today to some extent (excluding hardware which requires DMA from main memory to video memory), because it's relatively cheap and easy way to let operating systems to access video RAM directly without too much trouble.

Since the main PC x86 computer bus was the IBM ISA bus, there was no easy way to tell where the resources of each card were actually located at the IO space nor in the physical memory space. There were a couple of attempts to fix this - the most notable was the Plug-and-Play standard.

The real change came from a new computer bus in the mid 90s - the PCI bus. This new bus was PnP friendly - no more hardcoded resource allocations which means also that OS drivers can find where the firmware (BIOS) mapped the BAR (Base address registers) for the actual resources. This was also the era where SuperVGA video adapters started to appear, taking advantage of this new bus.

Since VGA was introduced, countless amount of vendors brought their own implementations and video adapters for usage in the PC market. By now, most of them are gone, leaving the major vendors (Intel, AMD and Nvidia) to still be able to manufacture video adapters which are today commonly known as Graphics Processing Unit (abbreviated as GPU) - due to the fact that today video adapters are not only outputting pixels to the computer screen, but have a whole set of processors to take care of heavy computational tasks of graphics assets, and even general processing tasks nowadays.

SuperVGA was only the first step into this direction, yet SuperVGA is not a standard, but a marketing name for is essentially each video adapters' vendor tried to do in the 90s - building an extension upon VGA. All of these vendors did that without creating a unified standard, like with VGA, which ensured everyone are conforming to well-known and expected video hardware behavior. To try to cope with the dire situation, the VBE (Video BIOS extensions) standard was created to help BIOS and operating system vendors to be able to get high resolution framebuffer from any hardware that complied to the standard. When UEFI came along, the vendors agreed to create the Graphics output protocol (known as UEFI GOP), to provide the same set of features that VBE had, but now is usable from 64-bit kernel code as long as the kernel didn't shutdown the UEFI services (which it really should do) after completing the boot process.

Then how does it all apply to the subsystem?

Glad you asked! Since hardware framebuffers are still relevant today, we use them to put pixels so the video encoder of a GPU can convert these bits into light, so you could actually see a picture from a computer screen. Each GPU implements its own internal functionality so it might vary from very simple devices (like the QEMU bochs-display device, which is nothing more than framebuffer region and a couple of registers to manage it) to very complex devices, such as bare metal devices (like Intel integrated GPUs, etc).

The Kernel graphics subsystem strives to manage all of these devices in a unified fashion as much as possible. Of course, actual implementations should vary internally in the amount of code to handle the actual device, but all basic API being exposed to userspace is the same.

The role of MMUs and virtual memory

One of the primary goals of the subsystem to is to allow userspace applications, like the WindowServer, to utilize the hardware framebuffers, so we can see the SerenityOS desktop, and to ensure the internal TTY subsystem in the Kernel can use the same framebuffers to put output from kernel virtual consoles when desired to (i.e. the user switched to the Virtual console from another console that is in graphics mode).

The SerenityOS kernel utilizes the MMU and virtual memory in a very neat trick to give the "feel of control" to whoever did the mmap syscall on a DisplayConnector device, while keeping the control to the Kernel to decide who accesses the actual VRAM at a given time. This works by working with the following assumptions:

1. Current usage of mmap is only for direct framebuffer manipulation. This means that if we add support for batch buffers or other objects that should reside in VRAM, this trick can lead to catastrophic incidents with the underlying hardware. This happens to be this way, due to the fact that we essentially can take VRAM access from the WindowServer at anytime we want, without the WindowServer being aware of this so it can still function in the background.
2. We need to know the maximum dimensions of the framebuffers when initializing the device and creating the DisplayConnector device at runtime. This happens because we map all the possible pages of VRAM framebuffer at that time, and also reserve the same amount of pages in usable physical memory space, so we could reserve the contents of VRAM between the switch from graphics mode to console mode and vice-versa.

The actual implementation is quite simple, yet powerful enough to let everyone live comfortably - each DisplayConnector device is backed by a special VMObject (VMObject is the base class for managing virtual memory scenarios easily) that is created when the DisplayConnector device is initialized - we need to find the physical address of the start of the framebuffer and the maximum resource size (this is where PCI BARs play their role, as we can determine with them the physical address by reading their values and also the maximum resource size, by doing a very simple write 1s-and-read trick that was introduced with the PCI bus when it was created). Then when the object is created, the code ensures we reserve for later usage the same amount of pages somewhere else to ensure we preserve the contents of VRAM between the switch from console and graphics mode and vice-versa. The special VMObject is tied to each Memory::Region object, so it can instruct each virtual-to-physical memory mapping to be actually re-mapped to wherever we want in physical address space, therefore, we do not interrupt any userspace application from drawing its pixels to the framebuffer in the background.

Do you plan supporting old VGA adapters?

Given the nature of the user experience SerenityOS strives to deliver to the users, a core requirement from the first day of this project was to only support 32 bit-per-pixel (also known as True-color framebuffer) hardware framebuffers. We do support hardware framebuffers that neglect the alpha-channel (essentially it's a 24 bit-per-pixel), as long as each pixel is aligned to 4 bytes. The QEMU std-vga (bochs-display with VGA capabilities) device was chosen as the first device to be supported in the project, and that was an excellent choice for that time to put up with the said requirement.

This hard requirement is due to the fact that supporting anything besides True-color framebuffers is a waste of time for a new modern kernel. Not only that, but relying on VGA with modern monitors is essentially settling for blurry, badly-shaped graphics on a computer monitor, due to unoptimized resolution scaling with modern screen ratios.

Old VGA adapters are certainly not capable of using high resolution framebuffers when operating in pure native VGA mode (i.e. not operating in an extension mode of the video adapter), therefore, if the Kernel cannot find a suitable framebuffer to work with or a video adapter it has a driver for, then the last resort is to use the old VGA text mode console. Therefore, the SerenityOS kernel will probably never support pure VGA functionality. That technology was good for operating systems in the 90s, but is not usable anymore.

By doing so, we ensure that legacy cruft is not introduced in the Kernel space. This indeed helps keeping the Graphics subsystem lean and flexible to future changes.

What about the Video BIOS Extensions? It can gives high resolution framebuffers without writing native drivers!

As for using Video BIOS extensions - this requires us to be able to call to BIOS 16-bit real mode code. The solutions for these are:

1. Drop to real mode, invoke the BIOS interrupt and return to our kernel.
2. Writing a Real-Mode 16-bit emulator, either in Kernel space or userspace.
3. Use Intel VT-x extensions to simulate a processor running in Real mode.
4. Use the old v8086 mode in x86 processors to get an hardware monitor of 16-bit tasks.

Neither of these options is suitable for us. Dropping to real mode is quite dangerous task, and breaks the concept of memory protection entirely. Writing a real mode emulator is the safest solution, yet can take a not negligible amount of effort to get something usable and correct. Using the hardware options such as Intel VT-x or the v8086 mode are almost equally equivalent to writing an emulator.

We will probably never support using the Video BIOS extensions because of these reasons:

1. Major part of this project is to maximize usability and fun on what we do, and turning into legacy-cruft to temporarily solve a solution is not the right thing to do.
2. VBE is not usable on machines that lack support of BIOS. As of 2022, this increasingly becomes a problem because many PC vendors dropped support for BIOS (known as CSM [Compatibility Support Module] in UEFI terms).
3. VBE is limited to whatever the vendor decided to hardcode in the OptionROM of the video adapter, which means it can limit us to a small set of resolutions and bits-per-pixel settings, some of these settings are not convenient for us, nor suitable for our needs.
4. VBE lacks the support of detecting if the screen actually supports the resolution settings, which means that the operating system has to use other methods to determine if screen output is working properly (e.g. waiting for a couple of seconds for user confirmation on the selected settings). This is because VBE lacks support of getting the screen EDID because most of the time, the EDID resides in a ROM in the computer screen, which is inaccessible without using specific methods to extract it (via the Display Data Channel), which are not encoded or implemented in the PCI OptionROM of the device. This is in contrast to native drivers which are able to do this, and VGA, that never relied on such methods and instead relied on all video adapters and computer screen to use an well-known specification-defined display modes.

What are the native drivers that are included in the kernel? what type of configurations are supported?

The kernel can be configured to operate in the following conditions:

1. Fully-enable the graphics subsystem, initialize every device being supported.
2. Only use the pre-initialized framebuffer from the bootloader, don't initialize anything else.
3. Don't use any framebuffer, don't initialize any device.

By default, we try to fully-initialize the graphics subsystem, which means we iterate over all PCI devices, searching for VGA compatible devices or Display Controller devices.

We currently natively support QEMU std-vga (and bochs-display) device, VirtIO GPU, VMWare SVGA II adapter, and Intel Graphics (Gen 4 only). We try our best to avoid using a pre-initialized framebuffer, so if we detect any of the said devices, we simply ignore the pre-initialized framebuffer from the bootloader.

The user can choose to use a different condition of the Graphics subsystem, but hardware limitations such as lack of supported hardware can either lead the Kernel to use a pre-initialized framebuffer or completely "abandon" graphics usage (as was mentioned in third condition), making the system usable only through a VGA 80x25 text mode console.

Userspace APIs

Unified Graphics IOCTLs

All graphics ioctls are currently unified and being implemented in one final method of the DisplayDevice class, to keep implementation consistent as much as possible.

Syscalls

The read and write syscalls are not supported and probably will never be. In the transition period from the old framebuffer code in the Kernel to the current design, the mmap syscall was quite dangerous and did not handle multiple userspace programs trying to use it on one device. Since that was resolved and mmap can be used safely, read and write syscalls are no longer needed and are considered obsolete for this device because no userspace program in Serenity will ever need to use them, or test them at the very least.

The ioctl syscall is used to control the DisplayConnector device - to invoke changing of the current mode-set of a framebuffer, flush the framebuffer, etc.

Major and minor numbering

The major number is fixed at 226. Minor number is allocated incrementally as instances are initialized.