Problem: cmake cannot handle gzip files (see
https://gitlab.kitware.com/cmake/cmake/-/issues/23054 for more
details).
Instead of downloading the compressed (*.gz) USB and PCI ids,
we not download the raw uncompressed files. The sizes we "loose"
due to downloading such files are meaningless.
This are the file sizes:
```
diego@debian:~/$ ls -lh pci.ids{,.gz} usb.ids{,.gz}
-rw-r--r-- 1 diego diego 1.3M Aug 7 04:15 pci.ids
-rw-r--r-- 1 diego diego 300K Aug 7 04:15 pci.ids.gz
-rw-r--r-- 1 diego diego 700K May 20 22:34 usb.ids
-rw-r--r-- 1 diego diego 245K May 20 22:34 usb.ids.gz
```
Newer cmake's have internal functions to un-compress files. These
functions will work on pure windows - as well as linux. This
eliminates the need to search for external tools (TAR,GZIP,ZIP) - and
helps fixing #9866.
In order to finally fix#9866 we need to decide to bump the cmake
version requirements and remove the checks. If we demand a newer cmake
version, we will loose Ubuntu 20.04 as a build target - as it ships
with CMake 3.16.
For now - we keep compatibility with CMake 3.16 - and only if CMake
3.18 as been found - we use its new functionality.
Remove the Corrosion dependency, and use the now-builtin
add_jakt_executable function from the Jakt install rules to build our
example application.
By using find_package(Jakt), we now have to set ENABLE_JAKT manually on
both serenity and Lagom at the same time, so the preferred method to do
this for now is:
cmake -B Build/superbuild<arch><toolchain> \
-S Meta/CMake/Superbuild \
-DENABLE_JAKT=ON \
-DJAKT_SOURCE_DIR=/path/to/jakt
Where omitting JAKT_SOURCE_DIR will still pull from the main branch of
SerenityOS/jakt. This can be done after runing Meta/serenity.sh run.
The FLAC "spec tests", or rather the test suite by xiph that exercises
weird FLAC features and edge cases, can be found at
https://github.com/ietf-wg-cellar/flac-test-files and is a good
challenge for our FLAC decoder to become more spec compliant. Running
these tests is similar to LibWasm spec tests, you need to pass
INCLUDE_FLAC_SPEC_TESTS to CMake.
As of integrating these tests, 23 out of 63 fail. :yakplus:
The current emoji_txt.cmake does not handle download errors (which were
a common source of issues in the build problems channel) or Unicode
versioning. These are both handled by unicode_data.cmake. Move the
download to unicode_data.cmake so that we can more easily handle next
month's Unicode 15 release.
Instead of manually updating emoji.txt whenever new emoji are added,
we use Unicode's emoji-test.txt to generate emoji.txt on each build,
including only the emojis that Serenity supports at that time.
By using emoji-test.txt, we can also include all forms of each emoji
(fully-qualified, minimally-qualified, and unqualified) which can be
helpful when double-checking how certain forms are handled.
Not sure why that happens and how it worked until now, but we need to be
more precise about the location of PCI and USB IDs when decompressing
them while building the OS.
In commit 02e8f29560 we started exporting
the `CMAKE_INSTALL_*DIR` variables without prefix in order to make
better use of the actual `PREFIX` settings.
However, commands like `file(MAKE_DIRECTORY ...)` don't understand the
GNUInstallDirs way of building paths, so we ended up creating
directories in our main project directory by accident.
Fix that by manually adding the correct prefix onto the path.
In preparation for future refactoring of Lagom, let's use the variables
from GNUInstallDirs as much as possible for the helper macros and other
scripts used by the main build already.
This release brings support for various C++23 constructs like `if
consteval` and multidimensional subscript operators. Vectorization is
now enabled for O2 too, and `-ftrivial-auto-var-init` has been added
which can help us find and prevent security issues coming from
uninitialized variables.
Toolchain/Patches/gcc.patch is now significanly smaller as some unused,
autoconf-generated code has been removed.
Besides a version bump, the following changes have been made to our
toolchain infrastructure:
- LLVM/Clang is now built with -march=native if the host compiler
supports it. An exception to this is CI, as the toolchain cache is
shared among many different machines there.
- The LLVM tarball is not re-extracted if the hash of the applied
patches doesn't differ.
- The patches have been split up into atomic chunks.
- Port-specific patches have been integrated into the main patches,
which will aid in the work towards self-hosting.
- <sysroot>/usr/local/lib is now appended to the linker's search path by
default.
- --pack-dyn-relocs=relr is appended to the linker command line by
default, meaning ports take advantage of RELR relocations without any
patches or additional compiler flags.
The formatting of LLVM port's package.sh has been bothering me, so I
also indented the arguments to the CMake invocation.
We have seen some cases where the build fails for folks, and they are
missing unzip/tar/gzip etc. We can catch some of these in CMake itself,
so lets make sure to handle that uniformly across the build system.
The REQUIRED flag to `find_program` was only added on in CMake 3.18 and
above, so we can't rely on that to actually halt the program execution.
With regular builds, the generated IPC headers exist inside the Build
directory. The path Userland/Services under the build directory is
added to the include path.
For in-system builds the IPC headers are installed at /usr/include/.
To support this, we add /usr/include/Userland/Services to the build path
when building from Hack Studio.
Co-Authored-By: Andrew Kaster <akaster@serenityos.org>
This variable was originally called USE_MOLD_LINKER, but it was changed
to ENABLE_MOLD_LINKER during review to be consistent with other
configuration options. I branched off the commits that added RELR
support before this change, and I failed to update the variable name
there.
While playing around with getting serenity to run on my main desktop
machine I wanted a way of easily updating my physical serenity
partition.
To use it you just need to:
- Create and format your local partition to ext4
- Set `SERENITY_TARGET_INSTALL_PARTITION` to the partition /dev path.
- Run the `install-native-partition` build target.
Example:
$ export SERENITY_TARGET_INSTALL_PARTITION=/dev/nvme1n1p3
$ cd serenity/Build/x86_64
$ ninja install-native-partition
This commit adds support for building the SerenityOS userland with the
new [mold linker].
This is not enabled by default yet; to link using mold, run the
`Toolchain/BuildMold.sh` script to build the latest release of mold, and
set the `ENABLE_MOLD_LINKER` CMake variable to ON. This option relies on
toolchain support that has been added just recently, so you might need
to rebuild your toolchain for mold to work.
[mold linker]: https://github.com/rui314/mold
If this option is set, we will not build all components.
Instead, we include an external CMake file passed in via a variable
named HACKSTUDIO_BUILD_CMAKE_FILE.
This will be used to build serenity components from Hack Studio.
The `--allow-shlib-undefined` option is a bit of a misnomer. It actually
controls whether we should be allowed to have undefined references after
symbols from all dependencies have been resolved, so it applies both to
shared libraries and executables.
LLD defaults to allowing undefined references in shared libraries, but
not in executables. Previously, we had to disable this check for
executables too, as it caused a build failure due to the
LibC-LibPthread-libc++ and the LibCore-LibCrypto circular dependencies.
Now that those have been resolved, we can enable this warning, in the
hopes that it will prevent us from introducing circular libraries and
missing dependencies that might cause unexpected breakage.
There's only two places where we're using the C99 feature of array
designated initalizers. This feature seemingly wasn't included with
C++20 designated initalizers for classes and structs. The only two
places we were using this feature are suitably old and isolated that
it makes sense to just suppress the warning at the usage sites while
discouraging future array designated intializers in new code.
Enable the warning project-wide. It catches when a non-virtual method
creates an overload set with a virtual method. This might cause
surprising overload resolution depending on how the method is invoked.
The Clang implementation of this warning protects against some undefined
pre-processor behavior while ignoring function-like macros. The gcc
implementation also warns on function-like macros, and is therefore
noisy.
These were removed in the Superbuild conversion. Re-add the checks that
make sure that if there's a toolchain update, developers re-build their
toolchain.
gzip -c is supported in both Linux and BSD flavors of gzip. The -o flag
was introduced in a previous commit which is present in OpenBSD, but not
other flavors of Linux. -c will write to stdout which is redirected to
the target files. As a side benefit, we no longer need to copy files
anywhere
OpenBSD gzip does not have the -k flag to keep the original after
extraction. Work around this by copying the original gzip to the dest
and then extracting. A bit of a hack, but only needs to be done for the
first-time or rebuilds
OpenBSD provides crypt in libc, not libcrypt. Adjust if/else to check
for either and proceed accordingly
Remove outdated OpenBSD checks when building the toolchain
This option is already enabled when building Lagom, so let's enable it
for the main build too. We will no longer be surprised by Lagom Clang
CI builds failing while everything compiles locally.
Furthermore, the stronger `-Wsuggest-override` warning is enabled in
this commit, which enforces the use of the `override` keyword in all
classes, not just those which already have some methods marked as
`override`. This works with both GCC and Clang.
This commit updates the Clang toolchain's version to 13.0.0, which comes
with better C++20 support and improved handling of new features by
clang-format. Due to the newly enabled `-Bsymbolic-functions` flag, our
Clang binaries will only be 2-4% slower than if we dynamically linked
them, but we save hundreds of megabytes of disk space.
The `BuildClang.sh` script has been reworked to build the entire
toolchain in just three steps: one for the compiler, one for GNU
binutils, and one for the runtime libraries. This reduces the complexity
of the build script, and will allow us to modify the CI configuration to
only rebuild the libraries when our libc headers change.
Most of the compile flags have been moved out to a separate CMake cache
file, similarly to how the Android and Fuchsia toolchains are
implemented within the LLVM repo. This provides a nicer interface than
the heaps of command-line arguments.
We no longer build separate toolchains for each architecture, as the
same Clang binary can compile code for multiple targets.
The horrible mess that `SERENITY_CLANG_ARCH` was, has been removed in
this commit. Clang happily accepts an `i686-pc-serenity` target triple,
which matches what our GCC toolchain accepts.
Replace the old logic where we would start with a host build, and swap
all the CMake compiler and target variables underneath it to trick
CMake into building for Serenity after we configured and built the Lagom
code generators.
The SuperBuild creates two ExternalProjects, one for Lagom and one for
Serenity. The Serenity project depends on the install stage for the
Lagom build. The SuperBuild also generates a CMakeToolchain file for the
Serenity build to use that replaces the old toolchain file that was only
used for Ports.
To ensure that code generators are rebuilt when core libraries such as
AK and LibCore are modified, developers will need to direct their manual
`ninja` invocations to the SuperBuild's binary directory instead of the
Serenity binary directory.
This commit includes warning coalescing and option style cleanup for the
affected CMakeLists in the Kernel, top level, and runtime support
libraries. A large part of the cleanup is replacing USE_CLANG_TOOLCHAIN
with the proper CMAKE_CXX_COMPILER_ID variable, which will no longer be
confused by a host clang compiler.
This common strategy of having a serenity_option() macro defined in
either the Lagom or top level CMakeLists.txt allows us to do two things:
First, we can more clearly see which options are Serenity-specific,
Lagom-specific, or common between the target and host builds.
Second, it enables the upcoming SuperBuild changes to set() the options
in the SuperBuild's CMake cache and forward each target's options to the
corresponding ExternalProject.
This makes it so we don't need to specify the full path to all the
helper scripts we include() from different places in the codebase and
feels a lot cleaner.
This prevents GCC and Clang from deleting null pointer checks for
optimization purposes. I think we're strictly better off crashing
in those cases instead of the compiler hiding errors from us.
This tells the linker to not combine read-only data and executable code,
instead favoring multiple PT_LOAD headers with more precise permissions.
This greatly reduces the amount of executable pages in all our programs
and libraries.
/usr/lib/libjs.so before:
Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align
LOAD 0x000000 0x00000000 0x00000000 0x2fc77c 0x2fc77c R E 0x1000
LOAD 0x2fc900 0x002fd900 0x002fd900 0x0c708 0x0dd1c RW 0x1000
/usr/lib/libjs.so after:
Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align
LOAD 0x000000 0x00000000 0x00000000 0x80e60 0x80e60 R 0x1000
LOAD 0x081000 0x00081000 0x00081000 0x25f6c9 0x25f6c9 R E 0x1000
LOAD 0x2e1000 0x002e1000 0x002e1000 0x1c27c 0x1c27c R 0x1000
LOAD 0x2fd900 0x002fe900 0x002fe900 0x0c708 0x0dd1c RW 0x1000
As you can see, we go from 0x2fc77c bytes of executable memory down to
0x25f6c9 (a ~20% reduction!) The memory that was previous executable is
now simply read-only instead. :^)
