Our TLS implementation relies on the TLS model being "initial-exec".
We previously enforced this by adding the '-ftls-model=initial-exec'
flag in the root CmakeLists file, but that did not affect ports - So
now we put that flag in the gcc spec files.
Closes#5366
realpath(1) is specific to coreutils and its behavior can be had
with readlink -f
Create the Toolchain Build directory if it doesn't exist before
calling readlink, since realpath(3) on at least OpenBSD will error
on a non-existent path
The current version of our Python port (3.6.0) is over four years old by
now and has (or had, I haven't actually tried it in a while) some
limitations - time for an upgrade! The latest Python release is 3.9.1,
so I used that version. It's a from-scratch port, no patches are taken
from the previous port to ensure the smallest possible amount of code is
patched. The BuildPython.sh script is useful so I kept it, with some
tweaks. I added a short document explaining each patch to ease judging
their underlying problem and necessity in the future.
Compared to the old Python port, this one does support both the time
module as well as threading (at least _thread) just fine. Importing
modules written in C (everything in /usr/local/lib/python3.9/lib-dynload)
currently asserts in Serenity's dynamic loader, which is unfortunate but
probably solvable. Possibly related to #4642. I didn't try building
Python statically, which might be one possibility to circumvent this
issue.
I also renamed the directory to just "python3", which is analogous to
the Python 3.x package most Linux distributions provide. That implicitly
means that we likely will not support multiple versions of the Python
port at any given time, but again, neither do many other systems by
default. Recent versions are usually backwards compatible anyway though,
so having the latest shouldn't be a problem.
On the other hand bumping the version should now be be as simple as
updating the variables in version.sh, given that no new patches are
required.
These core modules to currently not build - I chose to ignore that for
now rather than adding more patches to make them work somehow, which
means they're fully unavailable. This should probably be fixed in
Serenity itself.
_ctypes, _decimal, _socket, mmap, resource, termios
These optional modules requiring 3rd-party dependencies do currently not
build (even with depends="ncurses openssl zlib"). Especially the absence
of a readline port makes the REPL a bit painful to use. :^)
_bz2, _curses, _curses_panel, _dbm, _gdbm, _hashlib, _lzma, _sqlite3,
_ssl, _tkinter, _uuid, nis, ossaudiodev, readline, spwd, zlib
I did some work on LibC and LibM beforehand to add at least stubs of
missing required functions, it still encounters an ASSERT_NOT_REACHED()
/ TODO() every now and then, notably frexp() (implementations of that
can be found online easily if you want to get that working right now).
But then again that's our fault and not this port's. :^)
We now configure gcc to always use the -fno-exceptions flag.
This does not affect our code since we do not use exceptions, and also
fixes the gcc port.
RTTI is still disabled for the Kernel, and for the Dynamic Loader. This
allows for much less awkward navigation of class heirarchies in LibCore,
LibGUI, LibWeb, and LibJS (eventually). Measured RootFS size increase
was < 1%, and libgui.so binary size was ~3.3%. The small binary size
increase here seems worth it :^)
* Add SERENITY_ARCH option to CMake for selecting the target toolchain
* Port all build scripts but continue to use i686
* Update GitHub Actions cache to include BuildIt.sh
A good number of contributors use macOS. However, we have a bit of
a tendency of breaking the macOS build without realising it.
Luckily, GitHub Actions does actually supply macOS environments,
so let's use it.
We now configure the gcc spec files to use a different crt files for
static & PIE binaries.
This relieves us from the need to explicitly specify the desired crt0
file in cmake scripts.
This is necessary because cache reusability will be determined by Github Actions.
Note that we only cache if explicitly asked to do so,
which only happens on Github Actions.
When libstdc++ was added in 4977fd22b8, just calling
'make install' was the easiest way to install the headers. And the headers are all
that is needed for libstdc++ to determine the ABI. Since then, BuildIt.sh was
rewritten again and again, and somehow everyone just silently assumed that
libstdc++ also depends on libc.a and libm.a, because surely it does?
Turns out, it doesn't! This massively reduces the dependencies of libstdc++,
hopefully meaning that the Toolchain doesn't need to be rebuilt so often on Travis.
Furthermore, the old method of trying to determine the dependency tree with
bash/grep/etc. has finally broken anyways:
https://travis-ci.com/github/SerenityOS/serenity/builds/179805569#L567
In summary, this should eliminate most of the Toolchain rebuilds on Travis,
and therefore make Travis build blazingly fast! :^)
./configure generates about 3500 lines in a few seconds. Noone will ever read
those lines and they make loading the Travis webpage slower. And if there is
ever a problem, it will be because the Travis base image changed (which happens
only rarely) in a way that interferes with compiling gcc (which is incredibly
unlikely), or we update gcc (which happens very rarely) and gcc doesn't like
the Travis iamge (which again is incredibly unlikely). In all of these cases,
finding the culprit will be self-evident.
Empirically, every single push or PR has to download *and then upload*
about 3.6 GiB of "cache stuff", which takes up about 400 seconds:
https://travis-ci.com/github/SerenityOS/serenity/builds/177500795
On every single push/PR! No matter what!
Those 3.6 GB consist of:
- 3.2 GB Toolchain cache (around 260 MB per compressed item)
- 0.4 GB ccache, but is capped at 0.5 GB: https://travis-ci.com/github/BenWiederhake/serenity/builds/177528549
- (And 200 KB for some weird debian package? Dunno.)
Investigating in the size, the Toolchain consists mostly of *DEBUG SYMBOLS IN
THE COMPILER BINARIES* which comically misses the point. If we ever run into
compiler crashes, any stacktrace would be lost anyway as soon as the Travis VM
shuts down. Furthermore, Travis will only ever compile Serenity itself, and
Serenity forbids C in it's Contribution Guidelines. That's another 20 MB we
don't need to cache.
Stripping the binaries and deleting the C compiler reduces the uncompressed size
from 1200 MB down to 220 MB. The compressed size gets reduced from 260 MB to 70MB.
That's a reduction of 73%.
It'll take a while until the 'old' toolchains get deleted.
I guess it'll take less than a week.
From that point onward, the Travis cache will be 1.2 GB, consisting of:
- 0.7 GB Toolchain cache
- 0.5 GB ccache
- (And that weird 200 KB deb file)
If network speeds are linear, then this should reduce the "cache network
overhead time" from about 400 seconds to about 120 seconds.
tl;dr: Strip unnecessary debug infos, delete an unused files, and speed
everything up by two minutes. (Both Toolchain cache hits and Toolchain rebuilds!)
This change allows users to use CMAKE_GENERATOR=Ninja ./BuildIt.sh
BuildIt.sh assumes the default cmake generator is Make. However,
the user may specify CMAKE_GENERATOR=Ninja, for example, to set the
default generator. Therefore, instead of calling make to build the
LibC target we should call cmake --build to use the correct generated
files.
The SDL port failed to build because the CMake toolchain filed pointed
to the old root. Now the toolchain file assumes that the Root is in
Build/Root.
Additionally, the AK/ and Kernel/ headers need to be installed in the
root too.
We can do away with that shenanigans now that libstdc++ is gone.
Also, simplify the toolchain dependency hash calculation to only depend
on the toolchain build script(s) and the Patches files we use to modify
the toolchain itself.
This is __cxa_guard_acquire, __cxa_guard_release, and __cxa_guard_abort.
We put these symbols in a 'fake' libstdc++ to trick gcc into thinking it
has libstdc++. These symbols are necessary for C++ programs and not C
programs, so, seems file. There's no way to tell gcc that, for example,
the standard lib it should use is libc++ or libc. So, this is what we
have for now.
When threaded code enters a block that is trying to call the constructor
for a block-scope static, the compiler will emit calls to these methods
to handle the "call_once" nature of block-scope statics.
The compiler creates a 64-bit guard variable, which it checks the first
byte of to determine if the variable should be intialized or not.
If the compiler-generated code reads that byte as a 0, it will call
__cxa_guard_acquire to try and be the thread to call the constructor for
the static variable. If the first byte is 1, it will assume that the
variable's constructor was called, and go on to access it.
__cxa_guard_acquire uses one of the 7 implementation defined bytes of
the guard variable as an atomic 8 bit variable. To control a state
machine that lets each entering thread know if they gained
'initialization rights', someone is working on the varaible, someone is
working on the varaible and there's at least one thread waiting for it
to be intialized, or if the variable was initialized and it's time to
access it. We only store a 1 to the byte the compiler looks at in
__cxa_guard_release, and use a futex to handle waiting.
In order to remove libstdc++ completely, we need to give up on their
implementation of abi::__cxa_demangle. The demangler logic will actually
have to be quite complex, and included in both the kernel and userspace.
A definite fixme for the future, to parse the mangled names into real
deal names.
Back in 36ba0a35ee I thought that Travis would
automagically delete theoldest files. Apparently it does not.
Note that no dummy changes are needed, because BuildIt.sh lists itself
as a dependency for the Toolchain. Hooray for something that works!
It didn't feel right to have a "DHCPClient" in a "Servers" directory.
Rename this to Services to better reflect the type of programs we'll
be putting in there.
The toolchain builds just fine without the git repository (tested on
windows and linux). We can skip setting up the repo and apply the
patches directly when we aren't working on the toolchain itself. A
flag `--dev` has been added for cases when git repo is needed. Without
the flag, regular patch is applied. This significantly improves build
times for first time builds.
This should give a significant boost to Travis speeds, because most of the
compile time is spent building the toolchain over and over again.
However, the toolchain (or libc or libm) changes only rarely,
so most rebuilds can skip this step.
The hashing has been put into a separate file to keep it
as decoupled as possible from BuiltIt.sh.
Turns out the reason GCC wasn't as smart about startup code for
shared objects as we hoped is because nobody told it to be :D
Change the STARTFILE_SPEC and ENDFILE_SPEC in gcc/config/serenity.h to
skip crt0.o and to link the S variants of crtbegin
and crtend for shared objects.
Because we're using the crtbegin and crtend from libgcc, also tell
libgcc in libgcc/config.host to compile crtbeginS and crtendS from
crtstuff.c.
Toolchain build makes git repo out of toolchain to allow patching
Fix Makefiles to use new libstdc++
Parameterize BuildIt with default TARGET of i686 but arm is experimental
For python3 cross compilation, a native installation of python3 is
needed. This patch adds a build script for python3 to the toolchain
and informs the user to run that script if the python port is build
and no native python3 with the same major and minor version is
being found.
Added a script to build QEMU from source as part of the Toolchain.
The script content could be in BuildIt.sh but has been put in
a seperate file to make the build optional.
Added PATH=$PATH to sudo calls to hand over the Toolchain's PATH
setup by UseIt.sh. This enabled the script's to use the QEMU
contained in the SerenityOS toolchain.
Deleted old documentation in Meta and replaced it by a new
documentation in the Toolchain folder.
Ports/.port_include.sh, Toolchain/BuildIt.sh, Toolchain/UseIt.sh
have been left largely untouched due to use of Bash-exclusive
functions and variables such as $BASH_SOURCE, pushd and popd.
When we used "make install" in the past, the "install" target would pull
in the library targets as dependencies, and everything got built that way.
Now that we use "install.sh" instead, we have to build things manually.
This makes out-of-tree linking possible. And at the same time, add a
CMakeToolchain.txt file that can be used to build arbitrary cmake-using
applications on Serenity by pointing to the CMAKE_TOOLCHAIN_FILE when
running cmake:
-DCMAKE_TOOLCHAIN_FILE=~/code/serenity/Toolchain/CMakeToolchain.txt
