This only tests "can it be parsed", but the goal of this commit is to
provide a test framework that can be built upon :)
The conformance tests are downloaded, compiled* and installed only if
the INCLUDE_WASM_SPEC_TESTS cmake option is enabled.
(*) Since we do not yet have a wast parser, the compilation is delegated
to an external tool from binaryen, `wasm-as`, which is required for the
test suite download/install to succeed.
This *does* run the tests in CI, but it currently does not include the
spec conformance tests.
This had very bad interactions with ccache, often leading to rebuilds
with 100% cache misses, etc. Ali says it wasn't that big of a speedup
in the end anyway, so let's not bother with it.
We can always bring it back in the future if it seems like a good idea.
This program turns a description of a state machine that takes its input
byte-by-byte into C++ code. The state machine is described in a custom
format as specified below:
```
// Comments are started by two slashes, and cause the rest of the line
// to be ignored
@name ExampleStateMachine // sets the name of the generated class
@namespace Test // sets the namespace (optional)
@begin Begin // sets the state the parser will start in
// The rest of the file contains one or more states and an optional
// @anywhere directive. Each of these is a curly bracket delimited set
// of state transitions. State transitions contain a selector, the
// literal "=>" and a (new_state, action) tuple. Examples:
// 0x0a => (Begin, PrintLine)
// [0x00..0x1f] => (_, Warn) // '_' means no change
// [0x41..0x5a] => (BeginWord, _) // '_' means no action
// Rules common to all states. These take precedence over rules in the
// specific states.
@anywhere {
0x0a => (Begin, PrintLine)
[0x00..0x1f] => (_, Warn)
}
Begin {
[0x41..0x5a] => (Word, _)
[0x61..0x7a] => (Word, _)
// For missing values, the transition (_, _) is implied
}
Word {
// The entry action is run when we transition to this state from a
// *different* state. @anywhere can't have this
@entry IncreaseWordCount
0x09 => (Begin, _)
0x20 => (Begin, _)
// The exit action is run before we transition to any *other* state
// from here. @anywhere can't have this
@exit EndOfWord
}
```
The generated code consists of a single class which takes a
`Function<Action, u8>` as a parameter in its constructor. This gets
called whenever an action is to be done. This is because some input
might not produce an action, but others might produce up to 3 (exit,
state transition, entry). The actions allow us to build a more
advanced parser over the simple state machine.
The sole public method, `void advance(u8)`, handles the input
byte-by-byte, managing the state changes and requesting the appropriate
Action from the handler.
Internally, the state transitions are resolved via a lookup table. This
is a bit wasteful for more complex state machines, therefore the
generator is designed to be easily extendable with a switch-based
resolver; only the private `lookup_state_transition` method needs to be
re-implemented.
My goal for this tool is to use it for implementing a standard-compliant
ANSI escape sequence parser for LibVT, as described on
<https://vt100.net/emu/dec_ansi_parser>
Make messages which should be fatal, actually fail the build.
- FATAL is not a valid mode keyword. The full list is available in the
docs: https://cmake.org/cmake/help/v3.19/command/message.html
- SEND_ERROR doesn't immediately stop processing, FATAL_ERROR does.
We should immediately stop if the Toolchain is not present.
- The app icon size validation was just a WARNING that is easy to
overlook. We should promote it to a FATAL_ERROR so that people will
not overlook the issue when adding a new application. We can only make
the small icon message FATAL_ERROR, as there is currently one
violation of the medium app icon validation.
With the goal of centralizing all tests in the system, this is a
first step to establish a Tests sub-tree. It will contain all of
the unit tests and test harnesses for the various components in the
system.
When building libraries on macOS they'd be missing the SONAME
attribute which causes the linker to embed relative paths into
other libraries and executables:
Dynamic section at offset 0x52794 contains 28 entries:
Type Name/Value
(NEEDED) Shared library: [libgcc_s.so]
(NEEDED) Shared library: [Userland/Libraries/LibCrypt/libcrypt.so]
(NEEDED) Shared library: [Userland/Libraries/LibCrypto/libcrypto.so]
(NEEDED) Shared library: [Userland/Libraries/LibC/libc.so]
(NEEDED) Shared library: [libsystem.so]
(NEEDED) Shared library: [libm.so]
(NEEDED) Shared library: [libc.so]
The dynamic linker then fails to load those libraries which makes
the system unbootable.
Make this stuff a bit easier to maintain by using the
root level variables to build up the Toolchain paths.
Also leave a note for future editors of BuildIt.sh to
give them warning about the other changes they'll need
to make.
While this has a rather significant impact for me, it appears to have
very minimal build time improvements (or in some cases, regressions).
Also appears to cause some issues when building on macOS.
So disable it by default, but leave the option so people that get
something out of it (seems to mostly be a case of "is reading the
headers fast enough") can turn it on for their builds.
Until we get the goodness that C++ modules are supposed to be, let's try
to shave off some parse time using precompiled headers.
This commit only adds some very common AK headers, only to binaries,
libraries and the kernel (tests are not covered due to incompatibility
with AK/TestSuite.h).
This option is on by default, but can be disabled by passing
`-DPRECOMPILE_COMMON_HEADERS=OFF` to cmake, which will disable all
header precompilations.
This makes the build about 30 seconds faster on my machine (about 7%).
Problem:
- Newer versions of clang (ToT) have a similar `-Wliteral-suffix`
warning as GCC. A previous commit enabled it for all compilers. This
needs to be silenced for the entire build, but it currently only is
silenced for some directories.
Solution:
- Move the `-Wno-literal-suffix` option up in the CMakeLists.txt so
that it gets applied everywhere.
Problem:
- There are redundant options being set for some directories.
- Clang ToT fails to compile the project.
Solution:
- Remove redundancies.
- Fix clang error list.
This warning informs of float-to-double conversions. The best solution
seems to be to do math *either* in 32-bit *or* in 64-bit, and only to
cross over when absolutely necessary.
This flag warns on classes which have `virtual` functions but do not
have a `virtual` destructor.
This patch adds both the flag and missing destructors. The access level
of the destructors was determined by a two rules of thumb:
1. A destructor should have a similar or lower access level to that of a
constructor.
2. Having a `private` destructor implicitly deletes the default
constructor, which is probably undesirable for "interface" types
(classes with only virtual functions and no data).
In short, most of the added destructors are `protected`, unless the
compiler complained about access.
The following warnings do not occur anywhere in the codebase and so
enabling them is effectivly free:
* `-Wcast-align`
* `-Wduplicated-cond`
* `-Wformat=2`
* `-Wlogical-op`
* `-Wmisleading-indentation`
* `-Wunused`
These are taken as a strict subset of the list in #5487.
install-ports copys the necessary files from Ports/ to /usr/Ports. Also
refactor the compiler and destiation variables from .port_include.sh
into .hosted_defs.sh. .hosted_defs.sh does not exists when ports are
built in serenity
Prior to this patch there was some long line of unreadable compiler
options. Now the long lines are deduplicated and there is only one
option per line to ease reading/maintenance.
Build ELF executables with a zero length `GNU_STACK`
program header flagged non-executable.
The stack is never executable on SerenityOS regardless
of whether the `GNU_STACK` header is specified.
Specifically defining this header is more explicit,
as absence of this header implies an executable stack
on other systems (Linux).
These tests were never built for the serenity target. Move their Lagom
build steps to the Lagom CMakeLists.txt, and add serenity build steps
for them. Also, fix the build errors when building them with the
serenity cross-compiler :^)
A new operator, operator""sv was added as of C++17 to support
string_view literals. This allows string_views to be constructed
from string literals and with no runtime cost to find the string
length.
See: https://en.cppreference.com/w/cpp/string/basic_string_view/operator%22%22sv
This change implements that functionality in AK::StringView.
We do have to suppress some warnings about implementing reserved
operators as we are essentially implementing STL functions in AK
as we have no STL :).
For some reason I don't yet understand, building the kernel with -O2
produces a way-too-large kernel on some people's systems.
Since there are some really nice performance benefits from -O2 in
userspace, let's do a compromise and build Userland with -O2 but
put Kernel back into the -Os box for now.
Our TLS implementation relies on the TLS model being "initial-exec".
We previously enforced this by adding the '-ftls-model=initial-exec'
flag in the root CmakeLists file, but that did not affect ports - So
now we put that flag in the gcc spec files.
Closes#5366
This option causes GCC to generate code to prevent "stack clash" style
attacks where a very large stack allocation is used in to jump over the
stack guard page and into whatever's next to it.
To support this, I had to reorganize the "load_elf" function into two
passes. First we map all the dynamic objects, to get their symbols
into the global lookup table. Then we link all the dynamic objects.
So many read-only GOT's! :^)
The dynamic loader will now mark RELRO segments read-only after
performing relocations. This is pretty cool!
Note that this only applies to main executables so far,.
RELRO support for shared libraries will require some reorganizing
of the dynamic loader.
This removes some hard references to the toolchain, some unnecessary
uses of an external install command, and disables a -Werror flag (for
the time being) - only if run inside serenity.
With this, we can build and link the kernel :^)
Running 'ninja install && ninja image && ninja run` is kind of
annoying. I got tired, and came up with this instead, which does the
right thing and I don't have to type out the incantation.
KASAN is a dynamic analysis tool that finds memory errors. It focuses
mostly on finding use-after-free and out-of-bound read/writes bugs.
KASAN works by allocating a "shadow memory" region which is used to store
whether each byte of memory is safe to access. The compiler then instruments
the kernel code and a check is inserted which validates the state of the
shadow memory region on every memory access (load or store).
To fully integrate KASAN into the SerenityOS kernel we need to:
a) Implement the KASAN interface to intercept the injected loads/stores.
void __asan_load*(address);
void __asan_store(address);
b) Setup KASAN region and determine the shadow memory offset + translation.
This might be challenging since Serenity is only 32bit at this time.
Ex: Linux implements kernel address -> shadow address translation like:
static inline void *kasan_mem_to_shadow(const void *addr)
{
return ((unsigned long)addr >> KASAN_SHADOW_SCALE_SHIFT)
+ KASAN_SHADOW_OFFSET;
}
c) Integrating KASAN with Kernel allocators.
The kernel allocators need to be taught how to record allocation state
in the shadow memory region.
This commit only implements the initial steps of this long process:
- A new (default OFF) CMake build flag `ENABLE_KERNEL_ADDRESS_SANITIZER`
- Stubs out enough of the KASAN interface to allow the Kernel to link clean.
Currently the KASAN kernel crashes on boot (triple fault because of the crash
in strlen other sanitizer are seeing) but the goal here is to just get started,
and this should help others jump in and continue making progress on KASAN.
References:
* ASAN Paper: https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/37752.pdf
* KASAN Docs: https://github.com/google/kasan
* NetBSD KASAN Blog: https://blog.netbsd.org/tnf/entry/kernel_address_sanitizer_part_3
* LWN KASAN Article: https://lwn.net/Articles/612153/
* Tracking Issue #5351
This is an external file from https://pci-ids.ucw.cz that's being updated
daily, which was imported a while ago but probably shouldn't live in the
SerenityOS repository in the first place (or else would need manual
maintenance). The legal aspects of redistributing this file as we
currently do are not quite clear to me, they require either GPL (version
2 or later) or 3-clause BSD - Serenity is 2-clause BSD...
The current version we use is 2019.08.08, so quite outdated - and while
most of these devices are obviously not supported, we're still capable
of *listing* them, so having an up-to-date version with recent additions
and fixes would be nice.
This updates the root CMakeLists.txt to check for existence of the file
and download it if not found - effectively on every fresh build. Do note
that this is not a critical file, and the system runs just fine should
this ever fail. :^)
This achieves two things:
- Programs can now intentionally perform arbitrary syscalls by calling
syscall(). This allows us to work on things like syscall fuzzing.
- It restricts the ability of userspace to make syscalls to a single
4KB page of code. In order to call the kernel directly, an attacker
must now locate this page and call through it.
This was done with the help of several scripts, I dump them here to
easily find them later:
awk '/#ifdef/ { print "#cmakedefine01 "$2 }' AK/Debug.h.in
for debug_macro in $(awk '/#ifdef/ { print $2 }' AK/Debug.h.in)
do
find . \( -name '*.cpp' -o -name '*.h' -o -name '*.in' \) -not -path './Toolchain/*' -not -path './Build/*' -exec sed -i -E 's/#ifdef '$debug_macro'/#if '$debug_macro'/' {} \;
done
# Remember to remove WRAPPER_GERNERATOR_DEBUG from the list.
awk '/#cmake/ { print "set("$2" ON)" }' AK/Debug.h.in
Else, there's tons of "-- Set runtime path of" spam at build time,
with apparently no way of disabling the build noise other than turning
of rpaths. If the dynamic loader uses them at some point, we probably
want to set them through cflags/ldflags instead of through cmake's
built-in thing anyways, for that reason.
Modify the user mode runtime to insert stack canaries to find stack corruptions.
The `-fstack-protector-strong` variant was chosen because it catches more
issues than vanilla `-fstack-protector`, but doesn't have substantial
performance impact like `-fstack-protector-all`.
Details:
-fstack-protector enables stack protection for vulnerable functions that contain:
* A character array larger than 8 bytes.
* An 8-bit integer array larger than 8 bytes.
* A call to alloca() with either a variable size or a constant size bigger than 8 bytes.
-fstack-protector-strong enables stack protection for vulnerable functions that contain:
* An array of any size and type.
* A call to alloca().
* A local variable that has its address taken.
Example of it catching corrupting in the `stack-smash` test:
```
courage ~ $ ./user/Tests/LibC/stack-smash
[+] Starting the stack smash ...
Error: Stack protector failure, stack smashing detected!
Shell: Job 1 (/usr/Tests/LibC/stack-smash) Aborted
```
RTTI is still disabled for the Kernel, and for the Dynamic Loader. This
allows for much less awkward navigation of class heirarchies in LibCore,
LibGUI, LibWeb, and LibJS (eventually). Measured RootFS size increase
was < 1%, and libgui.so binary size was ~3.3%. The small binary size
increase here seems worth it :^)
* Add SERENITY_ARCH option to CMake for selecting the target toolchain
* Port all build scripts but continue to use i686
* Update GitHub Actions cache to include BuildIt.sh
Previosuly, generation of the SONAME attribute was disabled.
This caused libraries to have relative paths in DT_NEEDED attributes
(e.g "Libraries/libcore.so" instead of just "libcore.so"),
which caused build errors when the working directory during build was
not $SERENITY_ROOT/Build.
This caused the build of ports that use libraries other than libc.so
to fail (e.g the nesalizer port).
Closes#4457
We now configure the gcc spec files to use a different crt files for
static & PIE binaries.
This relieves us from the need to explicitly specify the desired crt0
file in cmake scripts.
Problem:
- These utility functions are only used in `AK`, but are being defined
in the top-level. This clutters the top-level.
Solution:
- Move the utility functions to `Meta/CMake/utils.cmake` and include
where needed.
- Also, move `all_the_debug_macros.cmake` into `Meta/CMake` directory
to consolidate the location of `*.cmake` script files.
Problem:
- Modifying CXXFLAGS directly is an old CMake style.
- The giant and ever-growing list of `*_DEBUG` macros clutters the
top-level CMakeLists.txt.
Solution:
- Use the more current `add_compile_definitions` function.
- Sort all the debug options so that they are easy to view.
- Move the `*_DEBUG` macros to their own file which can be included
directly.
Problem:
- Functions are duplicated in [PBM,PGM,PPM]Loader class
implementations. They are functionally equivalent. This does not
follow the DRY (Don't Repeat Yourself) principle.
Solution:
- Factor out the common functions into a separate file.
- Refactor common code to generic functions.
- Change `PPM_DEBUG` macro to be `PORTABLE_IMAGE_LOADER_DEBUG` to work
with all the supported types. This requires adding the image type to
the debug log messages for easier debugging.
Problem:
- Appending to CMAKE_CXX_FLAGS for everything is cumbersome.
Solution:
- Use the `add_compile_options` built-in function to handle adding
compiler options (and even de-duplicating).
Problem:
- Setting `CMAKE_CXX_FLAGS` directly to effect the version of the C++
standard being used is no longer the recommended best practice.
Solution:
- Set C++20 mode in the compiler by setting `CMAKE_CXX_STANDARD`.
- Force the build system generator not to fallback to the latest
standard supported by the compiler by enabling
`CMAKE_CXX_STANDARD_REQUIRED`. This shouldn't ever be a problem
though since the toolchain is tightly controlled.
- Disable GNU compiler extensions by disabling `CMAKE_CXX_EXTENSIONS`
to preserve the previous flags.
This new subsystem is somewhat replacing the IDE disk code we had with a
new flexible design.
StorageDevice is a generic class that represent a generic storage
device. It is meant that specific storage hardware will override the
interface. StorageController is a generic class that represent
a storage controller that can be found in a machine.
The IDEController class governs two IDEChannels. An IDEChannel is
responsible to manage the master & slave devices of the channel,
therefore an IDEChannel is an IRQHandler.
New serenity_app() targets can be defined which allows application
icons to be emedded directly into the executable. The embedded
icons will then be used when creating an icon for that file in
LibGUI.
This patch replaces the UI-from-JSON mechanism with a more
human-friendly DSL.
The current implementation simply converts the GML into a JSON object
that can be consumed by GUI::Widget::load_from_json(). The parser is
not very helpful if you make a mistake.
The language offers a very simple way to instantiate any registered
Core::Object class by simply saying @ClassName
@GUI::Label {
text: "Hello friends!"
tooltip: ":^)"
}
Layouts are Core::Objects and can be assigned to the "layout" property:
@GUI::Widget {
layout: @GUI::VerticalBoxLayout {
spacing: 2
margins: [8, 8, 8, 8]
}
}
And finally, child objects are simply nested within their parent:
@GUI::Widget {
layout: @GUI::HorizontalBoxLayout {
}
@GUI::Button {
text: "OK"
}
@GUI::Button {
text: "Cancel"
}
}
This feels a *lot* more pleasant to write than the JSON we had. The fact
that no new code was being written with the JSON mechanism was pretty
telling, so let's approach this with developer convenience in mind. :^)
We need to account for how many shared lock instances the current
thread owns, so that we can properly release such references when
yielding execution.
We also need to release the process lock when donating.
The dynamic loader exists as /usr/lib/Loader.so and is loaded by the
kernel when ET_DYN programs are executed.
The dynamic loader is responsible for loading the dependencies of the
main program, allocating TLS storage, preparing all loaded objects for
execution and finally jumping to the entry of the main program.
This prevents zombies created by multi-threaded applications and brings
our model back to closer to what other OSs do.
This also means that SIGSTOP needs to halt all threads, and SIGCONT needs
to resume those threads.
This changes the Thread::wait_on function to not enable interrupts
upon leaving, which caused some problems with page fault handlers
and in other situations. It may now be called from critical
sections, with interrupts enabled or disabled, and returns to the
same state.
This also requires some fixes to Lock. To aid debugging, a new
define LOCK_DEBUG is added that enables checking for Lock leaks
upon finalization of a Thread.
This commit is a mix of several commits, squashed into one because the
commits before 'Move regex to own Library and fix all the broken stuff'
were not fixable in any elegant way.
The commits are listed below for "historical" purposes:
- AK: Add options/flags and Errors for regular expressions
Flags can be provided for any possible flavour by adding a new scoped enum.
Handling of flags is done by templated Options class and the overloaded
'|' and '&' operators.
- AK: Add Lexer for regular expressions
The lexer parses the input and extracts tokens needed to parse a regular
expression.
- AK: Add regex Parser and PosixExtendedParser
This patchset adds a abstract parser class that can be derived to implement
different parsers. A parser produces bytecode to be executed within the
regex matcher.
- AK: Add regex matcher
This patchset adds an regex matcher based on the principles of the T-REX VM.
The bytecode pruduced by the respective Parser is put into the matcher and
the VM will recursively execute the bytecode according to the available OpCodes.
Possible improvement: the recursion could be replaced by multi threading capabilities.
To match a Regular expression, e.g. for the Posix standard regular expression matcher
use the following API:
```
Pattern<PosixExtendedParser> pattern("^.*$");
auto result = pattern.match("Well, hello friends!\nHello World!"); // Match whole needle
EXPECT(result.count == 1);
EXPECT(result.matches.at(0).view.starts_with("Well"));
EXPECT(result.matches.at(0).view.end() == "!");
result = pattern.match("Well, hello friends!\nHello World!", PosixFlags::Multiline); // Match line by line
EXPECT(result.count == 2);
EXPECT(result.matches.at(0).view == "Well, hello friends!");
EXPECT(result.matches.at(1).view == "Hello World!");
EXPECT(pattern.has_match("Well,....")); // Just check if match without a result, which saves some resources.
```
- AK: Rework regex to work with opcodes objects
This patchsets reworks the matcher to work on a more structured base.
For that an abstract OpCode class and derived classes for the specific
OpCodes have been added. The respective opcode logic is contained in
each respective execute() method.
- AK: Add benchmark for regex
- AK: Some optimization in regex for runtime and memory
- LibRegex: Move regex to own Library and fix all the broken stuff
Now regex works again and grep utility is also in place for testing.
This commit also fixes the use of regex.h in C by making `regex_t`
an opaque (-ish) type, which makes its behaviour consistent between
C and C++ compilers.
Previously, <regex.h> would've blown C compilers up, and even if it
didn't, would've caused a leak in C code, and not in C++ code (due to
the existence of `OwnPtr` inside the struct).
To make this whole ordeal easier to deal with (for now), this pulls the
definitions of `reg*()` into LibRegex.
pros:
- The circular dependency between LibC and LibRegex is broken
- Eaiser to test (without accidentally pulling in the host's libc!)
cons:
- Using any of the regex.h functions will require the user to link -lregex
- The symbols will be missing from libc, which will be a big surprise
down the line (especially with shared libs).
Co-Authored-By: Ali Mohammad Pur <ali.mpfard@gmail.com>
Problem:
- CMake is not outputting `compile_commands.json`.
- `compile_commands.json` is used by build integration tooling such as
`clang-tidy`.
Solution:
- Enable `CMAKE_EXPORT_COMPILE_COMMANDS` option so that the file is
output.
This is our first client of the new JSON GUI declaration thingy.
The skeleton of the TextEditor app GUI is now declared separately from
the C++ logic, and we use the Core::Object::name() of widgets to locate
them once they have been instantiated by the GUI builder.
I know, the tags don't actually matter. However, clang warns by default,
and instead of disabling the warning for clang I'd rather enable the warning
for gcc.
