This is similar to the LibJS test data that resides in
/home/anon/js-tests.
It's more convenient than storing the test programs as raw strings
in the code.
As the parser now flattens out the instructions and inserts synthetic
nesting/structured instructions where needed, we can treat the whole
thing as a simple parsed bytecode stream.
This currently knows how to execute the following instructions:
- unreachable
- nop
- local.get
- local.set
- {i,f}{32,64}.const
- block
- loop
- if/else
- branch / branch_if
- i32_add
- i32_and/or/xor
- i32_ne
This also extends the 'wasm' utility to optionally execute the first
function in the module with optionally user-supplied arguments.
This had very bad interactions with ccache, often leading to rebuilds
with 100% cache misses, etc. Ali says it wasn't that big of a speedup
in the end anyway, so let's not bother with it.
We can always bring it back in the future if it seems like a good idea.
As we removed the support of VBE modesetting that was done by GRUB early
on boot, we need to determine if we can modeset the resolution with our
drivers, and if not, we should enable text mode and ensure that
SystemServer knows about it too.
Also, SystemServer should first check if there's a framebuffer device
node, which is an indication that text mode was not even if it was
requested. Then, if it doesn't find it, it should check what boot_mode
argument the user specified (in case it's self-test). This way if we
try to use bochs-display device (which is not VGA compatible) and
request a text mode, it will not honor the request and will continue
with graphical mode.
Also try to print critical messages with mininum memory allocations
possible.
In LibVT, We make the implementation flexible for kernel-specific
methods that are implemented in ConsoleImpl class.
This commit replaces the former, hand-written parser with a new one that
can be generated automatically according to a state change diagram.
The new `EscapeSequenceParser` class provides a more ergonomic interface
to dealing with escape sequences. This interface has been inspired by
Alacritty's [vte library](https://github.com/alacritty/vte/).
I tried to avoid changing the application logic inside the `Terminal`
class. While this code has not been thoroughly tested, I can't find
regressions in the basic command line utilities or `vttest`.
`Terminal` now displays nicer debug messages when it encounters an
unknown escape sequence. Defensive programming and bounds checks have
been added where we access parameters, and as a result, we can now
endure 4-5 seconds of `cat /dev/urandom`. :D
We generate EscapeSequenceStateMachine.h when building the in-kernel
LibVT, and we assume that the file is already in place when the userland
library is being built. This will probably cause problems later on, but
I can't find a way to do it nicely.
As of ~April 2021 the Meta/build-image-grub.sh script no longer works
for me (on Fedora 34) and fails with the following error:
/usr/sbin/grub2-install: warning: ../grub-core/partmap/msdos.c:403:
your core.img is unusually large. It won't fit in the embedding
area.
/usr/sbin/grub2-install: warning: Embedding is not possible. GRUB
can only be installed in this setup by using blocklists. However,
blocklists are UNRELIABLE and their use is discouraged..
/usr/sbin/grub2-install: error: will not proceed with blocklists.
Changing the size of the boot partition from 32 kiB to 1 MiB (2048
sectors) fixes the issue. This is also described in the following Ubuntu
grub2 bug (as well as 40 duplicates!) from 2012, which suggests the same
fix: https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/1059827
The Arch Linux wiki also uses 1 MiB in their BIOS/MBR examples for
parted: https://wiki.archlinux.org/title/Parted
I'm not sure why this suddenly stopped working, however I was able to
boot with an image created with this change applied.
Ideally we would never allocate under a spinlock, as it has many
performance and potentially functionality (deadlock) pitfalls.
We violate that rule in many places today, but we need a tool to track
them all down and fix them. This change introduces a new macro option
named `KMALLOC_VERIFY_NO_SPINLOCK_HELD` which can catch these
situations at runtime via an assert.
Previously the CMake options for -fsanitize=address, thread and
undefined were gated behind clang, which was unecessary. Only
-fsanitize=fuzzer is clang-only.
Since the operations are already complicated and will become even more
so soon, let's split them into their own files. We can also integrate
the NumberTheory operations that would better fit there into this class
as well.
This commit doesn't change behaviors, but moves the allocation of some
variables into caller classes.
Make messages which should be fatal, actually fail the build.
- FATAL is not a valid mode keyword. The full list is available in the
docs: https://cmake.org/cmake/help/v3.19/command/message.html
- SEND_ERROR doesn't immediately stop processing, FATAL_ERROR does.
We should immediately stop if the Toolchain is not present.
- The app icon size validation was just a WARNING that is easy to
overlook. We should promote it to a FATAL_ERROR so that people will
not overlook the issue when adding a new application. We can only make
the small icon message FATAL_ERROR, as there is currently one
violation of the medium app icon validation.
This commit introduces the ability to parse the document catalog dict,
as well as the page tree and individual pages. Pages obviously aren't
fully parsed, as we won't care about most of the fields until we
start actually rendering PDFs.
One of the primary benefits of the PDF format is laziness. PDFs are
not meant to be parsed all at once, and the same is true for pages.
When a Document is constructed, it builds a map of page number to
object index, but it does not fetch and parse any of the pages. A page
is only parsed when a caller requests that particular page (and is
cached going forwards).
Additionally, this commit also adds an object_cast function which
logs bad casts if DEBUG_PDF is set. Additionally, utility functions
were added to ArrayObject and DictObject to get all types of objects
from the collections to avoid having to manually cast.
This can currently parse a really simple module.
Note that it cannot parse the DataCount section, and it's still missing
almost all of the instructions.
This commit also adds a 'wasm' test utility that tries to parse a given
webassembly binary file.
It currently does nothing but exit when the parse fails, but it's a
start :^)
This currently (obviously) doesn't support any actual 3D hardware,
hence all calls are done via software rendering.
Note that any modern constructs such as shaders are unsupported,
as this driver only implements Fixed Function Pipeline functionality.
The library is split into a base GLContext interface and a software
based renderer implementation of said interface. The global glXXX
functions serve as an OpenGL compatible c-style interface to the
currently bound context instance.
Co-authored-by: Stephan Unverwerth <s.unverwerth@gmx.de>
Index page:
- Change links from "Man 1" to "Section 1"
Section index pages:
- Change title from "1" to "Section 1 - SerenityOS man pages"
- Change links from "foo" to "foo(1)"
Man pages:
- Change title from "foo" to "foo(1) - SerenityOS man pages"
The data structures here were changed from sets to dicts at some point,
with the sets now having different names - this would crash trying to
subtract two dicts.
Previously the directions omitted that you have to specify
`-CMAKE_CXX_COMPILER` when building the Fuzzers. This
would cause all kinds of weird problems at compilation and
link time. You can't specify one or the other, they must
both be pointing at clang in order for things to work as
experted. Fix this by updating the documentation to specify
that the user should specify both the C and CXX compiler explicitly
to be safe, as well as forcing the cmake clang argument handling
to modify the CXX compiler variable instead of the C version.
This should help with getting commit messages tidy before they pass
through CI's commit linter :^)
For this hook to work pre-commit has to be explicitly installed via:
`pre-commit install --hook-type commit-msg`
Just 'github.actor' may or may not be the actual PR author, but
especially since action runs have to be approved for first-time
contributors, it's been giving us a lot of incorrect results.
Most of the existing lint-ing shell scripts have the ability
to only run on the files which have actually changed.
The new port lint-ing script doesn't have this functionality
unfortunately. This forces us to lint ALL the ports on every
single change to any other file in the system if you have
the pre-commit hook setup for your git clone locally.
Instead we can use pre-commit's feature to only run a hook
if certain files have changed to reduce the situations in
which we would run the Meta/lint-ports.py script.
Clang's default constexpr-steps limit is 1048576, which is not enough
for LibGfx's generation of the unicode bidirectional class lookup table
while GCC doesn't have any limit at all, so this patch increases the
limit to an arbitrarily larger value.
As many macros as possible are moved to Macros.h, while the
macros to create a test case are moved to TestCase.h. TestCase is now
the only user-facing header for creating a test case. TestSuite and its
helpers have moved into a .cpp file. Instead of requiring a TEST_MAIN
macro to be instantiated into the test file, a TestMain.cpp file is
provided instead that will be linked against each test. This has the
side effect that, if we wanted to have test cases split across multiple
files, it's as simple as adding them all to the same executable.
The test main should be portable to kernel mode as well, so if
there's a set of tests that should be run in self-test mode in kernel
space, we can accomodate that.
A new serenity_test CMake function streamlines adding a new test with
arguments for the test source file, subdirectory under /usr/Tests to
install the test application and an optional list of libraries to link
against the test application. To accomodate future test where the
provided TestMain.cpp is not suitable (e.g. test-js), a CUSTOM_MAIN
parameter can be passed to the function to not link against the
boilerplate main function.
Using rsync is significantly faster when we're using an existing
image because we only have to copy files which are new or have
been updated. This cuts down the image creation time to about
a second or two assuming an old image exists.
This adds support for re-using and re-sizing existing disk images.
Disk images are checked with e2fsck prior to re-use and a new disk
image is automatically created when that check fails.
This adds support for detecting incorrect version numbers and links
in the ports list.
Also, unlike before it doesn't parse the package.sh script but executes
it instead which allows us to detect syntax errors.
SPDX License Identifiers are a more compact / standardized
way of representing file license information.
See: https://spdx.dev/resources/use/#identifiers
This was done with the `ambr` search and replace tool.
ambr --no-parent-ignore --key-from-file --rep-from-file key.txt rep.txt *
This patch adds a basic Zsh completion script for the commands and
targets provided by Meta/serenity.sh. There's some room for improvement
here, e.g. we could provide completion for available CMake targets -
currently completion stops after serenity.sh <command> <target>.
You can enable it by adding this to your .zshrc before completions are
loaded:
fpath=($SERENITY_SOURCE_DIR/Meta/ShellCompletions/zsh $fpath)
Until we get the goodness that C++ modules are supposed to be, let's try
to shave off some parse time using precompiled headers.
This commit only adds some very common AK headers, only to binaries,
libraries and the kernel (tests are not covered due to incompatibility
with AK/TestSuite.h).
This option is on by default, but can be disabled by passing
`-DPRECOMPILE_COMMON_HEADERS=OFF` to cmake, which will disable all
header precompilations.
This makes the build about 30 seconds faster on my machine (about 7%).
Move LibCompress unit tests to LibCompress/Tests directory and register
them with CMake's add_test. This allows us to run these tests with
ninja test instead of running a separate executable.
Also split the existing tests in 3 test files that better follow the
source code structure (inspired by AK tests).
This was brought up as something that would be useful by
`RealKC` on the discord, and I happened to agree that it
would be useful. Especially given the abundance of Toolchain
changes recently.
The only icons we are currently warning about are designed
and rendered as small icons intentionally, as their only use
is in desktop applets, and thus are exempt to this rule.
This reduces build spam back down to a minimum.
I should have just done this in the first place, back in #4729
This is a very basic implementation that only requests 4096 bytes
of entropy from the host once, but its still high quality entropy
so it should be a good fix for #4490 (boot-time entropy starvation)
for virtualized environments.
Co-authored-by: Sahan <sahan.h.fernando@gmail.com>
This commit includes a lot of small changes and additions needed to
finalize the base implementation of VirtIOQueues and VirtDevices:
* The device specific driver implementation now has to handle setting
up the queues it needs before letting the base device class know it
finised initialization
* Supplying buffers to VirtQueues is now done via ScatterGatherLists
instead of arbitary buffer pointers - this ensures the pointers are
physical and allows us to follow the specification in regards to the
requirement that individual descriptors must point to physically
contiguous buffers. This can be further improved in the future by
implementating support for the Indirect-Descriptors feature (as
defined by the specification) to reduce descriptor usage for very
fragmented buffers.
* When supplying buffers to a VirtQueue the driver must supply a
(temporarily-)unique token (usually the supplied buffer's virtual
address) to ensure the driver can discern which buffer has finished
processing by the device in the case in which the device does not
offer the F_IN_ORDER feature.
* Device drivers now handle queue updates (supplied buffers being
returned from the device) by implementing a single pure virtual
method instead of setting a seperate callback for each queue
* Two new VirtQueue methods were added to allow the device driver
to either discard or get used/returned buffers from the device by
cleanly removing them off the descriptor chain (This also allows
the VirtQueue implementation to reuse those freed descriptors)
This also includes the necessary changes to the VirtIOConsole
implementation to match these interface changes.
Co-authored-by: Sahan <sahan.h.fernando@gmail.com>
This patch allocates a physical page for each of the virtqueues and
memcpys to it when receiving a buffer to get a physical, aligned
contiguous buffer as required by the virtio specification.
Co-authored-by: Sahan <sahan.h.fernando@gmail.com>
Based on pull #3236 by tomuta, this adds helper methods for generic
device initialization, and partily-broken virtqueue helper methods
Co-authored-by: Tom <tomut@yahoo.com>
Co-authored-by: Sahan <sahan.h.fernando@gmail.com>
This change fixes two bugs:
1) If you run `serenity.sh` outside of your serenity git clone, the
`get_top_dir()` function won't be able to auto detect the serenity
root dir and we'll error out. By allowing the script to use an
existing $SERENITY_ROOT vlaue if it exists, we can solve
this problem.
2) If tried to run a command which ends up calling `build_toolchain()`
and you were outside of the root of the project, it should just
fail. Fix this by utilizing `$SERENITY_ROOT`.
I have this symlinked into ~/bin, when looking at the help/usage
it would previously print the fully qualified path to the script
making the help very difficult to read.
install-ports copys the necessary files from Ports/ to /usr/Ports. Also
refactor the compiler and destiation variables from .port_include.sh
into .hosted_defs.sh. .hosted_defs.sh does not exists when ports are
built in serenity
The end goal of this commit is to allow to boot on bare metal with no
PS/2 device connected to the system. It turned out that the original
code relied on the existence of the PS/2 keyboard, so VirtualConsole
called it even though ACPI indicated the there's no i8042 controller on
my real machine because I didn't plug any PS/2 device.
The code is much more flexible, so adding HID support for other type of
hardware (e.g. USB HID) could be much simpler.
Briefly describing the change, we have a new singleton called
HIDManagement, which is responsible to initialize the i8042 controller
if exists, and to enumerate its devices. I also abstracted a bit
things, so now every Human interface device is represented with the
HIDDevice class. Then, there are 2 types of it - the MouseDevice and
KeyboardDevice classes; both are responsible to handle the interface in
the DevFS.
PS2KeyboardDevice, PS2MouseDevice and VMWareMouseDevice classes are
responsible for handling the hardware-specific interface they are
assigned to. Therefore, they are inheriting from the IRQHandler class.
Almost a year after first working on this, it's finally done: an
implementation of Promises for LibJS! :^)
The core functionality is working and closely following the spec [1].
I mostly took the pseudo code and transformed it into C++ - if you read
and understand it, you will know how the spec implements Promises; and
if you read the spec first, the code will look very familiar.
Implemented functions are:
- Promise() constructor
- Promise.prototype.then()
- Promise.prototype.catch()
- Promise.prototype.finally()
- Promise.resolve()
- Promise.reject()
For the tests I added a new function to test-js's global object,
runQueuedPromiseJobs(), which calls vm.run_queued_promise_jobs().
By design, queued jobs normally only run after the script was fully
executed, making it improssible to test handlers in individual test()
calls by default [2].
Subsequent commits include integrations into LibWeb and js(1) -
pretty-printing, running queued promise jobs when necessary.
This has an unusual amount of dbgln() statements, all hidden behind the
PROMISE_DEBUG flag - I'm leaving them in for now as they've been very
useful while debugging this, things can get quite complex with so many
asynchronously executed functions.
I've not extensively explored use of these APIs for promise-based
functionality in LibWeb (fetch(), Notification.requestPermission()
etc.), but we'll get there in due time.
[1]: https://tc39.es/ecma262/#sec-promise-objects
[2]: https://tc39.es/ecma262/#sec-jobs-and-job-queues
This utility traces the route packets take to a user specified host.
QEMU user networking (the type of networking we currently have setup)
does not support sending "real" raw IPv4 packets, and as such we cant
specify a custom TTL value. As a result, this utility will only work
on real hardware, qemu setup with tun networking (requires root) and
other hypervisors that support bridged adapters (VirtualBox/VMWare).
Running the tests will be moved to a separate test command which can
then leverage the availability of different targets and run either unit
tests on the host or the image in QEMU in self-test mode. :^)
- Only call ensure_toolchain for non-lagom targets
- Use host addr2line, we can't expect the i686 toolchain's addr2line to
support the host's binary executable format
- Don't export SERENITY_ARCH and TOOLCHAIN_DIR, don't need them anymore
It is possible to set the run.sh mode via the SERENITY_RUN environment
variable, but the SERENITY_DISK_IMAGE="grub_disk_image" override for
qgrub mode was only checking $1. This makes qgrub mode work via 'ninja
run' without explicitly setting SERENITY_DISK_IMAGE:
SERENITY_RUN=qgrub ninja run
The hierarchy is AHCIController, AHCIPortHandler, AHCIPort and
SATADiskDevice. Each AHCIController has at least one AHCIPortHandler.
An AHCIPortHandler is an interrupt handler that takes care of
enumeration of handled AHCI ports when an interrupt occurs. Each
AHCIPort takes care of one SATADiskDevice, and later on we can add
support for Port multiplier.
When we implement support of Message signalled interrupts, we can spawn
many AHCIPortHandlers, and allow each one of them to be responsible for
a set of AHCIPorts.
This makes them available for use by other language servers.
Also as a bonus, update the Shell language server to discover some
symbols and add go-to-definition functionality :^)
This commit removes the only 3rd party library (and its usages)
in serenity: puff, which is used for deflate decompression. and
replaces it with the existing original serenity implementation
in LibCompress. :^)
This allows us to remove the FAIL_REGEX logic from the CTest invocation
of AK and LibRegex tests, as they will return a non-zero exit code on
failure :^).
Also means that running a failing TestSuite-enabled test with the
run-test-and-shutdown script will actually print that the test failed.
Build a new version of Serenity in CI that doesn't have all the debug
symbols on, or we'd be waiting a very long time to boot.
Insert a TestRunner entry into SystemServer.ini that will run a shell
script that runs tests in /bin and /usr/Tests and shuts down the system
in the new self-test boot mode. Also make sure enough basic services are
started in self-test such that the tests will actually run properly.
These tests were never built for the serenity target. Move their Lagom
build steps to the Lagom CMakeLists.txt, and add serenity build steps
for them. Also, fix the build errors when building them with the
serenity cross-compiler :^)
This is already set in the root CMakeLists.txt as well as here for Clang
(-Wno-user-defined-literals), but was forgotten for GCC which made an
Lagom-only build (cmake ../Meta/Lagom [...]) fail.
This is basically just for consistency, it's quite strange to see
multiple AK container types next to each other, some with and some
without the namespace prefix - we're 'using AK::Foo;' a lot and should
leverage that. :^)
A new operator, operator""sv was added as of C++17 to support
string_view literals. This allows string_views to be constructed
from string literals and with no runtime cost to find the string
length.
See: https://en.cppreference.com/w/cpp/string/basic_string_view/operator%22%22sv
This change implements that functionality in AK::StringView.
We do have to suppress some warnings about implementing reserved
operators as we are essentially implementing STL functions in AK
as we have no STL :).
(...and ASSERT_NOT_REACHED => VERIFY_NOT_REACHED)
Since all of these checks are done in release builds as well,
let's rename them to VERIFY to prevent confusion, as everyone is
used to assertions being compiled out in release.
We can introduce a new ASSERT macro that is specifically for debug
checks, but I'm doing this wholesale conversion first since we've
accumulated thousands of these already, and it's not immediately
obvious which ones are suitable for ASSERT.
This patchset allows the editor to avoid redrawing the entire line when
the changes cause no unrecoverable style updates, and are at the end of
the line (this applies to most normal typing situations).
Cases that this does not resolve:
- When the cursor is not at the end of the buffer
- When a display refresh changes the styles on the already-drawn parts
of the line
- When the prompt has not yet been drawn, or has somehow changed
Fixes#5296.
realpath(1) is specific to coreutils and its behavior can be had
with readlink -f
Create the Toolchain Build directory if it doesn't exist before
calling readlink, since realpath(3) on at least OpenBSD will error
on a non-existent path
Detection broke when we moved from '#ifdef DEBUG_FOO dbgln()' to 'dbgln<DEBUG_FOO>()'.
This patch makes detection more general, which sadly runs into more false-positives.
No rotten code was found, hooray! :^)
This wrapper abstracts the watch_file setup and file handling, and
allows using the watch_file events as part of the event loop via the
Core::Notifier class.
Also renames the existing DirectoryWatcher class to BlockingFileWatcher,
and adds support for the Modified mode in this class.
Arbitrarily split up to make git bisect easier.
These unnecessary #include's were found by combining an automated tool (which
determined likely candidates) and some brain power (which decided whether
the #include is also semantically superfluous).
This caused some confusion: Apparently, clang has no trouble overriding Shell's
main, and this issue only surfaced when I tried to build the fuzzers with
wrong configuration (i.e., without the clang-injected 'main').
The diff is suggested by, and work of, @alimpfard.
This used to be in Kernel/, next to the build-root-filesystem.sh script,
which was then moved to Meta/ during the transition to CMake but has the
working directory set to Build/, effectively expecting it there - which
seems silly.
TL;DR: Very confusing. Use an explicit path relative to SERENITY_ROOT
instead and update the .gitignore files.
This fills in a bunch of the FIXMEs that was in prepare_script.
execute_script is almost finished, it's just missing the module side.
As an aside, let's not assert when inserting a script element with
innerHTML.
This parser will be used by the C++ langauge server to provide better
auto-complete (& maybe also other things in the future).
It is designed to be error tolerant, and keeps track of the position
spans of the AST nodes, which should be useful later for incremental
parsing.
This was done with the help of several scripts, I dump them here to
easily find them later:
awk '/#ifdef/ { print "#cmakedefine01 "$2 }' AK/Debug.h.in
for debug_macro in $(awk '/#ifdef/ { print $2 }' AK/Debug.h.in)
do
find . \( -name '*.cpp' -o -name '*.h' -o -name '*.in' \) -not -path './Toolchain/*' -not -path './Build/*' -exec sed -i -E 's/#ifdef '$debug_macro'/#if '$debug_macro'/' {} \;
done
# Remember to remove WRAPPER_GERNERATOR_DEBUG from the list.
awk '/#cmake/ { print "set("$2" ON)" }' AK/Debug.h.in
-fsanitize=fuzzer was being added to LINKER_FLAGS from Lagom/CMakeLists,
which we don't want with FuzzilliJs as we want to define the functions
it provides ourselves.
If Serenity is ever used for more than a few days, the user will be more likely to
want to interact with their home directory than just be dropped at '/'.
Also, we have a Welcome program. Spotlight it!
And finally, there was a missing newline in the build script.
Previously we had /bin/sh, which might be bash but is run in POSIX mode
on some systems, causing read -r to not work correctly and inserting
newlines when encountering literal "\n" in the source.
Fixes#5040.
There's no guarantee that the last executed command will have a zero
exit code, and so the shell exit code may or may not be zero, even if
all the tests pass.
Also changes the `test || echo fail && exit` to
`if not test { echo fail && exit }`, since that's nicer-looking.
This adds support for FUTEX_WAKE_OP, FUTEX_WAIT_BITSET, FUTEX_WAKE_BITSET,
FUTEX_REQUEUE, and FUTEX_CMP_REQUEUE, as well well as global and private
futex and absolute/relative timeouts against the appropriate clock. This
also changes the implementation so that kernel resources are only used when
a thread is blocked on a futex.
Global futexes are implemented as offsets in VMObjects, so that different
processes can share a futex against the same VMObject despite potentially
being mapped at different virtual addresses.
All users of this mechanism have been switched to anonymous files and
passing file descriptors with sendfd()/recvfd().
Shbufs got us where we are today, but it's time we say good-bye to them
and welcome a much more idiomatic replacement. :^)
I've reached out to all of these authors asking if they would like to
claim the bounty and no one did. Let's list them on the website anyway
since it's fun to read about them. :^)
