Commit graph

175 commits

Author SHA1 Message Date
Luke
5c911ad4b1 Meta: Add GitHub Actions workflow for Lagom with Fuzzers
There are cases where Lagom will build with GCC but not Clang.
This often goes unnoticed for a while as we don't often build with
Clang.

However, this is now important to test in CI because of the
OSS-Fuzz integration.

Note that this only tests the build, it does not run any tests.
Note that it also only builds LagomCore, Lagom and the fuzzers.
It does not build the other programs that use Lagom.
2020-11-29 20:22:56 +01:00
Nico Weber
29e4dc7634 Lagom: Mention OSS-Fuzz in ReadMe
We added OSS-Fuzz integration in #4154, but documentation about it
is spread across several pull requests, IRC, and issues. Let's collect
the important bits in the ReadMe.
2020-11-29 20:22:09 +01:00
AnotherTest
1ecea2f105 Lagom: Add a Shell parser fuzzer 2020-11-29 16:22:44 +01:00
Linus Groh
6a37285d93 Lagom: Add Regex fuzzers 2020-11-29 09:29:26 +01:00
DavidKorczynski
ff8ac60184
Lagom: Various fixes to make Lagom run on OSS-Fuzz (#4176) 2020-11-27 23:57:02 +01:00
Emanuel Sprung
55450055d8 LibRegex: Add a regular expression library
This commit is a mix of several commits, squashed into one because the
commits before 'Move regex to own Library and fix all the broken stuff'
were not fixable in any elegant way.
The commits are listed below for "historical" purposes:

- AK: Add options/flags and Errors for regular expressions

Flags can be provided for any possible flavour by adding a new scoped enum.
Handling of flags is done by templated Options class and the overloaded
'|' and '&' operators.

- AK: Add Lexer for regular expressions

The lexer parses the input and extracts tokens needed to parse a regular
expression.

- AK: Add regex Parser and PosixExtendedParser

This patchset adds a abstract parser class that can be derived to implement
different parsers. A parser produces bytecode to be executed within the
regex matcher.

- AK: Add regex matcher

This patchset adds an regex matcher based on the principles of the T-REX VM.
The bytecode pruduced by the respective Parser is put into the matcher and
the VM will recursively execute the bytecode according to the available OpCodes.
Possible improvement: the recursion could be replaced by multi threading capabilities.

To match a Regular expression, e.g. for the Posix standard regular expression matcher
use the following API:

```
Pattern<PosixExtendedParser> pattern("^.*$");
auto result = pattern.match("Well, hello friends!\nHello World!"); // Match whole needle

EXPECT(result.count == 1);
EXPECT(result.matches.at(0).view.starts_with("Well"));
EXPECT(result.matches.at(0).view.end() == "!");

result = pattern.match("Well, hello friends!\nHello World!", PosixFlags::Multiline); // Match line by line

EXPECT(result.count == 2);
EXPECT(result.matches.at(0).view == "Well, hello friends!");
EXPECT(result.matches.at(1).view == "Hello World!");

EXPECT(pattern.has_match("Well,....")); // Just check if match without a result, which saves some resources.
```

- AK: Rework regex to work with opcodes objects

This patchsets reworks the matcher to work on a more structured base.
For that an abstract OpCode class and derived classes for the specific
OpCodes have been added. The respective opcode logic is contained in
each respective execute() method.

- AK: Add benchmark for regex

- AK: Some optimization in regex for runtime and memory

- LibRegex: Move regex to own Library and fix all the broken stuff

Now regex works again and grep utility is also in place for testing.
This commit also fixes the use of regex.h in C by making `regex_t`
an opaque (-ish) type, which makes its behaviour consistent between
C and C++ compilers.
Previously, <regex.h> would've blown C compilers up, and even if it
didn't, would've caused a leak in C code, and not in C++ code (due to
the existence of `OwnPtr` inside the struct).

To make this whole ordeal easier to deal with (for now), this pulls the
definitions of `reg*()` into LibRegex.

pros:
- The circular dependency between LibC and LibRegex is broken
- Eaiser to test (without accidentally pulling in the host's libc!)

cons:
- Using any of the regex.h functions will require the user to link -lregex
- The symbols will be missing from libc, which will be a big surprise
  down the line (especially with shared libs).

Co-Authored-By: Ali Mohammad Pur <ali.mpfard@gmail.com>
2020-11-27 21:32:41 +01:00
Nico Weber
f160df123b Lagom: Rename FuzzBMP to FuzzBMPLoader 2020-11-26 09:59:46 +01:00
Nico Weber
ec4c2f971b Lagom: Make BMP fuzzer look like the other image loader fuzzers 2020-11-26 09:59:46 +01:00
Nico Weber
4f12efe191 Lagom: Add fuzzers for remaining image loaders: ICO, PNG, PBM, PGM 2020-11-26 09:59:46 +01:00
Ben Wiederhake
ec36388f40 Meta: Reduce IRC spam
Fixes #4145.
2020-11-25 20:50:22 +01:00
Linus Groh
4e68f179d6 Lagom: Fix FuzzJs build
This was broken with the JS::Parser::Error position changes, but I don't
actually see a reason to do anything with the parser errors here, so
let's remove it and consider simply not crashing a success. :^)
2020-11-25 20:00:23 +01:00
Nico Weber
f13a38aa4d Lagom: Add a gif loader fuzzer 2020-11-20 21:17:58 +01:00
Nico Weber
812892ef86 Lagom: Add a jpg fuzzer 2020-11-19 21:21:45 +01:00
Nico Weber
3f3a7d128b Lagom: Make fuzzer cmake less repetitive 2020-11-19 21:21:45 +01:00
Nico Weber
f5967c4745 Lagom: Add a PPM fuzzer
It finds the problem fixed in 69518bd178 but nothing else.
2020-11-19 14:04:35 +01:00
Ben Wiederhake
c4933f9bd7 Meta: Fix IRC notification script
What a silly mistake. How did I manage to do that?
2020-11-14 17:14:45 +01:00
Ben Wiederhake
ef9ac8a8a2 Meta: Use SerenityBot for IRC notifications
This avoids "useless" join/part notifications.
2020-11-14 15:30:29 +01:00
Nico Weber
39ce7251f7 Lagom: Use -fsanitize=fuzzer, not -fsanitize=fuzzer-no-link
Fuzzers don't link for me without this change.
2020-11-14 10:09:03 +01:00
Nico Weber
d775dea13c Lagom: Add a gemini fuzzer
Didn't find anything interesting, but might as well check it in.
2020-11-14 10:09:03 +01:00
Nico Weber
8ed83f9a1d Lagom: Augment fuzzing readme a bit
Recommend using asan, don't set the c compiler (c++ compiler is
sufficient), mention how to run on several cores, and how to get less
output.
2020-11-14 10:09:03 +01:00
Ben Wiederhake
cf9419fc4f Meta: Nicer IRC notifications 2020-11-13 08:56:56 +01:00
Nico Weber
5c2e8b6189 Lagom: Add ntpquery to lagom build 2020-11-12 21:21:33 +01:00
Nico Weber
5fcd34b810 Userland: Add an "adjtime" utility
It's a thin userland wrapper around adjtime(2). It can be used
to view current pending time adjustments, and root can use it to
smoothly adjust the system time.

As far as I can tell, other systems don't have a userland utility
for this, but it seems useful. Useful enough that I'm adding it to
the lagom build so I can use it on my linux box too :)
2020-11-10 19:03:08 +01:00
Linus Groh
46cd9157cd Lagom/Fuzzers: Add a Dockerfile for FuzzilliJs
Based on Fedora. This allows building and running FuzzilliJs and Fuzzilli
itself in a Docker/Podman container.
2020-11-08 11:01:07 +01:00
Ben Wiederhake
3cd39097f6 Meta: Add script that runs all lints 2020-11-08 09:58:55 +01:00
Linus Groh
2c9e6585f8 Documentation: Update required GCC version to >= 10
I initially thought as long as Lagom is not built >= 9 would be fine,
but LagomCore is always built for the code generators.
2020-11-07 18:22:18 +01:00
Luke
8928607455 Lagom/Fuzzers: Add Fuzzilli version of FuzzJs
Fuzzilli is a JavaScript engine fuzzer made by googleprojectzero.

https://github.com/googleprojectzero/fuzzilli/
2020-11-07 10:09:55 +01:00
Linus Groh
21912123c4 Travis: Update host GCC to version 10
This allows us to use the latest C++20 features in programs which we
compile and run as part of Lagom.
2020-11-02 13:12:35 +01:00
Brendan Coles
c1bfb8cb0e Meta: lint-shell-scripts: Exit if shellcheck is not installed 2020-11-01 10:29:33 +01:00
AnotherTest
37c089fb7b LibTLS: (Almost) verify certificate chain against root CA certificates
Also adds a very primitive systemwide ca_certs.ini file.
2020-10-30 23:42:03 +01:00
asynts
607931268e CMake: Use CONFIGURE_DEPENDS in existing globs. 2020-10-29 11:52:47 +01:00
Laurent Cimon
b4790010a8 Build: Modify various parts to allow the build to succeed on FreeBSD 2020-10-20 14:40:47 +02:00
Andreas Kling
75d5f436bc Toolchain: Upgrade to GCC 10.2.0 2020-10-12 19:53:25 +02:00
Linus Groh
bcfc6f0c57 Everywhere: Fix more typos 2020-10-03 12:36:49 +02:00
Ben Wiederhake
df4c9b729d Meta: Provide script to automatically flag bad formatting 2020-09-25 21:18:17 +02:00
Valtteri Koskivuori
f0fdbb1d83 Meta: Fix build-root-filesystem.sh on macOS 2020-09-20 21:45:00 +02:00
Tom
e317ee7541 Meta: Add env variable SERENITY_RUN to be able to choose qemu, bochs, etc
This allows picking for example bochs: SERENITY_RUN=b ninja run
2020-09-19 00:33:02 +02:00
AnotherTest
961661ea1d Build: Preserve hard/symbolic links when building root fs
This fixes the issue where there would not be enough space to copy
things when at least the git port and the gcc port are installed.
2020-09-16 15:56:35 +02:00
Andreas Kling
2b17d980cf Meta: Include .json files in the Qt Creator refresh script 2020-09-14 19:55:18 +02:00
Andreas Kling
2f97590409 Meta: Make the text-to-cpp-string thingy pass shellcheck 2020-09-14 16:33:53 +02:00
Andreas Kling
40892ac72e Meta: Add helper for generating a C++ string from a text file 2020-09-14 16:16:36 +02:00
Linus Groh
e9562e1920 Userland: Add {md5,sha1,sha256,sha512}sum 2020-09-13 11:37:56 +02:00
Ben Wiederhake
8498a5678d Meta: Avoid deprecated qemu option
Apparently "-soundhw pcspk" is deprecated too. However, I don't know which "name"
to insert, and I can't test it, hence I didn't touch it.
2020-09-12 13:46:15 +02:00
Ben Wiederhake
25ccd40d5a Meta: Describe how to analyze an LLVM fuzzer crash 2020-09-12 00:13:29 +02:00
Ben Wiederhake
98bfcb4b57 Meta+LibGfx: Fuzz BMP parsing 2020-09-12 00:13:29 +02:00
Ben Wiederhake
5d3c437cce LibJS: Fix start position of multi-line tokens
This broke in case of unterminated regular expressions, causing goofy location
numbers, and 'source_location_hint' to eat up all memory:

Unexpected token UnterminatedRegexLiteral. Expected statement (line: 2, column: 4294967292)
2020-09-12 00:13:29 +02:00
Andreas Kling
18ff9c3fc2 Meta: Fix shellcheck whines in check-symbols.sh 2020-09-07 00:00:12 +02:00
Itamar
1c20e684f3 Travis: Run script that checks for forbidden symbols in LibC
check-symbols.sh fails the build if undefined __cx_guard_* symbols are
found in  LibC.

This will help us catch port breakage sooner.
2020-09-06 21:36:36 +02:00
Andreas Kling
dcd47655d0 utmpupdate: Add a program for updating /var/run/utmp
To keep track of ongoing terminal sessions, we now have a sort-of
traditional /var/run/utmp file, like other Unix systems.
Unlike other Unix systems however, ours is of course JSON. :^)

The /bin/utmpupdate program is used to update the file, which is
not writable by regular user accounts. This helper program is
set-GID "utmp".
2020-09-06 16:10:27 +02:00
Ben Wiederhake
c8668e9b7c Meta: Fix wonky copyright headers in Tests 2020-09-05 14:19:38 +02:00