MAX_GENERATED_VALUES_PER_TEST is now the --randomized_runs flag:
$ ./Build/lagom/bin/TestGenerator --randomized_runs 1000
It's sometimes useful to try larger numbers for it instead of the
default of 100.
MAX_GEN_ATTEMPTS_PER_VALUE is now a constexpr. It's not usually needed
to tweak this value; we can recompile with a different value on the rare
occasion.
REJECT and ASSUME are useful for filtering out unwanted generated
values. While this is not ideal, it is ocassionally useful and so we
include it for convenience.
The main loop of RANDOMIZED_TEST_CASE runs the test case 100 times, each
time trying to generate a different set of values. Inside that loop, if
it sees a REJECT (ASSUME is implemented in terms of REJECT), it retries
up to 15 times before giving up (perhaps it's impossible or just very
improbable to generate a value that will survive REJECT or ASSUME).
REJECT("Reason for rejecting") will just outright fail, while
ASSUME(bool) is more of an equivalent of a .filter() method from
functional languages.
This will be very useful as we add the randomized test cases and their
two loops ("generate+test many times" and "shrink once failure is
found"), because without this failure reporting we'd get many FAIL error
messages while still searching for the minimal one.
So, inside randomized test cases we want to only turn the error
reporting on for one last time after all the generating and shrinking.
This will be a foundational part of bootstrapping generators: this is
the way they'll get prerecorded values from / record random values into
RandomRuns. (Generators don't get in contact with RandomRuns
themselves, they just interact with the RandomnessSource.)
This will be used in the randomized tests a lot more than it is in the
unit tests / benchmarks; randomized tests will run the test function
multiple times, check the result and optionally start shrinking the
failing input. Generators will also be able to fail, resulting in some
of the new TestResult variants.
Previously VERIFY et al. was redefined inside tests to not abort and
instead fail the test. This wouldn't apply to non-header code though,
and was not helpful, as it prevented you from easily attaching gdb near
the abort.
After this removal tests can still use the EXPECT family of macros, but
VERIFY will behave like it does in the rest of the codebase (abort
etc.).
AK's version should see better inlining behaviors, than the LibM one.
We avoid mixed usage for now though.
Also clean up some stale math includes and improper floatingpoint usage.
As many macros as possible are moved to Macros.h, while the
macros to create a test case are moved to TestCase.h. TestCase is now
the only user-facing header for creating a test case. TestSuite and its
helpers have moved into a .cpp file. Instead of requiring a TEST_MAIN
macro to be instantiated into the test file, a TestMain.cpp file is
provided instead that will be linked against each test. This has the
side effect that, if we wanted to have test cases split across multiple
files, it's as simple as adding them all to the same executable.
The test main should be portable to kernel mode as well, so if
there's a set of tests that should be run in self-test mode in kernel
space, we can accomodate that.
A new serenity_test CMake function streamlines adding a new test with
arguments for the test source file, subdirectory under /usr/Tests to
install the test application and an optional list of libraries to link
against the test application. To accomodate future test where the
provided TestMain.cpp is not suitable (e.g. test-js), a CUSTOM_MAIN
parameter can be passed to the function to not link against the
boilerplate main function.
