The headless-browser source is getting a bit unwieldy. The ordering of
class and method definitions is fragile; e.g. the application and web
view classes each require full definitions of each other. So it has
reached the point where it makes sense to give headless-browser some
better file structure.
To prepare for that, this patch simply moves its source to live along-
side the other browser chromes. This location is a bit better prepared
for creating more files, as the Utilities folder doesn't even have its
own CMakeLists.txt.
We have a bit of forgiveness around allowing tests to pass with varying
trailing newlines. Only write a rebaselined test to disk if it would not
have passed under those conditions.
We currently create a single WebView and run all 1400+ LibWeb tests in
serial over that WebView. Instead, let's create as many WebViews as
there are processes on the system, and run LibWeb tests concurrently
over those views.
To do this performantly requires that we never block the main thread of
the headless-browser process once the tests are running. Doing so will
effectively pause execution of all other tests. So test execution is now
Promise-based.
On my machine (with a hardware concurrency of 32), this reduces the run
time of LibWeb tests from 31.382s to 3.640s. CPU utilization increases
from 5% to 67%.
Instead, just log the view's current URL when the WebView crashes. It
won't make any sense to track the executing test this way once there are
many tests running concurrently.
We currently run all tests in a single WebView instance. That instance
owns the process-wide RequestClient / ImageDecoderClient, so if we were
to create a second instance, we'd run into trouble.
This migrates ownership of these services to the Application class, and
makes the Application own the WebView. In the future, this will let the
Application own a list of views.
We currently pass around the individual fields of the Application class
to a bunch of free functions. This makes adding a new field, and passing
it all the way to e.g. run_dump_test pretty annoying, as we have to go
through about 5 function calls.
This will get much worse in an upcoming patch to run LibWeb tests
concurrently. There, we will have to further pass these flags around as
async lambda value captures.
To make this nicer, just access the flags from Application::the(), which
is how the "real" UIs access their application objects as well.
The existing rebaseline script is a bit limiting in that it can only
rebaseline a single test at a time. When making sweeping changes, this
patch will let us rebaseline any number of tests at once.
For example, in the following abbreviated test HTML:
<span>some text</span>
<script>println("whf")</script>
We would have to craft the expectation file to include the "some text"
segment, usually with some leading whitespace. This is a bit annoying,
and makes it difficult to manually craft expectation files.
So instead of comparing the expectation against the entire DOM inner
text, we now send the inner text of just the <pre> element containing
the test output when we invoke `internals.signalTextTestIsDone`.
Previously, if you ran with a relative path, then everything would run
fine except that the test name would be blank in the output, eg:
1/1234:
instead of:
1/1234: Text/input/canvas/export.html
The IPCs to request a page's text, layout tree, etc. are currently all
synchronous. This can result in a deadlock when WebContent also makes
a synchronous IPC call, as both ends will be waiting on each other.
This replaces the page info IPCs with a single, asynchronous IPC. This
new IPC is promise-based, much like our screenshot IPC.
The main motivator here was noticing that --disable-sql-database did not
work with AppKit. Rather than re-implementing this there, move ownership
of these classes to WebView::Application, so that each UI does not need
to individually worry about it.
The identifier "Protocol" is claimed by Objective-C and Swift for use
by the language's built-in protocol conformance feature, which is
similar to Rust traits or Java interfaces.
Rename LibProtocol -> LibRequests, and its namespace from Protocol to
Requests to accomodate this.
Web specs do not return through javascript percent decoded URL path
components - but we were doing this in a number of places due to the
default behaviour of URL::serialize_path.
Since percent encoded URL paths may not contain valid UTF-8 - this was
resulting in us crashing in these places.
For example - on an HTMLAnchorElement when retrieving the pathname for
the URL of:
http://ladybird.org/foo%C2%91%91
To fix this make the URL class only return the percent encoded
serialized path, matching the URL spec. When the decoded path is
required instead explicitly call URL::percent_decode.
This fixes a crash running WPT URL tests for the anchor element on:
https://wpt.live/url/a-element.html
Currently, if we want to add a new e.g. WebContent command line option,
we have to add it to all of Qt, AppKit, and headless-browser. (Or worse,
we only add it to one of these, and we have feature disparity).
To prevent this, this moves command line flags to WebView::Application.
The flags are assigned to ChromeOptions and WebContentOptions structs.
Each chrome can still add its platform-specific options; for example,
the Qt chrome has a flag to enable Qt networking.
There should be no behavior change here, other than that AppKit will now
support command line flags that were previously only supported by Qt.
Skia painter is visibly faster than LibGfx painter and has more complete
CSS transforms support. With this change:
- On Linux, it will try to use Vulkan-backend with fallback to
CPU-backend
- On macOS it will try to use Metal-backend with fallback to
CPU-backend
- headless-browser always runs with CPU-backend in layout mode
LibGfx's output is consistent across different platforms, which allows
us to have one set of expectations for screenshot tests. This
consistency will not hold for Skia, where features like antialiasing and
gradient color interpolation vary slightly depending on the platform. In
upcoming changes, we are going to switch to using Skia as the default
painter, which leaves us with the following options:
- Have per-platform screenshot test expectations.
- Limit screenshot tests to run only on one platform and maintain a
single set of expectation files.
For now, I have decided to choose the latter option, using Linux as it
seems to be the most popular platform among developers.
This change will make it easier to disable screenshot comparison tests
on a specific platform or have per-platform expectations.
Additionally, it's nice to be able to tell if a ref-test uses a
screenshot as an expectation by looking at the test path.
This large commit also refactors LibWebView's process handling to use
a top-level Application class that uses a new WebView::Process class to
encapsulate the IPC-centric nature of each helper process.
This is the same behavior as RequestServer, with the added benefit that
we know how to gracefully reconnect ImageDecoder to all WebContent
processes on restart.
This makes WebView::Database wrap around sqlite3 instead of LibSQL. The
effect on outside callers is pretty minimal. The main consequences are:
1. We must ensure the Cookie table exists before preparing any SQL
statements involving that table.
2. We can use an INSERT OR REPLACE statement instead of separate INSERT
and UPDATE statements.
The main intention of this change is to have a consistent look and
behavior across all scrollbars, including elements with
`overflow: scroll` and `overflow: auto`, iframes, and a page.
Before:
- Page's scrollbar is painted by Browser (Qt/AppKit) using the
corresponding UI framework style,
- Both WebContent and Browser know the scroll position offset.
- WebContent uses did_request_scroll_to() IPC call to send updates.
- Browser uses set_viewport_rect() to send updates.
After:
- Page's scrollbar is painted on WebContent side using the same style as
currently used for elements with `overflow: scroll` and
`overflow: auto`. A nice side effects: scrollbars are now painted for
iframes, and page's scrollbar respects scrollbar-width CSS property.
- Only WebContent knows scroll position offset.
- did_request_scroll_to() is no longer used.
- set_viewport_rect() is changed to set_viewport_size().
This was no longer doing anything. We'll eventually want a way to pass
system default fonts to each WebContent process, but we don't need to
squeeze everything through this API that was really meant for Serenity's
very idiosyncratic font system.
Now that the chrome process is a singleton on all platforms, we can
safely add a cache to the CookieJar to greatly speed up access. The way
this works is we read all cookies upfront from the database. As cookies
are updated by the web, we store a list of "dirty" cookies that need to
be flushed to the database. We do that synchronization every 30 seconds
and at shutdown.
There's plenty of room for improvement here, some of which is marked
with FIXMEs in the CookieJar.
Before these changes, in a SQL database populated with 300 cookies,
browsing to https://twinings.co.uk/ WebContent spent:
19,806ms waiting for a get-cookie response
505ms waiting for a set-cookie response
With these changes, it spends:
24ms waiting for a get-cookie response
15ms waiting for a set-cookie response
For `AK_OS_SERENITY`, the root path of the resources folder is "/res";
but otherwise it should be the `s_serenity_resource_root` variable set
in `platform_init()`.
However, a path provided on the command line, will override the default
path in both of those cases.
This change also makes sure that `RequestServer` can find the
certificates file `serenity/Build/lagom/share/Lagom/ladybird/cacert.pem`
It previously resided in LibWebView to hide the details of launching a
singleton process. That functionality now lives in LibCore. By moving
this to Ladybird, we will be able to register the process with the task
manager.
For the normal non-test use case of headless-browser, the function
`load_page_for_screenshot_and_exit()` didn't actually load the requested
webpage in the `WebView`, resulting in an all white pixels screenshot.
This URL library ends up being a relatively fundamental base library of
the system, as LibCore depends on LibURL.
This change has two main benefits:
* Moving AK back more towards being an agnostic library that can
be used between the kernel and userspace. URL has never really fit
that description - and is not used in the kernel.
* URL _should_ depend on LibUnicode, as it needs punnycode support.
However, it's not really possible to do this inside of AK as it can't
depend on any external library. This change brings us a little closer
to being able to do that, but unfortunately we aren't there quite
yet, as the code generators depend on LibCore.
We had previous implemented some plumbing for file input elements in
commit 636602a54e.
This implements the return path for chromes to inform WebContent of the
file(s) the user selected. This patch includes a dummy implementation
for headless-browser to enable testing.
Previously, the check for `.html` meant that `.svg` tests were excluded.
This led to a few `.svg` with missing or bit-rotted expectations, which
have now been added/updated.
