The changes to tests are due to LibTimeZone incorrectly interpreting
time stamps in the TZDB. The TZDB will list zone transitions in either
UTC or the zone's local time (which is then subject to DST offsets).
LibTimeZone did not handle the latter at all.
For example:
The following rule is in effect until November 18, 6PM UTC.
America/Chicago -5:50:36 - LMT 1883 Nov 18 18:00u
The following rule is in effect until March 1, 2AM in Chicago time. But
at that time, a DST transition occurs, so the local time is actually
3AM.
America/Chicago -6:00 Chicago C%sT 1936 Mar 1 2:00
Multiple APIs have moved from the DOM Parsing and Serialization spec to
HTML.
Updates spec URLs and comments.
Delete InnerHTML file:
- Make parse_fragment a member of Element, matching serialize_fragment
on Node.
- Move inner_html_setter inline into Element and ShadowRoot as per the
spec.
Add FIXME to Range.idl for Trusted Types createContextualFragment
H.264 in Matroska can have blocks with unordered timestamps. Without
passing these as the presentation timestamp into the FFmpeg decoder,
the frames will not be returned in chronological order.
VideoFrame will now include a timestamp that is used by the
PlaybackManager, rather than assuming that it is the same timestamp
returned by the demuxer.
VP9 continues to function, but this also allows AV1 to be decoded. With
this commit, H.264 is still non-functional, as the decoder requires
some extra initial data from the track definition in the Matroska file.
It is not needed by code generators anymore, so it is not needed in the
Lagom tools build. And it is not needed as an object library anymore; it
was created this way so it could be included in Serenity's LibC.
LibLocale was split off from LibUnicode a couple years ago to reduce the
number of applications on SerenityOS that depend on CLDR data. Now that
we use ICU, both LibUnicode and LibLocale are actually linking in this
data. And since vcpkg gives us static libraries, both libraries are over
30MB in size.
This patch reverts the separation and merges LibLocale into LibUnicode
again. We now have just one library that includes the ICU data.
Further, this will let LibUnicode share the locale cache that previously
would only exist in LibLocale.
We only need LibCoreMinimal for the lagom-tools build. In particular, by
removing LibUnicode, we remove the lagom-tools dependence on the system
ICU package, as we do not have vcpkg hooked into this build. (We could
probably add vcpkg here, but since this libraries aren't even needed, we
don't need to bother).
This implements most of the CloseWatcher API from the html spec.
AbortSignal support is unimplemented.
Integration with dialogs and popovers is also unimplemented.
For SerenityOS, we parse emoji metadata from the UCD to learn emoji
groups, subgroups, names, etc. We used this information only in the
emoji picker dialog. It is entirely unused within Ladybird.
This removes our dependence on the UCD emoji file, as we no longer
need any of its information. All we need to know is the file path to
our custom emoji, which we get from Meta/emoji-file-list.txt.
There are a couple of differences here due to using ICU:
1. Titlecasing behaves slightly differently. We previously transformed
"123dollars" to "123Dollars", as we would use word segmentation to
split a string into words, then transform the first cased character
to titlecase. ICU doesn't go quite that far, and leaves the string
as "123dollars". While this is a behavior change, the only user of
this API is the `text-transform: capitalize;` CSS rule, and we now
match the behavior of other browsers.
2. There isn't an API to compare strings with case insensitivity without
allocating case-folded strings for both the left- and right-hand-side
strings. Our implementation was previously allocation-free; however,
in a benchmark, ICU is still ~1.4x faster.
This adds a motion preference to the browser UI similar to the existing
ones for color scheme and contrast.
Both AppKit UI and Qt UI has this new preference.
The auto value is currently the same as NoPreference, follow-ups can
address wiring that up to the actual preference for the OS.
This changes the Sanitizer configs to build all the vcpkg dependencies
with our specified CFLAGS and CXXFLAGS for ASAN and UBSAN.
Unfortunately, we can't yet enable actually compiling them with
sanitizers enabled, because this causes test failures that need to be
investigated.
This uses ICU for all of the Intl.PluralRules prototypes, which lets us
remove all data from our plural rules generator.
Plural rules depend directly on internal data from the number formatter,
so rather than creating a separate Locale::PluralRules class (which will
make accessing that data awkward), this adds plural rules APIs to the
existing Locale::NumberFormat.
...and shadow tree with TextNode for "value" attribute is created.
This means InlineFormattingContext is used, and button's text now
respects CSS text-decoration properties and unicode-ranges.
This uses ICU for the Intl.DateTimeFormat `format` `formatToParts`,
`formatRange`, and `formatRangeToParts`.
This lets us remove most data from our date-time format generator. All
that remains are time zone data and locale week info, which are relied
upon still for other interfaces. So they will be removed in a future
patch.
Note: All of the changes to the test files in this patch are now aligned
with other browsers. This includes:
* Some very incorrect formatting of Japanese symbols. (Looking at the
old results now, it's very obvious they were wrong.)
* Old FIXMEs regarding range formatting not including the start/end date
when only time fields were requested, but the dates differ.
* Day period inconsistencies.
Methods and attributes marked with [FIXME] are now implemented as
direct properties with the value `undefined` and are marked with the
[[Unimplemented]] attribute. This allows accesses to these properties
to be reported, while having no other side-effects.
This fixes an issue where [FIXME] methods broke feature detection on
some sites.
This uses ICU for the Intl.NumberFormat `format` and `formatToParts`
prototypes. It does not yet port the range formatter prototypes.
Most of the new code in LibLocale/NumberFormat is simply mapping from
ECMA-402 types to ICU types. Beyond that, the only algorithmic change is
that we have to mutate the output from ICU for `formatToParts` to match
what is expected by ECMA-402. This is explained in NumberFormat.cpp in
`flatten_partitions`.
This lets us remove most data from our number format generator. All that
remains are numbering system digits and symbols, which are relied upon
still for other interfaces (e.g. Intl.DateTimeFormat). So they will be
removed in a future patch.
Note: All of the changes to the test files in this patch are now aligned
with both Chrome and Safari.
Note: We keep locale parsing and syntactic validation as-is. ECMA-402
places additional restrictions on locales above what is required by the
Unicode spec. ICU doesn't provide methods that let us easily check those
restrictions, whereas LibLocale does. Other browsers also implement
their own validators here.
This introduces a locale cache to re-use parsed locale data and various
related structures (not doing so has a non-negligible performance impact
on Intl tests).
The existing APIs for canonicalization and display names are pretty
intertwined, so they must both be adapted at once here. The results of
canonicalization are slightly different on some edge cases. But the
changed results are actually now aligned with Chrome and Safari.
If we get a suggestion from fontconfig, we try those fonts first, before
falling back on the hard coded list of known suitable fonts for each
generic family.
This saves us the trouble of maintaining our own implementation,
and instantly brings us to full WOFF2 feature parity with others.
Co-Authored-By: Andrew Kaster <akaster@serenityos.org>
Also add a vcpkg command to ladybird.sh to ensure that vcpkg is setup,
and use a local binary cache for vcpkg build and install media to
avoid cluttering $XDG_CACHE_HOME.
We were already linking librt to LibCore for shm_open and friends.
Now that we build the code that uses POSIX shm into LibCoreMinimal, we
need to link librt into that as well.
And hook it into ladybird.sh for convenience. The script will set up
PATH and other environment variables automatically.
On CI, vcpkg is theoretically already installed on Linux machines, but
not with the right environment variables, and not on macOS. So this also
makes CI use this script to bootstrap vcpkg.
The Encoding specification maps ISO-8859-1 to windows-1252 and expects
the windows-1252 translation table to be used, which differs from
ISO-8859-1 for 0x80-0x9F.
Other contexts expect to get the actual ISO-8859-1 encoding, with 1-to-1
mapping to U+0000-U+00FF, when requesting it.
`decoder_for_exact_name` is introduced, which skips the mapping from
aliases to the encoding name done by `get_standardized_encoding`.
This was used to convert markdown into HTML for display in the browser,
but no other browser behaves this way, so let's simplify things by
removing it.
(Yes, we could implement all kinds of "convert to HTML and display" for
every file format out there, but that's far outside the scope of a
browser engine.)
For some reason, WebContent fails to load simple sites like xkcd.com
without the Qt event loop, even when using RequestServer instead of the
Qt networking stack. The CMake build on Linux has the same issue if we
skip installing the Qt event loop. It's not clear why this is - whether
something depends on the Qt event loop, or if there's a bug in the Unix
event loop implementation.
This is, however, also needed to use the --enable-qt-networking feature.
Implements `table.get`, `table.set`, `elem.drop`, `table.size`,
and `table.grow`. Also fixes a few issues when generating ref-related
spectests. Also changes the `TableInstance` type to use
`Vector<Reference>` instead of `Vector<Optional<Reference>>`, because
the ability to be null is already encoded in the `Reference` type.
On my Mac system with Homebrew SDL + self-built Clang, SDL2's include
directory is not in the library search path by default. Add it to
unbreak the build.
This allows searching for text with case-insensitivity. As this is
probably what most users expect, the default behavior is changes to
perform case-insensitive lookups. Chromes may add UI to change the
behavior as they see fit.
GC-allocated objects should never have JS::SafeFunction/JS::Handle
fields.
For now the plugin only emits warnings here, as there are many cases
of this occurring in the codebase that aren't trivial to fix. It is also
behind a CMake flag since it is a _very_ loud warning.
EventSource allows opening a persistent HTTP connection to a server over
which events are continuously streamed.
Unfortunately, our test infrastructure does not allow for automating any
tests of this feature yet. It only works with HTTP connections.
Supporting unbuffered fetches is actually part of the fetch spec in its
HTTP-network-fetch algorithm. We had previously implemented this method
in a very ad-hoc manner as a simple wrapper around ResourceLoader. This
is still the case, but we now implement a good amount of these steps
according to spec, using ResourceLoader's unbuffered API. The response
data is forwarded through to the fetch response using streams.
This will eventually let us remove the use of ResourceLoader's buffered
API, as all responses should just be streamed this way. The streams spec
then supplies ways to wait for completion, thus allowing fully buffered
responses. However, we have more work to do to make the other parts of
our fetch implementation (namely, Body::fully_read) use streams before
we can do this.
Download files to a temporary location, then only move the downloaded
file to the real location once the download is complete. This prevents
CMake from being confused about partially-downloaded files, e.g. if
someone presses ctrl+c in the middle of a download.
Note the GN build already behaves this way.
The logic in this script was *intended* to use the system's default
compiler if it was sufficiently new, and only start searching for the
latest installed if the default was not suitable.
However, the `cxx` program does not exist on Unixes, so the version
check always failed. We should be using the standard `c++` program name
instead.
After this change, the `CC` and `CXX` environment variables will have to
be used if someone wants to force a newer compiler version.
Now that the lambda capture plugin isn't full of false-positives, we can
make the jump and start halting builds for these errors. It also allows
these plugins to be useful in CI.
Instead of being opt-out with NOESCAPE, it is now opt-in with ESCAPING.
Opt-out is ideal, but unfortunately this was extremely noisy when
compiling the entire codebase. Escaping functions are rarer than non-
escaping ones, so let's just go with that for now.
This also allows us to gradually add heuristics for detecting missing
ESCAPING annotations and emitting them as errors. It also nicely matches
the spelling that Swift uses (@escaping), which is where this idea
originally came from.
No behavior change. No measurable performance different either.
(I tried `hyperfine 'Build/lagom/bin/image --no-output foo.webp'`
for a few input images before and after this change, and I didn't
see a difference. I also tried if moving both
Gfx::CanonicalCode::read_symbol() and
Compress::CanonicalCode::read_symbol() inline, and that didn't
help either.)
This allows readonly attributes and functions to have a 'FIXME' extended
attribute added to the IDL definition to stub out the function. This
makes debugging web compatibility issues on live sites much easier as a
FIXME message is logged whenever one of these functions or attributes
are called.
Support still needs to be extended to non-readonly attributes (and some
other special cases), but this should allow us to set a big percentage
of the commented out attributes/functions in IDL files to instead use
this extended attribute.
* Matches how the loader is organized
* `compress_VP8L_image_data()` will grow longer when we add actual
compression
* Maybe someone wants to write a lossy compressor one day
No behavior change.
The new baked image is a Prekernel and a Kernel baked together now, so
essentially we no longer need to pass the Prekernel as -kernel and the
actual kernel image as -initrd to QEMU, leaving the option to pass an
actual initrd or initramfs module later on with multiboot.
Nobody uses this functionality. I used this code on my old 2007 ICH7
test machine about a year ago, but bare metal is a small aspect of the
project, so it's safe to assume that nobody really tests this piece of
code.
Therefore, let's drop this for good and focus on more modern hardware.
Now both /bin/zcat and /bin/gunzip are symlinks to /bin/gzip, and we
essentially running it in decompression mode through these symlinks.
This ensures we don't maintain 2 versions of code to decompress Gzipped
data anymore, and handle the use case of gzipped-streaming input only
once in the codebase.
This is a more general and robust replacement of the LibJSGCVerifier.
We want to add more generic static analysis, and this new plugin will
be built in a way that integrates into the rest of the system.
🤽 - U+1F93D PERSON PLAYING WATER POLO
🤽♂️ - U+1F93D U+200D U+2642 MAN PLAYING WATER POLO
🤽♀️ - U+1F93D U+200D U+2640 WOMAN PLAYING WATER POLO
🦨 - U+1F9A8 SKUNK
The high-level design is that we have a static method on WebPWriter that
returns an AnimationWriter object. AnimationWriter has a virtual method
for writing individual frames. This allows streaming animations to disk,
without having to buffer up the entire animation in memory first.
The semantics of this function, add_frame(), are that data is flushed
to disk every time the function is called, so that no explicit `close()`
method is needed.
For some formats that store animation length at the start of the file,
including WebP, this means that this needs to write to a SeekableStream,
so that add_frame() can seek to the start and update the size when a
frame is written.
This design should work for GIF and APNG writing as well. We can move
AnimationWriter to a new header if we add writers for these.
Currently, `animation` can read any animated image format we can read
(apng, gif, webp) and convert it to an animated webp file.
The written animated webp file is not compressed whatsoever, so this
creates large output files at the moment.
Xcode clang doesn't understand the -std=c++23 spelling yet, and this
is what CMake's `set(CMAKE_CXX_STANDARD 23)` translates to too.
Unbreaks building with Xcode clang on macOS.
An AudioNode is the fundamental building block used in 'Audio
Contexts'. In our immediate case, the audio node we are working towards
implementing is an oscillating source node.
Previously the GML compiler did not support object properties such as
`content_widget: @GUI::Widget{}` for GUI::ScrollableContainerWidget;
this commit adds support for such properties by simply calling
`set_<key>(<TProperty>&)` on the object.
This commit also removes the previous hack where
ScrollableContainerWidget was special-cased to have its singular child
used as the content widget; the only GML file using this behaviour was
also changed to be in line with 'proper' GML as handled by the GML
Playground.
This makes it possible to use externally defined toplevel widgets that
have no C++ header defining them.
Note that this only allows widget-native properties on the object, as
the actual original definition is not available.
We already have required this version for quite a while for Lagom,
Ladybird and Serenity. Now that we require it in all of our CMakeLists,
let's scrub for better ways of writing things.
This allows main UI processes created while there is a currently
running one to request a new tab or a new window with the initial urls
provided on the command line. This matches (almost) the behavior of
Chromium and Firefox.
Add a new IPC protocol between two UI processes. The main UI process
will create an IPC server socket, while secondary UI processes will
connect to that socket and send over the URLs and action it wants the
main process to take.
This was resulting in a whole lot of rebuilding whenever a new IDL
interface was added.
Instead, just directly include the prototype in every C++ file which
needs it. While we only really need a forward declaration in each cpp
file; including the full prototype header (which itself only includes
LibJS/Object.h, which is already transitively brought in by
PlatformObject) - it seems like a small price to pay compared to what
feels like a full rebuild of LibWeb whenever a new IDL file is added.
Given all of these includes are only needed for the ::initialize
method, there is probably a smart way of avoiding this problem
altogether. I've considered both using some macro trickery or generating
these functions somehow instead.
The prototype header generation was getting a bit long.
This is also a step towards generating code for IDL files only
containing an enum definition without any interface. In that case we
can't put the enum definitions alongside the prototype - there is no
prototype to speak of.
We should never hit this case - so don't generate code for it, and
instead put in a VERIFY_NOT_REACHED.
Also improve the formatting of the generated code to closer match the
serenity code style.
Instead of a cryptic error that occurs due to an interface with no name,
fail early on by explicitly checking that an interface was parsed with a
name.
This change moves WebAssembly related data that was previously globally
accessible into the `WebAssemblyCache` object and creates one of these
per global object. This ensures that WebAssembly data cannot be
accessed across realms.
The following command was used to clang-format these files:
clang-format-18 -i $(find . \
-not \( -path "./\.*" -prune \) \
-not \( -path "./Base/*" -prune \) \
-not \( -path "./Build/*" -prune \) \
-not \( -path "./Toolchain/*" -prune \) \
-not \( -path "./Ports/*" -prune \) \
-type f -name "*.cpp" -o -name "*.mm" -o -name "*.h")
There are a couple of weird cases where clang-format now thinks that a
pointer access in an initializer list, e.g. `m_member(ptr->foo)`, is a
lambda return statement, and it puts spaces around the `->`.
Currently, if the prebuilt toolchain cache gets used, we will not try to
build the toolchain. Thus, the toolchain's ccache does not get used, and
is then pruned entirely at the end of the run.
So for now, let's just not prune the toolchain ccache. After a few years
it only reached 0.8 GB in size. And now that we are starting from empty
again, it would likely be a few more years before we reach 0.8 GB again.
If there is a cache miss while downloading the ccache from GitHub/Azure,
the .ccache directory won't exist when we try to update the modification
time of its contents. Configure the ccache size first, which will create
the .ccache directory if it doesn't exist.
Clang builds of ASAN+UBSAN on Linux take significantly less time on the
Azure CI runners. Measured times were 82 minutes for Clang 18 and
112 minutes for GCC 13, with no cache.
To keep our coverage of Ladybird builds + testing with GCC, add a
nightly job to run through the full test cycle on GCC 13.
It previously resided in LibWebView to hide the details of launching a
singleton process. That functionality now lives in LibCore. By moving
this to Ladybird, we will be able to register the process with the task
manager.
This just moves the code to launch a single process such as SQLServer to
LibCore. This will allow re-using this feature for other processes, and
will allow moving the launching of SQLServer to Ladybird.
This will be needed to collect statistics from processes that do not
have anything to do with LibWebView. The ProcessInfo structure must be
virtual to allow callers to add application-specific information.
This refactor eliminates the need for a second "fd passing socket" on
Lagom, as it uses SCM_RIGHTS in the expected fashion, to send fds along
with the data of our Unix socket message.
When launched with the new --enable-idl-tracing option, we now log
every call to web platform APIs declared via IDL, along with the
arguments passed.
This can be very helpful when trying to figure out what a site is
doing, especially if it's not doing what you'd expect.
On macOS, it's not trivial to get a Mach task port for your children.
This implementation registers the chrome process as a well-known
service with launchd based on its pid, and lets each child process
send over a reference to its mach_task_self() back to the chrome.
We'll need this Mach task port right to get process statistics.
