The previous behavior was to always VERIFY that the UTF-8 bytes were
valid when iterating over the code points of an UTF8View. This change
makes it so we instead output the 0xFFFD 'REPLACEMENT CHARACTER'
code point when encountering invalid bytes, and keep iterating the
view after skipping one byte.
Leaving the decision to the consumer would break symmetry with the
UTF32View API, which would in turn require heavy refactoring and/or
code duplication in generic code such as the one found in
Gfx::Painter and the Shell.
To make it easier for the consumers to detect the original bytes, we
provide a new method on the iterator that returns a Span over the
data that has been decoded. This method is immediately used in the
TextNode::compute_text_for_rendering method, which previously did
this in a ad-hoc waay.
This also add tests for the new behavior in TestUtf8.cpp, as well
as reinforcements to the existing tests to check if the underlying
bytes match up with their expected values.
This adds a peek method for Utf8CodepointIterator, which enables it to
be used in some parsing cases where peeking is necessary.
peek(0) is equivalent to operator*, expect that peek() does not contain
any assertions and will just return an empty Optional<u32>.
This also implements a test case for iterating UTF-8.
This patch implements a Unicode-safe substring method, which can be used
when offset and length should be specified in actual characters instead
of bytes.
This can be used to mitigate issues where a string is split in the
middle of a UTF-8 multi-byte character, which leads to invalid UTF-8.
Furthermore, it implements to common shorthands for substring methods
which take only an offset and return the substring until the end of the
string.
SPDX License Identifiers are a more compact / standardized
way of representing file license information.
See: https://spdx.dev/resources/use/#identifiers
This was done with the `ambr` search and replace tool.
ambr --no-parent-ignore --key-from-file --rep-from-file key.txt rep.txt *
Unlike String/StringView::starts_with this compares utf8 code points
instead of "characters" (bytes), which is important when handling
aribtary utf-8 input that could include overlong characters.
(...and ASSERT_NOT_REACHED => VERIFY_NOT_REACHED)
Since all of these checks are done in release builds as well,
let's rename them to VERIFY to prevent confusion, as everyone is
used to assertions being compiled out in release.
We can introduce a new ASSERT macro that is specifically for debug
checks, but I'm doing this wholesale conversion first since we've
accumulated thousands of these already, and it's not immediately
obvious which ones are suitable for ASSERT.
This was causing WindowServer and Taskbar to crash sometimes when the
stars aligned and we tried cutting off a string ending with "..." right
on top of an emoji. :^)
Problem:
- `(void)` simply casts the expression to void. This is understood to
indicate that it is ignored, but this is really a compiler trick to
get the compiler to not generate a warning.
Solution:
- Use the `[[maybe_unused]]` attribute to indicate the value is unused.
Note:
- Functions taking a `(void)` argument list have also been changed to
`()` because this is not needed and shows up in the same grep
command.
You can now #include <AK/Forward.h> to get most of the AK types as
forward declarations.
Header dependency explosion is one of the main contributors to compile
times at the moment, so this is a step towards smaller include graphs.
This changes copyright holder to myself for the source code files that I've
created or have (almost) completely rewritten. Not included are the files
that were significantly changed by others even though it was me who originally
created them (think HtmlView), or the many other files I've contributed code to.
As suggested by Joshua, this commit adds the 2-clause BSD license as a
comment block to the top of every source file.
For the first pass, I've just added myself for simplicity. I encourage
everyone to add themselves as copyright holders of any file they've
added or modified in some significant way. If I've added myself in
error somewhere, feel free to replace it with the appropriate copyright
holder instead.
Going forward, all new source files should include a license header.
Using int was a mistake. This patch changes String, StringImpl,
StringView and StringBuilder to use size_t instead of int for lengths.
Obviously a lot of code needs to change as a result of this.
The old implementation tried to move forward as long as the current
byte looks like a UTF-8 character continuation byte (has its two
most significant bits set to 10). This is correct as long as we assume
the string is actually valid UTF-8, which we do (we also have a separate
method that can check whether it is the case).
We can't, however, assume that the data after the end of our string
is also valid UTF-8 (in fact, we're not even allowed to look at data
outside out string, but it happens to a valid memory region most of
the time). If the byte after the end of our string also has its most
significant bits set to 10, we would move one byte forward, and then
fail the m_length > 0 assertion.
One way to fix this would be to add a length check inside the loop
condition. The other one, implemented in this commit, is to reimplement
the whole function in terms of decode_first_byte(), which gives us
the length as encoded in the first byte. This also brings it more
in line with the other functions around it that do UTF-8 decoding.
Utf8View wraps a StringView and implements begin() and end() that
return a Utf8CodepointIterator, which parses UTF-8-encoded Unicode
codepoints and returns them as 32-bit integers.
This is the first step towards supporting emojis in Serenity ^)
https://github.com/SerenityOS/serenity/issues/490
