This patch adds a 128-byte inline buffer that we use before switching
to using a dynamically growing ByteBuffer.
This allows us to avoid heap allocations in many cases, and totally
incidentally also speeds up @nico's favorite test, "disasm /bin/id"
more than 2x. :^)
Much like with Vector::append(), you may want to append multiple items in one
go. It's actually more important to do this for prepending, because you don't
want to copy the rest of items further each time.
first_matching returns the first item in the vector that matches
the given condition.
last_matching returns the last item in the vector that matches
the given condition.
Problem:
- Several files have missing includes. This results in complaints from
`clang-tidy`.
- `#ifdef` is followed by `#elif <value>` which evaluates to `0`.
Solution:
- Add missing includes.
- Change to `#elif defined(<value>)`.
Problem:
- It is difficult to refactor because there are no tests to bind the
functionality.
- Arguments are not forwarded correctly to the constructor.
Solution:
- Add tests.
- Change constructor to take forwarding references.
Problem:
- `is_zero()` is implemented by checking each value in the array by
hand. This is error-prone and less expressive than using an
algorithm.
Solution:
- Implement `is_zero()` in terms of `all_of`.
Problem:
- Raw loops are often written to validate that all values in a
container meet a predicate, but raw loops are not as expressive as
functions implementing well-named algorithms and are error-prone.
Solution:
- Implement a very generic form of `all_of`.
Problem:
- C-style arrays do not automatically provide bounds checking and are
less type safe overall.
- `__builtin_memcmp` is not a constant expression in the current gcc.
Solution:
- Change private m_data to be AK::Array.
- Eliminate constructor from C-style array.
- Change users of the C-style array constructor to use the default
constructor.
- Change `operator==()` to be a hand-written comparison loop and let
the optimizer figure out to use `memcmp`.
Problem:
- `MACAddress` class is not usable in a compile-time context.
- `__builtin_memcpy` is not constexpr in gcc.
Solution:
- Decorate functions with `constexpr` keyword.
- Use default constructors and destructors.
- Change `__builtin_memcpy` to a hand-written `for` loop and let the
compiler's optimizer take care of it.
- Add tests to ensure compile-time capabilities.
This fixes an OOB access when the last read/written chunk is empty (as we _just_
started on a new chunk).
Also adds a test case to TestMemoryStream.
Found via human fuzzing in the shell:
```sh
for $(cat /dev/urandom) {
clear
match $it {
?* as (x) {
echo $x
sleep 1
}
}
}
```
would assert at some point.
Personally I found this unintuitive at first, but it is in line with
strtol(), Python's int() or JavaScript's parseInt(), so I guess it makes
sense.
Fixes#4097.
Problem:
- There are no unit tests for `MACAddress` class. This makes it
difficult to refactor and ensure the same behavior.
- `m_data` private member variable is uninitialized leading to undefined
behavior of `is_zero()`.
Solution:
- Add unit tests to cover basic functionality.
- Initialize `m_data`.
Problem:
- C++20 changes the way equality operators are generated. This results
in overload ambiguity as reported by clang.
Solution:
- Remove `AK::Vector::operator!=` because it will be automatically
generated in terms of `AK::Vector::operator==`.
- Change `AK::Vector::operator==` to be a function template so that
overload resolution is not confused about `a == b` vs `b == a`.
- Add tests to ensure the behavior works.
Notes:
- There is more info available at
https://brevzin.github.io/c++/2019/07/28/comparisons-cpp20/ for
deeper discussion about overload resolution, operator rewriting, and
generated functions.
This gives the compiler enough information to optimize index validation
when using range-for to iterate over a Vector, drastically reducing the
cost of such loops.
Problem:
- `typedef` is a keyword which comes from C and carries with it old
syntax that is hard to read.
- Creating type aliases with the `using` keyword allows for easier
future maintenance because it supports template syntax.
- There is inconsistent use of `typedef` vs `using`.
Solution:
- Use `clang-tidy`'s checker called `modernize-use-using` to update
the syntax to use the newer syntax.
- Remove unused functions to make `clang-tidy` happy.
- This results in consistency within the codebase.
Problem:
- Building with clang is broken because of the `struct` vs `class`
mismatch between the definition and declaration.
Solution:
- Change `class` to `struct` in the forward declaration.
Problem:
- IPv4Address class cannot be used in a compile-time context.
- A union is used by initializing one of the members and reading the
non-active member. This is undefined behavior and not permitted in a
`constexpr` context.
Solution:
- Eliminate undefined behavior by changing to a simple `u32` for
storage instead of the union and performing mask/shift calculations
for obtaining the individual octets.
- Decorate functions with `constexpr` where possible. Currently string
formatting and optionals are not `constexpr`-capable so functions
using those are left out.
- Modify tests to validate functionality in a `constexpr` context in
addition to the run-time tests already being run. This ensures that
functionality is the same in both contexts.
This makes most operations thread safe, especially so that they
can safely be used in the Kernel. This includes obtaining a strong
reference from a weak reference, which now requires an explicit
call to WeakPtr::strong_ref(). Another major change is that
Weakable::make_weak_ref() may require the explicit target type.
Previously we used reinterpret_cast in WeakPtr, assuming that it
can be properly converted. But WeakPtr does not necessarily have
the knowledge to be able to do this. Instead, we now ask the class
itself to deliver a WeakPtr to the type that we want.
Also, WeakLink is no longer specific to a target type. The reason
for this is that we want to be able to safely convert e.g. WeakPtr<T>
to WeakPtr<U>, and before this we just reinterpret_cast the internal
WeakLink<T> to WeakLink<U>, which is a bold assumption that it would
actually produce the correct code. Instead, WeakLink now operates
on just a raw pointer and we only make those constructors/operators
available if we can verify that it can be safely cast.
In order to guarantee thread safety, we now use the least significant
bit in the pointer for locking purposes. This also means that only
properly aligned pointers can be used.
Problem:
- There is no direct unit testing of the IPv4Address functionality
which makes refactoring difficult.
Solution:
- Add unit tests to cover the current functionality of
IPv4Address. This will allow future refactorings with confidence.
We had competing inline definitions of the placement operators new.
Avoid this by having <AK/kmalloc.h> pull in <new> from the compiler
and always using their definitions instead.
I feel like there must be an elegant solution to this whole situation
with the operators, but I'm not sure what it is.
`AK::URL` will now check if the URL requires a port to be set using
`AK::URL.protocol_requires_port(protocol)`.
If the URL does not specify a port, and no default port for the URL
protocol is found with `AK::URL.default_port_for_protocol(protocol)`,
the URL is considered to be invalid.
This looks at three things:
- if the type has a typedef `AllowOwnPtr', respect that
- if not, disallow construction if both of `ref()' and `unref()' are
present.
Note that in the second case, if a type only defines `ref()' or only
defines `unref()', an OwnPtr can be created, as a RefPtr of that type
would be ill-formed.
Also marks a `Performance' to explicitly allow OwnPtrs.
This allows going back one character at a time, and then re-consume
previously consumed chars.
The code I need this for looks something like this:
ASSERT(lexer.consume_specific('\\'));
if (lexer.next_is("foo"))
...
lexer.retreat();
lexer.consume_escaped_character(); // This expects lexer.peek() == '\\'
Problem:
- `constexpr_sum` is implemented using `Array` which means the
function needs to be a function template so that the size can be
deduced.
Solution:
- Change the `Array` function argument to a `Span` since `Span` now is
`constexpr`.
Problem:
- Hash functions can be `constexpr`, but are not.
Solution:
- Change `inline` keyword to `constexpr`.
- Add `static_assert` tests to ensure the hash functions work in a
`constexpr` context.
Problem:
- The hash functions have no associated tests, so there is nothing
binding their behavior.
Solution:
- Bind the hash function behavior by adding tests.
- Use the existing behavior as "correct".
Problem:
- `constexpr` functions are decorated with the `inline` specifier
keyword. This is redundant because `constexpr` functions are
implicitly `inline`.
- [dcl.constexpr], §7.1.5/2 in the C++11 standard): "constexpr
functions and constexpr constructors are implicitly inline (7.1.2)".
Solution:
- Remove the redundant `inline` keyword.
Problem:
- `Checked` is not `constexpr`-aware.
Solution:
- Decorate member functions with `constexpr` keyword.
- Add tests to ensure the functionality where possible.
Problem:
- Compiler-generated functions are being defined which results in
extra code to maintain.
Solution:
- Switch to compiler-generated default functions for default
construction, copy assignment, move assignment, copy construction
and move construction.
Problem:
- There is no test which guarantees the CircularQueue does not
construct any objects of the value type. The goal is to have
uninitialized memory which can be used.
Solution:
- Add a test requiring that the constructor of the value type is never
called.
This commit also removes a few functions like raw_out and vwarn. If we
want to write raw output, we can do this as follows:
out("{}", "Hello, World!");
The vout stuff isn't really public API anyways, so no need for another
vwarn.
Problem:
- It is not possible to perform a binary search at compile-time
because `binary_search` is not `constexpr`-aware.
Solution:
- Add `constexpr` support.
This implements the transmit time suggestion in (abandoned?)
draft-ietf-ntp-data-minimization. (The other suggestions were already
implemented as far as I can tell.)
Problem:
- `Span` is not `constexpr` aware.
Solution:
- Add `constexpr` support for all parts that do not require
`reinterpret_cast`.
- Modify tests which use the `constexpr` functions.
Double the capacity when used+deleted buckets crosses 60% of capacity.
This appears to be a sweet spot for performance based on some ad-hoc
testing with test-js. :^)
Instead of each hash bucket being a SinglyLinkedList, switch to using
closed hashing (open addressing). Buckets are chained together via
double hashing (hashing the hash until we find an unused bucket.)
This greatly reduces malloc traffic, since each added element no longer
allocates a new linked list node.
Appears performance neutral on test-js. Can definitely be tuned and
could use proper management of load factor, etc.
There is no portable way to forward declare abort because the libc
implementations disagree on the signature.
Originally, I added a __portable_abort function with a "portable"
signature which just called abort. But I really don't like it and just
including <stdlib.h> is simpler.
Note that the headers we include in <AK/TestSuite.h> are no longer
commutative now, we have to include <stdlib.h> before anything else.
Problem:
- Output of decode and encode grow as the decode and encode
happen. This is inefficient because a large size will require many
reallocations.
- `const` qualifiers are missing on variables which are not intended
to change.
Solution:
- Since the size of the decoded or encoded message is known prior to
starting, calculate the size and set the output to that size
immediately. All appends will not incur the reallocation overhead.
- Add `const` qualifiers to show intent.
* AK: Add formatter for JsonValue.
* Inspector: Use new format functions.
* Profiler: Use new format functions.
* UserspaceEmulator: Use new format functions.
Problem:
- The Base64 alphabet and lookup table are initialized at
run-time. This results in an initial start-up cost as well as a
boolean evaluation and branch every time the function is called.
Solution:
- Provide `constexpr` functions which initialize the alphabet and
lookup table at compile-time. These can be called and assigned to a
`constexpr` variable so that there is no run-time cost associated
with the initialization or lookup.
We are adding the process name as prefix and a newline as suffix to any
message written to debug. Thus, the following doesn't make any sense:
for (u8 byte : bytes)
dbg("{:02x} ", byte);
dbgln();
Which function call would put the prefix? This doesn't make any sense,
thus these functions must go.
The example above could be converted to:
StringBuilder builder;
for (u8 byte : bytes)
builder.appendff("{:02x} ", byte);
dbgln("{}", builder.build());
Problem:
- Constructors and conversion operators are not `constexpr`,
but they can be.
- `constexpr` is needed here so that other classes can add `constexpr`
evaluation.
Solution:
- Add the `constexpr` keyword to the constructors and
conversion operators.
- Add `static_assert` tests which ensure the capability works.
Problem:
- m_data is being passed to the constructor of the parent class before
it is initialized. This is not really a problem because the compiler
knows the location and it is only a span being constructed, but it
triggers a warning in clang for use-before-init.
Solution:
- Initialize using a default constructed array and then overwrite it
inside the constructor after the member is initialized.
Formatter is specialized in the header file. The definition in the
implementation file is extraneous and has no effect. Simply removing
it so that there is no confusion.
String literals are just pointers to a constant character. It should be
possible to format them as such. (The default is to print them as
strings still.)
When we write the format specifier '{:#08x}' we are asking for eight
significant digits, zero padding and the prefix '0x'.
However, previously we got only six significant digits because the
prefix counted towards the width. (The number '8' here is the total
width and not the number of significant digits.)
Both fmtlib and printf shared this behaviour. However, I am introducing
a special case here because when we do zero padding we really only care
about the digits and not the width.
Notice that zero padding is a special case anyways, because zero padding
goes after the prefix as opposed to any other padding which goes before
it.
This would previously fail at runtime, and it would have zero indication
of what exactly went wrong.
Also adds `AK::DependentFalse<Ts...>', which is a...dependent false.
In the future all (normal) output should be written by any of the
following functions:
out (currently called new_out)
outln
dbg (currently called new_dbg)
dbgln
warn (currently called new_warn)
warnln
However, there are still a ton of uses of the old out/warn/dbg in the
code base so the new functions are called new_out/new_warn/new_dbg. I am
going to rename them as soon as all the other usages are gone (this
might take a while.)
I also added raw_out/raw_dbg/raw_warn which don't do any escaping,
this should be useful if no formatting is required and if the input
contains tons of curly braces. (I am not entirely sure if this function
will stay, but I am adding it for now.)
If we're sharing buffers, we only want to share trivial structures
as anything else could potentially share internal pointers, which
most likely is going to cause problems due to different address
spaces.
Fix the GUI::SystemTheme structure, which was not trivial, which
is now caught at compile time.
Fixes#3650
The problem with our test suite is that it can't detect if a test
failed. When a test fails we simply write 'FAIL ...' to stderr and move
on.
Previously, the test suite would list all tests as passing regardless
how many assertions failed. In the future it might be smart to implement
this properly but test suites for C++ are always hard to do nicely.
(Because C++ execution isn't meant to be embedded.)
It's now save to pass a signed integer as parameter and then use it as
replacement field (previously, this would just cast it to size_t which
would be bad.)
This finally takes care of the kind-of excessive boilerplate code that were the
ctype adapters. On the other hand, I had to link `LibC/ctype.cpp` to the Kernel
(for `AK/JsonParser.cpp` and `AK/Format.cpp`). The previous commit actually makes
sense now: the `string.h` includes in `ctype.{h,cpp}` would require to link more LibC
stuff to the Kernel when it only needs the `_ctype_` array of `ctype.cpp`, and there
wasn't any string stuff used in ctype.
Instead of all this I could have put static derivatives of `is_any_of()` in the
concerned AK files, however that would have meant more boilerplate and workarounds;
so I went for the Kernel approach.
Since commit 1ec59f28ce turns the ctype macros
into functions we can now feed them directly to a GenericLexer! This will lead to
removing the ctype adapters that were kind-of excessive boilerplate, but needed as
the Kernel doesn't compile with the LibC.
I put this into the <AK/PrintfImplementation.h> header in the hope that
it could be re-used by the printf implementation. That would not be
super trivial though, so I am not doing that now.
The `consume_quoted_string()` can now take an escape character. This allows it
(for example) to capture a string's enclosing quotes. The escape character is
optional by default.
You can also consume and unescape a quoted string with the eponymous method
`consume_and_unescape_string()`. It takes an escape character as parameter
(backslash by default). It builds a String in which common escape sequences
get... unescaped :^) (e.g. \n, \r, \t...).
Instead of just implementing format specifiers ad-hog this commit
implements the exact syntax std::format uses.
There are still a ton of features that are not supported by this
implementation, however, the format specifiers should be parsed
correctly.
In some cases however, the format specifiers aren't quite parsed
correctly, for example:
String::formatted("{:{}}", 42, 4)
should produce the string " 42" however an (unrelated) assertion fails.
This is because vformat doesn't consider nested parentheses. I have to
spend some time coming up with a simple way of doing this, I don't feel
like doing that right now.
The fundamental code for this already exists, by limiting the number of
format arguments (arbitrarily) to 256 large widths are used to encode
that these should be taken from other format parameters.
With this commit, <AK/Format.h> has a more supportive role and isn't
used directly.
Essentially, there now is a public 'vformat' function ('v' for vector)
which takes already type erased parameters. The name is choosen to
indicate that this function behaves similar to C-style functions taking
a va_list equivalent.
The interface for frontend users are now 'String::formatted' and
'StringBuilder::appendff'.
Two things I hate about C++:
1. 'int', 'signed int' and 'unsigned int' are two distinct types while
'char, 'signed char' and 'unsigned char' are *three* distinct types.
This is because 'signed int' is an alias for 'int' but 'signed char'
can't be an alias for 'char' because on some weird systems 'char' is
unsigned.
One might think why not do it the other way around, make 'int' an
alias for 'signed int' and 'char' an alias for whatever that is on
the platform, or make 'char' signed on all platforms. But who am I
to ask?
2. 'unsigned long' and 'unsigned long long' are always different types,
even if both are 64 bit numbers.
This commit fixes a few bugs that coming from this.
See Also: 1b3169f405.
This function is not avaliable in the kernel.
In the future it would be nice to have some sort of <charconv> header
that does this for all integer types and then call it in strtoull and et
cetera.
The difference would be that this function say 'from_chars' would return
an Optional and not just interpret anything invalid as zero.
There are three classes avaliable that share the functionality of
BufferStream:
1. InputMemoryStream is for reading from static buffers. Example:
Bytes input = /* ... */;
InputMemoryStream stream { input };
LittleEndian<u32> little_endian_value;
input >> little_endian_value;
u32 host_endian_value;
input >> host_endian_value;
SomeComplexStruct complex_struct;
input >> Bytes { &complex_struct, sizeof(complex_struct) };
2. OutputMemoryStream is for writing to static buffers. Example:
Array<u8, 4096> buffer;
OutputMemoryStream stream;
stream << LittleEndian<u32> { 42 };
stream << ReadonlyBytes { &complex_struct, sizeof(complex_struct) };
foo(stream.bytes());
3. DuplexMemoryStream for writing to dynamic buffers, can also be used
as an intermediate buffer by reading from it directly. Example:
DuplexMemoryStream stream;
stream << NetworkOrdered<u32> { 13 };
stream << NetowkrOrdered<u64> { 22 };
NetworkOrdered<u32> value;
stream >> value;
ASSERT(value == 13);
foo(stream.copy_into_contiguous_buffer());
Unlike BufferStream these streams do not use a fixed endianness
(BufferStream used little endian) these have to be explicitly specified.
There are helper types in <AK/Endian.h>.
OutputMemoryStream was originally a proxy for DuplexMemoryStream that
did not expose any reading API.
Now I need to add another class that is like OutputMemoryStream but only
for static buffers. My first idea was to make OutputMemoryStream do that
too, but I think it's much better to have a distinct class for that.
I originally wanted to call that class FixedOutputMemoryStream but that
name is really cumbersome and it's a bit unintuitive because
InputMemoryStream is already reading from a fixed buffer.
So let's just use DuplexMemoryStream instead of OutputMemoryStream for
any dynamic stuff and create a new OutputMemoryStream for static
buffers.
Consider the following snippet:
void foo(InputStream& stream) {
if(!stream.eof()) {
u8 byte;
stream >> byte;
}
}
There is a very subtle bug in this snippet, for some input streams eof()
might return false even if no more data can be read. In this case an
error flag would be set on the stream.
Until now I've always ensured that this is not the case, but this made
the implementation of eof() unnecessarily complicated.
InputFileStream::eof had to keep a ByteBuffer around just to make this
possible. That meant a ton of unnecessary copies just to get a reliable
eof().
In most cases it isn't actually necessary to have a reliable eof()
implementation.
In most other cases a reliable eof() is avaliable anyways because in
some cases like InputMemoryStream it is very easy to implement.
This makes PrintfImplementation usable with any sequence, provided that
a 'next element' function can be written for it.
Does not affect the behaviour of printf() and co.
It wasn't actually possible to call
const LogStream& operator<<(const LogStream&, ReadonlyBytes);
because it was shadowed by
template<typename T>
const LogStream& operator<<(const LogStream& stream, Span<T> span);
not sure how I didn't find this when I added the overload.
It would be possible to use SFINAE to disable the other overload,
however, I think it is better to use a different method entirely because
the output can be very verbose:
void dump_bytes(ReadonlyBytes);
Leverage constexpr and __builtin_ffs for Bitmap::find_first. Also add
a variant Bitmap::find_one_anywhere that can start scanning at a
provided hint.
Also, merge Bitmap::fill_range into the already existing Bitmap::set_range
The streaming operator doesn't short-circuit, consider the following
snippet:
void foo(InputStream& stream) {
int a, b;
stream >> a >> b;
}
If the first read fails, the second is called regardless. It should be
well defined what happens in this case: nothing.
This is a strcpy()-like method with actually sane semantics:
* It accepts a non-empty buffer along with its size in bytes.
* It copies as much of the string as fits into the buffer.
* It always null-terminates the result.
* It returns, as a non-discardable boolean, whether the whole string has been
copied.
Intended usage looks like this:
bool fits = string.copy_characters_to_buffer(buffer, sizeof(buffer));
and then either
if (!fits) {
fprintf(stderr, "The name does not fit!!11");
return nullptr;
}
or, if you're sure the buffer is large enough,
// I'm totally sure it fits because [reasons go here].
ASSERT(fits);
or if you're feeling extremely adventurous,
(void)fits;
but don't do that, please.
For some weird reason the C++ standard considers char, signed char and
unsigned char *three* different types. On the other hand int is just an
alias for signed int, meaning that int, signed int and unsigned int are
just *two* different types.
https://stackoverflow.com/a/32856568/8746648
Before, we had about these occurrence counts:
COPY: 13 without, 33 with
MOVE: 12 without, 28 with
Clearly, 'with' was the preferred way. However, this introduced double-semicolons
all over the place, and caused some warnings to trigger.
This patch *forces* the usage of a semi-colon when calling the macro,
by removing the semi-colon within the macro. (And thus also gets rid
of the double-semicolon.)
The implementation in LibC did a timestamp->day-of-week conversion
which looks like a valuable thing to have. But we only need it in
time_to_tm, where we already computed year/month/day -- so let's
consolidate on the day_of_week function in DateTime (which is
getting extracted to AK).
