We've finally gotten kmalloc to a point where it feels decent enough
to drop this comment.
There's still a lot of room for improvement, and we'll continue working
on it.
This was a premature optimization from the early days of SerenityOS.
The eternal heap was a simple bump pointer allocator over a static
byte array. My original idea was to avoid heap fragmentation and improve
data locality, but both ideas were rooted in cargo culting, not data.
We would reserve 4 MiB at boot and only ended up using ~256 KiB, wasting
the rest.
This patch replaces all kmalloc_eternal() usage by regular kmalloc().
This patch adds generic slab allocators to kmalloc. In this initial
version, the slab sizes are 16, 32, 64, 128, 256 and 512 bytes.
Slabheaps are backed by 64 KiB block-aligned blocks with freelists,
similar to what we do in LibC malloc and LibJS Heap.
There are no more users of the C-style kfree() API in the kernel,
so let's get rid of it and enjoy the new world where we always know
how much memory we are freeing. :^)
This patch does two things:
- Combines kmalloc_aligned() and kmalloc_aligned_cxx(). Templatizing
the alignment parameter doesn't seem like a valuable enough
optimization to justify having two almost-identical implementations.
- Stores the real allocation size of an aligned allocation along with
the other alignment metadata, and uses it to call kfree_sized()
instead of kfree().
Since we allocate the subheap in the first page of the given storage
let's assert that the subheap can actually fit in a single page, to
prevent the possible future headache of trying to debug the cause of
random kernel memory corruption :^)
This avoids getting caught with our pants down when heap expansion fails
due to missing page tables. It also avoids a circular dependency on
kmalloc() by way of HashMap::set() in MemoryManager::ensure_pte().
Previously, the heap expansion logic could end up calling kmalloc
recursively, which was quite messy and hard to reason about.
This patch redesigns heap expansion so that it's kmalloc-free:
- We make a single large virtual range allocation at startup
- When expanding, we bump allocate VM from that region
- When expanding, we populate page tables directly ourselves,
instead of going via MemoryManager.
This makes heap expansion a great deal simpler. However, do note that it
introduces two new flaws that we'll need to deal with eventually:
- The single virtual range allocation is limited to 64 MiB and once
exhausted, kmalloc() will fail. (Actually, it will PANIC for now..)
- The kmalloc heap can no longer shrink once expanded. Subheaps stay
in place once constructed.
cert-dcl50-cpp: No variadic functions, suppressed in RefCounted and
ThreadSafeRefCounted for implementing the magic one_ref_left and
will_be_destroyed functions.
cert-dcl58-cpp: No opening ::std, suppressed in the places we put names
in ::std to aid tools (move, forward, nullptr_t, align_val_t, etc).
SonarCloud flagged this "Code Smell", where we are accessing these
static methods as if they are instance methods. While it is technically
possible, it is very confusing to read when you realize they are static
functions.
PVS-Studio flagged these as uninitialized. While there is no bug here,
it is our policy to always initialize members to avoid potential bugs
in the future.
This expands the reach of error propagation greatly throughout the
kernel. Sadly, it also exposes the fact that we're allocating (and
doing other fallible things) in constructors all over the place.
This patch doesn't attempt to address that of course. That's work for
our future selves.
This is the idiomatic way to declare type aliases in modern C++.
Flagged by Sonar Cloud as a "Code Smell", but I happen to agree
with this particular one. :^)
The C++ standard specifies that `free` and `operator delete` should
be callable with nullptr. The non-aligned `kfree` already handles this,
but because of the pointer arithmetic to obtain the allocation start
pointer, the aligned version would produce undefined behavior.
In e7fb70b05, regular kmalloc was changed to return nullptr on
allocation failure instead of crashing. The `kmalloc_aligned_cxx`
wrapper used by the aligned operator new should do the same.
Now that we have a significant amount of code paths handling OOM, lets
enable kmalloc and friends to actually return nullptr. This way we can
start stressing these paths and validating all of they work as expected.
By making these functions static we close a window where we could get
preempted after calling Processor::current() and move to another
processor.
Co-authored-by: Tom <tomut@yahoo.com>
Kernels built with Clang seem to be quite allocation-heavy compared to
their GCC counterparts. We would sometimes end up crashing during boot
because the eternal ranges had no free capacity.
This is a much more ergonomic option than getting a
`VERIFY_NOT_REACHED()` failure at run-time. I encountered this issue
with Clang, where sized deallocation is not the default due to ABI
breakage concerns.
Note that we can't simply just not declare these functions, because the
C++ standard states:
> If this function with size parameter is defined, the program shall
> also define the version without the size parameter.
The compiler will use these to allocate objects that have alignment
requirements greater than that of our normal `operator new` (4/8 byte
aligned).
This means we can now use smart pointers for over-aligned types.
Fixes a FIXME.
The C++ standard says that it's legal to call the `delete` operator with
a null pointer argument, in which case it should be a no-op. I
encountered this issue when running a kernel that's compiled with Clang.
I assume this fact was used for some kind of optimization.
This was only used by a single class (AK::ByteBuffer) in the kernel
and not in an OOM-safe way.
Now that ByteBuffer no longer uses it, there's no need for the kernel
heap to burden itself with supporting this.
C++14 gave us sized operator delete, but we haven't been taking
advantage of it. Let's get to a point where it can help us by
adding kfree_sized(void*, size_t).
When creating uninitialized storage for variables, we need to make sure
that the alignment is correct. Fixes a KUBSAN failure when running
kernels compiled with Clang.
In `Syscalls/socket.cpp`, we can simply use local variables, as
`sockaddr_un` is a POD type.
Along with moving the `alignas` specifier to the correct member,
`AK::Optional`'s internal buffer has been made non-zeroed by default.
GCC emitted bogus uninitialized memory access warnings, so we now use
`__builtin_launder` to tell the compiler that we know what we are doing.
This might disable some optimizations, but judging by how GCC failed to
notice that the memory's initialization is dependent on `m_has_value`,
I'm not sure that's a bad thing.
