beenull/ladybird
1
0
Fork 0
mirror of https://github.com/LadybirdBrowser/ladybird.git synced 2024-11-30 03:20:28 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 7
ladybird/Kernel/VM/MemoryManager.cpp

788 lines
27 KiB
C++
Raw Normal View History

Kernel: Run clang-format on everything.
2019-06-07 09:43:58 +00:00
#include "CMOS.h"
#include "Process.h"
#include "StdLib.h"
Add basic paging. Only identity maps the bottom 4MB as a start.
2018-10-17 21:13:55 +00:00
#include <AK/Assertions.h>
#include <AK/kstdio.h>
Kernel: Move i386.{cpp,h} => Arch/i386/CPU.{cpp,h} There's a ton of work that would need to be done before we could spin up on another architecture, but let's at least try to separate things out a bit.
2019-06-07 18:02:01 +00:00
#include <Kernel/Arch/i386/CPU.h>
Kernel: Run clang-format on everything.
2019-06-07 09:43:58 +00:00
#include <Kernel/FileSystem/Inode.h>
Kernel: Use the Multiboot memory map info to inform our paging setup. This makes it possible to run Serenity with more than 64 MB of RAM. Because each physical page is represented by a PhysicalPage object, and such objects are allocated using kmalloc_eternal(), more RAM means more pressure on kmalloc_eternal(), so we're gonna need a better strategy for this. But for now, let's just celebrate that we can use the 128 MB of RAM we've been telling QEMU to run with. :^)
2019-06-09 09:48:58 +00:00
#include <Kernel/Multiboot.h>
Kernel: Run clang-format on everything.
2019-06-07 09:43:58 +00:00
#include <Kernel/VM/MemoryManager.h>
Add basic paging. Only identity maps the bottom 4MB as a start.
2018-10-17 21:13:55 +00:00
More work on per-process page directories. It basically works now! I spent some time stuck on a problem where processes would clobber each other's stacks. Took me a moment to figure out that their stacks were allocated in the sub-4MB linear address range which is shared between all processes. Oops!
2018-11-01 10:30:48 +00:00
//#define MM_DEBUG
Implement COW pages! :^) sys$fork() now clones all writable regions with per-page COW bits. The pages are then mapped read-only and we handle a PF by COWing the pages. This is quite delightful. Obviously there's lots of work to do still, and it needs better data structures, but the general concept works.
2018-11-05 12:48:07 +00:00
//#define PAGE_FAULT_DEBUG
More work on per-process page directories. It basically works now! I spent some time stuck on a problem where processes would clobber each other's stacks. Took me a moment to figure out that their stacks were allocated in the sub-4MB linear address range which is shared between all processes. Oops!
2018-11-01 10:30:48 +00:00
Add basic paging. Only identity maps the bottom 4MB as a start.
2018-10-17 21:13:55 +00:00
static MemoryManager* s_the;
Implement loading of linked ELF executables. This took me a couple hours. :^) The ELF loading code now allocates a single region for the entire file and creates virtual memory mappings for the sections as needed. Very nice!
2018-10-27 12:56:52 +00:00
MemoryManager& MM
Add basic paging. Only identity maps the bottom 4MB as a start.
2018-10-17 21:13:55 +00:00
{
return *s_the;
}
MemoryManager::MemoryManager()
{
Kernel: Refactor Region/PageDirectory ownership model. Make PageDirectory retainable and have each Region co-own the PageDirectory they're mapped into. When unmapped, Region has no associated PageDirectory. This allows Region to automatically unmap itself when destroyed.
2019-01-22 04:01:00 +00:00
m_kernel_page_directory = PageDirectory::create_at_fixed_address(PhysicalAddress(0x4000));
Yet more coding style fixes.
2018-12-03 00:38:22 +00:00
m_page_table_zero = (dword*)0x6000;
Kernel: Use the Multiboot memory map info to inform our paging setup. This makes it possible to run Serenity with more than 64 MB of RAM. Because each physical page is represented by a PhysicalPage object, and such objects are allocated using kmalloc_eternal(), more RAM means more pressure on kmalloc_eternal(), so we're gonna need a better strategy for this. But for now, let's just celebrate that we can use the 128 MB of RAM we've been telling QEMU to run with. :^)
2019-06-09 09:48:58 +00:00
m_page_table_one = (dword*)0x7000;
More work on per-process page directories. It basically works now! I spent some time stuck on a problem where processes would clobber each other's stacks. Took me a moment to figure out that their stacks were allocated in the sub-4MB linear address range which is shared between all processes. Oops!
2018-11-01 10:30:48 +00:00
Yet more coding style fixes.
2018-12-03 00:38:22 +00:00
initialize_paging();
Kernel: Introduce threads, and refactor everything in support of it. The scheduler now operates on threads, rather than on processes. Each process has a main thread, and can have any number of additional threads. The process exits when the main thread exits. This patch doesn't actually spawn any additional threads, it merely does all the plumbing needed to make it possible. :^)
2019-03-23 21:03:17 +00:00
kprintf("MM initialized.\n");
Add basic paging. Only identity maps the bottom 4MB as a start.
2018-10-17 21:13:55 +00:00
}
MemoryManager::~MemoryManager()
{
}
Let PageDirectory have a PhysicalPage for the PDB instead of being the PDB. This is more efficient than using the wasteful kmalloc_aligned() approach to allocation. It also allows much tighter encapsulation of the class.
2018-12-31 13:58:03 +00:00
void MemoryManager::populate_page_directory(PageDirectory& page_directory)
{
MM: Allocate page tables from a separate set of physical pages. The old approach only worked because of an overpermissive accident. There's now a concept of supervisor physical pages that can be allocated. They all sit in the low 4 MB of physical memory and are identity mapped, shared between all processes, and only ring 0 can access them.
2019-01-01 01:09:43 +00:00
page_directory.m_directory_page = allocate_supervisor_physical_page();
Let PageDirectory have a PhysicalPage for the PDB instead of being the PDB. This is more efficient than using the wasteful kmalloc_aligned() approach to allocation. It also allows much tighter encapsulation of the class.
2018-12-31 13:58:03 +00:00
page_directory.entries()[0] = kernel_page_directory().entries()[0];
Kernel: Use the Multiboot memory map info to inform our paging setup. This makes it possible to run Serenity with more than 64 MB of RAM. Because each physical page is represented by a PhysicalPage object, and such objects are allocated using kmalloc_eternal(), more RAM means more pressure on kmalloc_eternal(), so we're gonna need a better strategy for this. But for now, let's just celebrate that we can use the 128 MB of RAM we've been telling QEMU to run with. :^)
2019-06-09 09:48:58 +00:00
page_directory.entries()[1] = kernel_page_directory().entries()[1];
Kernel: More work on bringing up E1000 support.
2019-03-11 10:11:29 +00:00
// Defer to the kernel page tables for 0xC0000000-0xFFFFFFFF
for (int i = 768; i < 1024; ++i)
page_directory.entries()[i] = kernel_page_directory().entries()[i];
Give each task its own page directory. This isn't finished but I'll commit as I go. We need to get to where context switching only needs to change CR3 and everything's ready to go. My basic idea is: - The first 4 kB is off-limits. This catches null dereferences. - Up to the 4 MB mark is identity-mapped and kernel-only. - The rest is available to everyone! While the first 4 MB is only available to the kernel, it's still mapped in every process, for convenience when entering the kernel.
2018-11-01 08:01:51 +00:00
}
Yet more coding style fixes.
2018-12-03 00:38:22 +00:00
void MemoryManager::initialize_paging()
Add basic paging. Only identity maps the bottom 4MB as a start.
2018-10-17 21:13:55 +00:00
{
static_assert(sizeof(MemoryManager::PageDirectoryEntry) == 4);
static_assert(sizeof(MemoryManager::PageTableEntry) == 4);
Yet more coding style fixes.
2018-12-03 00:38:22 +00:00
memset(m_page_table_zero, 0, PAGE_SIZE);
Kernel: Use the Multiboot memory map info to inform our paging setup. This makes it possible to run Serenity with more than 64 MB of RAM. Because each physical page is represented by a PhysicalPage object, and such objects are allocated using kmalloc_eternal(), more RAM means more pressure on kmalloc_eternal(), so we're gonna need a better strategy for this. But for now, let's just celebrate that we can use the 128 MB of RAM we've been telling QEMU to run with. :^)
2019-06-09 09:48:58 +00:00
memset(m_page_table_one, 0, PAGE_SIZE);
Add basic paging. Only identity maps the bottom 4MB as a start.
2018-10-17 21:13:55 +00:00
Snazz up the kprintf() output a bit by giving it its own color.
2018-10-31 19:10:39 +00:00
#ifdef MM_DEBUG
Let PageDirectory have a PhysicalPage for the PDB instead of being the PDB. This is more efficient than using the wasteful kmalloc_aligned() approach to allocation. It also allows much tighter encapsulation of the class.
2018-12-31 13:58:03 +00:00
dbgprintf("MM: Kernel page directory @ %p\n", kernel_page_directory().cr3());
Snazz up the kprintf() output a bit by giving it its own color.
2018-10-31 19:10:39 +00:00
#endif
More paging stuff. The test userspace process now runs at linear address 0x300000 which is mapped to a dynamically allocated page from the MemoryManager. Cool!
2018-10-18 11:05:00 +00:00
Let PageDirectory have a PhysicalPage for the PDB instead of being the PDB. This is more efficient than using the wasteful kmalloc_aligned() approach to allocation. It also allows much tighter encapsulation of the class.
2018-12-31 13:58:03 +00:00
#ifdef MM_DEBUG
dbgprintf("MM: Protect against null dereferences\n");
#endif
Protect the first 4 KB of memory. This makes null pointers crashy, tremendously useful :^)
2018-10-21 19:57:59 +00:00
// Make null dereferences crash.
Kernel: Rename LinearAddress => VirtualAddress.
2019-06-07 10:56:50 +00:00
map_protected(VirtualAddress(0), PAGE_SIZE);
Protect the first 4 KB of memory. This makes null pointers crashy, tremendously useful :^)
2018-10-21 19:57:59 +00:00
Let PageDirectory have a PhysicalPage for the PDB instead of being the PDB. This is more efficient than using the wasteful kmalloc_aligned() approach to allocation. It also allows much tighter encapsulation of the class.
2018-12-31 13:58:03 +00:00
#ifdef MM_DEBUG
Kernel: Use the Multiboot memory map info to inform our paging setup. This makes it possible to run Serenity with more than 64 MB of RAM. Because each physical page is represented by a PhysicalPage object, and such objects are allocated using kmalloc_eternal(), more RAM means more pressure on kmalloc_eternal(), so we're gonna need a better strategy for this. But for now, let's just celebrate that we can use the 128 MB of RAM we've been telling QEMU to run with. :^)
2019-06-09 09:48:58 +00:00
dbgprintf("MM: Identity map bottom 5MB\n");
Let PageDirectory have a PhysicalPage for the PDB instead of being the PDB. This is more efficient than using the wasteful kmalloc_aligned() approach to allocation. It also allows much tighter encapsulation of the class.
2018-12-31 13:58:03 +00:00
#endif
Kernel: Use the Multiboot memory map info to inform our paging setup. This makes it possible to run Serenity with more than 64 MB of RAM. Because each physical page is represented by a PhysicalPage object, and such objects are allocated using kmalloc_eternal(), more RAM means more pressure on kmalloc_eternal(), so we're gonna need a better strategy for this. But for now, let's just celebrate that we can use the 128 MB of RAM we've been telling QEMU to run with. :^)
2019-06-09 09:48:58 +00:00
// The bottom 5 MB (except for the null page) are identity mapped & supervisor only.
MM: Allocate page tables from a separate set of physical pages. The old approach only worked because of an overpermissive accident. There's now a concept of supervisor physical pages that can be allocated. They all sit in the low 4 MB of physical memory and are identity mapped, shared between all processes, and only ring 0 can access them.
2019-01-01 01:09:43 +00:00
// Every process shares these mappings.
Kernel: Use the Multiboot memory map info to inform our paging setup. This makes it possible to run Serenity with more than 64 MB of RAM. Because each physical page is represented by a PhysicalPage object, and such objects are allocated using kmalloc_eternal(), more RAM means more pressure on kmalloc_eternal(), so we're gonna need a better strategy for this. But for now, let's just celebrate that we can use the 128 MB of RAM we've been telling QEMU to run with. :^)
2019-06-09 09:48:58 +00:00
create_identity_mapping(kernel_page_directory(), VirtualAddress(PAGE_SIZE), (5 * MB) - PAGE_SIZE);
Tidy up memory map a bit and write out the general map in MemoryManager. There was a bug that given enough supervisor page allocation, we would eventually start dipping into the kmalloc range.
2019-01-15 23:44:09 +00:00
// Basic memory map:
// 0 -> 512 kB Kernel code. Root page directory & PDE 0.
Kernel: Actually zero-fill eagerly committed regions. Previously, calling Region::commit() would make sure to allocate any missing physical pages, but they would contain uninitialized data. This was very obvious when allocating GraphicsBitmaps as they would have garbage pixels rather than being filled with black. The MM quickmap mechanism didn't work when operating on a non-active page directory (which happens when we're in the middle of exec, for example.) This patch changes quickmap to reside in the shared kernel page directory. Also added some missing clobber lists to inline asm that I stumbled on.
2019-01-31 02:57:06 +00:00
// (last page before 1MB) Used by quickmap_page().
Kernel: Use the Multiboot memory map info to inform our paging setup. This makes it possible to run Serenity with more than 64 MB of RAM. Because each physical page is represented by a PhysicalPage object, and such objects are allocated using kmalloc_eternal(), more RAM means more pressure on kmalloc_eternal(), so we're gonna need a better strategy for this. But for now, let's just celebrate that we can use the 128 MB of RAM we've been telling QEMU to run with. :^)
2019-06-09 09:48:58 +00:00
// 1 MB -> 3 MB kmalloc_eternal() space.
// 3 MB -> 4 MB kmalloc() space.
// 4 MB -> 5 MB Supervisor physical pages (available for allocation!)
// 5 MB -> 0xc0000000 Userspace physical pages (available for allocation!)
Kernel: Replace the last "linear" with "virtual".
2019-06-13 19:42:12 +00:00
// 0xc0000000-0xffffffff Kernel-only virtual address space
Kernel: Allocate kernel stacks for threads using the region allocator. This patch moves away from using kmalloc memory for thread kernel stacks. This reduces pressure on kmalloc (16 KB per thread adds up fast) and prevents kernel stack overflow from scribbling all over random unrelated kernel memory.
2019-05-14 09:51:00 +00:00
Kernel: Refactor MemoryManager to use a Bitmap rather than a Vector This significantly reduces the pressure on the kernel heap when allocating a lot of pages. Previously at about 250MB allocated, the free page list would outgrow the kernel's heap. Given that there is no longer a page list, this does not happen. The next barrier will be the kernel memory used by the page records for in-use memory. This kicks in at about 1GB.
2019-06-11 11:13:02 +00:00
#ifdef MM_DEBUG
dbgprintf("MM: Quickmap will use %p\n", m_quickmap_addr.get());
#endif
m_quickmap_addr = VirtualAddress((1 * MB) - PAGE_SIZE);
RetainPtr<PhysicalRegion> region = nullptr;
bool region_is_super = false;
Kernel: Use the Multiboot memory map info to inform our paging setup. This makes it possible to run Serenity with more than 64 MB of RAM. Because each physical page is represented by a PhysicalPage object, and such objects are allocated using kmalloc_eternal(), more RAM means more pressure on kmalloc_eternal(), so we're gonna need a better strategy for this. But for now, let's just celebrate that we can use the 128 MB of RAM we've been telling QEMU to run with. :^)
2019-06-09 09:48:58 +00:00
for (auto* mmap = (multiboot_memory_map_t*)multiboot_info_ptr->mmap_addr; (unsigned long)mmap < multiboot_info_ptr->mmap_addr + multiboot_info_ptr->mmap_length; mmap = (multiboot_memory_map_t*)((unsigned long)mmap + mmap->size + sizeof(mmap->size))) {
kprintf("MM: Multiboot mmap: base_addr = 0x%x%08x, length = 0x%x%08x, type = 0x%x\n",
(dword)(mmap->addr >> 32),
(dword)(mmap->addr & 0xffffffff),
(dword)(mmap->len >> 32),
(dword)(mmap->len & 0xffffffff),
(dword)mmap->type);
if (mmap->type != MULTIBOOT_MEMORY_AVAILABLE)
continue;
Kernel: Refactor MemoryManager to use a Bitmap rather than a Vector This significantly reduces the pressure on the kernel heap when allocating a lot of pages. Previously at about 250MB allocated, the free page list would outgrow the kernel's heap. Given that there is no longer a page list, this does not happen. The next barrier will be the kernel memory used by the page records for in-use memory. This kicks in at about 1GB.
2019-06-11 11:13:02 +00:00
Kernel: Use the Multiboot memory map info to inform our paging setup. This makes it possible to run Serenity with more than 64 MB of RAM. Because each physical page is represented by a PhysicalPage object, and such objects are allocated using kmalloc_eternal(), more RAM means more pressure on kmalloc_eternal(), so we're gonna need a better strategy for this. But for now, let's just celebrate that we can use the 128 MB of RAM we've been telling QEMU to run with. :^)
2019-06-09 09:48:58 +00:00
// FIXME: Maybe make use of stuff below the 1MB mark?
if (mmap->addr < (1 * MB))
continue;
Kernel: Refactor MemoryManager to use a Bitmap rather than a Vector This significantly reduces the pressure on the kernel heap when allocating a lot of pages. Previously at about 250MB allocated, the free page list would outgrow the kernel's heap. Given that there is no longer a page list, this does not happen. The next barrier will be the kernel memory used by the page records for in-use memory. This kicks in at about 1GB.
2019-06-11 11:13:02 +00:00
#ifdef MM_DEBUG
kprintf("MM: considering memory at %p - %p\n",
(dword)mmap->addr, (dword)(mmap->addr + mmap->len));
#endif
Kernel: Use the Multiboot memory map info to inform our paging setup. This makes it possible to run Serenity with more than 64 MB of RAM. Because each physical page is represented by a PhysicalPage object, and such objects are allocated using kmalloc_eternal(), more RAM means more pressure on kmalloc_eternal(), so we're gonna need a better strategy for this. But for now, let's just celebrate that we can use the 128 MB of RAM we've been telling QEMU to run with. :^)
2019-06-09 09:48:58 +00:00
for (size_t page_base = mmap->addr; page_base < (mmap->addr + mmap->len); page_base += PAGE_SIZE) {
Kernel: Refactor MemoryManager to use a Bitmap rather than a Vector This significantly reduces the pressure on the kernel heap when allocating a lot of pages. Previously at about 250MB allocated, the free page list would outgrow the kernel's heap. Given that there is no longer a page list, this does not happen. The next barrier will be the kernel memory used by the page records for in-use memory. This kicks in at about 1GB.
2019-06-11 11:13:02 +00:00
auto addr = PhysicalAddress(page_base);
if (page_base < 4 * MB) {
// nothing
} else if (page_base >= 4 * MB && page_base < 5 * MB) {
if (region.is_null() || !region_is_super || region->upper().offset(PAGE_SIZE) != addr) {
m_super_physical_regions.append(PhysicalRegion::create(addr, addr));
region = m_super_physical_regions.last();
region_is_super = true;
} else {
region->expand(region->lower(), addr);
}
} else {
if (region.is_null() || region_is_super || region->upper().offset(PAGE_SIZE) != addr) {
m_user_physical_regions.append(PhysicalRegion::create(addr, addr));
region = m_user_physical_regions.last();
region_is_super = false;
} else {
region->expand(region->lower(), addr);
}
Kernel: Use the Multiboot memory map info to inform our paging setup. This makes it possible to run Serenity with more than 64 MB of RAM. Because each physical page is represented by a PhysicalPage object, and such objects are allocated using kmalloc_eternal(), more RAM means more pressure on kmalloc_eternal(), so we're gonna need a better strategy for this. But for now, let's just celebrate that we can use the 128 MB of RAM we've been telling QEMU to run with. :^)
2019-06-09 09:48:58 +00:00
}
}
}
Let PageDirectory have a PhysicalPage for the PDB instead of being the PDB. This is more efficient than using the wasteful kmalloc_aligned() approach to allocation. It also allows much tighter encapsulation of the class.
2018-12-31 13:58:03 +00:00
Kernel: Refactor MemoryManager to use a Bitmap rather than a Vector This significantly reduces the pressure on the kernel heap when allocating a lot of pages. Previously at about 250MB allocated, the free page list would outgrow the kernel's heap. Given that there is no longer a page list, this does not happen. The next barrier will be the kernel memory used by the page records for in-use memory. This kicks in at about 1GB.
2019-06-11 11:13:02 +00:00
for (auto& region : m_super_physical_regions)
m_super_physical_pages += region->finalize_capacity();
for (auto& region : m_user_physical_regions)
m_user_physical_pages += region->finalize_capacity();
Let PageDirectory have a PhysicalPage for the PDB instead of being the PDB. This is more efficient than using the wasteful kmalloc_aligned() approach to allocation. It also allows much tighter encapsulation of the class.
2018-12-31 13:58:03 +00:00
#ifdef MM_DEBUG
dbgprintf("MM: Installing page directory\n");
#endif
Kernel: More work on Ethernet support.
2019-03-10 22:40:09 +00:00
Kernel: Run clang-format on everything.
2019-06-07 09:43:58 +00:00
asm volatile("movl %%eax, %%cr3" ::"a"(kernel_page_directory().cr3()));
Add basic paging. Only identity maps the bottom 4MB as a start.
2018-10-17 21:13:55 +00:00
asm volatile(
Kernel: Actually zero-fill eagerly committed regions. Previously, calling Region::commit() would make sure to allocate any missing physical pages, but they would contain uninitialized data. This was very obvious when allocating GraphicsBitmaps as they would have garbage pixels rather than being filled with black. The MM quickmap mechanism didn't work when operating on a non-active page directory (which happens when we're in the middle of exec, for example.) This patch changes quickmap to reside in the shared kernel page directory. Also added some missing clobber lists to inline asm that I stumbled on.
2019-01-31 02:57:06 +00:00
"movl %%cr0, %%eax\n"
"orl $0x80000001, %%eax\n"
Kernel: Run clang-format on everything.
2019-06-07 09:43:58 +00:00
"movl %%eax, %%cr0\n" ::
: "%eax", "memory");
Kernel: Introduce threads, and refactor everything in support of it. The scheduler now operates on threads, rather than on processes. Each process has a main thread, and can have any number of additional threads. The process exits when the main thread exits. This patch doesn't actually spawn any additional threads, it merely does all the plumbing needed to make it possible. :^)
2019-03-23 21:03:17 +00:00
#ifdef MM_DEBUG
dbgprintf("MM: Paging initialized.\n");
#endif
Add basic paging. Only identity maps the bottom 4MB as a start.
2018-10-17 21:13:55 +00:00
}
Replace zones with individually tracked physical pages. It's just a simple struct { ref_count, paddr }. This will allow me to implement lazy zeroing and COW pages.
2018-11-05 09:23:00 +00:00
RetainPtr<PhysicalPage> MemoryManager::allocate_page_table(PageDirectory& page_directory, unsigned index)
Implement loading of linked ELF executables. This took me a couple hours. :^) The ELF loading code now allocates a single region for the entire file and creates virtual memory mappings for the sections as needed. Very nice!
2018-10-27 12:56:52 +00:00
{
Make PageDirectory store physical pages in a HashMap. This container is really just there to keep a retain on the individual PhysicalPages for each page table. A HashMap does the job with far greater space efficiency.
2018-12-31 14:10:12 +00:00
ASSERT(!page_directory.m_physical_pages.contains(index));
MM: Allocate page tables from a separate set of physical pages. The old approach only worked because of an overpermissive accident. There's now a concept of supervisor physical pages that can be allocated. They all sit in the low 4 MB of physical memory and are identity mapped, shared between all processes, and only ring 0 can access them.
2019-01-01 01:09:43 +00:00
auto physical_page = allocate_supervisor_physical_page();
Remove MM::allocate_physical_pages() since it wasn't used. Everyone was allocating one page at a time with allocate_physical_page().
2018-11-10 21:14:41 +00:00
if (!physical_page)
return nullptr;
Big, possibly complete sweep of naming changes.
2019-01-31 16:31:23 +00:00
page_directory.m_physical_pages.set(index, physical_page.copy_ref());
Make PageDirectory store physical pages in a HashMap. This container is really just there to keep a retain on the individual PhysicalPages for each page table. A HashMap does the job with far greater space efficiency.
2018-12-31 14:10:12 +00:00
return physical_page;
Fix dumb-but-hard-to-find bug in paging. This was the fix: -process.m_page_directory[0] = m_kernel_page_directory[0]; -process.m_page_directory[1] = m_kernel_page_directory[1]; +process.m_page_directory->entries[0] = m_kernel_page_directory->entries[0]; +process.m_page_directory->entries[1] = m_kernel_page_directory->entries[1]; I spent a good two hours scratching my head, not being able to figure out why user process page directories felt they had ownership of page tables in the kernel page directory. It was because I was copying the entire damn kernel page directory into the process instead of only sharing the two first PDE's. Dang!
2018-11-02 23:31:42 +00:00
}
Kernel: Rename LinearAddress => VirtualAddress.
2019-06-07 10:56:50 +00:00
void MemoryManager::remove_identity_mapping(PageDirectory& page_directory, VirtualAddress vaddr, size_t size)
Fix dumb-but-hard-to-find bug in paging. This was the fix: -process.m_page_directory[0] = m_kernel_page_directory[0]; -process.m_page_directory[1] = m_kernel_page_directory[1]; +process.m_page_directory->entries[0] = m_kernel_page_directory->entries[0]; +process.m_page_directory->entries[1] = m_kernel_page_directory->entries[1]; I spent a good two hours scratching my head, not being able to figure out why user process page directories felt they had ownership of page tables in the kernel page directory. It was because I was copying the entire damn kernel page directory into the process instead of only sharing the two first PDE's. Dang!
2018-11-02 23:31:42 +00:00
{
InterruptDisabler disabler;
Kernel: Rename LinearAddress => VirtualAddress.
2019-06-07 10:56:50 +00:00
// FIXME: ASSERT(vaddr is 4KB aligned);
Fix dumb-but-hard-to-find bug in paging. This was the fix: -process.m_page_directory[0] = m_kernel_page_directory[0]; -process.m_page_directory[1] = m_kernel_page_directory[1]; +process.m_page_directory->entries[0] = m_kernel_page_directory->entries[0]; +process.m_page_directory->entries[1] = m_kernel_page_directory->entries[1]; I spent a good two hours scratching my head, not being able to figure out why user process page directories felt they had ownership of page tables in the kernel page directory. It was because I was copying the entire damn kernel page directory into the process instead of only sharing the two first PDE's. Dang!
2018-11-02 23:31:42 +00:00
for (dword offset = 0; offset < size; offset += PAGE_SIZE) {
Kernel: Rename LinearAddress => VirtualAddress.
2019-06-07 10:56:50 +00:00
auto pte_address = vaddr.offset(offset);
MM: Allocate page tables from a separate set of physical pages. The old approach only worked because of an overpermissive accident. There's now a concept of supervisor physical pages that can be allocated. They all sit in the low 4 MB of physical memory and are identity mapped, shared between all processes, and only ring 0 can access them.
2019-01-01 01:09:43 +00:00
auto pte = ensure_pte(page_directory, pte_address);
Yet more coding style fixes.
2018-12-03 00:38:22 +00:00
pte.set_physical_page_base(0);
pte.set_user_allowed(false);
pte.set_present(true);
pte.set_writable(true);
flush_tlb(pte_address);
Fix dumb-but-hard-to-find bug in paging. This was the fix: -process.m_page_directory[0] = m_kernel_page_directory[0]; -process.m_page_directory[1] = m_kernel_page_directory[1]; +process.m_page_directory->entries[0] = m_kernel_page_directory->entries[0]; +process.m_page_directory->entries[1] = m_kernel_page_directory->entries[1]; I spent a good two hours scratching my head, not being able to figure out why user process page directories felt they had ownership of page tables in the kernel page directory. It was because I was copying the entire damn kernel page directory into the process instead of only sharing the two first PDE's. Dang!
2018-11-02 23:31:42 +00:00
}
Implement loading of linked ELF executables. This took me a couple hours. :^) The ELF loading code now allocates a single region for the entire file and creates virtual memory mappings for the sections as needed. Very nice!
2018-10-27 12:56:52 +00:00
}
Kernel: Rename LinearAddress => VirtualAddress.
2019-06-07 10:56:50 +00:00
auto MemoryManager::ensure_pte(PageDirectory& page_directory, VirtualAddress vaddr) -> PageTableEntry
Add basic paging. Only identity maps the bottom 4MB as a start.
2018-10-17 21:13:55 +00:00
{
Selectively disable interrupts in MM. Also added an ASSERT_INTERRUPTS_DISABLED() macro.
2018-10-25 08:15:28 +00:00
ASSERT_INTERRUPTS_DISABLED();
Kernel: Rename LinearAddress => VirtualAddress.
2019-06-07 10:56:50 +00:00
dword page_directory_index = (vaddr.get() >> 22) & 0x3ff;
dword page_table_index = (vaddr.get() >> 12) & 0x3ff;
Add basic paging. Only identity maps the bottom 4MB as a start.
2018-10-17 21:13:55 +00:00
Let PageDirectory have a PhysicalPage for the PDB instead of being the PDB. This is more efficient than using the wasteful kmalloc_aligned() approach to allocation. It also allows much tighter encapsulation of the class.
2018-12-31 13:58:03 +00:00
PageDirectoryEntry pde = PageDirectoryEntry(&page_directory.entries()[page_directory_index]);
Yet more coding style fixes.
2018-12-03 00:38:22 +00:00
if (!pde.is_present()) {
Snazz up the kprintf() output a bit by giving it its own color.
2018-10-31 19:10:39 +00:00
#ifdef MM_DEBUG
Kernel: Rename LinearAddress => VirtualAddress.
2019-06-07 10:56:50 +00:00
dbgprintf("MM: PDE %u not present (requested for L%x), allocating\n", page_directory_index, vaddr.get());
Snazz up the kprintf() output a bit by giving it its own color.
2018-10-31 19:10:39 +00:00
#endif
Free physical pages allocated for a process's page directory on exit. Also use a ProcessPagingScope instead of region aliasing to implement create-process ELF loading.
2018-11-01 22:04:34 +00:00
if (page_directory_index == 0) {
AK: Improve smart pointer ergonomics a bit.
2019-04-14 00:36:06 +00:00
ASSERT(&page_directory == m_kernel_page_directory);
Big, possibly complete sweep of naming changes.
2019-01-31 16:31:23 +00:00
pde.set_page_table_base((dword)m_page_table_zero);
Yet more coding style fixes.
2018-12-03 00:38:22 +00:00
pde.set_user_allowed(false);
pde.set_present(true);
pde.set_writable(true);
Kernel: Use the Multiboot memory map info to inform our paging setup. This makes it possible to run Serenity with more than 64 MB of RAM. Because each physical page is represented by a PhysicalPage object, and such objects are allocated using kmalloc_eternal(), more RAM means more pressure on kmalloc_eternal(), so we're gonna need a better strategy for this. But for now, let's just celebrate that we can use the 128 MB of RAM we've been telling QEMU to run with. :^)
2019-06-09 09:48:58 +00:00
} else if (page_directory_index == 1) {
ASSERT(&page_directory == m_kernel_page_directory);
pde.set_page_table_base((dword)m_page_table_one);
pde.set_user_allowed(false);
pde.set_present(true);
pde.set_writable(true);
Add basic paging. Only identity maps the bottom 4MB as a start.
2018-10-17 21:13:55 +00:00
} else {
Kernel: Start adding support for E1000 network adapters.
2019-03-10 14:25:33 +00:00
//ASSERT(&page_directory != m_kernel_page_directory.ptr());
Let PageDirectory have a PhysicalPage for the PDB instead of being the PDB. This is more efficient than using the wasteful kmalloc_aligned() approach to allocation. It also allows much tighter encapsulation of the class.
2018-12-31 13:58:03 +00:00
auto page_table = allocate_page_table(page_directory, page_directory_index);
More work on per-process page directories. It basically works now! I spent some time stuck on a problem where processes would clobber each other's stacks. Took me a moment to figure out that their stacks were allocated in the sub-4MB linear address range which is shared between all processes. Oops!
2018-11-01 10:30:48 +00:00
#ifdef MM_DEBUG
MM: Allocate page tables from a separate set of physical pages. The old approach only worked because of an overpermissive accident. There's now a concept of supervisor physical pages that can be allocated. They all sit in the low 4 MB of physical memory and are identity mapped, shared between all processes, and only ring 0 can access them.
2019-01-01 01:09:43 +00:00
dbgprintf("MM: PD K%x (%s) at P%x allocated page table #%u (for L%x) at P%x\n",
Let PageDirectory have a PhysicalPage for the PDB instead of being the PDB. This is more efficient than using the wasteful kmalloc_aligned() approach to allocation. It also allows much tighter encapsulation of the class.
2018-12-31 13:58:03 +00:00
&page_directory,
AK: Improve smart pointer ergonomics a bit.
2019-04-14 00:36:06 +00:00
&page_directory == m_kernel_page_directory ? "Kernel" : "User",
MM: Allocate page tables from a separate set of physical pages. The old approach only worked because of an overpermissive accident. There's now a concept of supervisor physical pages that can be allocated. They all sit in the low 4 MB of physical memory and are identity mapped, shared between all processes, and only ring 0 can access them.
2019-01-01 01:09:43 +00:00
page_directory.cr3(),
Fix dumb-but-hard-to-find bug in paging. This was the fix: -process.m_page_directory[0] = m_kernel_page_directory[0]; -process.m_page_directory[1] = m_kernel_page_directory[1]; +process.m_page_directory->entries[0] = m_kernel_page_directory->entries[0]; +process.m_page_directory->entries[1] = m_kernel_page_directory->entries[1]; I spent a good two hours scratching my head, not being able to figure out why user process page directories felt they had ownership of page tables in the kernel page directory. It was because I was copying the entire damn kernel page directory into the process instead of only sharing the two first PDE's. Dang!
2018-11-02 23:31:42 +00:00
page_directory_index,
Kernel: Rename LinearAddress => VirtualAddress.
2019-06-07 10:56:50 +00:00
vaddr.get(),
Replace zones with individually tracked physical pages. It's just a simple struct { ref_count, paddr }. This will allow me to implement lazy zeroing and COW pages.
2018-11-05 09:23:00 +00:00
page_table->paddr().get());
More work on per-process page directories. It basically works now! I spent some time stuck on a problem where processes would clobber each other's stacks. Took me a moment to figure out that their stacks were allocated in the sub-4MB linear address range which is shared between all processes. Oops!
2018-11-01 10:30:48 +00:00
#endif
Replace zones with individually tracked physical pages. It's just a simple struct { ref_count, paddr }. This will allow me to implement lazy zeroing and COW pages.
2018-11-05 09:23:00 +00:00
Big, possibly complete sweep of naming changes.
2019-01-31 16:31:23 +00:00
pde.set_page_table_base(page_table->paddr().get());
Yet more coding style fixes.
2018-12-03 00:38:22 +00:00
pde.set_user_allowed(true);
pde.set_present(true);
pde.set_writable(true);
Make PageDirectory store physical pages in a HashMap. This container is really just there to keep a retain on the individual PhysicalPages for each page table. A HashMap does the job with far greater space efficiency.
2018-12-31 14:10:12 +00:00
page_directory.m_physical_pages.set(page_directory_index, move(page_table));
Add basic paging. Only identity maps the bottom 4MB as a start.
2018-10-17 21:13:55 +00:00
}
}
Big, possibly complete sweep of naming changes.
2019-01-31 16:31:23 +00:00
return PageTableEntry(&pde.page_table_base()[page_table_index]);
Add basic paging. Only identity maps the bottom 4MB as a start.
2018-10-17 21:13:55 +00:00
}
Kernel: Rename LinearAddress => VirtualAddress.
2019-06-07 10:56:50 +00:00
void MemoryManager::map_protected(VirtualAddress vaddr, size_t length)
Protect the first 4 KB of memory. This makes null pointers crashy, tremendously useful :^)
2018-10-21 19:57:59 +00:00
{
Selectively disable interrupts in MM. Also added an ASSERT_INTERRUPTS_DISABLED() macro.
2018-10-25 08:15:28 +00:00
InterruptDisabler disabler;
Kernel: Replace the last "linear" with "virtual".
2019-06-13 19:42:12 +00:00
ASSERT(vaddr.is_page_aligned());
Fix dumb-but-hard-to-find bug in paging. This was the fix: -process.m_page_directory[0] = m_kernel_page_directory[0]; -process.m_page_directory[1] = m_kernel_page_directory[1]; +process.m_page_directory->entries[0] = m_kernel_page_directory->entries[0]; +process.m_page_directory->entries[1] = m_kernel_page_directory->entries[1]; I spent a good two hours scratching my head, not being able to figure out why user process page directories felt they had ownership of page tables in the kernel page directory. It was because I was copying the entire damn kernel page directory into the process instead of only sharing the two first PDE's. Dang!
2018-11-02 23:31:42 +00:00
for (dword offset = 0; offset < length; offset += PAGE_SIZE) {
Kernel: Rename LinearAddress => VirtualAddress.
2019-06-07 10:56:50 +00:00
auto pte_address = vaddr.offset(offset);
Big, possibly complete sweep of naming changes.
2019-01-31 16:31:23 +00:00
auto pte = ensure_pte(kernel_page_directory(), pte_address);
pte.set_physical_page_base(pte_address.get());
Yet more coding style fixes.
2018-12-03 00:38:22 +00:00
pte.set_user_allowed(false);
pte.set_present(false);
pte.set_writable(false);
Big, possibly complete sweep of naming changes.
2019-01-31 16:31:23 +00:00
flush_tlb(pte_address);
Protect the first 4 KB of memory. This makes null pointers crashy, tremendously useful :^)
2018-10-21 19:57:59 +00:00
}
}
Kernel: Rename LinearAddress => VirtualAddress.
2019-06-07 10:56:50 +00:00
void MemoryManager::create_identity_mapping(PageDirectory& page_directory, VirtualAddress vaddr, size_t size)
Add basic paging. Only identity maps the bottom 4MB as a start.
2018-10-17 21:13:55 +00:00
{
Selectively disable interrupts in MM. Also added an ASSERT_INTERRUPTS_DISABLED() macro.
2018-10-25 08:15:28 +00:00
InterruptDisabler disabler;
Kernel: Rename LinearAddress => VirtualAddress.
2019-06-07 10:56:50 +00:00
ASSERT((vaddr.get() & ~PAGE_MASK) == 0);
Fix dumb-but-hard-to-find bug in paging. This was the fix: -process.m_page_directory[0] = m_kernel_page_directory[0]; -process.m_page_directory[1] = m_kernel_page_directory[1]; +process.m_page_directory->entries[0] = m_kernel_page_directory->entries[0]; +process.m_page_directory->entries[1] = m_kernel_page_directory->entries[1]; I spent a good two hours scratching my head, not being able to figure out why user process page directories felt they had ownership of page tables in the kernel page directory. It was because I was copying the entire damn kernel page directory into the process instead of only sharing the two first PDE's. Dang!
2018-11-02 23:31:42 +00:00
for (dword offset = 0; offset < size; offset += PAGE_SIZE) {
Kernel: Rename LinearAddress => VirtualAddress.
2019-06-07 10:56:50 +00:00
auto pte_address = vaddr.offset(offset);
Big, possibly complete sweep of naming changes.
2019-01-31 16:31:23 +00:00
auto pte = ensure_pte(page_directory, pte_address);
pte.set_physical_page_base(pte_address.get());
Yet more coding style fixes.
2018-12-03 00:38:22 +00:00
pte.set_user_allowed(false);
pte.set_present(true);
pte.set_writable(true);
Big, possibly complete sweep of naming changes.
2019-01-31 16:31:23 +00:00
page_directory.flush(pte_address);
Add basic paging. Only identity maps the bottom 4MB as a start.
2018-10-17 21:13:55 +00:00
}
}
void MemoryManager::initialize()
{
s_the = new MemoryManager;
}
Kernel: Rename LinearAddress => VirtualAddress.
2019-06-07 10:56:50 +00:00
Region* MemoryManager::region_from_vaddr(Process& process, VirtualAddress vaddr)
Implement COW pages! :^) sys$fork() now clones all writable regions with per-page COW bits. The pages are then mapped read-only and we handle a PF by COWing the pages. This is quite delightful. Obviously there's lots of work to do still, and it needs better data structures, but the general concept works.
2018-11-05 12:48:07 +00:00
{
ASSERT_INTERRUPTS_DISABLED();
Kernel: Rename LinearAddress => VirtualAddress.
2019-06-07 10:56:50 +00:00
if (vaddr.get() >= 0xc0000000) {
Kernel: Allocate kernel stacks for threads using the region allocator. This patch moves away from using kmalloc memory for thread kernel stacks. This reduces pressure on kmalloc (16 KB per thread adds up fast) and prevents kernel stack overflow from scribbling all over random unrelated kernel memory.
2019-05-14 09:51:00 +00:00
for (auto& region : MM.m_kernel_regions) {
Kernel: Rename LinearAddress => VirtualAddress.
2019-06-07 10:56:50 +00:00
if (region->contains(vaddr))
Kernel: Allocate kernel stacks for threads using the region allocator. This patch moves away from using kmalloc memory for thread kernel stacks. This reduces pressure on kmalloc (16 KB per thread adds up fast) and prevents kernel stack overflow from scribbling all over random unrelated kernel memory.
2019-05-14 09:51:00 +00:00
return region;
}
}
Implement COW pages! :^) sys$fork() now clones all writable regions with per-page COW bits. The pages are then mapped read-only and we handle a PF by COWing the pages. This is quite delightful. Obviously there's lots of work to do still, and it needs better data structures, but the general concept works.
2018-11-05 12:48:07 +00:00
// FIXME: Use a binary search tree (maybe red/black?) or some other more appropriate data structure!
for (auto& region : process.m_regions) {
Kernel: Rename LinearAddress => VirtualAddress.
2019-06-07 10:56:50 +00:00
if (region->contains(vaddr))
Implement COW pages! :^) sys$fork() now clones all writable regions with per-page COW bits. The pages are then mapped read-only and we handle a PF by COWing the pages. This is quite delightful. Obviously there's lots of work to do still, and it needs better data structures, but the general concept works.
2018-11-05 12:48:07 +00:00
return region.ptr();
}
Kernel: Rename LinearAddress => VirtualAddress.
2019-06-07 10:56:50 +00:00
dbgprintf("%s(%u) Couldn't find region for L%x (CR3=%x)\n", process.name().characters(), process.pid(), vaddr.get(), process.page_directory().cr3());
Kernel: Fix incorrect EFAULTs when syscall would write into COW pages.
2019-01-25 00:39:15 +00:00
return nullptr;
}
Kernel: Rename LinearAddress => VirtualAddress.
2019-06-07 10:56:50 +00:00
const Region* MemoryManager::region_from_vaddr(const Process& process, VirtualAddress vaddr)
Kernel: Fix incorrect EFAULTs when syscall would write into COW pages.
2019-01-25 00:39:15 +00:00
{
Kernel: Rename LinearAddress => VirtualAddress.
2019-06-07 10:56:50 +00:00
if (vaddr.get() >= 0xc0000000) {
Kernel: Allocate kernel stacks for threads using the region allocator. This patch moves away from using kmalloc memory for thread kernel stacks. This reduces pressure on kmalloc (16 KB per thread adds up fast) and prevents kernel stack overflow from scribbling all over random unrelated kernel memory.
2019-05-14 09:51:00 +00:00
for (auto& region : MM.m_kernel_regions) {
Kernel: Rename LinearAddress => VirtualAddress.
2019-06-07 10:56:50 +00:00
if (region->contains(vaddr))
Kernel: Allocate kernel stacks for threads using the region allocator. This patch moves away from using kmalloc memory for thread kernel stacks. This reduces pressure on kmalloc (16 KB per thread adds up fast) and prevents kernel stack overflow from scribbling all over random unrelated kernel memory.
2019-05-14 09:51:00 +00:00
return region;
}
}
Kernel: Fix incorrect EFAULTs when syscall would write into COW pages.
2019-01-25 00:39:15 +00:00
// FIXME: Use a binary search tree (maybe red/black?) or some other more appropriate data structure!
for (auto& region : process.m_regions) {
Kernel: Rename LinearAddress => VirtualAddress.
2019-06-07 10:56:50 +00:00
if (region->contains(vaddr))
Kernel: Fix incorrect EFAULTs when syscall would write into COW pages.
2019-01-25 00:39:15 +00:00
return region.ptr();
}
Kernel: Rename LinearAddress => VirtualAddress.
2019-06-07 10:56:50 +00:00
dbgprintf("%s(%u) Couldn't find region for L%x (CR3=%x)\n", process.name().characters(), process.pid(), vaddr.get(), process.page_directory().cr3());
Add a VMO pointer to VNode. This way, if anyone tries to map an already mapped file, we share the VMO.
2018-11-08 14:39:26 +00:00
return nullptr;
Implement COW pages! :^) sys$fork() now clones all writable regions with per-page COW bits. The pages are then mapped read-only and we handle a PF by COWing the pages. This is quite delightful. Obviously there's lots of work to do still, and it needs better data structures, but the general concept works.
2018-11-05 12:48:07 +00:00
}
Kernel: Refactor Region/PageDirectory ownership model. Make PageDirectory retainable and have each Region co-own the PageDirectory they're mapped into. When unmapped, Region has no associated PageDirectory. This allows Region to automatically unmap itself when destroyed.
2019-01-22 04:01:00 +00:00
bool MemoryManager::zero_page(Region& region, unsigned page_index_in_region)
Add basic zero faults. mmap() will now map uncommitted pages that get allocated and zeroed upon the first access. I also made /proc/PID/vm show number of "committed" bytes in each region. This is so cool! :^)
2018-11-19 01:17:20 +00:00
{
ASSERT_INTERRUPTS_DISABLED();
auto& vmo = region.vmo();
Kernel: If someone else zero-fills a shared VMO page, don't freak out. Just map the new page and move on.
2019-02-20 20:33:07 +00:00
auto& vmo_page = vmo.physical_pages()[region.first_page_index() + page_index_in_region];
sti();
LOCKER(vmo.m_paging_lock);
cli();
if (!vmo_page.is_null()) {
#ifdef PAGE_FAULT_DEBUG
dbgprintf("MM: zero_page() but page already present. Fine with me!\n");
#endif
remap_region_page(region, page_index_in_region, true);
return true;
}
Kernel: Refactor MemoryManager to use a Bitmap rather than a Vector This significantly reduces the pressure on the kernel heap when allocating a lot of pages. Previously at about 250MB allocated, the free page list would outgrow the kernel's heap. Given that there is no longer a page list, this does not happen. The next barrier will be the kernel memory used by the page records for in-use memory. This kicks in at about 1GB.
2019-06-11 11:13:02 +00:00
auto physical_page = allocate_user_physical_page(ShouldZeroFill::Yes);
Add basic zero faults. mmap() will now map uncommitted pages that get allocated and zeroed upon the first access. I also made /proc/PID/vm show number of "committed" bytes in each region. This is so cool! :^)
2018-11-19 01:17:20 +00:00
#ifdef PAGE_FAULT_DEBUG
dbgprintf(" >> ZERO P%x\n", physical_page->paddr().get());
#endif
Kernel: Encapsulate the Region's COW map a bit better.
2019-05-14 15:31:57 +00:00
region.set_should_cow(page_index_in_region, false);
Add basic zero faults. mmap() will now map uncommitted pages that get allocated and zeroed upon the first access. I also made /proc/PID/vm show number of "committed" bytes in each region. This is so cool! :^)
2018-11-19 01:17:20 +00:00
vmo.physical_pages()[page_index_in_region] = move(physical_page);
Kernel: Refactor Region/PageDirectory ownership model. Make PageDirectory retainable and have each Region co-own the PageDirectory they're mapped into. When unmapped, Region has no associated PageDirectory. This allows Region to automatically unmap itself when destroyed.
2019-01-22 04:01:00 +00:00
remap_region_page(region, page_index_in_region, true);
Add basic zero faults. mmap() will now map uncommitted pages that get allocated and zeroed upon the first access. I also made /proc/PID/vm show number of "committed" bytes in each region. This is so cool! :^)
2018-11-19 01:17:20 +00:00
return true;
}
Kernel: Refactor Region/PageDirectory ownership model. Make PageDirectory retainable and have each Region co-own the PageDirectory they're mapped into. When unmapped, Region has no associated PageDirectory. This allows Region to automatically unmap itself when destroyed.
2019-01-22 04:01:00 +00:00
bool MemoryManager::copy_on_write(Region& region, unsigned page_index_in_region)
Implement COW pages! :^) sys$fork() now clones all writable regions with per-page COW bits. The pages are then mapped read-only and we handle a PF by COWing the pages. This is quite delightful. Obviously there's lots of work to do still, and it needs better data structures, but the general concept works.
2018-11-05 12:48:07 +00:00
{
Only COW on fault if the physical page has retain_count > 1. This makes COW pages lazily-but-transparently revert back to read/write.
2018-11-05 13:10:18 +00:00
ASSERT_INTERRUPTS_DISABLED();
Refactor the virtual memory object model a bit: Process now has a number of Regions. Each Region is backed by a VMObject. A VMObject can be file-backed or anonymous. These can be shared.
2018-11-08 13:35:30 +00:00
auto& vmo = region.vmo();
if (vmo.physical_pages()[page_index_in_region]->retain_count() == 1) {
Only COW on fault if the physical page has retain_count > 1. This makes COW pages lazily-but-transparently revert back to read/write.
2018-11-05 13:10:18 +00:00
#ifdef PAGE_FAULT_DEBUG
dbgprintf(" >> It's a COW page but nobody is sharing it anymore. Remap r/w\n");
#endif
Kernel: Encapsulate the Region's COW map a bit better.
2019-05-14 15:31:57 +00:00
region.set_should_cow(page_index_in_region, false);
Kernel: Refactor Region/PageDirectory ownership model. Make PageDirectory retainable and have each Region co-own the PageDirectory they're mapped into. When unmapped, Region has no associated PageDirectory. This allows Region to automatically unmap itself when destroyed.
2019-01-22 04:01:00 +00:00
remap_region_page(region, page_index_in_region, true);
Only COW on fault if the physical page has retain_count > 1. This makes COW pages lazily-but-transparently revert back to read/write.
2018-11-05 13:10:18 +00:00
return true;
}
Implement COW pages! :^) sys$fork() now clones all writable regions with per-page COW bits. The pages are then mapped read-only and we handle a PF by COWing the pages. This is quite delightful. Obviously there's lots of work to do still, and it needs better data structures, but the general concept works.
2018-11-05 12:48:07 +00:00
#ifdef PAGE_FAULT_DEBUG
dbgprintf(" >> It's a COW page and it's time to COW!\n");
#endif
Refactor the virtual memory object model a bit: Process now has a number of Regions. Each Region is backed by a VMObject. A VMObject can be file-backed or anonymous. These can be shared.
2018-11-08 13:35:30 +00:00
auto physical_page_to_copy = move(vmo.physical_pages()[page_index_in_region]);
Kernel: Refactor MemoryManager to use a Bitmap rather than a Vector This significantly reduces the pressure on the kernel heap when allocating a lot of pages. Previously at about 250MB allocated, the free page list would outgrow the kernel's heap. Given that there is no longer a page list, this does not happen. The next barrier will be the kernel memory used by the page records for in-use memory. This kicks in at about 1GB.
2019-06-11 11:13:02 +00:00
auto physical_page = allocate_user_physical_page(ShouldZeroFill::No);
Remove MM::allocate_physical_pages() since it wasn't used. Everyone was allocating one page at a time with allocate_physical_page().
2018-11-10 21:14:41 +00:00
byte* dest_ptr = quickmap_page(*physical_page);
Kernel: Rename LinearAddress => VirtualAddress.
2019-06-07 10:56:50 +00:00
const byte* src_ptr = region.vaddr().offset(page_index_in_region * PAGE_SIZE).as_ptr();
Implement COW pages! :^) sys$fork() now clones all writable regions with per-page COW bits. The pages are then mapped read-only and we handle a PF by COWing the pages. This is quite delightful. Obviously there's lots of work to do still, and it needs better data structures, but the general concept works.
2018-11-05 12:48:07 +00:00
#ifdef PAGE_FAULT_DEBUG
Remove MM::allocate_physical_pages() since it wasn't used. Everyone was allocating one page at a time with allocate_physical_page().
2018-11-10 21:14:41 +00:00
dbgprintf(" >> COW P%x <- P%x\n", physical_page->paddr().get(), physical_page_to_copy->paddr().get());
Implement COW pages! :^) sys$fork() now clones all writable regions with per-page COW bits. The pages are then mapped read-only and we handle a PF by COWing the pages. This is quite delightful. Obviously there's lots of work to do still, and it needs better data structures, but the general concept works.
2018-11-05 12:48:07 +00:00
#endif
memcpy(dest_ptr, src_ptr, PAGE_SIZE);
Remove MM::allocate_physical_pages() since it wasn't used. Everyone was allocating one page at a time with allocate_physical_page().
2018-11-10 21:14:41 +00:00
vmo.physical_pages()[page_index_in_region] = move(physical_page);
Implement COW pages! :^) sys$fork() now clones all writable regions with per-page COW bits. The pages are then mapped read-only and we handle a PF by COWing the pages. This is quite delightful. Obviously there's lots of work to do still, and it needs better data structures, but the general concept works.
2018-11-05 12:48:07 +00:00
unquickmap_page();
Kernel: Encapsulate the Region's COW map a bit better.
2019-05-14 15:31:57 +00:00
region.set_should_cow(page_index_in_region, false);
Kernel: Refactor Region/PageDirectory ownership model. Make PageDirectory retainable and have each Region co-own the PageDirectory they're mapped into. When unmapped, Region has no associated PageDirectory. This allows Region to automatically unmap itself when destroyed.
2019-01-22 04:01:00 +00:00
remap_region_page(region, page_index_in_region, true);
Implement COW pages! :^) sys$fork() now clones all writable regions with per-page COW bits. The pages are then mapped read-only and we handle a PF by COWing the pages. This is quite delightful. Obviously there's lots of work to do still, and it needs better data structures, but the general concept works.
2018-11-05 12:48:07 +00:00
return true;
}
Kernel: Refactor Region/PageDirectory ownership model. Make PageDirectory retainable and have each Region co-own the PageDirectory they're mapped into. When unmapped, Region has no associated PageDirectory. This allows Region to automatically unmap itself when destroyed.
2019-01-22 04:01:00 +00:00
bool MemoryManager::page_in_from_inode(Region& region, unsigned page_index_in_region)
Support basic mmap'ing of a file! All right, we can now mmap() a file and it gets magically paged in from fs in response to an NP page fault. This is really cool :^) I need to refactor this to support sharing of read-only file-backed pages, but it's cool to just have something working.
2018-11-08 11:59:16 +00:00
{
Kernel: Finally stop exposing Region members to the public.
2019-01-24 17:09:46 +00:00
ASSERT(region.page_directory());
Refactor the virtual memory object model a bit: Process now has a number of Regions. Each Region is backed by a VMObject. A VMObject can be file-backed or anonymous. These can be shared.
2018-11-08 13:35:30 +00:00
auto& vmo = region.vmo();
ASSERT(!vmo.is_anonymous());
Get rid of Vnode concept. We already have an abstraction between Process and Inode/CharacterDevice/FIFO and it's called FileDescriptor. :^)
2019-01-16 11:57:07 +00:00
ASSERT(vmo.inode());
Make font loading use mmap(). This exposed a serious race condition in page_in_from_inode(). Reordered the logic and added a paging lock to VMObject. Now, only one process can page in from a VMObject at a time. There are definitely ways to optimize this, for instance by making the locking be per-page instead. It's not something that I'm going to worry about right now though.
2019-02-03 00:36:25 +00:00
Teach Process::exec() about the magic of file-backed VMO's. This is really sweet! :^) The four instances of /bin/sh spawned at startup now share their read-only text pages. There are problems and limitations here, and plenty of room for improvement. But it kinda works.
2018-11-08 20:20:09 +00:00
auto& vmo_page = vmo.physical_pages()[region.first_page_index() + page_index_in_region];
Make font loading use mmap(). This exposed a serious race condition in page_in_from_inode(). Reordered the logic and added a paging lock to VMObject. Now, only one process can page in from a VMObject at a time. There are definitely ways to optimize this, for instance by making the locking be per-page instead. It's not something that I'm going to worry about right now though.
2019-02-03 00:36:25 +00:00
Kernel: Add an InterruptFlagSaver helper class. This is useful instead of InterruptDisabler in some cases.
2019-02-05 10:13:34 +00:00
InterruptFlagSaver saver;
Make font loading use mmap(). This exposed a serious race condition in page_in_from_inode(). Reordered the logic and added a paging lock to VMObject. Now, only one process can page in from a VMObject at a time. There are definitely ways to optimize this, for instance by making the locking be per-page instead. It's not something that I'm going to worry about right now though.
2019-02-03 00:36:25 +00:00
Kernel: Add an InterruptFlagSaver helper class. This is useful instead of InterruptDisabler in some cases.
2019-02-05 10:13:34 +00:00
sti();
Make font loading use mmap(). This exposed a serious race condition in page_in_from_inode(). Reordered the logic and added a paging lock to VMObject. Now, only one process can page in from a VMObject at a time. There are definitely ways to optimize this, for instance by making the locking be per-page instead. It's not something that I'm going to worry about right now though.
2019-02-03 00:36:25 +00:00
LOCKER(vmo.m_paging_lock);
Kernel: Add an InterruptFlagSaver helper class. This is useful instead of InterruptDisabler in some cases.
2019-02-05 10:13:34 +00:00
cli();
Make font loading use mmap(). This exposed a serious race condition in page_in_from_inode(). Reordered the logic and added a paging lock to VMObject. Now, only one process can page in from a VMObject at a time. There are definitely ways to optimize this, for instance by making the locking be per-page instead. It's not something that I'm going to worry about right now though.
2019-02-03 00:36:25 +00:00
if (!vmo_page.is_null()) {
Clean up some uninteresting log spam.
2019-02-06 10:32:23 +00:00
dbgprintf("MM: page_in_from_inode() but page already present. Fine with me!\n");
Make font loading use mmap(). This exposed a serious race condition in page_in_from_inode(). Reordered the logic and added a paging lock to VMObject. Now, only one process can page in from a VMObject at a time. There are definitely ways to optimize this, for instance by making the locking be per-page instead. It's not something that I'm going to worry about right now though.
2019-02-03 00:36:25 +00:00
remap_region_page(region, page_index_in_region, true);
return true;
Support basic mmap'ing of a file! All right, we can now mmap() a file and it gets magically paged in from fs in response to an NP page fault. This is really cool :^) I need to refactor this to support sharing of read-only file-backed pages, but it's cool to just have something working.
2018-11-08 11:59:16 +00:00
}
Make font loading use mmap(). This exposed a serious race condition in page_in_from_inode(). Reordered the logic and added a paging lock to VMObject. Now, only one process can page in from a VMObject at a time. There are definitely ways to optimize this, for instance by making the locking be per-page instead. It's not something that I'm going to worry about right now though.
2019-02-03 00:36:25 +00:00
Rage hacking to get bash to run. It finally runs. So cool! :^)
2018-11-11 14:36:40 +00:00
#ifdef MM_DEBUG
mmap all the font files! Font now uses the same in-memory format as the font files we have on disk. This allows us to simply mmap() the font files and not use any additional memory for them. Very cool! :^) Hacking on this exposed a bug in file-backed VMObjects where the first client to instantiate a VMObject for a specific inode also got to decide its size. Since file-backed VMObjects always have the same size as the underlying file, this made no sense, so I removed the ability to even set a size in that case.
2019-02-05 05:43:33 +00:00
dbgprintf("MM: page_in_from_inode ready to read from inode\n");
Rage hacking to get bash to run. It finally runs. So cool! :^)
2018-11-11 14:36:40 +00:00
#endif
Kernel: Add an InterruptFlagSaver helper class. This is useful instead of InterruptDisabler in some cases.
2019-02-05 10:13:34 +00:00
sti();
Make font loading use mmap(). This exposed a serious race condition in page_in_from_inode(). Reordered the logic and added a paging lock to VMObject. Now, only one process can page in from a VMObject at a time. There are definitely ways to optimize this, for instance by making the locking be per-page instead. It's not something that I'm going to worry about right now though.
2019-02-03 00:36:25 +00:00
byte page_buffer[PAGE_SIZE];
auto& inode = *vmo.inode();
auto nread = inode.read_bytes(vmo.inode_offset() + ((region.first_page_index() + page_index_in_region) * PAGE_SIZE), PAGE_SIZE, page_buffer, nullptr);
Support basic mmap'ing of a file! All right, we can now mmap() a file and it gets magically paged in from fs in response to an NP page fault. This is really cool :^) I need to refactor this to support sharing of read-only file-backed pages, but it's cool to just have something working.
2018-11-08 11:59:16 +00:00
if (nread < 0) {
Get rid of Vnode concept. We already have an abstraction between Process and Inode/CharacterDevice/FIFO and it's called FileDescriptor. :^)
2019-01-16 11:57:07 +00:00
kprintf("MM: page_in_from_inode had error (%d) while reading!\n", nread);
Support basic mmap'ing of a file! All right, we can now mmap() a file and it gets magically paged in from fs in response to an NP page fault. This is really cool :^) I need to refactor this to support sharing of read-only file-backed pages, but it's cool to just have something working.
2018-11-08 11:59:16 +00:00
return false;
}
if (nread < PAGE_SIZE) {
// If we read less than a page, zero out the rest to avoid leaking uninitialized data.
Make font loading use mmap(). This exposed a serious race condition in page_in_from_inode(). Reordered the logic and added a paging lock to VMObject. Now, only one process can page in from a VMObject at a time. There are definitely ways to optimize this, for instance by making the locking be per-page instead. It's not something that I'm going to worry about right now though.
2019-02-03 00:36:25 +00:00
memset(page_buffer + nread, 0, PAGE_SIZE - nread);
Support basic mmap'ing of a file! All right, we can now mmap() a file and it gets magically paged in from fs in response to an NP page fault. This is really cool :^) I need to refactor this to support sharing of read-only file-backed pages, but it's cool to just have something working.
2018-11-08 11:59:16 +00:00
}
Kernel: Add an InterruptFlagSaver helper class. This is useful instead of InterruptDisabler in some cases.
2019-02-05 10:13:34 +00:00
cli();
Kernel: Refactor MemoryManager to use a Bitmap rather than a Vector This significantly reduces the pressure on the kernel heap when allocating a lot of pages. Previously at about 250MB allocated, the free page list would outgrow the kernel's heap. Given that there is no longer a page list, this does not happen. The next barrier will be the kernel memory used by the page records for in-use memory. This kicks in at about 1GB.
2019-06-11 11:13:02 +00:00
vmo_page = allocate_user_physical_page(ShouldZeroFill::No);
Make font loading use mmap(). This exposed a serious race condition in page_in_from_inode(). Reordered the logic and added a paging lock to VMObject. Now, only one process can page in from a VMObject at a time. There are definitely ways to optimize this, for instance by making the locking be per-page instead. It's not something that I'm going to worry about right now though.
2019-02-03 00:36:25 +00:00
if (vmo_page.is_null()) {
kprintf("MM: page_in_from_inode was unable to allocate a physical page\n");
return false;
}
remap_region_page(region, page_index_in_region, true);
Kernel: Rename LinearAddress => VirtualAddress.
2019-06-07 10:56:50 +00:00
byte* dest_ptr = region.vaddr().offset(page_index_in_region * PAGE_SIZE).as_ptr();
Make font loading use mmap(). This exposed a serious race condition in page_in_from_inode(). Reordered the logic and added a paging lock to VMObject. Now, only one process can page in from a VMObject at a time. There are definitely ways to optimize this, for instance by making the locking be per-page instead. It's not something that I'm going to worry about right now though.
2019-02-03 00:36:25 +00:00
memcpy(dest_ptr, page_buffer, PAGE_SIZE);
Support basic mmap'ing of a file! All right, we can now mmap() a file and it gets magically paged in from fs in response to an NP page fault. This is really cool :^) I need to refactor this to support sharing of read-only file-backed pages, but it's cool to just have something working.
2018-11-08 11:59:16 +00:00
return true;
}
Tidy up the page fault code a bit in preparation.
2018-11-05 09:29:19 +00:00
PageFaultResponse MemoryManager::handle_page_fault(const PageFault& fault)
More paging stuff. The test userspace process now runs at linear address 0x300000 which is mapped to a dynamically allocated page from the MemoryManager. Cool!
2018-10-18 11:05:00 +00:00
{
Selectively disable interrupts in MM. Also added an ASSERT_INTERRUPTS_DISABLED() macro.
2018-10-25 08:15:28 +00:00
ASSERT_INTERRUPTS_DISABLED();
Kernel: Introduce threads, and refactor everything in support of it. The scheduler now operates on threads, rather than on processes. Each process has a main thread, and can have any number of additional threads. The process exits when the main thread exits. This patch doesn't actually spawn any additional threads, it merely does all the plumbing needed to make it possible. :^)
2019-03-23 21:03:17 +00:00
ASSERT(current);
Implement COW pages! :^) sys$fork() now clones all writable regions with per-page COW bits. The pages are then mapped read-only and we handle a PF by COWing the pages. This is quite delightful. Obviously there's lots of work to do still, and it needs better data structures, but the general concept works.
2018-11-05 12:48:07 +00:00
#ifdef PAGE_FAULT_DEBUG
Kernel: Rename LinearAddress => VirtualAddress.
2019-06-07 10:56:50 +00:00
dbgprintf("MM: handle_page_fault(%w) at L%x\n", fault.code(), fault.vaddr().get());
Implement COW pages! :^) sys$fork() now clones all writable regions with per-page COW bits. The pages are then mapped read-only and we handle a PF by COWing the pages. This is quite delightful. Obviously there's lots of work to do still, and it needs better data structures, but the general concept works.
2018-11-05 12:48:07 +00:00
#endif
Kernel: Rename LinearAddress => VirtualAddress.
2019-06-07 10:56:50 +00:00
ASSERT(fault.vaddr() != m_quickmap_addr);
auto* region = region_from_vaddr(current->process(), fault.vaddr());
Add a VMO pointer to VNode. This way, if anyone tries to map an already mapped file, we share the VMO.
2018-11-08 14:39:26 +00:00
if (!region) {
Kernel: Rename LinearAddress => VirtualAddress.
2019-06-07 10:56:50 +00:00
kprintf("NP(error) fault at invalid address L%x\n", fault.vaddr().get());
Add a VMO pointer to VNode. This way, if anyone tries to map an already mapped file, we share the VMO.
2018-11-08 14:39:26 +00:00
return PageFaultResponse::ShouldCrash;
}
Kernel: Rename LinearAddress => VirtualAddress.
2019-06-07 10:56:50 +00:00
auto page_index_in_region = region->page_index_from_address(fault.vaddr());
Tidy up the page fault code a bit in preparation.
2018-11-05 09:29:19 +00:00
if (fault.is_not_present()) {
Get rid of Vnode concept. We already have an abstraction between Process and Inode/CharacterDevice/FIFO and it's called FileDescriptor. :^)
2019-01-16 11:57:07 +00:00
if (region->vmo().inode()) {
Kernel: Put page fault logspam behind PAGE_FAULT_DEBUG.
2019-02-10 10:37:59 +00:00
#ifdef PAGE_FAULT_DEBUG
Get rid of Vnode concept. We already have an abstraction between Process and Inode/CharacterDevice/FIFO and it's called FileDescriptor. :^)
2019-01-16 11:57:07 +00:00
dbgprintf("NP(inode) fault in Region{%p}[%u]\n", region, page_index_in_region);
Kernel: Put page fault logspam behind PAGE_FAULT_DEBUG.
2019-02-10 10:37:59 +00:00
#endif
Kernel: Refactor Region/PageDirectory ownership model. Make PageDirectory retainable and have each Region co-own the PageDirectory they're mapped into. When unmapped, Region has no associated PageDirectory. This allows Region to automatically unmap itself when destroyed.
2019-01-22 04:01:00 +00:00
page_in_from_inode(*region, page_index_in_region);
Support basic mmap'ing of a file! All right, we can now mmap() a file and it gets magically paged in from fs in response to an NP page fault. This is really cool :^) I need to refactor this to support sharing of read-only file-backed pages, but it's cool to just have something working.
2018-11-08 11:59:16 +00:00
return PageFaultResponse::Continue;
} else {
Kernel: Put page fault logspam behind PAGE_FAULT_DEBUG.
2019-02-10 10:37:59 +00:00
#ifdef PAGE_FAULT_DEBUG
Add basic zero faults. mmap() will now map uncommitted pages that get allocated and zeroed upon the first access. I also made /proc/PID/vm show number of "committed" bytes in each region. This is so cool! :^)
2018-11-19 01:17:20 +00:00
dbgprintf("NP(zero) fault in Region{%p}[%u]\n", region, page_index_in_region);
Kernel: Put page fault logspam behind PAGE_FAULT_DEBUG.
2019-02-10 10:37:59 +00:00
#endif
Kernel: Refactor Region/PageDirectory ownership model. Make PageDirectory retainable and have each Region co-own the PageDirectory they're mapped into. When unmapped, Region has no associated PageDirectory. This allows Region to automatically unmap itself when destroyed.
2019-01-22 04:01:00 +00:00
zero_page(*region, page_index_in_region);
Add basic zero faults. mmap() will now map uncommitted pages that get allocated and zeroed upon the first access. I also made /proc/PID/vm show number of "committed" bytes in each region. This is so cool! :^)
2018-11-19 01:17:20 +00:00
return PageFaultResponse::Continue;
Support basic mmap'ing of a file! All right, we can now mmap() a file and it gets magically paged in from fs in response to an NP page fault. This is really cool :^) I need to refactor this to support sharing of read-only file-backed pages, but it's cool to just have something working.
2018-11-08 11:59:16 +00:00
}
Tidy up the page fault code a bit in preparation.
2018-11-05 09:29:19 +00:00
} else if (fault.is_protection_violation()) {
Kernel: Encapsulate the Region's COW map a bit better.
2019-05-14 15:31:57 +00:00
if (region->should_cow(page_index_in_region)) {
Kernel: Put page fault logspam behind PAGE_FAULT_DEBUG.
2019-02-10 10:37:59 +00:00
#ifdef PAGE_FAULT_DEBUG
Support basic mmap'ing of a file! All right, we can now mmap() a file and it gets magically paged in from fs in response to an NP page fault. This is really cool :^) I need to refactor this to support sharing of read-only file-backed pages, but it's cool to just have something working.
2018-11-08 11:59:16 +00:00
dbgprintf("PV(cow) fault in Region{%p}[%u]\n", region, page_index_in_region);
Kernel: Put page fault logspam behind PAGE_FAULT_DEBUG.
2019-02-10 10:37:59 +00:00
#endif
Kernel: Refactor Region/PageDirectory ownership model. Make PageDirectory retainable and have each Region co-own the PageDirectory they're mapped into. When unmapped, Region has no associated PageDirectory. This allows Region to automatically unmap itself when destroyed.
2019-01-22 04:01:00 +00:00
bool success = copy_on_write(*region, page_index_in_region);
Implement COW pages! :^) sys$fork() now clones all writable regions with per-page COW bits. The pages are then mapped read-only and we handle a PF by COWing the pages. This is quite delightful. Obviously there's lots of work to do still, and it needs better data structures, but the general concept works.
2018-11-05 12:48:07 +00:00
ASSERT(success);
return PageFaultResponse::Continue;
}
Kernel: Rename LinearAddress => VirtualAddress.
2019-06-07 10:56:50 +00:00
kprintf("PV(error) fault in Region{%p}[%u] at L%x\n", region, page_index_in_region, fault.vaddr().get());
Implement COW pages! :^) sys$fork() now clones all writable regions with per-page COW bits. The pages are then mapped read-only and we handle a PF by COWing the pages. This is quite delightful. Obviously there's lots of work to do still, and it needs better data structures, but the general concept works.
2018-11-05 12:48:07 +00:00
} else {
ASSERT_NOT_REACHED();
More paging stuff. The test userspace process now runs at linear address 0x300000 which is mapped to a dynamically allocated page from the MemoryManager. Cool!
2018-10-18 11:05:00 +00:00
}
Implement COW pages! :^) sys$fork() now clones all writable regions with per-page COW bits. The pages are then mapped read-only and we handle a PF by COWing the pages. This is quite delightful. Obviously there's lots of work to do still, and it needs better data structures, but the general concept works.
2018-11-05 12:48:07 +00:00
More paging stuff. The test userspace process now runs at linear address 0x300000 which is mapped to a dynamically allocated page from the MemoryManager. Cool!
2018-10-18 11:05:00 +00:00
return PageFaultResponse::ShouldCrash;
}
Kernel: Allocate kernel stacks for threads using the region allocator. This patch moves away from using kmalloc memory for thread kernel stacks. This reduces pressure on kmalloc (16 KB per thread adds up fast) and prevents kernel stack overflow from scribbling all over random unrelated kernel memory.
2019-05-14 09:51:00 +00:00
RetainPtr<Region> MemoryManager::allocate_kernel_region(size_t size, String&& name)
{
InterruptDisabler disabler;
ASSERT(!(size % PAGE_SIZE));
Kernel: Let PageDirectory own the associated RangeAllocator. Since we transition to a new PageDirectory on exec(), we need a matching RangeAllocator to go with the new directory. Instead of juggling this in Process and MemoryManager, simply attach the RangeAllocator to the PageDirectory instead. Fixes #61.
2019-05-20 02:46:29 +00:00
auto range = kernel_page_directory().range_allocator().allocate_anywhere(size);
Kernel: Use a RangeAllocator for kernel-only virtual space allocation too.
2019-05-17 02:02:29 +00:00
ASSERT(range.is_valid());
Kernel: Make the Process allocate_region* API's understand "int prot". Instead of having to inspect 'prot' at every call site, make the Process API's take care of that so we can just pass it through.
2019-05-30 14:14:37 +00:00
auto region = adopt(*new Region(range, move(name), PROT_READ | PROT_WRITE | PROT_EXEC, false));
Kernel: Use a RangeAllocator for kernel-only virtual space allocation too.
2019-05-17 02:02:29 +00:00
MM.map_region_at_address(*m_kernel_page_directory, *region, range.base(), false);
Kernel: Make allocate_kernel_region() commit the region automatically. This means that kernel regions will eagerly get physical pages allocated. It would be nice to zero-fill these on demand instead, but that would require a bunch of MemoryManager changes.
2019-05-14 13:36:24 +00:00
// FIXME: It would be cool if these could zero-fill on demand instead.
region->commit();
Kernel: Allocate kernel stacks for threads using the region allocator. This patch moves away from using kmalloc memory for thread kernel stacks. This reduces pressure on kmalloc (16 KB per thread adds up fast) and prevents kernel stack overflow from scribbling all over random unrelated kernel memory.
2019-05-14 09:51:00 +00:00
return region;
}
VM: Pass a PhysicalPage by rvalue reference when returning it to the freelist. This makes no functional difference, but it makes it clear that MemoryManager and PhysicalRegion take over the actual physical page represented by this PhysicalPage instance.
2019-06-14 11:56:21 +00:00
void MemoryManager::deallocate_user_physical_page(PhysicalPage&& page)
Kernel: Refactor MemoryManager to use a Bitmap rather than a Vector This significantly reduces the pressure on the kernel heap when allocating a lot of pages. Previously at about 250MB allocated, the free page list would outgrow the kernel's heap. Given that there is no longer a page list, this does not happen. The next barrier will be the kernel memory used by the page records for in-use memory. This kicks in at about 1GB.
2019-06-11 11:13:02 +00:00
{
for (auto& region : m_user_physical_regions) {
if (!region->contains(page)) {
kprintf(
"MM: deallocate_user_physical_page: %p not in %p -> %p\n",
page.paddr(), region->lower().get(), region->upper().get());
continue;
}
VM: Pass a PhysicalPage by rvalue reference when returning it to the freelist. This makes no functional difference, but it makes it clear that MemoryManager and PhysicalRegion take over the actual physical page represented by this PhysicalPage instance.
2019-06-14 11:56:21 +00:00
region->return_page(move(page));
Kernel: Refactor MemoryManager to use a Bitmap rather than a Vector This significantly reduces the pressure on the kernel heap when allocating a lot of pages. Previously at about 250MB allocated, the free page list would outgrow the kernel's heap. Given that there is no longer a page list, this does not happen. The next barrier will be the kernel memory used by the page records for in-use memory. This kicks in at about 1GB.
2019-06-11 11:13:02 +00:00
m_user_physical_pages_used--;
return;
}
kprintf("MM: deallocate_user_physical_page couldn't figure out region for user page @ %p\n", page.paddr());
ASSERT_NOT_REACHED();
}
RetainPtr<PhysicalPage> MemoryManager::allocate_user_physical_page(ShouldZeroFill should_zero_fill)
Support basic mmap'ing of a file! All right, we can now mmap() a file and it gets magically paged in from fs in response to an NP page fault. This is really cool :^) I need to refactor this to support sharing of read-only file-backed pages, but it's cool to just have something working.
2018-11-08 11:59:16 +00:00
{
InterruptDisabler disabler;
Kernel: Refactor MemoryManager to use a Bitmap rather than a Vector This significantly reduces the pressure on the kernel heap when allocating a lot of pages. Previously at about 250MB allocated, the free page list would outgrow the kernel's heap. Given that there is no longer a page list, this does not happen. The next barrier will be the kernel memory used by the page records for in-use memory. This kicks in at about 1GB.
2019-06-11 11:13:02 +00:00
RetainPtr<PhysicalPage> page = nullptr;
for (auto& region : m_user_physical_regions) {
page = region->take_free_page(false);
if (page.is_null())
continue;
}
if (!page) {
if (m_user_physical_regions.is_empty()) {
kprintf("MM: no user physical regions available (?)\n");
}
kprintf("MM: no user physical pages available\n");
WindowServer: Support resizing windows. This is pretty limited and not entirely stable, but it does work! :^)
2019-02-20 14:34:55 +00:00
ASSERT_NOT_REACHED();
Kernel: Run clang-format on everything.
2019-06-07 09:43:58 +00:00
return {};
WindowServer: Support resizing windows. This is pretty limited and not entirely stable, but it does work! :^)
2019-02-20 14:34:55 +00:00
}
Kernel: Refactor MemoryManager to use a Bitmap rather than a Vector This significantly reduces the pressure on the kernel heap when allocating a lot of pages. Previously at about 250MB allocated, the free page list would outgrow the kernel's heap. Given that there is no longer a page list, this does not happen. The next barrier will be the kernel memory used by the page records for in-use memory. This kicks in at about 1GB.
2019-06-11 11:13:02 +00:00
Support basic mmap'ing of a file! All right, we can now mmap() a file and it gets magically paged in from fs in response to an NP page fault. This is really cool :^) I need to refactor this to support sharing of read-only file-backed pages, but it's cool to just have something working.
2018-11-08 11:59:16 +00:00
#ifdef MM_DEBUG
Kernel: Refactor MemoryManager to use a Bitmap rather than a Vector This significantly reduces the pressure on the kernel heap when allocating a lot of pages. Previously at about 250MB allocated, the free page list would outgrow the kernel's heap. Given that there is no longer a page list, this does not happen. The next barrier will be the kernel memory used by the page records for in-use memory. This kicks in at about 1GB.
2019-06-11 11:13:02 +00:00
dbgprintf("MM: allocate_user_physical_page vending P%p\n", page->paddr().get());
Support basic mmap'ing of a file! All right, we can now mmap() a file and it gets magically paged in from fs in response to an NP page fault. This is really cool :^) I need to refactor this to support sharing of read-only file-backed pages, but it's cool to just have something working.
2018-11-08 11:59:16 +00:00
#endif
Kernel: Refactor MemoryManager to use a Bitmap rather than a Vector This significantly reduces the pressure on the kernel heap when allocating a lot of pages. Previously at about 250MB allocated, the free page list would outgrow the kernel's heap. Given that there is no longer a page list, this does not happen. The next barrier will be the kernel memory used by the page records for in-use memory. This kicks in at about 1GB.
2019-06-11 11:13:02 +00:00
Kernel: Actually zero-fill eagerly committed regions. Previously, calling Region::commit() would make sure to allocate any missing physical pages, but they would contain uninitialized data. This was very obvious when allocating GraphicsBitmaps as they would have garbage pixels rather than being filled with black. The MM quickmap mechanism didn't work when operating on a non-active page directory (which happens when we're in the middle of exec, for example.) This patch changes quickmap to reside in the shared kernel page directory. Also added some missing clobber lists to inline asm that I stumbled on.
2019-01-31 02:57:06 +00:00
if (should_zero_fill == ShouldZeroFill::Yes) {
Kernel: Refactor MemoryManager to use a Bitmap rather than a Vector This significantly reduces the pressure on the kernel heap when allocating a lot of pages. Previously at about 250MB allocated, the free page list would outgrow the kernel's heap. Given that there is no longer a page list, this does not happen. The next barrier will be the kernel memory used by the page records for in-use memory. This kicks in at about 1GB.
2019-06-11 11:13:02 +00:00
auto* ptr = (dword*)quickmap_page(*page);
Kernel: Actually zero-fill eagerly committed regions. Previously, calling Region::commit() would make sure to allocate any missing physical pages, but they would contain uninitialized data. This was very obvious when allocating GraphicsBitmaps as they would have garbage pixels rather than being filled with black. The MM quickmap mechanism didn't work when operating on a non-active page directory (which happens when we're in the middle of exec, for example.) This patch changes quickmap to reside in the shared kernel page directory. Also added some missing clobber lists to inline asm that I stumbled on.
2019-01-31 02:57:06 +00:00
fast_dword_fill(ptr, 0, PAGE_SIZE / sizeof(dword));
unquickmap_page();
}
Kernel: Refactor MemoryManager to use a Bitmap rather than a Vector This significantly reduces the pressure on the kernel heap when allocating a lot of pages. Previously at about 250MB allocated, the free page list would outgrow the kernel's heap. Given that there is no longer a page list, this does not happen. The next barrier will be the kernel memory used by the page records for in-use memory. This kicks in at about 1GB.
2019-06-11 11:13:02 +00:00
m_user_physical_pages_used++;
return page;
}
VM: Pass a PhysicalPage by rvalue reference when returning it to the freelist. This makes no functional difference, but it makes it clear that MemoryManager and PhysicalRegion take over the actual physical page represented by this PhysicalPage instance.
2019-06-14 11:56:21 +00:00
void MemoryManager::deallocate_supervisor_physical_page(PhysicalPage&& page)
Kernel: Refactor MemoryManager to use a Bitmap rather than a Vector This significantly reduces the pressure on the kernel heap when allocating a lot of pages. Previously at about 250MB allocated, the free page list would outgrow the kernel's heap. Given that there is no longer a page list, this does not happen. The next barrier will be the kernel memory used by the page records for in-use memory. This kicks in at about 1GB.
2019-06-11 11:13:02 +00:00
{
for (auto& region : m_super_physical_regions) {
if (!region->contains(page)) {
kprintf(
"MM: deallocate_supervisor_physical_page: %p not in %p -> %p\n",
page.paddr(), region->lower().get(), region->upper().get());
continue;
}
VM: Pass a PhysicalPage by rvalue reference when returning it to the freelist. This makes no functional difference, but it makes it clear that MemoryManager and PhysicalRegion take over the actual physical page represented by this PhysicalPage instance.
2019-06-14 11:56:21 +00:00
region->return_page(move(page));
Kernel: Refactor MemoryManager to use a Bitmap rather than a Vector This significantly reduces the pressure on the kernel heap when allocating a lot of pages. Previously at about 250MB allocated, the free page list would outgrow the kernel's heap. Given that there is no longer a page list, this does not happen. The next barrier will be the kernel memory used by the page records for in-use memory. This kicks in at about 1GB.
2019-06-11 11:13:02 +00:00
m_super_physical_pages_used--;
return;
}
kprintf("MM: deallocate_supervisor_physical_page couldn't figure out region for super page @ %p\n", page.paddr());
ASSERT_NOT_REACHED();
Support basic mmap'ing of a file! All right, we can now mmap() a file and it gets magically paged in from fs in response to an NP page fault. This is really cool :^) I need to refactor this to support sharing of read-only file-backed pages, but it's cool to just have something working.
2018-11-08 11:59:16 +00:00
}
MM: Allocate page tables from a separate set of physical pages. The old approach only worked because of an overpermissive accident. There's now a concept of supervisor physical pages that can be allocated. They all sit in the low 4 MB of physical memory and are identity mapped, shared between all processes, and only ring 0 can access them.
2019-01-01 01:09:43 +00:00
RetainPtr<PhysicalPage> MemoryManager::allocate_supervisor_physical_page()
More work on per-process page directories. It basically works now! I spent some time stuck on a problem where processes would clobber each other's stacks. Took me a moment to figure out that their stacks were allocated in the sub-4MB linear address range which is shared between all processes. Oops!
2018-11-01 10:30:48 +00:00
{
InterruptDisabler disabler;
Kernel: Refactor MemoryManager to use a Bitmap rather than a Vector This significantly reduces the pressure on the kernel heap when allocating a lot of pages. Previously at about 250MB allocated, the free page list would outgrow the kernel's heap. Given that there is no longer a page list, this does not happen. The next barrier will be the kernel memory used by the page records for in-use memory. This kicks in at about 1GB.
2019-06-11 11:13:02 +00:00
RetainPtr<PhysicalPage> page = nullptr;
for (auto& region : m_super_physical_regions) {
page = region->take_free_page(true);
if (page.is_null())
continue;
}
if (!page) {
if (m_super_physical_regions.is_empty()) {
kprintf("MM: no super physical regions available (?)\n");
}
kprintf("MM: no super physical pages available\n");
WindowServer: Support resizing windows. This is pretty limited and not entirely stable, but it does work! :^)
2019-02-20 14:34:55 +00:00
ASSERT_NOT_REACHED();
Kernel: Run clang-format on everything.
2019-06-07 09:43:58 +00:00
return {};
WindowServer: Support resizing windows. This is pretty limited and not entirely stable, but it does work! :^)
2019-02-20 14:34:55 +00:00
}
Kernel: Refactor MemoryManager to use a Bitmap rather than a Vector This significantly reduces the pressure on the kernel heap when allocating a lot of pages. Previously at about 250MB allocated, the free page list would outgrow the kernel's heap. Given that there is no longer a page list, this does not happen. The next barrier will be the kernel memory used by the page records for in-use memory. This kicks in at about 1GB.
2019-06-11 11:13:02 +00:00
MM: Allocate page tables from a separate set of physical pages. The old approach only worked because of an overpermissive accident. There's now a concept of supervisor physical pages that can be allocated. They all sit in the low 4 MB of physical memory and are identity mapped, shared between all processes, and only ring 0 can access them.
2019-01-01 01:09:43 +00:00
#ifdef MM_DEBUG
Kernel: Refactor MemoryManager to use a Bitmap rather than a Vector This significantly reduces the pressure on the kernel heap when allocating a lot of pages. Previously at about 250MB allocated, the free page list would outgrow the kernel's heap. Given that there is no longer a page list, this does not happen. The next barrier will be the kernel memory used by the page records for in-use memory. This kicks in at about 1GB.
2019-06-11 11:13:02 +00:00
dbgprintf("MM: allocate_supervisor_physical_page vending P%p\n", page->paddr().get());
MM: Allocate page tables from a separate set of physical pages. The old approach only worked because of an overpermissive accident. There's now a concept of supervisor physical pages that can be allocated. They all sit in the low 4 MB of physical memory and are identity mapped, shared between all processes, and only ring 0 can access them.
2019-01-01 01:09:43 +00:00
#endif
Kernel: Refactor MemoryManager to use a Bitmap rather than a Vector This significantly reduces the pressure on the kernel heap when allocating a lot of pages. Previously at about 250MB allocated, the free page list would outgrow the kernel's heap. Given that there is no longer a page list, this does not happen. The next barrier will be the kernel memory used by the page records for in-use memory. This kicks in at about 1GB.
2019-06-11 11:13:02 +00:00
fast_dword_fill((dword*)page->paddr().as_ptr(), 0, PAGE_SIZE / sizeof(dword));
m_super_physical_pages_used++;
return page;
More work on per-process page directories. It basically works now! I spent some time stuck on a problem where processes would clobber each other's stacks. Took me a moment to figure out that their stacks were allocated in the sub-4MB linear address range which is shared between all processes. Oops!
2018-11-01 10:30:48 +00:00
}
Process now maps regions immediately when they are allocated. This avoids having to do a separate MM.mapRegionsForTask() pass. Also, more Task => Process renaming that I apparently hadn't saved yet.
2018-11-01 12:15:46 +00:00
void MemoryManager::enter_process_paging_scope(Process& process)
More work on per-process page directories. It basically works now! I spent some time stuck on a problem where processes would clobber each other's stacks. Took me a moment to figure out that their stacks were allocated in the sub-4MB linear address range which is shared between all processes. Oops!
2018-11-01 10:30:48 +00:00
{
Kernel: Introduce threads, and refactor everything in support of it. The scheduler now operates on threads, rather than on processes. Each process has a main thread, and can have any number of additional threads. The process exits when the main thread exits. This patch doesn't actually spawn any additional threads, it merely does all the plumbing needed to make it possible. :^)
2019-03-23 21:03:17 +00:00
ASSERT(current);
More work on per-process page directories. It basically works now! I spent some time stuck on a problem where processes would clobber each other's stacks. Took me a moment to figure out that their stacks were allocated in the sub-4MB linear address range which is shared between all processes. Oops!
2018-11-01 10:30:48 +00:00
InterruptDisabler disabler;
Kernel: Introduce threads, and refactor everything in support of it. The scheduler now operates on threads, rather than on processes. Each process has a main thread, and can have any number of additional threads. The process exits when the main thread exits. This patch doesn't actually spawn any additional threads, it merely does all the plumbing needed to make it possible. :^)
2019-03-23 21:03:17 +00:00
current->tss().cr3 = process.page_directory().cr3();
Kernel: Run clang-format on everything.
2019-06-07 09:43:58 +00:00
asm volatile("movl %%eax, %%cr3" ::"a"(process.page_directory().cr3())
: "memory");
More work on per-process page directories. It basically works now! I spent some time stuck on a problem where processes would clobber each other's stacks. Took me a moment to figure out that their stacks were allocated in the sub-4MB linear address range which is shared between all processes. Oops!
2018-11-01 10:30:48 +00:00
}
Yet more coding style fixes.
2018-12-03 00:38:22 +00:00
void MemoryManager::flush_entire_tlb()
Flush the TLB after modifying page tables. This is a very inefficient naive implementation, but it gets us going. Mapping tasks in and out of a shared address space now totally works.
2018-10-23 09:03:56 +00:00
{
asm volatile(
Kernel: Actually zero-fill eagerly committed regions. Previously, calling Region::commit() would make sure to allocate any missing physical pages, but they would contain uninitialized data. This was very obvious when allocating GraphicsBitmaps as they would have garbage pixels rather than being filled with black. The MM quickmap mechanism didn't work when operating on a non-active page directory (which happens when we're in the middle of exec, for example.) This patch changes quickmap to reside in the shared kernel page directory. Also added some missing clobber lists to inline asm that I stumbled on.
2019-01-31 02:57:06 +00:00
"mov %%cr3, %%eax\n"
Kernel: Run clang-format on everything.
2019-06-07 09:43:58 +00:00
"mov %%eax, %%cr3\n" ::
: "%eax", "memory");
Flush the TLB after modifying page tables. This is a very inefficient naive implementation, but it gets us going. Mapping tasks in and out of a shared address space now totally works.
2018-10-23 09:03:56 +00:00
}
Kernel: Rename LinearAddress => VirtualAddress.
2019-06-07 10:56:50 +00:00
void MemoryManager::flush_tlb(VirtualAddress vaddr)
Teach MM to flush individual TLB entries only.
2018-10-23 13:53:11 +00:00
{
Kernel: Run clang-format on everything.
2019-06-07 09:43:58 +00:00
asm volatile("invlpg %0"
:
Kernel: Rename LinearAddress => VirtualAddress.
2019-06-07 10:56:50 +00:00
: "m"(*(char*)vaddr.get())
Kernel: Run clang-format on everything.
2019-06-07 09:43:58 +00:00
: "memory");
Teach MM to flush individual TLB entries only.
2018-10-23 13:53:11 +00:00
}
Kernel: Rename LinearAddress => VirtualAddress.
2019-06-07 10:56:50 +00:00
void MemoryManager::map_for_kernel(VirtualAddress vaddr, PhysicalAddress paddr)
Kernel: Start adding support for E1000 network adapters.
2019-03-10 14:25:33 +00:00
{
Kernel: Rename LinearAddress => VirtualAddress.
2019-06-07 10:56:50 +00:00
auto pte = ensure_pte(kernel_page_directory(), vaddr);
Kernel: Start adding support for E1000 network adapters.
2019-03-10 14:25:33 +00:00
pte.set_physical_page_base(paddr.get());
pte.set_present(true);
pte.set_writable(true);
pte.set_user_allowed(false);
Kernel: Rename LinearAddress => VirtualAddress.
2019-06-07 10:56:50 +00:00
flush_tlb(vaddr);
Kernel: Start adding support for E1000 network adapters.
2019-03-10 14:25:33 +00:00
}
Implement COW pages! :^) sys$fork() now clones all writable regions with per-page COW bits. The pages are then mapped read-only and we handle a PF by COWing the pages. This is quite delightful. Obviously there's lots of work to do still, and it needs better data structures, but the general concept works.
2018-11-05 12:48:07 +00:00
byte* MemoryManager::quickmap_page(PhysicalPage& physical_page)
{
ASSERT_INTERRUPTS_DISABLED();
Kernel: Actually zero-fill eagerly committed regions. Previously, calling Region::commit() would make sure to allocate any missing physical pages, but they would contain uninitialized data. This was very obvious when allocating GraphicsBitmaps as they would have garbage pixels rather than being filled with black. The MM quickmap mechanism didn't work when operating on a non-active page directory (which happens when we're in the middle of exec, for example.) This patch changes quickmap to reside in the shared kernel page directory. Also added some missing clobber lists to inline asm that I stumbled on.
2019-01-31 02:57:06 +00:00
ASSERT(!m_quickmap_in_use);
m_quickmap_in_use = true;
Kernel: Rename LinearAddress => VirtualAddress.
2019-06-07 10:56:50 +00:00
auto page_vaddr = m_quickmap_addr;
auto pte = ensure_pte(kernel_page_directory(), page_vaddr);
Yet more coding style fixes.
2018-12-03 00:38:22 +00:00
pte.set_physical_page_base(physical_page.paddr().get());
MM: Allocate page tables from a separate set of physical pages. The old approach only worked because of an overpermissive accident. There's now a concept of supervisor physical pages that can be allocated. They all sit in the low 4 MB of physical memory and are identity mapped, shared between all processes, and only ring 0 can access them.
2019-01-01 01:09:43 +00:00
pte.set_present(true);
Yet more coding style fixes.
2018-12-03 00:38:22 +00:00
pte.set_writable(true);
Kernel: Actually zero-fill eagerly committed regions. Previously, calling Region::commit() would make sure to allocate any missing physical pages, but they would contain uninitialized data. This was very obvious when allocating GraphicsBitmaps as they would have garbage pixels rather than being filled with black. The MM quickmap mechanism didn't work when operating on a non-active page directory (which happens when we're in the middle of exec, for example.) This patch changes quickmap to reside in the shared kernel page directory. Also added some missing clobber lists to inline asm that I stumbled on.
2019-01-31 02:57:06 +00:00
pte.set_user_allowed(false);
Kernel: Rename LinearAddress => VirtualAddress.
2019-06-07 10:56:50 +00:00
flush_tlb(page_vaddr);
MM: Allocate page tables from a separate set of physical pages. The old approach only worked because of an overpermissive accident. There's now a concept of supervisor physical pages that can be allocated. They all sit in the low 4 MB of physical memory and are identity mapped, shared between all processes, and only ring 0 can access them.
2019-01-01 01:09:43 +00:00
ASSERT((dword)pte.physical_page_base() == physical_page.paddr().get());
Implement COW pages! :^) sys$fork() now clones all writable regions with per-page COW bits. The pages are then mapped read-only and we handle a PF by COWing the pages. This is quite delightful. Obviously there's lots of work to do still, and it needs better data structures, but the general concept works.
2018-11-05 12:48:07 +00:00
#ifdef MM_DEBUG
Kernel: Rename LinearAddress => VirtualAddress.
2019-06-07 10:56:50 +00:00
dbgprintf("MM: >> quickmap_page L%x => P%x @ PTE=%p\n", page_vaddr, physical_page.paddr().get(), pte.ptr());
Implement COW pages! :^) sys$fork() now clones all writable regions with per-page COW bits. The pages are then mapped read-only and we handle a PF by COWing the pages. This is quite delightful. Obviously there's lots of work to do still, and it needs better data structures, but the general concept works.
2018-11-05 12:48:07 +00:00
#endif
Kernel: Rename LinearAddress => VirtualAddress.
2019-06-07 10:56:50 +00:00
return page_vaddr.as_ptr();
Implement COW pages! :^) sys$fork() now clones all writable regions with per-page COW bits. The pages are then mapped read-only and we handle a PF by COWing the pages. This is quite delightful. Obviously there's lots of work to do still, and it needs better data structures, but the general concept works.
2018-11-05 12:48:07 +00:00
}
void MemoryManager::unquickmap_page()
{
ASSERT_INTERRUPTS_DISABLED();
Kernel: Actually zero-fill eagerly committed regions. Previously, calling Region::commit() would make sure to allocate any missing physical pages, but they would contain uninitialized data. This was very obvious when allocating GraphicsBitmaps as they would have garbage pixels rather than being filled with black. The MM quickmap mechanism didn't work when operating on a non-active page directory (which happens when we're in the middle of exec, for example.) This patch changes quickmap to reside in the shared kernel page directory. Also added some missing clobber lists to inline asm that I stumbled on.
2019-01-31 02:57:06 +00:00
ASSERT(m_quickmap_in_use);
Kernel: Rename LinearAddress => VirtualAddress.
2019-06-07 10:56:50 +00:00
auto page_vaddr = m_quickmap_addr;
auto pte = ensure_pte(kernel_page_directory(), page_vaddr);
Implement COW pages! :^) sys$fork() now clones all writable regions with per-page COW bits. The pages are then mapped read-only and we handle a PF by COWing the pages. This is quite delightful. Obviously there's lots of work to do still, and it needs better data structures, but the general concept works.
2018-11-05 12:48:07 +00:00
#ifdef MM_DEBUG
MM: Fix bug when mapping a region with a VMO with non-zero offset.
2018-12-29 02:34:24 +00:00
auto old_physical_address = pte.physical_page_base();
Implement COW pages! :^) sys$fork() now clones all writable regions with per-page COW bits. The pages are then mapped read-only and we handle a PF by COWing the pages. This is quite delightful. Obviously there's lots of work to do still, and it needs better data structures, but the general concept works.
2018-11-05 12:48:07 +00:00
#endif
Yet more coding style fixes.
2018-12-03 00:38:22 +00:00
pte.set_physical_page_base(0);
pte.set_present(false);
pte.set_writable(false);
Kernel: Rename LinearAddress => VirtualAddress.
2019-06-07 10:56:50 +00:00
flush_tlb(page_vaddr);
Implement COW pages! :^) sys$fork() now clones all writable regions with per-page COW bits. The pages are then mapped read-only and we handle a PF by COWing the pages. This is quite delightful. Obviously there's lots of work to do still, and it needs better data structures, but the general concept works.
2018-11-05 12:48:07 +00:00
#ifdef MM_DEBUG
Kernel: Rename LinearAddress => VirtualAddress.
2019-06-07 10:56:50 +00:00
dbgprintf("MM: >> unquickmap_page L%x =/> P%x\n", page_vaddr, old_physical_address);
Implement COW pages! :^) sys$fork() now clones all writable regions with per-page COW bits. The pages are then mapped read-only and we handle a PF by COWing the pages. This is quite delightful. Obviously there's lots of work to do still, and it needs better data structures, but the general concept works.
2018-11-05 12:48:07 +00:00
#endif
Kernel: Actually zero-fill eagerly committed regions. Previously, calling Region::commit() would make sure to allocate any missing physical pages, but they would contain uninitialized data. This was very obvious when allocating GraphicsBitmaps as they would have garbage pixels rather than being filled with black. The MM quickmap mechanism didn't work when operating on a non-active page directory (which happens when we're in the middle of exec, for example.) This patch changes quickmap to reside in the shared kernel page directory. Also added some missing clobber lists to inline asm that I stumbled on.
2019-01-31 02:57:06 +00:00
m_quickmap_in_use = false;
Implement COW pages! :^) sys$fork() now clones all writable regions with per-page COW bits. The pages are then mapped read-only and we handle a PF by COWing the pages. This is quite delightful. Obviously there's lots of work to do still, and it needs better data structures, but the general concept works.
2018-11-05 12:48:07 +00:00
}
Kernel: Refactor Region/PageDirectory ownership model. Make PageDirectory retainable and have each Region co-own the PageDirectory they're mapped into. When unmapped, Region has no associated PageDirectory. This allows Region to automatically unmap itself when destroyed.
2019-01-22 04:01:00 +00:00
void MemoryManager::remap_region_page(Region& region, unsigned page_index_in_region, bool user_allowed)
Implement COW pages! :^) sys$fork() now clones all writable regions with per-page COW bits. The pages are then mapped read-only and we handle a PF by COWing the pages. This is quite delightful. Obviously there's lots of work to do still, and it needs better data structures, but the general concept works.
2018-11-05 12:48:07 +00:00
{
Kernel: Finally stop exposing Region members to the public.
2019-01-24 17:09:46 +00:00
ASSERT(region.page_directory());
Implement COW pages! :^) sys$fork() now clones all writable regions with per-page COW bits. The pages are then mapped read-only and we handle a PF by COWing the pages. This is quite delightful. Obviously there's lots of work to do still, and it needs better data structures, but the general concept works.
2018-11-05 12:48:07 +00:00
InterruptDisabler disabler;
Kernel: Rename LinearAddress => VirtualAddress.
2019-06-07 10:56:50 +00:00
auto page_vaddr = region.vaddr().offset(page_index_in_region * PAGE_SIZE);
auto pte = ensure_pte(*region.page_directory(), page_vaddr);
Refactor the virtual memory object model a bit: Process now has a number of Regions. Each Region is backed by a VMObject. A VMObject can be file-backed or anonymous. These can be shared.
2018-11-08 13:35:30 +00:00
auto& physical_page = region.vmo().physical_pages()[page_index_in_region];
Implement COW pages! :^) sys$fork() now clones all writable regions with per-page COW bits. The pages are then mapped read-only and we handle a PF by COWing the pages. This is quite delightful. Obviously there's lots of work to do still, and it needs better data structures, but the general concept works.
2018-11-05 12:48:07 +00:00
ASSERT(physical_page);
Yet more coding style fixes.
2018-12-03 00:38:22 +00:00
pte.set_physical_page_base(physical_page->paddr().get());
pte.set_present(true); // FIXME: Maybe we should use the is_readable flag here?
Kernel: Encapsulate the Region's COW map a bit better.
2019-05-14 15:31:57 +00:00
if (region.should_cow(page_index_in_region))
Yet more coding style fixes.
2018-12-03 00:38:22 +00:00
pte.set_writable(false);
Implement COW pages! :^) sys$fork() now clones all writable regions with per-page COW bits. The pages are then mapped read-only and we handle a PF by COWing the pages. This is quite delightful. Obviously there's lots of work to do still, and it needs better data structures, but the general concept works.
2018-11-05 12:48:07 +00:00
else
Kernel: Finally stop exposing Region members to the public.
2019-01-24 17:09:46 +00:00
pte.set_writable(region.is_writable());
Kernel: Let's try disabling the CPU's page-level caching for framebuffers.
2019-02-07 08:24:41 +00:00
pte.set_cache_disabled(!region.vmo().m_allow_cpu_caching);
pte.set_write_through(!region.vmo().m_allow_cpu_caching);
Yet more coding style fixes.
2018-12-03 00:38:22 +00:00
pte.set_user_allowed(user_allowed);
Kernel: Rename LinearAddress => VirtualAddress.
2019-06-07 10:56:50 +00:00
region.page_directory()->flush(page_vaddr);
Implement COW pages! :^) sys$fork() now clones all writable regions with per-page COW bits. The pages are then mapped read-only and we handle a PF by COWing the pages. This is quite delightful. Obviously there's lots of work to do still, and it needs better data structures, but the general concept works.
2018-11-05 12:48:07 +00:00
#ifdef MM_DEBUG
Kernel: Rename LinearAddress => VirtualAddress.
2019-06-07 10:56:50 +00:00
dbgprintf("MM: >> remap_region_page (PD=%x, PTE=P%x) '%s' L%x => P%x (@%p)\n", region.page_directory()->cr3(), pte.ptr(), region.name().characters(), page_vaddr.get(), physical_page->paddr().get(), physical_page.ptr());
Implement COW pages! :^) sys$fork() now clones all writable regions with per-page COW bits. The pages are then mapped read-only and we handle a PF by COWing the pages. This is quite delightful. Obviously there's lots of work to do still, and it needs better data structures, but the general concept works.
2018-11-05 12:48:07 +00:00
#endif
}
Kernel: Invalidate file-backed VMO's when inodes are written. The current strategy is simply to nuke all physical pages and force reload them from disk. This is obviously not optimal and should eventually be optimized. It should be fairly straightforward.
2019-02-05 07:17:46 +00:00
void MemoryManager::remap_region(PageDirectory& page_directory, Region& region)
Implement COW pages! :^) sys$fork() now clones all writable regions with per-page COW bits. The pages are then mapped read-only and we handle a PF by COWing the pages. This is quite delightful. Obviously there's lots of work to do still, and it needs better data structures, but the general concept works.
2018-11-05 12:48:07 +00:00
{
InterruptDisabler disabler;
Kernel: Invalidate file-backed VMO's when inodes are written. The current strategy is simply to nuke all physical pages and force reload them from disk. This is obviously not optimal and should eventually be optimized. It should be fairly straightforward.
2019-02-05 07:17:46 +00:00
ASSERT(region.page_directory() == &page_directory);
Kernel: Rename LinearAddress => VirtualAddress.
2019-06-07 10:56:50 +00:00
map_region_at_address(page_directory, region, region.vaddr(), true);
Implement COW pages! :^) sys$fork() now clones all writable regions with per-page COW bits. The pages are then mapped read-only and we handle a PF by COWing the pages. This is quite delightful. Obviously there's lots of work to do still, and it needs better data structures, but the general concept works.
2018-11-05 12:48:07 +00:00
}
Kernel: Rename LinearAddress => VirtualAddress.
2019-06-07 10:56:50 +00:00
void MemoryManager::map_region_at_address(PageDirectory& page_directory, Region& region, VirtualAddress vaddr, bool user_allowed)
More work on per-process page directories. It basically works now! I spent some time stuck on a problem where processes would clobber each other's stacks. Took me a moment to figure out that their stacks were allocated in the sub-4MB linear address range which is shared between all processes. Oops!
2018-11-01 10:30:48 +00:00
{
InterruptDisabler disabler;
Kernel: Finally stop exposing Region members to the public.
2019-01-24 17:09:46 +00:00
region.set_page_directory(page_directory);
Refactor the virtual memory object model a bit: Process now has a number of Regions. Each Region is backed by a VMObject. A VMObject can be file-backed or anonymous. These can be shared.
2018-11-08 13:35:30 +00:00
auto& vmo = region.vmo();
Rage hacking to get bash to run. It finally runs. So cool! :^)
2018-11-11 14:36:40 +00:00
#ifdef MM_DEBUG
Teach Process::exec() about the magic of file-backed VMO's. This is really sweet! :^) The four instances of /bin/sh spawned at startup now share their read-only text pages. There are problems and limitations here, and plenty of room for improvement. But it kinda works.
2018-11-08 20:20:09 +00:00
dbgprintf("MM: map_region_at_address will map VMO pages %u - %u (VMO page count: %u)\n", region.first_page_index(), region.last_page_index(), vmo.page_count());
Rage hacking to get bash to run. It finally runs. So cool! :^)
2018-11-11 14:36:40 +00:00
#endif
MM: Fix bug when mapping a region with a VMO with non-zero offset.
2018-12-29 02:34:24 +00:00
for (size_t i = 0; i < region.page_count(); ++i) {
Kernel: Rename LinearAddress => VirtualAddress.
2019-06-07 10:56:50 +00:00
auto page_vaddr = vaddr.offset(i * PAGE_SIZE);
auto pte = ensure_pte(page_directory, page_vaddr);
MM: Fix bug when mapping a region with a VMO with non-zero offset.
2018-12-29 02:34:24 +00:00
auto& physical_page = vmo.physical_pages()[region.first_page_index() + i];
Replace zones with individually tracked physical pages. It's just a simple struct { ref_count, paddr }. This will allow me to implement lazy zeroing and COW pages.
2018-11-05 09:23:00 +00:00
if (physical_page) {
Yet more coding style fixes.
2018-12-03 00:38:22 +00:00
pte.set_physical_page_base(physical_page->paddr().get());
pte.set_present(true); // FIXME: Maybe we should use the is_readable flag here?
MM: Fix bug when mapping a region with a VMO with non-zero offset.
2018-12-29 02:34:24 +00:00
// FIXME: It seems wrong that the *region* cow map is essentially using *VMO* relative indices.
Kernel: Encapsulate the Region's COW map a bit better.
2019-05-14 15:31:57 +00:00
if (region.should_cow(region.first_page_index() + i))
Yet more coding style fixes.
2018-12-03 00:38:22 +00:00
pte.set_writable(false);
Implement COW pages! :^) sys$fork() now clones all writable regions with per-page COW bits. The pages are then mapped read-only and we handle a PF by COWing the pages. This is quite delightful. Obviously there's lots of work to do still, and it needs better data structures, but the general concept works.
2018-11-05 12:48:07 +00:00
else
Kernel: Finally stop exposing Region members to the public.
2019-01-24 17:09:46 +00:00
pte.set_writable(region.is_writable());
Kernel: Let's try disabling the CPU's page-level caching for framebuffers.
2019-02-07 08:24:41 +00:00
pte.set_cache_disabled(!region.vmo().m_allow_cpu_caching);
pte.set_write_through(!region.vmo().m_allow_cpu_caching);
Replace zones with individually tracked physical pages. It's just a simple struct { ref_count, paddr }. This will allow me to implement lazy zeroing and COW pages.
2018-11-05 09:23:00 +00:00
} else {
Yet more coding style fixes.
2018-12-03 00:38:22 +00:00
pte.set_physical_page_base(0);
pte.set_present(false);
Kernel: Finally stop exposing Region members to the public.
2019-01-24 17:09:46 +00:00
pte.set_writable(region.is_writable());
Replace zones with individually tracked physical pages. It's just a simple struct { ref_count, paddr }. This will allow me to implement lazy zeroing and COW pages.
2018-11-05 09:23:00 +00:00
}
Yet more coding style fixes.
2018-12-03 00:38:22 +00:00
pte.set_user_allowed(user_allowed);
Kernel: Rename LinearAddress => VirtualAddress.
2019-06-07 10:56:50 +00:00
page_directory.flush(page_vaddr);
More work on per-process page directories. It basically works now! I spent some time stuck on a problem where processes would clobber each other's stacks. Took me a moment to figure out that their stacks were allocated in the sub-4MB linear address range which is shared between all processes. Oops!
2018-11-01 10:30:48 +00:00
#ifdef MM_DEBUG
Kernel: Rename LinearAddress => VirtualAddress.
2019-06-07 10:56:50 +00:00
dbgprintf("MM: >> map_region_at_address (PD=%x) '%s' L%x => P%x (@%p)\n", &page_directory, region.name().characters(), page_vaddr, physical_page ? physical_page->paddr().get() : 0, physical_page.ptr());
More work on per-process page directories. It basically works now! I spent some time stuck on a problem where processes would clobber each other's stacks. Took me a moment to figure out that their stacks were allocated in the sub-4MB linear address range which is shared between all processes. Oops!
2018-11-01 10:30:48 +00:00
#endif
}
}
Kernel: Refactor Region/PageDirectory ownership model. Make PageDirectory retainable and have each Region co-own the PageDirectory they're mapped into. When unmapped, Region has no associated PageDirectory. This allows Region to automatically unmap itself when destroyed.
2019-01-22 04:01:00 +00:00
bool MemoryManager::unmap_region(Region& region)
Lots of hacking: - Turn Keyboard into a CharacterDevice (85,1) at /dev/keyboard. - Implement MM::unmapRegionsForTask() and MM::unmapRegion() - Save SS correctly on interrupt. - Add a simple Spawn syscall for launching another process. - Move a bunch of IO syscall debug output behind DEBUG_IO. - Have ASSERT do a "cli" immediately when failing. This makes the output look proper every time. - Implement a bunch of syscalls in LibC. - Add a simple shell ("sh"). All it can do now is read a line of text from /dev/keyboard and then try launching the specified executable by calling spawn(). There are definitely bugs in here, but we're moving on forward.
2018-10-23 08:12:50 +00:00
{
Kernel: Finally stop exposing Region members to the public.
2019-01-24 17:09:46 +00:00
ASSERT(region.page_directory());
Selectively disable interrupts in MM. Also added an ASSERT_INTERRUPTS_DISABLED() macro.
2018-10-25 08:15:28 +00:00
InterruptDisabler disabler;
Teach Process::exec() about the magic of file-backed VMO's. This is really sweet! :^) The four instances of /bin/sh spawned at startup now share their read-only text pages. There are problems and limitations here, and plenty of room for improvement. But it kinda works.
2018-11-08 20:20:09 +00:00
for (size_t i = 0; i < region.page_count(); ++i) {
Kernel: Rename LinearAddress => VirtualAddress.
2019-06-07 10:56:50 +00:00
auto vaddr = region.vaddr().offset(i * PAGE_SIZE);
auto pte = ensure_pte(*region.page_directory(), vaddr);
Yet more coding style fixes.
2018-12-03 00:38:22 +00:00
pte.set_physical_page_base(0);
pte.set_present(false);
pte.set_writable(false);
pte.set_user_allowed(false);
Kernel: Rename LinearAddress => VirtualAddress.
2019-06-07 10:56:50 +00:00
region.page_directory()->flush(vaddr);
More work on per-process page directories. It basically works now! I spent some time stuck on a problem where processes would clobber each other's stacks. Took me a moment to figure out that their stacks were allocated in the sub-4MB linear address range which is shared between all processes. Oops!
2018-11-01 10:30:48 +00:00
#ifdef MM_DEBUG
Teach Process::exec() about the magic of file-backed VMO's. This is really sweet! :^) The four instances of /bin/sh spawned at startup now share their read-only text pages. There are problems and limitations here, and plenty of room for improvement. But it kinda works.
2018-11-08 20:20:09 +00:00
auto& physical_page = region.vmo().physical_pages()[region.first_page_index() + i];
Kernel: Rename LinearAddress => VirtualAddress.
2019-06-07 10:56:50 +00:00
dbgprintf("MM: >> Unmapped L%x => P%x <<\n", vaddr, physical_page ? physical_page->paddr().get() : 0);
More work on per-process page directories. It basically works now! I spent some time stuck on a problem where processes would clobber each other's stacks. Took me a moment to figure out that their stacks were allocated in the sub-4MB linear address range which is shared between all processes. Oops!
2018-11-01 10:30:48 +00:00
#endif
Implement loading of linked ELF executables. This took me a couple hours. :^) The ELF loading code now allocates a single region for the entire file and creates virtual memory mappings for the sections as needed. Very nice!
2018-10-27 12:56:52 +00:00
}
Kernel: Finally stop exposing Region members to the public.
2019-01-24 17:09:46 +00:00
region.release_page_directory();
Implement loading of linked ELF executables. This took me a couple hours. :^) The ELF loading code now allocates a single region for the entire file and creates virtual memory mappings for the sections as needed. Very nice!
2018-10-27 12:56:52 +00:00
return true;
}
Yet more coding style fixes.
2018-12-03 00:38:22 +00:00
bool MemoryManager::map_region(Process& process, Region& region)
Launching an arbitrary ELF executable from disk works! :^) This is so cool! It's a bit messy now with two Task constructors, but eventually they should fold into a single constructor somehow.
2018-10-22 13:42:39 +00:00
{
Kernel: Rename LinearAddress => VirtualAddress.
2019-06-07 10:56:50 +00:00
map_region_at_address(process.page_directory(), region, region.vaddr(), true);
Launching an arbitrary ELF executable from disk works! :^) This is so cool! It's a bit messy now with two Task constructors, but eventually they should fold into a single constructor somehow.
2018-10-22 13:42:39 +00:00
return true;
Implement address validation by querying the task's page directory. This is way better than walking the region lists. I suppose we could even let the hardware trigger a page fault and handle that. That'll be the next step in the evolution here I guess.
2018-11-01 11:45:51 +00:00
}
Kernel: Rename LinearAddress => VirtualAddress.
2019-06-07 10:56:50 +00:00
bool MemoryManager::validate_user_read(const Process& process, VirtualAddress vaddr) const
Implement address validation by querying the task's page directory. This is way better than walking the region lists. I suppose we could even let the hardware trigger a page fault and handle that. That'll be the next step in the evolution here I guess.
2018-11-01 11:45:51 +00:00
{
Kernel: Rename LinearAddress => VirtualAddress.
2019-06-07 10:56:50 +00:00
auto* region = region_from_vaddr(process, vaddr);
Kernel: Fix incorrect EFAULTs when syscall would write into COW pages.
2019-01-25 00:39:15 +00:00
return region && region->is_readable();
Implement address validation by querying the task's page directory. This is way better than walking the region lists. I suppose we could even let the hardware trigger a page fault and handle that. That'll be the next step in the evolution here I guess.
2018-11-01 11:45:51 +00:00
}
Kernel: Rename LinearAddress => VirtualAddress.
2019-06-07 10:56:50 +00:00
bool MemoryManager::validate_user_write(const Process& process, VirtualAddress vaddr) const
Implement address validation by querying the task's page directory. This is way better than walking the region lists. I suppose we could even let the hardware trigger a page fault and handle that. That'll be the next step in the evolution here I guess.
2018-11-01 11:45:51 +00:00
{
Kernel: Rename LinearAddress => VirtualAddress.
2019-06-07 10:56:50 +00:00
auto* region = region_from_vaddr(process, vaddr);
Kernel: Fix incorrect EFAULTs when syscall would write into COW pages.
2019-01-25 00:39:15 +00:00
return region && region->is_writable();
Launching an arbitrary ELF executable from disk works! :^) This is so cool! It's a bit messy now with two Task constructors, but eventually they should fold into a single constructor somehow.
2018-10-22 13:42:39 +00:00
}
Implement fork()! This is quite cool! The syscall entry point plumbs the register dump down to sys$fork(), which uses it to set up the child process's TSS in order to resume execution right after the int 0x80 fork() call. :^) This works pretty well, although there is some problem with the kernel alias mappings used to clone the parent process's regions. If I disable the MM::release_page_directory() code, there's no problem. Probably there's a premature freeing of a physical page somehow.
2018-11-02 19:41:58 +00:00
Teach Process::exec() about the magic of file-backed VMO's. This is really sweet! :^) The four instances of /bin/sh spawned at startup now share their read-only text pages. There are problems and limitations here, and plenty of room for improvement. But it kinda works.
2018-11-08 20:20:09 +00:00
void MemoryManager::register_vmo(VMObject& vmo)
{
Fix VMO leak in Process::exec(). Gotta make sure things get cleaned up before we yield-teleport in exec(). Also VMOs and regions are now viewable through /proc/mm and /proc/regions.
2018-11-08 21:24:02 +00:00
InterruptDisabler disabler;
Teach Process::exec() about the magic of file-backed VMO's. This is really sweet! :^) The four instances of /bin/sh spawned at startup now share their read-only text pages. There are problems and limitations here, and plenty of room for improvement. But it kinda works.
2018-11-08 20:20:09 +00:00
m_vmos.set(&vmo);
}
void MemoryManager::unregister_vmo(VMObject& vmo)
{
Fix VMO leak in Process::exec(). Gotta make sure things get cleaned up before we yield-teleport in exec(). Also VMOs and regions are now viewable through /proc/mm and /proc/regions.
2018-11-08 21:24:02 +00:00
InterruptDisabler disabler;
Teach Process::exec() about the magic of file-backed VMO's. This is really sweet! :^) The four instances of /bin/sh spawned at startup now share their read-only text pages. There are problems and limitations here, and plenty of room for improvement. But it kinda works.
2018-11-08 20:20:09 +00:00
m_vmos.remove(&vmo);
}
Fix VMO leak in Process::exec(). Gotta make sure things get cleaned up before we yield-teleport in exec(). Also VMOs and regions are now viewable through /proc/mm and /proc/regions.
2018-11-08 21:24:02 +00:00
void MemoryManager::register_region(Region& region)
{
InterruptDisabler disabler;
Kernel: Rename LinearAddress => VirtualAddress.
2019-06-07 10:56:50 +00:00
if (region.vaddr().get() >= 0xc0000000)
Kernel: Allocate kernel stacks for threads using the region allocator. This patch moves away from using kmalloc memory for thread kernel stacks. This reduces pressure on kmalloc (16 KB per thread adds up fast) and prevents kernel stack overflow from scribbling all over random unrelated kernel memory.
2019-05-14 09:51:00 +00:00
m_kernel_regions.set(&region);
else
m_user_regions.set(&region);
Fix VMO leak in Process::exec(). Gotta make sure things get cleaned up before we yield-teleport in exec(). Also VMOs and regions are now viewable through /proc/mm and /proc/regions.
2018-11-08 21:24:02 +00:00
}
void MemoryManager::unregister_region(Region& region)
{
InterruptDisabler disabler;
Kernel: Rename LinearAddress => VirtualAddress.
2019-06-07 10:56:50 +00:00
if (region.vaddr().get() >= 0xc0000000)
Kernel: Allocate kernel stacks for threads using the region allocator. This patch moves away from using kmalloc memory for thread kernel stacks. This reduces pressure on kmalloc (16 KB per thread adds up fast) and prevents kernel stack overflow from scribbling all over random unrelated kernel memory.
2019-05-14 09:51:00 +00:00
m_kernel_regions.remove(&region);
else
m_user_regions.remove(&region);
Fix VMO leak in Process::exec(). Gotta make sure things get cleaned up before we yield-teleport in exec(). Also VMOs and regions are now viewable through /proc/mm and /proc/regions.
2018-11-08 21:24:02 +00:00
}
Fix some paging related bugs exposed by the spawn stress test. - Process::exec() needs to restore the original paging scope when called on a non-current process. - Add missing InterruptDisabler guards around g_processes access. - Only flush the TLB when modifying the active page tables.
2018-11-09 00:25:31 +00:00
Kernel: Introduce threads, and refactor everything in support of it. The scheduler now operates on threads, rather than on processes. Each process has a main thread, and can have any number of additional threads. The process exits when the main thread exits. This patch doesn't actually spawn any additional threads, it merely does all the plumbing needed to make it possible. :^)
2019-03-23 21:03:17 +00:00
ProcessPagingScope::ProcessPagingScope(Process& process)
{
ASSERT(current);
MM.enter_process_paging_scope(process);
}
ProcessPagingScope::~ProcessPagingScope()
{
MM.enter_process_paging_scope(current->process());
}
Reference in a new issue Copy permalink