| Age | Commit message (Collapse) | Author |
|---|
|
When creating uninitialized storage for variables, we need to make sure
that the alignment is correct. Fixes a KUBSAN failure when running
kernels compiled with Clang.
In `Syscalls/socket.cpp`, we can simply use local variables, as
`sockaddr_un` is a POD type.
Along with moving the `alignas` specifier to the correct member,
`AK::Optional`'s internal buffer has been made non-zeroed by default.
GCC emitted bogus uninitialized memory access warnings, so we now use
`__builtin_launder` to tell the compiler that we know what we are doing.
This might disable some optimizations, but judging by how GCC failed to
notice that the memory's initialization is dependent on `m_has_value`,
I'm not sure that's a bad thing.
|
|
Also make AllocationHeader acquisition from pointers more verbose
|
|
Also fixes a spelling mistake in the same file.
|
|
The kernel wouldn't boot reliably on x86_64 with just 2MiB.
|
|
|
|
Our types are getting a tiny bit larger for x86_64 so we need another
slab allocator to deal with that.
|
|
In standard C++, operators `new` and `new[]` are guaranteed to return a
valid (non-null) pointer and throw an exception if the allocation
couldn't be performed. Based on this, compilers did not check the
returned pointer before attempting to use them for object construction.
To avoid this, the allocator operators were changed to be `noexcept` in
PR #7026, which made GCC emit the desired null checks. Unfortunately,
this is a non-standard feature which meant that Clang would not accept
these function definitions, as it did not match its expected
declaration.
To make compiling using Clang possible, the special "nothrow" versions
of `new` are implemented in this commit. These take a tag type of
`std::nothrow_t` (used for disambiguating from placement new/etc.), and
are allowed by the standard to return null. There is a global variable,
`std::nothrow`, declared with this type, which is also exported into the
global namespace.
To perform fallible allocations, the following syntax should be used:
```cpp
auto ptr = new (nothrow) T;
```
As we don't support exceptions in the kernel, the only way of uphold the
"throwing" new's guarantee is to abort if the allocation couldn't be
performed. Once we have proper OOM handling in the kernel, this should
only be used for critical allocations, where we wouldn't be able to
recover from allocation failures anyway.
|
|
This also removes a lot of CPU.h includes infavor for Sections.h
|
|
We have that information in LibC, lets use that instead
|
|
|
|
There were a few cases where we could end up logging profiling events
before or after the associated process or thread exists in the profile:
After enabling profiling we might end up with CPU samples before we
had a chance to synthesize process/thread creation events.
After a thread exits we would still log associated kmalloc/kfree
events. Instead we now just ignore those events.
|
|
The allocation_size_in_chunks field contains the bytes necessary for
the AllocationHeader so we need to subtract that when we try to figure
out how much user data we have to copy.
Fixes #7549.
|
|
|
|
Problem:
- `BitmapView` permits changing the underlying `Bitmap`. This violates
the idea of a "view" since views are simply overlays which can
themselves change but do not change the underlying data.
Solution:
- Migrate all non-`const` member functions to Bitmap.
|
|
This reverts commit f25209113fcd15df5778938c4accf13c5139d278.
|
|
Problem:
- `BitmapView` permits changing the underlying `Bitmap`. This violates
the idea of a "view" since views are simply overlays which can
themselves change but do not change the underlying data.
Solution:
- Migrate all non-`const` member functions to Bitmap.
|
|
This implements the macOS API malloc_good_size() which returns the
true allocation size for a given requested allocation size. This
allows us to make use of all the available memory in a malloc chunk.
For example, for a malloc request of 35 bytes our malloc would
internally use a chunk of size 64, however the remaining 29 bytes
would be unused.
Knowing the true allocation size allows us to request more usable
memory that would otherwise be wasted and make that available for
Vector, HashTable and potentially other callers in the future.
|
|
Ideally we would never allocate under a spinlock, as it has many
performance and potentially functionality (deadlock) pitfalls.
We violate that rule in many places today, but we need a tool to track
them all down and fix them. This change introduces a new macro option
named `KMALLOC_VERIFY_NO_SPINLOCK_HELD` which can catch these
situations at runtime via an assert.
|
|
|
|
|
|
For Kernel OOM hardening to work correctly, we need to be able to
call a "nothrow" version of operator new. Unfortunately the default
"throwing" version of operator new assumes that the allocation will
never return on failure and will always throw an exception. This isn't
true in the Kernel, as we don't have exceptions. So if we call the
normal/throwing new and kmalloc returns NULL, the generated code will
happily go and dereference that NULL pointer by invoking the constructor
before we have a chance to handle the failure.
To fix this we declare operator new as noexcept in the Kernel headers,
which will allow the caller to actually handle allocation failure.
The delete implementations need to match the prototype of the new which
allocated them, so we need define delete as noexcept as well. GCC then
errors out declaring that you should implement sized delete as well, so
this change provides those stubs in order to compile cleanly.
Finally the new operator definitions have been standardized as being
declared with [[nodiscard]] to avoid potential memory leaks. So lets
declares the kernel versions that way as well.
|
|
We had some inconsistencies before:
- Sometimes "The", sometimes "the"
- Sometimes trailing ".", sometimes no trailing "."
I picked the most common one (lowecase "the", trailing ".") and applied
it to all copyright headers.
By using the exact same string everywhere we can ensure nothing gets
missed during a global search (and replace), and that these
inconsistencies are not spread any further (as copyright headers are
commonly copied to new files).
|
|
SPDX License Identifiers are a more compact / standardized
way of representing file license information.
See: https://spdx.dev/resources/use/#identifiers
This was done with the `ambr` search and replace tool.
ambr --no-parent-ignore --key-from-file --rep-from-file key.txt rep.txt *
|
|
Use BitmapView::set_range_and_verify_that_all_bits_flip() to validate
the heap chunk metadata bits as we go through them in kmalloc/kfree.
|
|
Double kfree() is exceedingly rare in our kernel since we use automatic
memory management and smart pointers for almost all code. However, it
doesn't hurt to do some basic checking that might one day catch bugs.
This patch makes us VERIFY that we don't already consider the first
chunk of a kmalloc() allocation free when kfree()'ing it.
|
|
Alot of code is shared between i386/i686/x86 and x86_64
and a lot probably will be used for compatability modes.
So we start by moving the headers into one Directory.
We will probalby be able to move some cpp files aswell.
|
|
The system is extremely sensitive to heap allocations during heap
expansion. This was causing frequent OOM panics under various loads.
Work around the issue for now by putting the logging behind
KMALLOC_DEBUG. Ideally dmesgln() & friends would not reqiure any
heap allocations, but we're not there right now.
Fixes #5724.
|
|
This macro inserts operator new/delete into a class, allowing you
to very easily specify a specific heap alignment.
|
|
This allows us to get kmalloc() memory aligned to the VM page size.
|
|
|
|
|
|
This was a thing back when the system was so little that any kernel
allocation above 1 MiB was basically guaranteed to be a bug. :^)
|
|
|
|
This may seem like a no-op change, however it shrinks down the Kernel by a bit:
.text -432
.unmap_after_init -60
.data -480
.debug_info -673
.debug_aranges 8
.debug_ranges -232
.debug_line -558
.debug_str -308
.debug_frame -40
With '= default', the compiler can do more inlining, hence the savings.
I intentionally omitted some opportunities for '= default', because they
would increase the Kernel size.
|
|
Make more of the kernel compile in 64-bit mode, and make some things
pointer-size-agnostic (by using FlatPtr.)
There's a lot of work to do here before the kernel will even compile.
|
|
(...and ASSERT_NOT_REACHED => VERIFY_NOT_REACHED)
Since all of these checks are done in release builds as well,
let's rename them to VERIFY to prevent confusion, as everyone is
used to assertions being compiled out in release.
We can introduce a new ASSERT macro that is specifically for debug
checks, but I'm doing this wholesale conversion first since we've
accumulated thousands of these already, and it's not immediately
obvious which ones are suitable for ASSERT.
|
|
|
|
There's no real system here, I just added it to various functions
that I don't believe we ever want to call after initialization
has finished.
With these changes, we're able to unmap 60 KiB of kernel text
after init. :^)
|
|
|
|
If we try to align a number above 0xfffff000 to the next multiple of
the page size (4 KiB), it would wrap around to 0. This is most likely
never what we want, so let's assert if that happens.
|
|
|
|
Now that we no longer need to support the signal trampolines being
user-accessible inside the kernel memory range, we can get rid of the
"kernel" and "user-accessible" flags on Region and simply use the
address of the region to determine whether it's kernel or user.
This also tightens the page table mapping code, since it can now set
user-accessibility based solely on the virtual address of a page.
|
|
The following script was used to make these changes:
#!/bin/bash
set -e
tmp=$(mktemp -d)
echo "tmp=$tmp"
find Kernel \( -name '*.cpp' -o -name '*.h' \) | sort > $tmp/Kernel.files
find . \( -path ./Toolchain -prune -o -path ./Build -prune -o -path ./Kernel -prune \) -o \( -name '*.cpp' -o -name '*.h' \) -print | sort > $tmp/EverythingExceptKernel.files
cat $tmp/Kernel.files | xargs grep -Eho '[A-Z0-9_]+_DEBUG' | sort | uniq > $tmp/Kernel.macros
cat $tmp/EverythingExceptKernel.files | xargs grep -Eho '[A-Z0-9_]+_DEBUG' | sort | uniq > $tmp/EverythingExceptKernel.macros
comm -23 $tmp/Kernel.macros $tmp/EverythingExceptKernel.macros > $tmp/Kernel.unique
comm -1 $tmp/Kernel.macros $tmp/EverythingExceptKernel.macros > $tmp/EverythingExceptKernel.unique
cat $tmp/Kernel.unique | awk '{ print "#cmakedefine01 "$1 }' > $tmp/Kernel.header
cat $tmp/EverythingExceptKernel.unique | awk '{ print "#cmakedefine01 "$1 }' > $tmp/EverythingExceptKernel.header
for macro in $(cat $tmp/Kernel.unique)
do
cat $tmp/Kernel.files | xargs grep -l $macro >> $tmp/Kernel.new-includes ||:
done
cat $tmp/Kernel.new-includes | sort > $tmp/Kernel.new-includes.sorted
for macro in $(cat $tmp/EverythingExceptKernel.unique)
do
cat $tmp/Kernel.files | xargs grep -l $macro >> $tmp/Kernel.old-includes ||:
done
cat $tmp/Kernel.old-includes | sort > $tmp/Kernel.old-includes.sorted
comm -23 $tmp/Kernel.new-includes.sorted $tmp/Kernel.old-includes.sorted > $tmp/Kernel.includes.new
comm -13 $tmp/Kernel.new-includes.sorted $tmp/Kernel.old-includes.sorted > $tmp/Kernel.includes.old
comm -12 $tmp/Kernel.new-includes.sorted $tmp/Kernel.old-includes.sorted > $tmp/Kernel.includes.mixed
for file in $(cat $tmp/Kernel.includes.new)
do
sed -i -E 's/#include <AK\/Debug\.h>/#include <Kernel\/Debug\.h>/' $file
done
for file in $(cat $tmp/Kernel.includes.mixed)
do
echo "mixed include in $file, requires manual editing."
done
|
|
|
|
It would be tempting to uncomment these statements, but that won't work
with the new changes.
This was done with the following commands:
find . \( -name '*.cpp' -o -name '*.h' -o -name '*.in' \) -not -path './Toolchain/*' -not -path './Build/*' -exec awk -i inplace '$0 !~ /\/\/#define/ { if (!toggle) { print; } else { toggle = !toggle } } ; $0 ~/\/\/#define/ { toggle = 1 }' {} \;
find . \( -name '*.cpp' -o -name '*.h' -o -name '*.in' \) -not -path './Toolchain/*' -not -path './Build/*' -exec awk -i inplace '$0 !~ /\/\/ #define/ { if (!toggle) { print; } else { toggle = !toggle } } ; $0 ~/\/\/ #define/ { toggle = 1 }' {} \;
|
|
The kernel ignored the first 8 MiB of RAM while parsing the memory map
because the kmalloc heaps and the super physical pages lived here. Move
all that stuff inside the .bss segment so that those memory regions are
accounted for, otherwise we risk overwriting boot modules placed next
to the kernel.
|
|
These changes are arbitrarily divided into multiple commits to make it
easier to find potentially introduced bugs with git bisect.Everything:
|
|
|
|
This implements memory commitments and lazy-allocation of committed
memory.
|
|
This adds the ability for a Region to define volatile/nonvolatile
areas within mapped memory using madvise(). This also means that
memory purging takes into account all views of the PurgeableVMObject
and only purges memory that is not needed by all of them. When calling
madvise() to change an area to nonvolatile memory, return whether
memory from that area was purged. At that time also try to remap
all memory that is requested to be nonvolatile, and if insufficient
pages are available notify the caller of that fact.
|
