summaryrefslogtreecommitdiff
path: root/Documentation/RunningTests.md
diff options
context:
space:
mode:
authorAndrew Kaster <andrewdkaster@gmail.com>2021-05-15 20:13:31 -0600
committerLinus Groh <mail@linusgroh.de>2021-05-16 21:58:14 +0100
commitcfdd231a4d6036e27da584bf633bd60ff215193e (patch)
treee37a4fd0d17f1517bd8ff4c4c1cefda3e438897c /Documentation/RunningTests.md
parentac1d87b9905ba9e321642c9915e78c2a7bbee557 (diff)
downloadserenity-cfdd231a4d6036e27da584bf633bd60ff215193e.zip
Documentation: Add Sanitizer section to RunningTests
Also, add link to RunningTests BuildInstructions, and clean up stale commands in RunningTests to align to current build strategies.
Diffstat (limited to 'Documentation/RunningTests.md')
-rw-r--r--Documentation/RunningTests.md77
1 files changed, 57 insertions, 20 deletions
diff --git a/Documentation/RunningTests.md b/Documentation/RunningTests.md
index 5da6004127..d922d6dfe4 100644
--- a/Documentation/RunningTests.md
+++ b/Documentation/RunningTests.md
@@ -1,10 +1,10 @@
-## Running SerenityOS Tests
+# Running SerenityOS Tests
-There are two classes of tests built during a Serenity build: host tests and target tests. Host tests run on the build
-machine, and use [Lagom](../Meta/Lagom/ReadMe.md) to build Serenity userspace libraries for the host platform. Target
-tests run on the Serenity machine, either emulated or bare metal.
+There are two classes of tests built during a SerenityOS build: host tests and target tests. Host tests run on the build
+machine, and use [Lagom](../Meta/Lagom/ReadMe.md) to build SerenityOS userspace libraries for the host platform. Target
+tests run on the SerenityOS machine, either emulated or bare metal.
-### Running Host Tests
+## Running Host Tests
There are two ways to build host tests: from a full build, or from a Lagom-only build. The only difference is the CMake
command used to initialize the build directory.
@@ -12,41 +12,78 @@ command used to initialize the build directory.
For a full build, pass `-DBUILD_LAGOM=ON` to the CMake command.
```sh
-mkdir Build
-cd Build
-cmake .. -GNinja -DBUILD_LAGOM=ON
+mkdir -p Build/lagom
+cd Build/lagom
+cmake ../.. -GNinja -DBUILD_LAGOM=ON
```
-For a Lagom-only build, pass the Lagom directory to CMake.
+For a Lagom-only build, pass the Lagom directory to CMake. The `BUILD_LAGOM` CMake option is still required.
```sh
mkdir BuildLagom
cd BuildLagom
-cmake ../Meta/Lagom -GNinja
+cmake ../Meta/Lagom -GNinja -DBUILD_LAGOM=ON
```
-In both cases, the tests can be run via ninja after doing a build:
+In both cases, the tests can be run via ninja after doing a build. Note that `test-js` requires the `SERENITY_SOURCE_DIR` environment variable to be set
+to the root of the serenity source tree when running on a non-SerenityOS host.
```sh
+# /path/to/serenity repository
+export SERENITY_SOURCE_DIR=${PWD}/..
ninja && ninja test
```
-### Running Target Tests
+To see the stdout/stderr output of failing tests, the reccommended way is to set the environment variable [`CTEST_OUTPUT_ON_FAILURE`](https://cmake.org/cmake/help/latest/manual/ctest.1.html#options) to 1.
-Tests built for the Serenity target get installed either into `/usr/Tests` or `/bin`. `/usr/Tests` is preferred, but
+```sh
+CTEST_OUTPUT_ON_FAILURE=1 ninja test
+
+# or, using ctest directly...
+ctest --output-on-failure
+```
+
+### Running with Sanitizers
+
+CI runs host tests with Address Sanitizer and Undefined Sanitizer instrumentation enabled. These tools catch many
+classes of common C++ errors, including memory leaks, out of bounds access to stack and heap allocations, and
+signed integer overflow. For more info on the sanitizers, check out the Address Sanitizer [wiki page](https://github.com/google/sanitizers/wiki),
+or the Undefined Sanitizer [documentation](https://clang.llvm.org/docs/UndefinedBehaviorSanitizer.html) from clang.
+
+Note that a sanitizer build will take significantly longer than a non-santizer build, and will mess with caches in tools such as `ccache`.
+The sanitizers can be enabled with the `-DENABLE_FOO_SANITIZER` set of flags. Sanitizers are only supported for Lagom tests, as SerenityOS support
+for gcc's `libsanitizer` is not yet implemented.
+
+```sh
+mkdir BuildLagom
+cd BuildLagom
+cmake ../Meta/Lagom -GNinja -DBUILD_LAGOM=ON -DENABLE_ADDRESS_SANITIZER=ON -DENABLE_UNDEFINED_SANITIZER=ON
+ninja
+CTEST_OUTPUT_ON_FAILURE=1 SERENITY_SOURCE_DIR=${PWD}/.. ninja test
+```
+
+To ensure that Undefined Sanitizer errors fail the test, the `halt_on_error` flag should be set to 1 in the environment variable `UBSAN_OPTIONS`.
+
+```sh
+UBSAN_OPTIONS=halt_on_error=1 CTEST_OUTPUT_ON_FAILURE=1 SERENITY_SOURCE_DIR=${PWD}/.. ninja test
+```
+
+## Running Target Tests
+
+Tests built for the SerenityOS target get installed either into `/usr/Tests` or `/bin`. `/usr/Tests` is preferred, but
some system tests are installed into `/bin` for historical reasons.
The easiest way to run all of the known tests in the system is to use the `run-tests-and-shutdown.sh` script that gets
installed into `/home/anon/tests`. When running in CI, the environment variable `$DO_SHUTDOWN_AFTER_TESTS` is set, which
will run `shutdown -n` after running all the tests.
-For completeness, a basic on-target test run will need the Serenity image built and run via QEMU.
+For completeness, a basic on-target test run will need the SerenityOS image built and run via QEMU.
```sh
-mkdir Build
-cd Build
-cmake .. -GNinja
-ninja && ninja install && ninja image && ninja run
+mkdir Build/i686
+cd Build/i686
+cmake ../.. -GNinja
+ninja install && ninja image && ninja run
```
In the initial terminal, one can easily run the test runner script:
@@ -77,8 +114,8 @@ BootModes=self-test
the serial debug output to `./debug.log` so that both stdout of the tests and the dbgln from the kernel/tests can be
captured.
-To run with CI's TestRunner system server entry, Serenity needs booted in self-test mode. Running the following shell
-lines will boot Serenity in self-test mode, run tests, and exit.
+To run with CI's TestRunner system server entry, SerenityOS needs booted in self-test mode. Running the following shell
+lines will boot SerenityOS in self-test mode, run tests, and exit.
```sh
export SERENITY_RUN=ci