diff options
author | Andrew Kaster <andrewdkaster@gmail.com> | 2021-05-15 20:13:31 -0600 |
---|---|---|
committer | Linus Groh <mail@linusgroh.de> | 2021-05-16 21:58:14 +0100 |
commit | cfdd231a4d6036e27da584bf633bd60ff215193e (patch) | |
tree | e37a4fd0d17f1517bd8ff4c4c1cefda3e438897c /Documentation/RunningTests.md | |
parent | ac1d87b9905ba9e321642c9915e78c2a7bbee557 (diff) | |
download | serenity-cfdd231a4d6036e27da584bf633bd60ff215193e.zip |
Documentation: Add Sanitizer section to RunningTests
Also, add link to RunningTests BuildInstructions, and clean up stale
commands in RunningTests to align to current build strategies.
Diffstat (limited to 'Documentation/RunningTests.md')
-rw-r--r-- | Documentation/RunningTests.md | 77 |
1 files changed, 57 insertions, 20 deletions
diff --git a/Documentation/RunningTests.md b/Documentation/RunningTests.md index 5da6004127..d922d6dfe4 100644 --- a/Documentation/RunningTests.md +++ b/Documentation/RunningTests.md @@ -1,10 +1,10 @@ -## Running SerenityOS Tests +# Running SerenityOS Tests -There are two classes of tests built during a Serenity build: host tests and target tests. Host tests run on the build -machine, and use [Lagom](../Meta/Lagom/ReadMe.md) to build Serenity userspace libraries for the host platform. Target -tests run on the Serenity machine, either emulated or bare metal. +There are two classes of tests built during a SerenityOS build: host tests and target tests. Host tests run on the build +machine, and use [Lagom](../Meta/Lagom/ReadMe.md) to build SerenityOS userspace libraries for the host platform. Target +tests run on the SerenityOS machine, either emulated or bare metal. -### Running Host Tests +## Running Host Tests There are two ways to build host tests: from a full build, or from a Lagom-only build. The only difference is the CMake command used to initialize the build directory. @@ -12,41 +12,78 @@ command used to initialize the build directory. For a full build, pass `-DBUILD_LAGOM=ON` to the CMake command. ```sh -mkdir Build -cd Build -cmake .. -GNinja -DBUILD_LAGOM=ON +mkdir -p Build/lagom +cd Build/lagom +cmake ../.. -GNinja -DBUILD_LAGOM=ON ``` -For a Lagom-only build, pass the Lagom directory to CMake. +For a Lagom-only build, pass the Lagom directory to CMake. The `BUILD_LAGOM` CMake option is still required. ```sh mkdir BuildLagom cd BuildLagom -cmake ../Meta/Lagom -GNinja +cmake ../Meta/Lagom -GNinja -DBUILD_LAGOM=ON ``` -In both cases, the tests can be run via ninja after doing a build: +In both cases, the tests can be run via ninja after doing a build. Note that `test-js` requires the `SERENITY_SOURCE_DIR` environment variable to be set +to the root of the serenity source tree when running on a non-SerenityOS host. ```sh +# /path/to/serenity repository +export SERENITY_SOURCE_DIR=${PWD}/.. ninja && ninja test ``` -### Running Target Tests +To see the stdout/stderr output of failing tests, the reccommended way is to set the environment variable [`CTEST_OUTPUT_ON_FAILURE`](https://cmake.org/cmake/help/latest/manual/ctest.1.html#options) to 1. -Tests built for the Serenity target get installed either into `/usr/Tests` or `/bin`. `/usr/Tests` is preferred, but +```sh +CTEST_OUTPUT_ON_FAILURE=1 ninja test + +# or, using ctest directly... +ctest --output-on-failure +``` + +### Running with Sanitizers + +CI runs host tests with Address Sanitizer and Undefined Sanitizer instrumentation enabled. These tools catch many +classes of common C++ errors, including memory leaks, out of bounds access to stack and heap allocations, and +signed integer overflow. For more info on the sanitizers, check out the Address Sanitizer [wiki page](https://github.com/google/sanitizers/wiki), +or the Undefined Sanitizer [documentation](https://clang.llvm.org/docs/UndefinedBehaviorSanitizer.html) from clang. + +Note that a sanitizer build will take significantly longer than a non-santizer build, and will mess with caches in tools such as `ccache`. +The sanitizers can be enabled with the `-DENABLE_FOO_SANITIZER` set of flags. Sanitizers are only supported for Lagom tests, as SerenityOS support +for gcc's `libsanitizer` is not yet implemented. + +```sh +mkdir BuildLagom +cd BuildLagom +cmake ../Meta/Lagom -GNinja -DBUILD_LAGOM=ON -DENABLE_ADDRESS_SANITIZER=ON -DENABLE_UNDEFINED_SANITIZER=ON +ninja +CTEST_OUTPUT_ON_FAILURE=1 SERENITY_SOURCE_DIR=${PWD}/.. ninja test +``` + +To ensure that Undefined Sanitizer errors fail the test, the `halt_on_error` flag should be set to 1 in the environment variable `UBSAN_OPTIONS`. + +```sh +UBSAN_OPTIONS=halt_on_error=1 CTEST_OUTPUT_ON_FAILURE=1 SERENITY_SOURCE_DIR=${PWD}/.. ninja test +``` + +## Running Target Tests + +Tests built for the SerenityOS target get installed either into `/usr/Tests` or `/bin`. `/usr/Tests` is preferred, but some system tests are installed into `/bin` for historical reasons. The easiest way to run all of the known tests in the system is to use the `run-tests-and-shutdown.sh` script that gets installed into `/home/anon/tests`. When running in CI, the environment variable `$DO_SHUTDOWN_AFTER_TESTS` is set, which will run `shutdown -n` after running all the tests. -For completeness, a basic on-target test run will need the Serenity image built and run via QEMU. +For completeness, a basic on-target test run will need the SerenityOS image built and run via QEMU. ```sh -mkdir Build -cd Build -cmake .. -GNinja -ninja && ninja install && ninja image && ninja run +mkdir Build/i686 +cd Build/i686 +cmake ../.. -GNinja +ninja install && ninja image && ninja run ``` In the initial terminal, one can easily run the test runner script: @@ -77,8 +114,8 @@ BootModes=self-test the serial debug output to `./debug.log` so that both stdout of the tests and the dbgln from the kernel/tests can be captured. -To run with CI's TestRunner system server entry, Serenity needs booted in self-test mode. Running the following shell -lines will boot Serenity in self-test mode, run tests, and exit. +To run with CI's TestRunner system server entry, SerenityOS needs booted in self-test mode. Running the following shell +lines will boot SerenityOS in self-test mode, run tests, and exit. ```sh export SERENITY_RUN=ci |