CYCLES_TEST_DEVICES is a list of devices (CPU, CUDA, OPTIX, OPENCL). It is set
to CPU only by default.
Test output is now writen to build/tests/cycles/<device>, and the HTML report
has separate report pages for the different devices, with option to compare
between CPU and GPU renders.
Various GPU tests are still failing due to CPU/GPU differences, these are to be
fixed or blacklisted still.
Ref T82193
This adds a new `--debug-exit-on-error` flag. When it is set, Blender
will abort with a non-zero exit code when there are internal errors.
Currently, "internal errors" includes memory leaks detected by
guardedalloc and error/fatal log entries in clog.
The new flag is passed to Blender in various places where automated
tests are run. Furthermore, the `--debug-memory` flag is used in tests,
because that makes the verbose output more useful, when dealing
with memory leaks.
Reviewers: brecht, sergey
Differential Revision: https://developer.blender.org/D8665
CentOS on the buildbot still runs Python 3.6, which is also used for the
unit tests. This means that the tests can't use language features that
are available to Blender itself. And testing with a different version of
Python than will be used by the actual code seems like a bad idea to me.
This commit adds `TEST_PYTHON_EXECUTABLE` as advanced CMake option. This
will allow us to set a specific Python executable when we need it. When
not set, a platform-specific default will be used:
- On Windows, the `python….exe` from the installation directory. This is
just like before this patch, except that this patch adds the
overridability.
- On macOS/Linux, the `${PYTHON_EXECUTABLE}` as found by CMake.
Every platform should now have a value (configured by the user or
detected by CMake) for `TEST_PYTHON_EXE`, so there is no need to allow
running without. This also removes the need to have some Python files
marked as executable.
If `TEST_PYTHON_EXE` is not user-configured, and thus the above default
is used, a status message is logged by CMake. I've seen this a lot in
other projects, and I like that it shows which values are auto-detected.
However, it's not common in Blender, so if we want we can either remove
it now, or remove it after the buildbot has been set up correctly.
Differential Revision: https://developer.blender.org/D7395
Reviewed by: campbellbarton, mont29, sergey
Blender startup time and shader compilation is a big factor when running
hundreds of tests, so now all renders in the same ctest run in the same
process.
This was previously reverted due to skipping other tests when one test
crashed. Now if a test crashes, Blender is re-run with the remaining
tests so we get results from them still.
Blender startup time and shader compilation is a big factor when running
hundreds of tests, so now all renders in the same ctest run in the same
process. If a test crashes, the remaining tests in the same category will
be marked as skipped.
Benchmarked on a quad core with ctest -j8.
cycles: 118.1s -> 94.3s
eevee: 66.2s -> 29.2s
workbench: 31.7s -> 8.6s
Being able to compare Eevee reference images is useful for refactoring I'm
working on so might as well add them now, even if we can still improve them.
Workbench tests are just rendering the same files as Cycles and Eevee. This
doesn't really tests many workbench settings until we add tests specifically
for them, but does cover how it it handles the different object types.
This adds Eevee render tests using the Cycles files. Currently it must
be enabled by setting WITH_OPENGL_RENDER_TESTS=ON. Once we have reference
images we can enable it by default.
Some of the Cycles and Eevee tests are also currently broken due to
modifier and particle changes.
Differential Revision: https://developer.blender.org/D3182
Shows new, reference and diff renders, with mouse hover to flip between
new and ref for easy comparison. This generates a report.html in
build_dir/tests/cycles, stored along with the new and diff images.
Differential Revision: https://developer.blender.org/D2770
Works similar to regular Cycles tests, just does OpenGL render to
get output image.
Seems to work fine with the only funny effect: Blender window will
pop up for each of the tests. This is current limitation of our
OpenGL context. Might be changed in the future.
Made them closer to how GTest shows the output, so reading test logs
is easier now (at least feels more uniform).
Additionally now we know how much time tests are taking so can tweak
samples/resolution to reduce render time of slow tests.
It is now also possible to enable colored messages using magic
CYCLESTEST_COLOR environment variable. This makes it even easier to
visually grep failed/passed tests using `ctest -R cycles -V`.
This reverts commit d390e24c49.
Forcing tests to success is really bad idea. It'll only lead to cases when
you see PASSED and will think everything is OK.
Long story short: never force tests to pass!
The idea is to use the set of really small images from the lib folder
and run Cycles render on them comparing render output to reference
images in the tests repository.
For sure same thing could become more generic for BI or Freestyle
render engines.
Thanks Campbell for review and code tweaks!
