Benchmarks

Minimal test example

A minimal test example was written specifically for each framework. You can find them in the jctest/test/comparisons folder. The tests were compiled with clang++ -O2

Comparisons are done using no configurations of the testing framework. Also, since the goal was to eliminate a precompiled library, the header only version was chosen where available.

#define JC_TEST_IMPLEMENTATION
#include <jc_test.h>

TEST(MyTest, Multiplication) {
    ASSERT_EQ(4, 2 * 2);
}
TEST(MyTest, Division) {
    ASSERT_EQ(2, 4 / 2);
}

int main(int argc, char *argv[]) {
    // Parses and removes any jc_test specific arguments
    jc_test_init(&argc, argv);
    // ... Do your test initialization
    return jc_test_run_all();
}

Benchmark

	jc_test	gtest	greatest	utest	doctest	catch2	snow 2
Size of program	27292	414608	19228	18280	146348	829572	23144
Compile time	217ms	600ms	141ms	86ms	1890ms	10662ms	216ms
Run time*	3ms	3ms	6ms	5ms	3ms	4ms	3ms

*) Picking the fastest time.

Real life examples

In our case (the Defold game engine), we wanted to replace GTEST with something new.

Here’s a comparison between the GTEST and jctest frameworks when building the game engine tests. We see a decrease of the compile time, ending up at ~54% of the original time.

The footprint on disc is also important, as it takes up both space but also affects writing/reading times. In this scenario the file sizes ended up at ~45% of the original sizes.