Using theories

Theories are a powerful tool for test-driven development, allowing you to test a specific behaviour against all permutations of a set of user-defined parameters known as “data points”.

Adding theories

group TheoryBase

Defines

Theory(Params, Suite, Name, ...) internal

Defines a new theory test.

The parameters are selected from a cartesian product defined by a TheoryDataPoints macro.

Example:

Theory((int arg0, double arg1), suite, test) {
    // function body
};

Parameters
  • Params: A list of function parameters.
  • Suite: The name of the test suite containing this test.
  • Name: The name of the test.
  • ...: An optional sequence of designated initializer key/value pairs as described in the criterion_test_extra_data structure (see criterion/types.h). Example: .exit_code = 1

TheoryDataPoints(Suite, Name) CR_TH_INTERNAL_TDPS(Suite, Name)

Defines an array of data points.

The types of the specified data points must match the types of the associated theory.

Each entry in the array must be the result of the DataPoints macro.

Example:

TheoryDataPoints(suite, test) = {
    DataPoints(int, 1, 2, 3),               // first theory parameter
    DataPoints(double, 4.2, 0, -INFINITY),  // second theory parameter
};

Parameters
  • Suite: The name of the test suite containing this test.
  • Name: The name of the test.

DataPoints(Type, ...) CR_EXPAND(CR_TH_INTERNAL_DP(Type, __VA_ARGS__))

Defines a new set of data points.

Parameters
  • Type: The type of each data point in the set.
  • ...: The data points in the set.

Adding theories is done by defining data points and a theory function:

#include <criterion/theories.h>

TheoryDataPoints(suite_name, test_name) = {
    DataPoints(Type0, val0, val1, val2, ..., valN),
    DataPoints(Type1, val0, val1, val2, ..., valN),
    ...
    DataPoints(TypeN, val0, val1, val2, ..., valN),
}

Theory((Type0 arg0, Type1 arg1, ..., TypeN argN), suite_name, test_name) {
}

suite_name and test_name are the identifiers of the test suite and the test, respectively. These identifiers must follow the language identifier format.

Type0/arg0 through TypeN/argN are the parameter types and names of theory theory function and are available in the body of the function.

Datapoints are declared in the same number, type, and order than the parameters inside the TheoryDataPoints macro, with the DataPoints macro. Beware! It is undefined behaviour to not have a matching number and type of theory parameters and datatypes.

Each DataPoints must then specify the values that will be used for the theory parameter it is linked to (val0 through valN).

Assertions and invariants

You can use any cr_assert or cr_expect macro functions inside the body of a theory function.

Theory invariants are enforced through the cr_assume(Condition) macro function: if Condition is false, then the current theory iteration aborts without making the test fail.

On top of those, more assume macro functions are available for common operations:

group TheoryInvariants

Defines

cr_assume(Condition) do { \ if (!(Condition)) \ cr_theory_abort(); \ } while (0)

Assumes Condition is true.

Evaluates Condition and continues execution if it is true. Otherwise the current theory iteration aborts without marking the test as failure.

Parameters
  • Condition: Condition to test

cr_assume_not(Condition) cr_assume(!(Condition))

Assumes Condition is false.

Evaluates Condition and continues execution if it is false. Otherwise the current theory iteration aborts without marking the test as failure.

Parameters
  • Condition: Condition to test

cr_assume_eq(Actual, Expected) cr_assume_op_(==, Actual, Expected)

Assumes Actual is equal to Expected

Continues execution if Actual is equal to Expected. Otherwise the current theory iteration aborts without marking the test as failure.

Parameters
  • Actual: Value to test
  • Expected: Expected value

cr_assume_neq(Actual, Unexpected) cr_assume_op_(!=, Actual, Unexpected)

Assumes Actual is not equal to Unexpected

Continues execution if Actual is not equal to Unexpected. Otherwise the current theory iteration aborts without marking the test as failure.

Parameters
  • Actual: Value to test
  • Unexpected: Unexpected value

cr_assume_gt(Actual, Reference) cr_assume_op_(>, Actual, Reference)

Assumes Actual is greater than Reference

Continues execution if Actual is greater than Reference. Otherwise the current theory iteration aborts without marking the test as failure.

Parameters
  • Actual: Value to test
  • Reference: Reference value

cr_assume_geq(Actual, Reference) cr_assume_op_(>=, Actual, Reference)

Assumes Actual is greater or equal to Reference

Continues execution if Actual is greater or equal to Reference. Otherwise the current theory iteration aborts without marking the test as failure.

Parameters
  • Actual: Value to test
  • Reference: Reference value

cr_assume_lt(Actual, Reference) cr_assume_op_(<, Actual, Reference)

Assumes Actual is less than Reference

Continues execution if Actual is less than Reference. Otherwise the current theory iteration aborts without marking the test as failure.

Parameters
  • Actual: Value to test
  • Reference: Reference value

cr_assume_leq(Actual, Reference) cr_assume_op_(<=, Actual, Reference)

Assumes Actual is less or equal to Reference

Continues execution if Actual is less or equal to Reference. Otherwise the current theory iteration aborts without marking the test as failure.

Parameters
  • Actual: Value to test
  • Reference: Reference value

cr_assume_null(Value) cr_assume_eq(Value, NULL)

Assumes Value is NULL.

Continues execution if Value is NULL. Otherwise the current theory iteration aborts without marking the test as failure.

Parameters
  • Value: Value to test

cr_assume_not_null(Value) cr_assume_neq(Value, NULL)

Assumes Value is not NULL.

Continues execution if Value is not NULL. Otherwise the current theory iteration aborts without marking the test as failure.

Parameters
  • Value: Value to test

cr_assume_float_eq(Actual, Expected, Epsilon) cr_assume((Expected) - (Actual) <= (Epsilon) \ && (Actual) - (Expected) <= (Epsilon))

Assumes Actual is equal to Expected with a tolerance of Epsilon

Continues execution if Actual is equal to Expected with a tolerance of Epsilon. Otherwise the current theory iteration aborts without marking the test as failure.

Note
Use this to test equality between floats
Parameters
  • Actual: Value to test
  • Expected: Expected value
  • Epsilon: Tolerance between Actual and Expected

cr_assume_float_neq(Actual, Expected, Epsilon) cr_assume((Expected) - (Actual) > (Epsilon) \ || (Actual) - (Expected) > (Epsilon))

Assumes Actual is not equal to Expected with a tolerance of Epsilon

Continues execution if Actual is not equal to Expected with a tolerance of Epsilon. Otherwise the current theory iteration aborts without marking the test as failure.

Note
Use this to test equality between floats
Parameters
  • Actual: Value to test
  • Expected: Expected value
  • Epsilon: Tolerance between Actual and Expected

cr_assume_str_eq(Actual, Expected) cr_assume_str_op_(==, Actual, Expected)

Assumes Actual is lexicographically equal to Expected

Continues execution if Actual is lexicographically equal to Expected. Otherwise the current theory iteration aborts without marking the test as failure.

Parameters
  • Actual: String to test
  • Expected: Expected string

cr_assume_str_neq(Actual, Unexpected) cr_assume_str_op_(!=, Actual, Unexpected)

Assumes Actual is not lexicographically equal to Unexpected

Continues execution if Actual is not lexicographically equal to Unexpected. Otherwise the current theory iteration aborts without marking the test as failure.

Parameters
  • Actual: String to test
  • Unexpected: Unexpected string

cr_assume_str_lt(Actual, Reference) cr_assume_str_op_(<, Actual, Reference)

Assumes Actual is lexicographically less than Reference

Continues execution if Actual is lexicographically less than Reference. Otherwise the current theory iteration aborts without marking the test as failure.

Parameters
  • Actual: Value to test
  • Reference: Reference value

cr_assume_str_leq(Actual, Reference) cr_assume_str_op_(<=, Actual, Reference)

Assumes Actual is lexicographically less or equal to Reference

Continues execution if Actual is lexicographically less or equal to Reference. Otherwise the current theory iteration aborts without marking the test as failure.

Parameters
  • Actual: Value to test
  • Reference: Reference value

cr_assume_str_gt(Actual, Reference) cr_assume_str_op_(>, Actual, Reference)

Assumes Actual is lexicographically greater than Reference

Continues execution if Actual is lexicographically greater than Reference. Otherwise the current theory iteration aborts without marking the test as failure.

Parameters
  • Actual: Value to test
  • Reference: Reference value

cr_assume_str_geq(Actual, Reference) cr_assume_str_op_(>=, Actual, Reference)

Assumes Actual is lexicographically greater or equal to Reference

Continues execution if Actual is lexicographically greater or equal to Reference. Otherwise the current theory iteration aborts without marking the test as failure.

Parameters
  • Actual: Value to test
  • Reference: Reference value

cr_assume_arr_eq(Actual, Expected, Size) cr_assume(!memcmp((Actual), (Expected), (Size)))

Assumes Actual is byte-to-byte equal to Expected

Continues execution if Actual is byte-to-byte equal to Expected. Otherwise the current theory iteration aborts without marking the test as failure.

Warning
This should not be used on struct arrays
Parameters
  • Actual: Array to test
  • Expected: Expected array
  • Size: The size of both arrays

cr_assume_arr_neq(Actual, Unexpected, Size) cr_assume(memcmp((Actual), (Unexpected), (Size)))

Assumes Actual is not byte-to-byte equal to Unexpected

Continues execution if Actual is not byte-to-byte equal to Unexpected. Otherwise the current theory iteration aborts without marking the test as failure.

Warning
This should not be used on struct arrays
Parameters
  • Actual: Array to test
  • Unexpected: Unexpected array
  • Size: The size of both arrays

Configuring theories

Theories can optionally recieve configuration parameters to alter the behaviour of the underlying test; as such, those parameters are the same ones as the ones of the Test macro function (c.f. Configuration reference).

Full sample & purpose of theories

We will illustrate how useful theories are with a simple example using Criterion:

The basics of theories

Let us imagine that we want to test if the algebraic properties of integers, and specifically concerning multiplication, are respected by the C language:

int my_mul(int lhs, int rhs) {
    return lhs * rhs;
}

Now, we know that multiplication over integers is commutative, so we first test that:

#include <criterion/criterion.h>

Test(algebra, multiplication_is_commutative) {
    cr_assert_eq(my_mul(2, 3), my_mul(3, 2));
}

However, this test is imperfect, because there is not enough triangulation to insure that my_mul is indeed commutative. One might be tempted to add more assertions on other values, but this will never be good enough: commutativity should work for any pair of integers, not just an arbitrary set, but, to be fair, you cannot just test this behaviour for every integer pair that exists.

Theories purposely bridge these two issues by introducing the concept of “data point” and by refactoring the repeating logic into a dedicated function:

#include <criterion/theories.h>

TheoryDataPoints(algebra, multiplication_is_commutative) = {
    DataPoints(int, [...]),
    DataPoints(int, [...]),
};

Theory((int lhs, int rhs), algebra, multiplication_is_commutative) {
    cr_assert_eq(my_mul(lhs, rhs), my_mul(rhs, lhs));
}

As you can see, we refactored the assertion into a theory taking two unspecified integers.

We first define some data points in the same order and type the parameters have, from left to right: the first DataPoints(int, ...) will define the set of values passed to the int lhs parameter, and the second will define the one passed to int rhs.

Choosing the values of the data point is left to you, but we might as well use “interesting” values: 0, -1, 1, -2, 2, INT_MAX, and INT_MIN:

#include <limits.h>

TheoryDataPoints(algebra, multiplication_is_commutative) = {
    DataPoints(int, 0, -1, 1, -2, 2, INT_MAX, INT_MIN),
    DataPoints(int, 0, -1, 1, -2, 2, INT_MAX, INT_MIN),
};

Using theory invariants

The second thing we can test on multiplication is that it is the inverse function of division. Then, given the division operation:

int my_div(int lhs, int rhs) {
    return lhs / rhs;
}

The associated theory is straight-forward:

#include <criterion/theories.h>

TheoryDataPoints(algebra, multiplication_is_inverse_of_division) = {
    DataPoints(int, 0, -1, 1, -2, 2, INT_MAX, INT_MIN),
    DataPoints(int, 0, -1, 1, -2, 2, INT_MAX, INT_MIN),
};

Theory((int lhs, int rhs), algebra, multiplication_is_inverse_of_division) {
    cr_assert_eq(lhs, my_div(my_mul(lhs, rhs), rhs));
}

However, we do have a problem because you cannot have the theory function divide by 0. For this purpose, we can assume than rhs will never be 0:

Theory((int lhs, int rhs), algebra, multiplication_is_inverse_of_division) {
    cr_assume(rhs != 0);
    cr_assert_eq(lhs, my_div(my_mul(lhs, rhs), rhs));
}

cr_assume will abort the current theory iteration if the condition is not fulfiled.

Running the test at that point will raise a big problem with the current implementation of my_mul and my_div:

[----] theories.c:24: Assertion failed: (a) == (bad_div(bad_mul(a, b), b))
[----]   Theory algebra::multiplication_is_inverse_of_division failed with the following parameters: (2147483647, 2)
[----] theories.c:24: Assertion failed: (a) == (bad_div(bad_mul(a, b), b))
[----]   Theory algebra::multiplication_is_inverse_of_division failed with the following parameters: (-2147483648, 2)
[----] theories.c:24: Unexpected signal caught below this line!
[FAIL] algebra::multiplication_is_inverse_of_division: CRASH!

The theory shows that my_div(my_mul(INT_MAX, 2), 2) and my_div(my_mul(INT_MIN, 2), 2) does not respect the properties for multiplication: it happens that the behaviour of these two functions is undefined because the operation overflows.

Similarly, the test crashes at the end; debugging shows that the source of the crash is the divison of INT_MAX by -1, which is undefined.

Fixing this is as easy as changing the prototypes of my_mul and my_div to operate on long long rather than int.

What’s the difference between theories and parameterized tests ?

While it may at first seem that theories and parameterized tests are the same, just because they happen to take multiple parameters does not mean that they logically behave in the same manner.

Parameterized tests are useful to test a specific logic against a fixed, finite set of examples that you need to work.

Theories are, well, just that: theories. They represent a test against an universal truth, regardless of the input data matching its predicates.

Implementation-wise, Criterion also marks the separation by the way that both are executed:

Each parameterized test iteration is run in its own test; this means that one parameterized test acts as a collection of many tests, and gets reported as such.

On the other hand, a theory act as one single test, since the size and contents of the generated data set is not relevant. It does not make sense to say that an universal truth is “partially true”, so if one of the iteration fails, then the whole test fails.