Using theories¶
Theories are a powerful tool for testdriven development, allowing you to test a specific behaviour against all permutations of a set of userdefined parameters known as “data points”.
Adding theories¶
 group TheoryBase
Defines

Theory(Params, Suite, Name, ...)¶
Defines a new theory test.
The parameters are selected from a cartesian product defined by a TheoryDataPoints macro.
Example:
Theory((int arg0, double arg1), suite, test) { // function body };
 Parameters
Params – A list of function parameters.
Suite – The name of the test suite containing this test.
Name – The name of the test.
... – An optional sequence of designated initializer key/value pairs as described in the
criterion_test_extra_data
structure (see criterion/types.h). Example: .exit_code = 1

TheoryDataPoints(Suite, Name)¶
Defines an array of data points.
The types of the specified data points must match the types of the associated theory.
Each entry in the array must be the result of the
DataPoints
macro.Example:
TheoryDataPoints(suite, test) = { DataPoints(int, 1, 2, 3), // first theory parameter DataPoints(double, 4.2, 0, INFINITY), // second theory parameter };
 Parameters
Suite – The name of the test suite containing this test.
Name – The name of the test.

DataPoints(Type, ...)¶
Defines a new set of data points.
 Parameters
Type – The type of each data point in the set.
... – The data points in the set.

Theory(Params, Suite, Name, ...)¶
Adding theories is done by defining data points and a theory function:
#include <criterion/theories.h>
TheoryDataPoints(suite_name, test_name) = {
DataPoints(Type0, val0, val1, val2, ..., valN),
DataPoints(Type1, val0, val1, val2, ..., valN),
...
DataPoints(TypeN, val0, val1, val2, ..., valN),
}
Theory((Type0 arg0, Type1 arg1, ..., TypeN argN), suite_name, test_name) {
}
suite_name
and test_name
are the identifiers of the test suite and
the test, respectively. These identifiers must follow the language
identifier format.
Type0/arg0
through TypeN/argN
are the parameter types and names of theory
theory function and are available in the body of the function.
Datapoints are declared in the same number, type, and order than the parameters
inside the TheoryDataPoints
macro, with the DataPoints
macro.
Beware! It is undefined behaviour to not have a matching number and type of
theory parameters and datatypes.
Each DataPoints
must then specify the values that will be used for the
theory parameter it is linked to (val0
through valN
).
Assertions and invariants¶
You can use any cr_assert
or cr_expect
macro functions inside the body of a
theory function.
Theory invariants are enforced through the cr_assume(Condition)
macro function:
if Condition
is false, then the current theory iteration aborts without
making the test fail.
On top of those, more assume
macro functions are available for common operations:
 group TheoryInvariants
Defines

cr_assume(Condition)¶
Assumes
Condition
is true.Evaluates
Condition
and continues execution if it is true. Otherwise the current theory iteration aborts without marking the test as failure. Parameters
Condition – [in] Condition to test

cr_assume_not(Condition)¶
Assumes
Condition
is false.Evaluates
Condition
and continues execution if it is false. Otherwise the current theory iteration aborts without marking the test as failure. Parameters
Condition – [in] Condition to test

cr_assume_eq(Actual, Expected)¶
Assumes
Actual
is equal toExpected
Continues execution if
Actual
is equal toExpected
. Otherwise the current theory iteration aborts without marking the test as failure. Parameters
Actual – [in] Value to test
Expected – [in] Expected value

cr_assume_neq(Actual, Unexpected)¶
Assumes
Actual
is not equal toUnexpected
Continues execution if
Actual
is not equal toUnexpected
. Otherwise the current theory iteration aborts without marking the test as failure. Parameters
Actual – [in] Value to test
Unexpected – [in] Unexpected value

cr_assume_gt(Actual, Reference)¶
Assumes
Actual
is greater thanReference
Continues execution if
Actual
is greater thanReference
. Otherwise the current theory iteration aborts without marking the test as failure. Parameters
Actual – [in] Value to test
Reference – [in] Reference value

cr_assume_geq(Actual, Reference)¶
Assumes
Actual
is greater or equal toReference
Continues execution if
Actual
is greater or equal toReference
. Otherwise the current theory iteration aborts without marking the test as failure. Parameters
Actual – [in] Value to test
Reference – [in] Reference value

cr_assume_lt(Actual, Reference)¶
Assumes
Actual
is less thanReference
Continues execution if
Actual
is less thanReference
. Otherwise the current theory iteration aborts without marking the test as failure. Parameters
Actual – [in] Value to test
Reference – [in] Reference value

cr_assume_leq(Actual, Reference)¶
Assumes
Actual
is less or equal toReference
Continues execution if
Actual
is less or equal toReference
. Otherwise the current theory iteration aborts without marking the test as failure. Parameters
Actual – [in] Value to test
Reference – [in] Reference value

cr_assume_null(Value)¶
Assumes
Value
is NULL.Continues execution if
Value
is NULL. Otherwise the current theory iteration aborts without marking the test as failure. Parameters
Value – [in] Value to test

cr_assume_not_null(Value)¶
Assumes
Value
is not NULL.Continues execution if
Value
is not NULL. Otherwise the current theory iteration aborts without marking the test as failure. Parameters
Value – [in] Value to test

cr_assume_float_eq(Actual, Expected, Epsilon)¶
Assumes
Actual
is equal toExpected
with a tolerance ofEpsilon
Continues execution if
Actual
is equal toExpected
with a tolerance of Epsilon. Otherwise the current theory iteration aborts without marking the test as failure.Note
Use this to test equality between floats
 Parameters
Actual – [in] Value to test
Expected – [in] Expected value
Epsilon – [in] Tolerance between Actual and Expected

cr_assume_float_neq(Actual, Expected, Epsilon)¶
Assumes
Actual
is not equal toExpected
with a tolerance ofEpsilon
Continues execution if
Actual
is not equal toExpected
with a tolerance of Epsilon. Otherwise the current theory iteration aborts without marking the test as failure.Note
Use this to test equality between floats
 Parameters
Actual – [in] Value to test
Expected – [in] Expected value
Epsilon – [in] Tolerance between Actual and Expected

cr_assume_str_eq(Actual, Expected)¶
Assumes
Actual
is lexicographically equal toExpected
Continues execution if
Actual
is lexicographically equal toExpected
. Otherwise the current theory iteration aborts without marking the test as failure. Parameters
Actual – [in] String to test
Expected – [in] Expected string

cr_assume_str_neq(Actual, Unexpected)¶
Assumes
Actual
is not lexicographically equal toUnexpected
Continues execution if
Actual
is not lexicographically equal toUnexpected
. Otherwise the current theory iteration aborts without marking the test as failure. Parameters
Actual – [in] String to test
Unexpected – [in] Unexpected string

cr_assume_str_lt(Actual, Reference)¶
Assumes
Actual
is lexicographically less thanReference
Continues execution if
Actual
is lexicographically less thanReference
. Otherwise the current theory iteration aborts without marking the test as failure. Parameters
Actual – [in] Value to test
Reference – [in] Reference value

cr_assume_str_leq(Actual, Reference)¶
Assumes
Actual
is lexicographically less or equal toReference
Continues execution if
Actual
is lexicographically less or equal toReference
. Otherwise the current theory iteration aborts without marking the test as failure. Parameters
Actual – [in] Value to test
Reference – [in] Reference value

cr_assume_str_gt(Actual, Reference)¶
Assumes
Actual
is lexicographically greater thanReference
Continues execution if
Actual
is lexicographically greater thanReference
. Otherwise the current theory iteration aborts without marking the test as failure. Parameters
Actual – [in] Value to test
Reference – [in] Reference value

cr_assume_str_geq(Actual, Reference)¶
Assumes
Actual
is lexicographically greater or equal toReference
Continues execution if
Actual
is lexicographically greater or equal toReference
. Otherwise the current theory iteration aborts without marking the test as failure. Parameters
Actual – [in] Value to test
Reference – [in] Reference value

cr_assume_arr_eq(Actual, Expected, Size)¶
Assumes
Actual
is bytetobyte equal toExpected
Continues execution if
Actual
is bytetobyte equal toExpected
. Otherwise the current theory iteration aborts without marking the test as failure.Warning
This should not be used on struct arrays
 Parameters
Actual – [in] Array to test
Expected – [in] Expected array
Size – [in] The size of both arrays

cr_assume_arr_neq(Actual, Unexpected, Size)¶
Assumes
Actual
is not bytetobyte equal toUnexpected
Continues execution if
Actual
is not bytetobyte equal toUnexpected
. Otherwise the current theory iteration aborts without marking the test as failure.Warning
This should not be used on struct arrays
 Parameters
Actual – [in] Array to test
Unexpected – [in] Unexpected array
Size – [in] The size of both arrays

cr_assume(Condition)¶
Configuring theories¶
Theories can optionally recieve configuration parameters to alter the behaviour
of the underlying test; as such, those parameters are the same ones as the ones
of the Test
macro function (c.f. Configuration reference).
Full sample & purpose of theories¶
We will illustrate how useful theories are with a simple example using Criterion:
The basics of theories¶
Let us imagine that we want to test if the algebraic properties of integers, and specifically concerning multiplication, are respected by the C language:
int my_mul(int lhs, int rhs) {
return lhs * rhs;
}
Now, we know that multiplication over integers is commutative, so we first test that:
#include <criterion/criterion.h>
Test(algebra, multiplication_is_commutative) {
cr_assert_eq(my_mul(2, 3), my_mul(3, 2));
}
However, this test is imperfect, because there is not enough triangulation to insure that my_mul is indeed commutative. One might be tempted to add more assertions on other values, but this will never be good enough: commutativity should work for any pair of integers, not just an arbitrary set, but, to be fair, you cannot just test this behaviour for every integer pair that exists.
Theories purposely bridge these two issues by introducing the concept of “data point” and by refactoring the repeating logic into a dedicated function:
#include <criterion/theories.h>
TheoryDataPoints(algebra, multiplication_is_commutative) = {
DataPoints(int, [...]),
DataPoints(int, [...]),
};
Theory((int lhs, int rhs), algebra, multiplication_is_commutative) {
cr_assert_eq(my_mul(lhs, rhs), my_mul(rhs, lhs));
}
As you can see, we refactored the assertion into a theory taking two unspecified integers.
We first define some data points in the same order and type the parameters have,
from left to right: the first DataPoints(int, ...)
will define the set of values passed
to the int lhs
parameter, and the second will define the one passed to int rhs
.
Choosing the values of the data point is left to you, but we might as well use
“interesting” values: 0
, 1
, 1
, 2
, 2
, INT_MAX
, and INT_MIN
:
#include <limits.h>
TheoryDataPoints(algebra, multiplication_is_commutative) = {
DataPoints(int, 0, 1, 1, 2, 2, INT_MAX, INT_MIN),
DataPoints(int, 0, 1, 1, 2, 2, INT_MAX, INT_MIN),
};
Using theory invariants¶
The second thing we can test on multiplication is that it is the inverse function of division. Then, given the division operation:
int my_div(int lhs, int rhs) {
return lhs / rhs;
}
The associated theory is straightforward:
#include <criterion/theories.h>
TheoryDataPoints(algebra, multiplication_is_inverse_of_division) = {
DataPoints(int, 0, 1, 1, 2, 2, INT_MAX, INT_MIN),
DataPoints(int, 0, 1, 1, 2, 2, INT_MAX, INT_MIN),
};
Theory((int lhs, int rhs), algebra, multiplication_is_inverse_of_division) {
cr_assert_eq(lhs, my_div(my_mul(lhs, rhs), rhs));
}
However, we do have a problem because you cannot have the theory function divide
by 0. For this purpose, we can assume
than rhs
will never be 0:
Theory((int lhs, int rhs), algebra, multiplication_is_inverse_of_division) {
cr_assume(rhs != 0);
cr_assert_eq(lhs, my_div(my_mul(lhs, rhs), rhs));
}
cr_assume
will abort the current theory iteration if the condition is not
fulfiled.
Running the test at that point will raise a big problem with the current
implementation of my_mul
and my_div
:
[] theories.c:24: Assertion failed: (a) == (bad_div(bad_mul(a, b), b))
[] Theory algebra::multiplication_is_inverse_of_division failed with the following parameters: (2147483647, 2)
[] theories.c:24: Assertion failed: (a) == (bad_div(bad_mul(a, b), b))
[] Theory algebra::multiplication_is_inverse_of_division failed with the following parameters: (2147483648, 2)
[] theories.c:24: Unexpected signal caught below this line!
[FAIL] algebra::multiplication_is_inverse_of_division: CRASH!
The theory shows that my_div(my_mul(INT_MAX, 2), 2)
and my_div(my_mul(INT_MIN, 2), 2)
does not respect the properties for multiplication: it happens that the
behaviour of these two functions is undefined because the operation overflows.
Similarly, the test crashes at the end; debugging shows that the source of the crash is the divison of INT_MAX by 1, which is undefined.
Fixing this is as easy as changing the prototypes of my_mul
and my_div
to operate on long long
rather than int
.
What’s the difference between theories and parameterized tests ?¶
While it may at first seem that theories and parameterized tests are the same, just because they happen to take multiple parameters does not mean that they logically behave in the same manner.
Parameterized tests are useful to test a specific logic against a fixed, finite set of examples that you need to work.
Theories are, well, just that: theories. They represent a test against an universal truth, regardless of the input data matching its predicates.
Implementationwise, Criterion also marks the separation by the way that both are executed:
Each parameterized test iteration is run in its own test; this means that one parameterized test acts as a collection of many tests, and gets reported as such.
On the other hand, a theory act as one single test, since the size and contents of the generated data set is not relevant. It does not make sense to say that an universal truth is “partially true”, so if one of the iteration fails, then the whole test fails.