Criterion¶
Introduction¶
Criterion is a dead-simple, non-intrusive testing framework for the C programming language.
Philosophy¶
Most test frameworks for C require a lot of boilerplate code to set up tests and test suites – you need to create a main, then register new test suites, then register the tests within these suits, and finally call the right functions.
This gives the user great control, at the unfortunate cost of simplicity.
Criterion follows the KISS principle, while keeping the control the user would have with other frameworks.
Features¶
- Tests are automatically registered when declared.
- A default entry point is provided, no need to declare a main unless you want to do special handling.
- Test are isolated in their own process, crashes and signals can be reported and tested.
- Progress and statistics can be followed in real time with report hooks.
- TAP output format can be enabled with an option.
- Runs on Linux, FreeBSD, Mac OS X, and Windows (compiles only with Cygwin for the moment).
- xUnit framework structure
Setup¶
Prerequisites¶
Currently, this library only works under *nix systems.
To compile the static library and its dependencies, GCC 4.9+ is needed.
To use the static library, any GNU-C compatible compiler will suffice (GCC, Clang/LLVM, ICC, MinGW-GCC, ...).
Installation¶
$ git clone https://github.com/Snaipe/Criterion.git $ cd Criterion $ ./autogen.sh && ./configure && make && sudo make installUsage¶
Given a test file named test.c, compile it with -lcriterion:
$ gcc -o test test.c -lcriterionGetting started¶
Adding tests¶
Adding tests is done using the
Test
macro:#include <criterion/criterion.h> Test(suite_name, test_name) { // test contents }
suite_name
andtest_name
are the identifiers of the test suite and the test, respectively. These identifiers must follow the language identifier format.Tests are automatically sorted by suite, then by name using the alphabetical order.
Asserting things¶
Assertions come in two kinds:
cr_assert*
are assertions that are fatal to the current test if failed; in other words, if the condition evaluates tofalse
, the test is marked as a failure and the execution of the function is aborted.cr_expect*
are, in the other hand, assertions that are not fatal to the test. Execution will continue even if the condition evaluates tofalse
, but the test will be marked as a failure.
cr_assert()
andcr_expect()
are the most simple kinds of assertions criterion has to offer. They both take a mandatory condition as a first parameter, and an optional failure message:#include <string.h> #include <criterion/criterion.h> Test(sample, test) { cr_expect(strlen("Test") == 4, "Expected \"Test\" to have a length of 4."); cr_expect(strlen("Hello") == 4, "This will always fail, why did I add this?"); cr_assert(strlen("") == 0); }On top of those, more assertions are available for common operations:
cr_assert_null(Ptr, [Message])
: passes if Ptr is NULL.cr_assert_eq(Actual, Expected, [Message])
: passes if Actual == Expected.cr_assert_lt(Actual, Expected, [Message])
: passes if Actual < Expected.cr_assert_leq(Actual, Expected, [Message])
: passes if Actual <= Expected.cr_assert_gt(Actual, Expected, [Message])
: passes if Actual > Expected.cr_assert_geq(Actual, Expected, [Message])
: passes if Actual >= Expected.cr_assert_float_eq(Actual, Expected, Epsilon, [Message])
: passes if Actual == Expected with an error of Epsilon.cr_assert_arrays_eq(Actual, Expected, Size, [Message])
: passes if all elements of Actual (from 0 to Size - 1) are equals to those of Expected.cr_assert_arrays_eq_cmp(Actual, Expected, Size, Cmp, [Message])
: Same asarrays_eq
but equality is defined by the result of the binary Cmp function.Equality and lexical comparison assertions are also available for strings:
cr_assert_strings_eq(Actual, Expected, [Message])
cr_assert_strings_lt(Actual, Expected, [Message])
cr_assert_strings_leq(Actual, Expected, [Message])
cr_assert_strings_gt(Actual, Expected, [Message])
cr_assert_strings_geq(Actual, Expected, [Message])
And some assertions have a logical negative counterpart:
cr_assert_not(Condition, [Message])
cr_assert_not_null(Ptr, [Message])
cr_assert_neq(Actual, Unexpected, [Message])
cr_assert_float_neq(Actual, Unexpected, Epsilon, [Message])
cr_assert_strings_neq(Actual, Unexpected, [Message])
cr_assert_arrays_neq(Actual, Unexpected, Size, [Message])
cr_assert_arrays_neq_cmp(Actual, Unexpected, Size, Cmp, [Message])
Of course, every
assert
has anexpect
counterpart.Please note that
arrays_(n)eq
assertions should not be used on padded structures – please usearrays_(n)eq_cmp
instead.Configuring tests¶
Tests may receive optional configuration parameters to alter their behaviour or provide additional metadata.
Fixtures¶
Tests that need some setup and teardown can register functions that will run before and after the test function:
#include <stdio.h> #include <criterion/criterion.h> void setup(void) { puts("Runs before the test"); } void teardown(void) { puts("Runs after the test"); } Test(suite_name, test_name, .init = setup, .fini = teardown) { // test contents }If a setup crashes, you will get a warning message, and the test will be aborted and marked as a failure. Is a teardown crashes, you will get a warning message, and the test will keep its result.
Testing signals¶
If a test receives a signal, it will by default be marked as a failure. You can, however, expect a test to only pass if a special kind of signal is received:
#include <stddef.h> #include <signal.h> #include <criterion/criterion.h> // This test will fail Test(sample, failing) { int *ptr = NULL; *ptr = 42; } // This test will pass Test(sample, passing, .signal = SIGSEGV) { int *ptr = NULL; *ptr = 42; }This feature will of course not work on Windows.
Configuration reference¶
Here is an exhaustive list of all possible configuration parameters you can pass:
Parameter Type Description .description const char * Adds a description. Cannot be NULL
..init void (*)(void) Adds a setup function the be executed before the test. .fini void (*)(void) Adds a teardown function the be executed after the test. .disabled bool Disables the test. .signal int Expect the test to raise the specified signal. Setting up suite-wise configuration¶
Tests under the same suite can have a suite-wise configuration – this is done using the
TestSuite
macro:#include <criterion/criterion.h> TestSuite(suite_name, [params...]); Test(suite_name, test_1) { } Test(suite_name, test_2) { }Configuration parameters are the same as above, but applied to the suite itself.
Suite fixtures are run along with test fixtures.
Report Hooks¶
Report hooks are functions that are called at key moments during the testing process. These are useful to report statistics gathered during the execution.
A report hook can be declared using the
ReportHook
macro:#include <criterion/criterion.h> #include <criterion/hooks.h> ReportHook(Phase)() { }The macro takes a Phase parameter that indicates the phase at which the function shall be run. Valid phases are described below.
Note: there are no guarantees regarding the order of execution of report hooks on the same phase. In other words, all report hooks of a specific phase could be executed in any order.
Testing Phases¶
The flow of the test process goes as follows:
PRE_ALL
: occurs before running the tests.PRE_SUITE
: occurs before a suite is initialized.PRE_INIT
: occurs before a test is initialized.PRE_TEST
: occurs after the test initialization, but before the test is run.ASSERT
: occurs when an assertion is hitTEST_CRASH
: occurs when a test crashes unexpectedly.POST_TEST
: occurs after a test ends, but before the test finalization.POST_FINI
: occurs after a test finalization.POST_SUITE
: occurs before a suite is finalized.POST_ALL
: occurs after all the tests are done.Hook Parameters¶
A report hook may take zero or one parameter. If a parameter is given, it is undefined behaviour if it is not a pointer type and not of the proper pointed type for that phase.
Valid types for each phases are:
struct criterion_test_set *
forPRE_ALL
.struct criterion_suite_set *
forPRE_SUITE
.struct criterion_test *
forPRE_INIT
andPRE_TEST
.struct criterion_assert_stats *
forASSERT
.struct criterion_test_stats *
forPOST_TEST
,POST_FINI
, andTEST_CRASH
.struct criterion_suite_stats *
forPOST_SUITE
.struct criterion_global_stats *
forPOST_ALL
.For instance, these are valid report hook declarations for the
PRE_TEST
phase:#include <criterion/criterion.h> #include <criterion/hooks.h> ReportHook(PRE_TEST)() { // not using the parameter } ReportHook(PRE_TEST)(struct criterion_test *test) { // using the parameter }Environment and CLI¶
Tests built with Criterion expose by default various command line switchs and environment variables to alter their runtime behaviour.
Command line arguments¶
-h or --help
: Show a help message with the available switches.-v or --version
: Prints the version of criterion that has been linked against.-l or --list
: Print all the tests in a list.-f or --fail-fast
: Exit after the first test failure.--ascii
: Don’t use fancy unicode symbols or colors in the output.--pattern [PATTERN]
: Run tests whose string identifier matches the given shell wildcard pattern (see dedicated section below). (*nix only)--no-early-exit
: The test workers shall not prematurely exit when done and will properly return from the main, cleaning up their process space. This is useful when tracking memory leaks withvalgrind --tool=memcheck
.--always-succeed
: The process shall exit with a status of0
.--tap
: Enables the TAP (Test Anything Protocol) output format.--verbose[=level]
: Makes the output verbose. When provided with an integer, sets the verbosity level to that integer.Shell Wildcard Pattern¶
Patterns in criterion are matched against a test’s string identifier with
fnmatch
. This feature is only available on *nix systems wherefnmatch
is provided.Special characters used in shell-style wildcard patterns are:
Pattern Meaning *
matches everything ?
matches any character [seq]
matches any character in seq [!seq]
matches any character not in seq A test string identifier is of the form
suite-name/test-name
, so a pattern ofsimple/*
matches every tests in thesimple
suite,*/passing
matches all tests namedpassing
regardless of the suite, and*
matches every possible test.Environment Variables¶
Environment variables are alternatives to command line switches when set to 1.
CRITERION_ALWAYS_SUCCEED
: Same as--always-succeed
.CRITERION_NO_EARLY_EXIT
: Same as--no-early-exit
.CRITERION_ENABLE_TAP
: Same as--tap
.CRITERION_FAIL_FAST
: Same as--fail-fast
.CRITERION_USE_ASCII
: Same as--ascii
.CRITERION_VERBOSITY_LEVEL
: Same as--verbose
. Sets the verbosity level to its value.CRITERION_TEST_PATTERN
: Same as--pattern
. Sets the test pattern to its value. (*nix only)Changing the internals¶
Providing your own main¶
If you are not satisfied with the default CLI or environment variables, you can define your own main function.
Configuring the test runner¶
You’d usually want to configure the test runner before calling it. Configuration is done by setting fields in a global variable named
criterion_options
(include criterion/options.h).Here is an exhaustive list of these fields:
Field Type Description logging_threshold enum criterion_logging_level The logging level output_provider struct criterion_output_provider * The output provider (see below) no_early_exit bool True iff the test worker should exit early always_succeed bool True iff criterion_run_all_tests should always returns 1 use_ascii bool True iff the outputs should use the ASCII charset fail_fast bool True iff the test runner should abort after the first failure pattern const char * The pattern of the tests that should be executed Starting the test runner¶
The test runner can be called with
criterion_run_all_tests
. The function returns 0 if one test or more failed, 1 otherwise.Implementing your own output provider¶
In case you are not satisfied by the default output provider, you can implement yours. To do so, simply set the
output_provider
option to your custom output provider.Each function contained in the structure is called during one of the standard phase of the criterion runner.
For more insight on how to implement this, see other existing output providers in
src/log/
.F.A.Q¶
Q. When running the test suite in Windows’ cmd.exe, the test executable prints weird characters, how do I fix that?
A. Windows’
cmd.exe
is not an unicode ANSI-compatible terminal emulator. There are plenty of ways to fix that behaviour:
- Pass
--ascii
to the test suite when executing.- Define the
CRITERION_USE_ASCII
environment variable to1
.- Get a better terminal emulator, such as the one shipped with Git or Cygwin.
Q. I’m having an issue with the library, what can I do ?
A. Open a new issue on the github issue tracker, and describe the problem you are experiencing, along with the platform you are running criterion on.