tfel-check is a tool designed to automate unit tests for
scientific applications and provide utilities to compare their outputs
against reference results. It is particularly useful for applications
that lack built-in validation capabilities.
The simplest way to use tfel-check is to invoke it
without arguments:
$ tfel-checkThis executes all .check files in the current directory
and its subdirectories. The syntax of .check files is
described in Section 2.
To run specific test files, list them explicitly:
$ tfel-check test1.check test2.checkA configuration file can be specified using the --config
option:
$ tfel-check --config=test.configConfiguration files are described in Section 4.
Substitution strings can be defined in configuration files or via the command line:
$ tfel-check --@python@=python3.5This replaces all occurrences of @python@ with
python3.5.
Commands in different .check files can be executed in
parallel processes. Commands within a single .check file
are always run sequentially.
By default, tfel-check uses a single process.
The --jobs (-j) option specifies the number
of parallel jobs. If no value is given, tfel-check uses the
number of available CPU cores.
If no option is given to --jobs, the number of available
cores is used, if the system provide this information.
By default, this number must be lower than the number of cores
available. This limit can be discarded by passing
--discard-jobs-limit=true to tfel-check.
Ensure that parallel execution is safe for your test suite:
For example, running two .check files that invoke
mfront to build shared libraries in the same directory may
cause race conditions, as both processes could write to
src/ or include/ simultaneously.
Similarly, improper use of @CleanDirectories or
@CleanFiles may delete data required by other
.check files.
By default, terminal output is not synchronized, meaning that individual steps (command execution, comparisons, etc.) are displayed immediately.
To synchronize output, use
--synchronize-terminal-output=true. This buffers each job’s
output and displays it only after the job completes.
Consider the following simple .check file:
@Requires {"tfel::madnex","mfront::python"};
@Command "@python@ OverridableImplementation.py";
@Command "mfront-query --author --material=A316LN --behaviour=Plasticity_SRMA2022 Plasticity.madnex"{
expected_output : "John Mac Enroe"
};The @Requires keyword lists required components
(tfel::madnex and mfront::python). If these
are unavailable, tests are skipped. Components can be defined in
configuration files (see Section 4). Default components are listed in
Section 5.
The @Command keyword executes a command. The
@python@ syntax is a substitution string that must be
defined (e.g., via --@python@=python3.5).
The second @Command includes a test that validates
the command’s output against the expected string
"John Mac Enroe".
To run this file:
$ tfel-check --@python@=python3.5 madnex.check
entering directory '/tmp/madnex'
* beginning of test './madnex.check'
** Exec-1 python3.5 OverridableImplementation.py [SUCCESS]
** Exec-2 mfront-query --author --material=A316LN --behaviour=Plast... [SUCCESS]
* end of test './madnex.check' [SUCCESS]
======This generates the following files:
tfel-check.log: Global log file in the invocation
directory.madnex.checklog: Summary of command executions and
tests for madnex.check.madnex-Exec-1.out, madnex-Exec-2.out:
Output and execution time for each command.TEST-madnex.xml: JUnit-compatible XML file for
integration with Jenkins.If requirements are unmet, tests are skipped:
$ tfel-check
entering directory '/tmp/madnex'
* beginning of test './madnex.check'
** Exec-1 @python@ OverridableImplementation.py [SKIPPED]
** Exec-2 mfront-query --author --material=A316LN --behaviour=Plast... [SKIPPED]
* end of test './madnex.check' [SUCCESS]
======Command failures are reported as follows:
$ tfel-check --@python@=python2.7
entering directory '/tmp/madnex'
* beginning of test './madnex.check'
** Exec-1 toto OverridableImplementation.py [ FAILED]
** Exec-2 mfront-query --author --material=A316LN --behaviour=Plast... [ FAILED]
* end of test './madnex.check' [ FAILED]
======Consider a tensile test simulated with the Cast3M finite
element solver. An MFront file defines the material
behaviour, and the simulation generates a result file
(ImplicitSimoMieheElastoPlasticityUniaxialTesting-castem.res)
containing the stress evolution along the tensile axis over time.
The ImplicitSimoMieheElastoPlasticity.check file:
@Requires {"Cast3M", "mfront::behaviour::castem_interface"};
@Command "mfront --obuild --interface=castem ImplicitSimoMieheElastoPlasticity.mfront";
@Command "@castem@ ImplicitSimoMieheElastoPlasticityUniaxialTesting.dgibi";
@Precision 100; // absolute tolerance: 100 Pa
@Test "ImplicitSimoMieheElastoPlasticityUniaxialTesting-castem.ref"
"ImplicitSimoMieheElastoPlasticityUniaxialTesting-castem.res" 2;@Requires keyword ensures Cast3M and
its MFront interface are available.@Command compiles the behaviour.@Command runs the Cast3M
simulation, using the @castem@ substitution.@Precision 100 sets an absolute tolerance of 100 Pa for
comparisons.@Test keyword compares the second column of the
reference and result files.A configuration file (castem.config) defines the
substitution:
components : {"Cast3M"};
substitutions : {"castem": "castem2019_PLEIADES"};Run the test with:
$ tfel-check --config=castem.config
entering directory '/tmp/tfel-check'
* beginning of test './ImplicitSimoMieheElastoPlasticity.check'
** Exec-1 mfront --obuild --interface=castem ImplicitSimoMieheElast... [SUCCESS]
** Exec-2 castem2019_PLEIADES ImplicitSimoMieheElastoPlasticityUnia... [SUCCESS]
** Compare-1 'ImplicitSimoMieheElastoPlasticityUniaxialTesting-cast... [SUCCESS]
* end of test './ImplicitSimoMieheElastoPlasticity.check' [SUCCESS]By default, tfel-check ignores command failures if at
least one test is defined. This accounts for cases where a command
produces correct results but exits with a non-zero status.
Use --discard-commands-failure to control this
behaviour:
$ tfel-check --discard-commands-failure=false
entering directory '/tmp/tests'
* beginning of test './test.check'
** Exec-1 ./main [ FAILED]
** Compare-1 'results.res' and 'results.res', column '1' [SUCCESS]
* end of test './test.check' [ FAILED]
======
$ tfel-check --discard-commands-failure=true
entering directory '/tmp/tests'
* beginning of test './test.check'
** Exec-1 ./main [ FAILED]
** Compare-1 'results.res' and 'results.res', column '1' [SUCCESS]
* end of test './test.check' [SUCCESS]
====== .check files describe tests and comparisons using a set
of keywords.
@Requires
keywordThe @Requires keyword lists the components required to
run the tests in the file. If any component is unavailable, all tests in
the file are skipped.
@Environment
keywordThe @Environment keyword defines environment variables
for commands:
@Environment {"VAR1": "value1", "VAR2": "value2"};If a variable is already defined, in the environment of
tfel-check or earlier in the test file, the value is
prepended to it, separated by :, as for search paths
(PATH, LD_LIBRARY_PATH, etc.).
@Command
keywordThe @Command keyword executes a command. It supports the
following options:
expected_output: Expected output as a string (single
line) or an array of strings (multiline). The test fails if the actual
output differs.expected_numerical_output: Tests numerical output.
Requires a map with:
value: Expected numerical value.criterion_value: Tolerance for the comparison.output_validation_regex: Regular expression to validate
the command output. The output is concatenated into a single string for
validation.shall_fail: Boolean. If true, the command
must fail for the test to pass. If an output check
(expected_output, expected_numerical_output,
empty_output, output_validation_regex) is also
given, it is applied to the output of the failed command, so that the
test also checks the reason of the failure.timeout: Integer. Maximum allowed duration in
seconds.Example:
@Command "echo 42" {
expected_output: "42"
};
@Command "./my_script" {
shall_fail: true
};
@Command "sleep 10"{
timeout: 1
};@Precision
keywordThe @Precision keyword sets the tolerance(s) for value
comparisons.
Absolute, Relative, or
Area comparisons: a single value.RelativeAndAbsolute or Mixed
comparisons: two values (relative and absolute).Example:
@Precision 1.e-6; // single precision for Absolute/Relative
@Precision 1.e-3 1.; // relative and absolute precision for Mixed@Precision with one value keeps the second value of a
previous @Precision. A warning is written in the log file
of the test (.checklog) when a
RelativeAndAbsolute or Mixed comparison uses
this second value.
@Test
keywordThe @Test keyword compares two files (reference and
current results).
Syntax:
@Test <reference_file> <results_file> <columns>;reference_file: Path to the file containing reference
values.results_file: Path to the file containing current
results.columns: List of column numbers or names to compare. At
least one column is required. Each column defines a separate test.The order of the files matters:
@Interpolation keyword),
the values of the second file are interpolated at the abscissas of the
first one,AllowLessResults option allows the second file to
have fewer values than the first one,Area comparison is normalized by the values of the
first file,Mixed comparison is
scaled by the values of the second file.Columns can be designated by their names only if the first line of
the file gives the names of the columns. This line must not start with
#: a line starting with # is a comment. A
column can also be given by a formula of column numbers, such as
'$2*1.e-6'.
Examples:
@Test 'reference.res' 'results.res' 2; // compare column 2
@Test 'reference.res' 'results.res' 'B'; // compare column named 'B'
@Test 'reference.res' 'results.res' 1 3 5; // compare columns 1, 3, and 5
@Test 'reference.res' 'results.res' 'A' 'B'; // compare columns named 'A' and 'B'Different columns can be compared with the following syntax:
// compare column 2 of 'reference.res' with column 3 of 'results.res'
@Test {'reference.res' : 2} {'results.res' : 3};@TestType
keywordThe @TestType keyword defines the comparison method for
values. Available types:
Absolute: Absolute difference between values must be ≤
precision.Relative: Relative difference between values must be ≤
precision.RelativeAndAbsolute: Combines relative and absolute
criteria. Requires two precision values.Mixed: Mixed relative/absolute comparison. Requires two
precision values.Area: Integral of the absolute difference between the
two curves, normalized by the maximum absolute value of the reference.
Requires a single precision value. See Section 3.1.5.L2: Same as Area, with the L2 norm of the
difference instead of the integral of its absolute value. See Section
3.1.6.For all types, a NaN value makes the test fail.
Example:
@TestType Absolute;
@Precision 1.e-6;
@Test 'reference.res' 'results.res' 1;
@TestType RelativeAndAbsolute;
@Precision 1.e-3 1.e-6; // relative and absolute tolerances
@Test 'reference.res' 'results.res' 2;@Interpolation keywordThe @Interpolation keyword specifies how the values of
the results are interpolated at the abscissas of the reference. This is
useful when the reference and result files have different abscissa
values (e.g., time steps).
Available interpolation methods:
None: No interpolation. The values are compared line by
line: both files must have the same number of values. If a column is
given with using, the abscissas of both files must be the
same. Otherwise, they are not checked.Linear: Linear interpolation between reference
points.Spline: Cubic spline interpolation.LocalSpline: Local spline interpolation (avoids global
smoothness constraints).The using clause specifies the column to use as the
interpolation abscissa.
If the results have at least as many distinct abscissas as the
reference, they are interpolated at the abscissas of the reference.
Otherwise, the AllowLessResults option is required, and the
values of the reference are interpolated at the abscissas of the
results. In both cases, the results must reach the bounds of the
abscissas of the reference: results stopping early make the test
fail.
No extrapolation is made: the abscissas of the interpolated file must cover the abscissas at which it is evaluated, otherwise the test fails. Both files must have values. The abscissas are sorted and, if an abscissa appears more than once in a file, only its first point is used.
Examples:
@Interpolation Linear using 'time';
@Interpolation Spline using 'tps' AllowLessResults;
@Interpolation LocalSpline using 'x';For Area and L2 comparisons, the
interpolation and the abscissa column used for the integration are given
by the @TestType keyword:
@TestType <Area|L2> interpolation <None|Linear|Spline|LocalSpline> using <column>;For example:
@TestType Area interpolation Linear using 'time';
@Precision 1.e-2;
@Test 'reference.res' 'results.res' 'y';This interpolation is independent of the one given by the
@Interpolation keyword. See Section 3.1.5.
tfel-check provides powerful tools to compare numerical
results against reference data. This section details the available
comparison types, interpolation methods, and precision settings.
The @TestType keyword defines how values are compared.
Each type has specific use cases:
Checks if the absolute difference between corresponding values is within the specified precision.
@TestType Absolute;
@Precision 1.e-6;
@Test 'reference.res' 'results.res' 1;Use case: Suitable for quantities with a known absolute tolerance (e.g., stress in Pa).
Checks if the relative difference between corresponding values is within the specified precision.
@TestType Relative;
@Precision 1.e-3; // 0.1% tolerance
@Test 'reference.res' 'results.res' 1;Use case: Ideal for quantities where relative accuracy matters (e.g., normalized values).
Combines both criteria: the test passes if either the relative or the absolute difference is within tolerance.
@TestType RelativeAndAbsolute;
@Precision 1.e-3 1.e-6; // relative and absolute tolerances
@Test 'reference.res' 'results.res' 1;Use case: Flexible comparison for quantities that may have small absolute values (where relative comparison fails) or large values (where absolute comparison is too strict).
Combines both tolerances: the test passes if \(\left|a-b\right| \leq p_{r}\,\left|b\right| + p_{a}\), where \(a\) and \(b\) are the values of the first and second files, and \(p_{r}\) and \(p_{a}\) the relative and absolute precisions.
@TestType Mixed;
@Precision 1.e-3 1.e-6; // relative and absolute tolerances
@Test 'reference.res' 'results.res' 1;Use case: Tolerance with a relative part and an absolute part, the latter being useful for values close to zero.
Computes the area between the two curves, i.e. the integral of the absolute value of their difference, normalized by the maximum absolute value of the reference:
\[ e = \frac{1}{\max_{i}\left|y^{\mathrm{ref}}_{i}\right|}\int\left|y^{\mathrm{ref}}\left(x\right)-y^{\mathrm{res}}\left(x\right)\right|\,\mathrm{d}x \]
where \(x\) is the abscissa column
given after the using keyword. The test succeeds if \(e\) is lower than the precision. This error
is a normalized \(L^{1}\) norm of the
difference.
The integral is computed by the trapezoidal rule on the union of the abscissas of both files. Each curve is evaluated:
Abscissas are sorted. If an abscissa appears more than once in a
file, only its first point is used. At least two distinct abscissas are
required. With the None interpolation, both files must have
the same abscissas.
If all the values of the reference are null, the area is not normalized.
The error \(e\) has the unit of the abscissa: it is the product of the length of the range of the abscissas and the mean deviation relative to the maximum of the reference. For example, for a test lasting 100 s, a precision of 1 corresponds to a mean deviation of 1 % of the maximum of the reference.
@TestType Area interpolation Linear using 'time';
@Precision 1; // time range of 100 s: mean deviation of 1 %
@Test 'reference.res' 'results.res' 'stress';Use case: Global comparison of two curves, possibly sampled differently, when local differences (for instance a slight time shift of a sharp transition) are acceptable.
Same as the Area comparison, with the L2 norm of the
difference between the two curves:
\[ e = \frac{1}{\max_{i}\left|y^{\mathrm{ref}}_{i}\right|}\sqrt{\int\left(y^{\mathrm{ref}}\left(x\right)-y^{\mathrm{res}}\left(x\right)\right)^{2}\,\mathrm{d}x} \]
The integral is computed by the trapezoidal rule on the same points
as for the Area comparison, with the same rules. The error
\(e\) has the unit of the square root
of the abscissa.
@TestType L2 interpolation Linear using 'time';
@Precision 1.e-2;
@Test 'reference.res' 'results.res' 'stress';Use case: Same as the Area comparison,
with more weight given to large local differences.
Interpolation is required when the abscissa values (e.g., time,
strain) in the result and reference files do not match. The
@Interpolation keyword specifies how the values of one file
are interpolated at the abscissas of the other one.
No interpolation is performed. The values are compared line by line,
so both files must have the same number of values. If a column is given
with using, the abscissas of both files must be the same,
up to a relative tolerance of \(10^{-10}\). Otherwise, the abscissas are
not checked.
@Interpolation None using 'time'; // the times must be the same
@Test 'reference.res' 'results.res' 'stress';Use case: Fastest method when both files use identical abscissa values.
Linear interpolation between reference points.
@Interpolation Linear using 'time';
@Test 'reference.res' 'results.res' 'stress';Use case: Simple and robust for most smooth data.
Cubic spline interpolation for smooth curves.
@Interpolation Spline using 'strain';
@Test 'reference.res' 'results.res' 'stress';Use case: High-quality interpolation for smooth data with continuous second derivatives.
Local spline interpolation avoids global smoothness constraints, making it more robust for noisy or non-smooth data.
@Interpolation LocalSpline using 'time' AllowLessResults;
@Test 'reference.res' 'results.res' 'stress';Use case: Preferred for data with sharp transitions or when the result file has fewer points than the reference.
The AllowLessResults option allows the result file to
have fewer data points than the reference file. This is useful when the
simulation uses adaptive time-stepping or when reference data is highly
resolved. The results must still reach the bounds of the abscissas of
the reference.
@Interpolation Linear using 'time' AllowLessResults;
@Test 'reference.res' 'results.res' 1;Multiple columns can be compared in a single @Test
directive by listing their indices or names.
@TestType Absolute;
@Precision 1.e-6;
@Test 'reference.res' 'results.res' 1 2 3; // compare columns 1, 2, and 3
@Test 'reference.res' 'results.res' 'A' 'B'; // compare columns named 'A' and 'B'Note: All specified columns are compared using the
same @TestType and @Precision.
Compare the second column of two files with an absolute tolerance of 1e-6:
@TestType Absolute;
@Precision 1.e-6;
@Test 'reference.res' 'results.res' 2;Compare stress-strain curves where the strain values (column 1) differ between files:
@TestType Relative;
@Precision 1.e-3;
@Interpolation Spline using 1;
@Test 'reference.res' 'results.res' 2;Strict comparison of temperature evolution with local spline interpolation:
@TestType Mixed;
@Precision 1.e-4 1.e-6; // relative and absolute tolerances
@Interpolation LocalSpline using 'time' AllowLessResults;
@Test 'reference.res' 'results.res' 'temperature';Compare two force-displacement curves sampled differently:
@TestType Area interpolation Linear using 'displacement';
// displacement in mm, range of 2 mm: mean deviation of 0.5 % of the
// maximum force
@Precision 1.e-2;
@Test 'reference.res' 'results.res' 'force';Compare multiple columns with different precisions (requires separate
@Test directives):
@TestType Absolute;
@Precision 1.e-3;
@Test 'reference.res' 'results.res' 'stress';
@TestType Relative;
@Precision 1.e-4;
@Test 'reference.res' 'results.res' 'strain';Configuration files define:
@python@ →
python3.5).Cast3M,
tfel::madnex).The syntax is JSON-inspired:
components: {"Cast3M", "tfel::madnex"};
substitutions: {"python": "python3.5", "castem": "castem2019"};
environment_variables: {"CXXFLAGS": "-Wall", "OMP_NUM_THREADS": "4"};The environment_variables section defines a map of
environment variables:
environment_variables: {"CXXFLAGS": "-Wall", "MPI_ROOT": "/path/to/mpi"};@Environment), the test file’s
definition takes precedence.@Environment keyword, the values given in
configuration files are used as given: they are not prepended to the
value of the variable in the environment of
tfel-check.tfel-check automatically defines components based on the
enabled features:
TFEL’s Python bindings are enabled:
tfel::python and mfront::python.madnex support is enabled:
tfel::madnex.MFront material property interface:
mfront::material_property::<name>_interface.MFront behaviour interface:
mfront::behaviour::<name>_interface.MFront model interface:
mfront::model::<name>_interface.To list all default components, use:
$ tfel-check --list-default-components
- tfel::python
- mfront::python
- tfel::madnex
- mfront::material_property::C++_interface
- mfront::material_property::c_interface
- mfront::behaviour::generic_interface
- mfront::model::mfront_interface
...tfel-check automatically declares substitutions for
TFEL executables:
@mfront@@mfront-query@@mtest@@mfront-doc@@mfm-test-generator@If Python bindings are enabled, @python@ is also
declared.
These default substitutions are applied after reading configuration files and command-line arguments, so they can be overridden by the user.
For a full list of command-line options, run:
$ tfel-check --helpThe options whose value is noted <boolean> require
the value true or false after an equal sign,
for example --discard-jobs-limit=true: they are rejected
without value.
| Option | Description |
|---|---|
--clean-up=<boolean> |
clean generated files and directories listed by the
@CleanFiles and @CleanDirectories keywords in
case of success (default: true). |
--discard-clean-up-on-failure=<boolean> |
don’t clean generated files and directories listed by the
@CleanFiles and @CleanDirectories keywords in
case of failure (default: true). |
--config, -c |
Add a configuration file. |
--discard-commands-failure=<boolean> |
Ignore command failures if comparisons pass (default:
true). If no comparisons are defined, failures are never
ignored. |
--discard-jobs-limit=<boolean> |
Disable the limit on parallel jobs (default:
false). |
--help, -h |
Display help message. |
--jobs, -j |
Number of parallel jobs (default: 1; number of CPU cores if no value
is given). Without value, -j must be the last argument or
be followed by another option: tfel-check -j a.check reads
a.check as the number of jobs, whereas
tfel-check --jobs a.check and
tfel-check a.check -j use the number of CPU cores. |
--list-default-components |
List all default components. |
--synchronize-terminal-output=<boolean> |
Synchronize terminal output in parallel (default:
false). |
--use-terminal-colors=<boolean> |
Use terminal colors for output (default: true). |
--version, -v |
Display version information. |
Run all .check files in the current directory:
$ tfel-checkRun specific files with 4 parallel jobs:
$ tfel-check -j 4 test1.check test2.checkList default components:
$ tfel-check --list-default-componentsOverride a substitution and enable terminal colors:
$ tfel-check --@python@=python3.10 --use-terminal-colors=true