Go to the first, previous, next, last section, table of contents.


Using greg

The Greg framework is designed to be used in two ways - as an embedded system from within any application which is linked with the Guile library, or stand-alone using the command-line greg driver script. For both of these methods of usage the test cases are written the same way and the expected output is the same.

Output

While Greg may produce more verbose output in response to various settings, the basic output of a test run is a series of lines describing the success/failure state of each testcase encountered, followed by a summary of all test cases.

In `normal' mode, only unexpected results are displayed, but in `verbose' output mode, results for all results are displayed.

greg flags the outcome of each test case with a line consisting of one of the following codes followed by a colon, a space, and then the testcase description.

PASS
The most desirable outcome: the test succeeded, and was expected to succeed.
UPASS
A pleasant kind of failure: a test was expected to fail, but succeeded. This may indicate progress; inspect the test case to determine whether you should amend it to stop expecting failure.
FAIL
A test failed, although it was expected to succeed. This may indicate regress; inspect the test case and the failing software to locate the bug.
XFAIL
A test failed, but it was expected to fail. This result indicates no change in a known bug. If a test fails because the operating system where the test runs lacks some facility required by the test, the outcome is UNSUPPORTED instead.
UNRESOLVED
Output from a test requires manual inspection; the test suite could not automatically determine the outcome. For example, your tests can report this outcome is when a test does not complete as expected.
UNTESTED
A test case is not yet complete, and in particular cannot yet produce a PASS or FAIL. You can also use this outcome in dummy "tests" that note explicitly the absence of a real test case for a particular property.
UNSUPPORTED
A test depends on a conditionally available feature that does not exist (in the configured testing environment). For example, you can use this outcome to report on a test case that does not work on a particular target because its operating system support does not include a required subroutine.

Files and directories

A Greg test run expects to find files and directories in a certain layout (modeled on that used by DejaGNU).

The test source directory (normally your current directory) is expected to contain one or more tool directories. Each tool directory should contain one or more test scripts. In fact any file in a tool directory which has a .scm extension is assumed to be a Guile test script.

When a normal Greg test run is done, Greg goes through each tool directory in turn and loads each test script in turn.

You may set the Guile variable greg-tools or use the --tools ... command-line option to specify a list of tools directories to use rather than assuming that all subdirectories are tool directories. If you do this, the tools are tested in the order in which they appear in the list rather than the default order (ASCII sorted by name).

You may set the Guile variable greg-files or use the --files ... command-line option to specify a list of file names to use rather than assuming that all .scm files in each tool directory are test scripts. If you do this, the files are loaded in the order in which they appear in the list.

As a (minor) complication to this simple layout, Greg permits the use of begin.grg and end.grg scripts in both the main source directory and in each tool directory. These scripts permit you to add any initialisation and cleanup code you want. Typically (for non-embedded testing) you would use a begin.grg script to start the application to be tested.

If begin.grg exists in the main source directory, it will be loaded before any tools are tested.

If end.grg exists in the main source directory, it will be loaded after all the tools are tested.

If begin.grg exists in a tool directory, it will be loaded before any test scripts in that directory are loaded.

If end.grg exists in a tool directory, it will be loaded after all the test scripts in that directory are loaded.

Embedded usage

Greg is designed primarily for embedded usage - any application that uses Guile as it's scripting language should be able to use Greg to test itself.

To this end - Greg provides a Guile module containing definitions of various procedures (used to run tests) and variables (used to modify the behavior of a test run).

Before trying to use any part of Greg, You need to load the Greg module with (use-modules (ice-9 greg))

The main procedure to run Greg tests is (greg-test-run). You can use this to run tests in much the same way as the greg script is used to run tests from the command-line. The behavior of this procedure is modified by setting the following top-level variables -

Command-line usage

greg is the executable test driver for Greg. This is a Guile script that you can use to run tests from the command line. The command-line options that you can pass to greg are listed below.

greg returns an exit code of 1 if any test has an unexpected result; otherwise (if all tests pass or fail as expected) it returns 0 as the exit code.

This is the full set of command line options that greg recognizes.

greg --tool tool ...
[ --debug ]
[ --file script ... ]
[ --help ]
[ --objdir path ]
[ --outdir path ]
[ --posix ]
[ --srcdir path ]
[ -v | --verbose ]  [ -V | --version ]
--tool tool ...
tool specifies what set of tests to run. It provides a list of subdirectories (each corresponding to a tool) in which test scripts can be found. Initialisation code (including executable tool startup) for each tool may be placed in `begin.grg' in the appropriate tool subdirectory. Cleanup code may be placed in `end.grg'. For example, including --tool gcc on the greg command line runs tests from the gcc subdirectory. The order in which the tools are tested will be the same as the order in which the tool names occur on the command line.
--file script ...
Specify the names of testsuites to run. By default, greg runs all tests for the tool, but you can restrict it to particular testsuites by giving the names of the .scm guile scripts that control them. You do not need to supply the .scm file extension - it will be assumed. testsuite may not include path information; use plain filenames. The order in which the test scripts are run will be the same as the order in which the script names occur on the command line.
--debug
Turns on the expect internal debugging output. Debugging output is displayed as part of the greg output, and logged to a file called `tests.dbg'. The extra debugging output does not normally appear on standard output.
--help
Prints out a short summary of the greg options, then exits (even if you also specify other options).
--objdir path
Use path as the top directory containing any auxiliary compiled test code. This defaults to `.'. Use this option to locate pre-compiled test code. You can normally prepare any auxiliary files needed with make.
--outdir path
Write output logs in directory path. The default is ., the directory where you start greg. This option affects only the log and the debug files `tool.log' and `tool.dbg'.
--srcdir path
Use path as the top directory for test scripts to run. greg looks in this directory for any subdirectory whose name matches any toolname (specified with --tool).
--verbose
-v
By default, greg shows only the output of tests that produce unexpected results; that is, tests with status FAIL (unexpected failure), UPASS (unexpected success), or ERROR (a severe error in the test case itself). Specifying --verbose to see output for tests with status PASS (success, as expected) and XFAIL (failure, as expected). It also causes a more detailed summary to be displayed.
Specifying --verbose more than once causes more detail to be displayed.
--version
-V
Prints out the version numbers of Greg, and Guile, then exits without running any tests.


Go to the first, previous, next, last section, table of contents.