Go to the first, previous, next, last section, table of contents.


Greg in brief

Greg is a framework for testing other programs and libraries. Its purpose is to provide a single front end for all tests and to be a small, simple framework for writing tests. Greg leverages off the Guile language to provide all the power (and more) of other test frameworks with greater simplicity and ease of use.

The simplicity of the Greg framework makes it easy to write tests for any program, but it was specifically written for use with GNUstep-Guile to permit direct testing of the GNUstep libraries without the necessity to run a separate driver program.

The core functionality of Greg is a Guile module which can be loaded into any software with an embedded Guile interpreter. Any program which uses Guile as it's scripting language can therefore use Greg to test itself directly!

For testing external programs, Greg provides a compiled module that may be dynamically linked into Guile to permit you to run an application as a child process on a pseudo-terminal. In conjunction with the standard Guile `expect' module, this lets you test external programs.

Also provided is greg - a Guile script to invoke the Greg test framework in much the same way that runtest is used in DejaGNU.

All tests have the same output format (enforced by the greg-testcase procedure). Greg's output is designed to be both readable and readily parsed by other software, so that it can be used as input to customised testing processes.

Greg provides most of the functionality of DejaGNU but is rather simpler. It omits specific support for cross-platform/remote testing since this is really rather trivial to add where required and tends to vary from site to site so much that an attempt at a generic solution is pretty pointless. What Greg does do, is provide hooks to let you easily introduce site specific code for handling those sorts of situations.

The current version of Greg can normally be found on GNU ftp sites, with documentation online at http://www.tiptree.demon.co.uk/gstep/guile/gregdoc_toc.html

or, for the bleeding edge - availably by anonymous cvs as part of the GNUstep-Guile package in the GNUstep project -

CVSROOT=":pserver:anoncvs@cvs.net-community.com:/gnustep"
export CVSROOT
cvs login (password is `anoncvs')
cvs -z3 checkout guile

How to run a Greg testsuite

To run tests from an existing collection, try running

make check

If the check target exists, it usually saves you some trouble--for instance, it can set up any auxiliary programs or other files needed by the tests.

Alternatively, if you are in the top-level source directory of an existing testsuite (ie. there are subdirectories containing files with a .scm extension), you can get the greg script to test all the tools in the directory by typing -

greg

If you have a test suite that is intended to be used for `embedded' testing - You need to start the application to be tested, gain access to it's Guile command line (or other guile interface) and enter the commands -

(use-modules (ice-9 greg))
(greg-test-all)

A trivial example of a testcase

Each Greg test is a Guile script; the tests vary widely in complexity, depending on the nature of the tool and the feature tested.

;
; GNUstep-guile interface library test
;
; Create an object using the NSString [stringWithCString:] method and
; check that the resulting object is of the correct class.
;
(greg-testcase "The `stringWithCString:' method creates an NSString object" #t
(lambda ()
  (define obj ([] "NSString" stringWithCString: "Hello world"))
  (gstep-bool ([] obj isKindOfClass: ([] "NSString" class)))
))

Though brief, this example is a complete test. It illustrates some of the main features of Greg test scripts:

Why Greg does what it does

Greg was written to support regression testing for the GNUstep libraries. It was inspired by an earlier test framework (by Ovidiu Predescu) that used DejaGNU along with a `driver' program (to make the calls to the library) and a suite of TcL scripts to control the driver.

There were three main problems (inherent in the nature of DejaGNU) with that approach -

So, something different was required, a test framework in a safer, simpler language that made it easy to create thin interfaces to libraries, so simplifying the task of producing testcases.

Of course, the good points of DejaGNU needed to be retained - clear output, Posix compliance, the ability to test separate programs as well as libraries.

A couple of additional goals seemed worthwhile -

Hopefully, Greg meets all its goals.

A POSIX conforming test framework

This section is copied almost directly from the DejaGNU documentation with minor modifications -

Greg is believed to conform to the POSIX standard for test frameworks.

POSIX standard 1003.3 defines what a testing framework needs to provide, in order to permit the creation of POSIX conformance test suites. This standard is primarily oriented to running POSIX conformance tests, but its requirements also support testing of features not related to POSIX conformance.

The POSIX documentation refers to assertions. An assertion is a description of behavior. For example, if a standard says "The sun shall shine", a corresponding assertion might be "The sun is shining." A test based on this assertion would pass or fail depending on whether it is daytime or nighttime. It is important to note that the standard being tested is never 1003.3; the standard being tested is some other standard, for which the assertions were written.

As there is no test suite to test testing frameworks for POSIX 1003.3 conformance, verifying conformance to this standard is done by repeatedly reading the standard and experimenting. One of the main things 1003.3 does specify is the set of allowed output messages, and their definitions. Four messages are supported for a required feature of POSIX conforming systems, and a fifth for a conditional feature. Greg supports the use of all five output messages; in this sense a test suite that uses exactly these messages can be considered POSIX conforming. These definitions specify the output of a test case:

PASS
A test has succeeded. That is, it demonstrated that the assertion is true.
UPASS
POSIX 1003.3 does not incorporate the notion of unexpected passes, so for strict POSIX, PASS, instead of UPASS, is returned for test cases which were not expected to pass but did. This means that PASS is in some sense more ambiguous than if UPASS is also used.
FAIL
A test has produced the bug it was intended to capture. That is, it has demonstrated that the assertion is false. The FAIL message is based on the test case only. Other messages are used to indicate a failure of the framework.
XFAIL
POSIX 1003.3 does not incorporate the notion of expected failures, so for strict POSIX, FAIL, instead of XFAIL, is returned for test cases which were expected to fail and did not. This means that FAIL is in some sense more ambiguous than if XFAIL is also used.
UNRESOLVED
A test produced indeterminate results. Usually, this means the test executed in an unexpected fashion; this outcome requires that a human being go over results, to determine if the test should have passed or failed. This message is also used for any test that requires human intervention because it is beyond the abilities of the testing framework. Any unresolved test should resolved to PASS or FAIL before a test run can be considered finished. Note that for POSIX, each assertion must produce a test result code. If the test isn't actually run, it must produce UNRESOLVED rather than just leaving that test out of the output. With Greg this is not a problem - any unexpected termination of a greg-testcase procedure will produce UNRESOLVED. Here are some of the ways a test may wind up UNRESOLVED:
UNTESTED
A test was not run. This is a placeholder, used when there is no real test case yet.

The only remaining output message left is intended to test features that are specified by the applicable POSIX standard as conditional:

UNSUPPORTED
There is no support for the tested case. This may mean that a conditional feature of an operating system, or of a compiler, is not implemented.

Greg uses the same output procedures to produce these messages for all test suites, and these procedures are already known to conform to POSIX 1003.3. For a Greg test suite to conform to POSIX 1003.3, you must set the variable greg-posix to be true (or run the greg command with the --posix flag). Doing this will ensure that non-posix extensions are not used.


Go to the first, previous, next, last section, table of contents.