Unit Testing

Unit testing means running automated tests for the certain block of the code. These tests are run either by scripts or unit testing framework. The result is a list os passed and/or failed tests.

Unit testing framework used with WinMerge development is the Google Test, available from http://code.google.com/p/googletest/. Other frameworks can be used when need arises. However we don't want to use every known framework so the reason for adding new framework needs to be good, like new technology or language used that current framework does not support.

Earlier we tried to start using CppUnit but unfortunately even compiling CppUnit seemed to be too hard for many people. And creating tests is just a mess. So CppUnit usage is discouraged nowadays.

Policy

Existing unit tests must be run without errors. For every error there must be a known explanation. It is a good practice to document errors into unit tests so others know what is going on. It is also recommended to submit bug reports of failing tests if the fix is not imminent.

When changing the code which has unit tests, test must be always run before the patch is submitted to patch tracker and before the patch is committed to SVN. Possible known failures must be explained. Failing unit can be a blocker preventing committing the patch.

Failing unit tests may be (and most likely are) blockers for the beta and stable releases. Experimental releases can be released with known errors.

Adding new tests

Unit tests by nature rarely are and can be complete. Adding more tests and improving current tests is highly encouraged. It is the easiest way to improve WinMerge quality!

Modfying existing tests

There must be some care when altering existing tests though. It may be that there was a reason the existing test were done as it was. If that reason is not obvious, it is better to create new improved test and leave the old test untouched.

Fixing a failing unit testing by modifying the test is highly questionable. The test must be buggy for this to be allowed.

Quick guide to Google Test

Google Test is an external library/framework that must be compiled before using with unit tests.

Unzip and compile Google Test library

Unzip the Google Test sources to some folder. Pick a folder you can keep around (not some temp folder) because you'll need the header files every time you compile unit tests.

Open solution file msvc\gtest.sln to the Visual Studio. Depending on Visual Studio version the project/solution file may need to be updated for later version before usage. Visual Studio asks about the conversion when opening the solution file.

Build both Debug and Release targets of gtest project. You don't need to build other projects. Indeed, rebuild all for the solution does not work. Some of the projects in the solution are unit tests for the framework itself.

Copy the compiled gtest.lib and gtestd.lib files from msvc\Release and msvc\Debug subfolders to Testing\GoogleTest - folder in WinMerge source tree.

Setup Visual Studio for Google Test

As mentioned earlier Google Test header files are needed to compile the tests. As the Google Test was unzipped to separate folder we must first tell Visual Studio where to find those headers.

Add the gtest-[version]\include -folder to Visual Studio's include file folders:

Open the test project into Visual Studio and compile

Open the testing project to Visual Studio (VS 2003 and later work) and compile Debug or Release -target of the project.

Run the tests

Open the Command Prompt and CD to folder where executable were build, usually [projectname]/Debug or [projectname]/Release.

Run the executable.

While the testing runs it prints progress information and info about passed/failed tests.

Creating new test project

Initial creation of new test project takes some effort and time for the first time. It is still pretty straightforward so nobody should have any real problems with it. If somebody has, please tell Kimmo about it and we'll improve the documentation!

Create new VS project

Open the Visual Studio and select creating new project. Select Win32 Console Project as the project type. You can create some other type too and then just fix the properties. But console project selects many properties correctly for you.

Note: If you are creating tests for class depending on MFC, then remember to include MFC (libraries) to the project.

Add new test projects into subfolders to folder Testing\GoogleTest . E.g. Testing\GoogleTest\CommandLine. The project (folder) name should somehow relate the to the code/functionality/class you are testing.

Set the project properties

Several project settings need to be changed to make test project compile (properly). There might be better choises and practices, but this is the best setup we currently know. If you have ideas for improving, let us know!

Set the C/C++ properties according list below:

Set the Linker properties according list below:

Add test file(s)

Next you need to add new C++ code (.cpp) file to the project. Name the file as [projectname]_test.cpp and make sure the file name does not overlap with the C++ files you are testing! _test as part of the filen name helps.

Copy/paste the beginning of existing test file to the new .cpp -file. Copy e.g. from Testing\GoogleTest\Paths\paths_test.cpp . Copy the content to the first test function (TEST_F()). Don't copy the test functions, but copy the namespace closing brace and main() function from the end of the file.

Rename the test class to desribe your tests (e.g. CommandLineTest ). Remember to rename constructor/destructor methods too!

Add tests

Now you can start adding tests. Look Google Test documentation and existing unit tests for help on that.

The simple test method looks like this:

	TEST_F(PathTest, Create_abspath1)
	{
		EXPECT_TRUE(paths_CreateIfNeeded("c:\\Temp"));
	}

The TEST_F macro defines a new test function, having test class (PathTest) and test name (Create_abspath1) as parameters. Then there is a function call done inside test macro. The framework defines several macros for the tests. The EXPECT_TRUE macro makes the test succeed if the parameter (funtion return value) is true, otherwise the test fails.

Split tests to several cases/files

GoogleTests allows splitting tests to several cases and several files. Adding new test case is easy: just add new file and create new test class there. Don't copy main() function. Eatch test project should have only one main().