Copyright (2000-2009) Ivan Moore.
Jester finds code that is not covered by tests. Jester makes some change to your code, runs your build (which runs your tests), and if the tests pass Jester displays a message saying what it changed. (This is called "mutation testing"). This version of Jester is a simplification of the “standard” one. The aim for this version of Jester is for it to be much simpler to run.
Jester includes a script for generating web pages that show the changes made that did not cause the tests to fail.
Jester is different than code coverage tools, because it can find code that is executed by the running of tests but not actually tested. However, Jester is not meant as a replacement for code coverage tools, merely as a complementary approach.
Jester is available only under the terms of the license agreement.
This version of Jester has been tried on Windows, using Sun's JDK 1.5. There is absolutely no warranty whatsoever (see license agreement ); it might work on those or other platforms - see the Jester web page for latest known bugs, platforms that it has been tried on, FAQs, etc). Older versions of Jester might work with older versions of Java.
The most important bits are in green, for the impatient.
unzip the jester.zip file (e.g. simple-jester-1.0.zip) in an empty directory
Java (>=1.5) must already be installed
For a simple example (which currently only runs under Windows and requires Java to be on the path), run the batch file 'test.bat'.
Some text should be output (and a progress dialog shown), and after less than 2 minutes, the execution should finish with following text output:
17 mutations survived out of 19 changes. Score = 11
took
0 minutes
If the message above does not appear, see the FAQ referenced at the end of this document.
This example also needs Python >= 2.0 installed in order to generate a file "jester.html" which shows the changes Jester made that did not cause the tests to fail. (If you don't have python installed, then 'test.bat' will still run but won't produce "jester.html" and will say something about python not being recognized.
Jester has a progress window - the progress bar shows the proportion of source files that have been mutated; the text area shows the most recent mutation that has been made; the progress bar is green if the last mutation caused the tests to fail (i.e. good) and red if the tests still passed despite the mutation (i.e. bad). Note that the progress bar does not show progress while making different mutations to the same file - that is shown by text in the text area changing - because that was simpler to implement.
Very IMPORTANT - only run Jester on a copy of your source files - as Jester changes the source code that it works on. If you use source control (which everyone should!) then you can leave your source code where it is (making sure everything you want to be in source control is checked in) and overwrite all your source from source control after running Jester. If you don't use source control, only run Jester on a copy of your source in a different directory, otherwise you'll regret it!
To run Jester, execute 'java -jar simple-jester.jar -buildCommand BUILD_COMMAND -source SOURCE_DIRECTORY' where BUILD_COMMAND is your build command and SOURCE_DIRECTORY is the directory (or a single source file) that contains the source code that you want Jester to mutation test.
BUILD_COMMAND should be the build command that is expected to show up any changes to code in the SOURCE_DIRECTORY. Typically, BUILD_COMMAND would compile and run the unit tests for the code in SOURCE_DIRECTORY. (e.g. "ant.bat -f my-build.xml compile run-unit-tests"). NOTE that on windows, for running ant, you need to say "ant.bat" not just "ant". NOTE that you need the double quotes around the complete build command if it contains spaces (as it often will).
For any change that Jester was able to make without the build failing, it prints the name of the file changed, the position in the file of the change (the character index), and some of the original source file from roughly 30 characters before to 30 characters after so that the change can be identified within the source file. Note that Jester can take a long time to run - try it on a small part of your system before trying it on everything.
Jester also produces a file called "jesterReport.xml" that contains all the changes that Jester made that did not cause the tests to fail.
Execute "python makeWebView.py" to generate copies of the original source files as ".html" pages, with all the changes Jester made that did not cause the tests to fail shown in red, and the code it changed shown with a line through it. A file "jester.html" is generated which links to all these files. These files are in the same directories, and have the same names, as the original source files, except have a ".html" extension. This script has options:
-z to ignore files that Jester didn't make any changes to
-p to ignore files that Jester didn't make any changes to where the tests still passed
-s to sort the files by file name in the "jester.html" file.
For other options, try "python makeWebView.py -h".
Execute "python makeAllChangesFiles.py jesterReport.xml" to generate copies of the original source files with all the changes Jester made that did not cause the tests to fail. These files are in the same directories, and have the same names, exept have a ".jester" extension (configurable). These files are useful for comparing against the original source code using whatever you usually use to compare files, to easily find the changes Jester made that did not cause the tests to fail.
Jester prints out warnings when it can't find it's config files. In such circumstances, Jester uses default values which are OK in many cases (i.e. Java code with an "ant" build command).
Jester writes out a file "jesterTimeout.txt" (in the directory where it is executed) whose contents is the number of milliseconds the tests took to run the first time they were run (i.e. before Jester made any changes). This is used by Jester to stop running the tests if they are taking too long (e.g. in case a change made by Jester causes the code to enter an infinite loop).
If you kill Jester during a run, by CTRL-C, there is a risk that your source code will not be reverted back to its original state and hence the tests might fail and hence Jester will not be able to run against your code unless you revert the effected file. This might be fixed in a future version, but might not - Jester cannot be held responsible for messing up your source code, it's up to you to look after it.
For the following features, edit the file called "jester.cfg" from the installation directory and specify it when running Jester using “-config jester.cfg”. For example, buildPassString=BUILD SUCCESSFUL refers to a line of text in the jester.cfg file.
Jester can be used for builds using any build tool that produces output to standard output that indicates whether the build was successful or not, by setting buildPassString to whatever indicates to Jester that the build was successful. By default, this is set to buildPassString=BUILD SUCCESSFUL which means that Jester will work with ant builds "out of the box".
If your source code uses a different source file extension than .java then set sourceFileExtension as appropriate.
Jester can also be made to print the results as it goes rather than waiting until it has finished to print an amalgamated report of all the changes it has made per file, by setting shouldReportEagerly=true. This can be useful if Jester is failing to complete it's run.
The mutations that Jester applies to the source code are configurable, by editing the file "mutations.cfg" from the installation directory and specify it when running Jester using “-mutations mutations.cfg”. Each line of this file must have a format like:
%if(%if(true || where % can be any single character as a delimiter, and the text if( will be replaced by if(true ||.
This simple scheme of replacing text without parsing has proven adequate so far, but may change in a future version of Jester.
Jester can be made to ignore parts of every source file, by editing the ignorelist.cfg file and specify it when running Jester using “-ignore ignorelist.cfg”. This file contains lines starting with a delimiter character that specify regions of the source code to ignore; i.e. regions where it will not try to make a mutation. The first string is where to start ignoring the source and the second is where to stop ignoring the source. For example, to ignore comments in Java code, the ignorelist.cfg file would contain:
%/*%*/ %//%\n
This is not entirely correct but is close enough; e.g. a string "http://jester.sf.net" up to the end of the line would not be considered for mutation by Jester using this ignorelist.cfg file, but that's not so bad. (Note - these are included in the ignorelist.cfg file in the standard build).
This feature can be used by Pester (Jester on Python code) too, e.g. to ignore python comments the ignorelist.cfg file would contain:
%#%\n
Note - the standard build also includes the line %//stopJesting%//resumeJesting so that you can easily tell Jester to ignore parts if a source file if it is causing problems for Jester. i.e. put a comment //stopJesting on the line before the code you want Jester to ignore and //resumeJesting on the line after the code you want Jester to ignore.
If the code will not compile (either because of classes missing from the class path before Jester tried to compile any classes, or as a result of a change to a source file by Jester) then Jester might hang indefinitely.
NOTE that this version of Jester will not work with any version of Pester, but can be used instead of Pester. This version of Jester is relatively easy to get to work for any language as long as you can give Jester some command to run (to compile if necessary, and run the tests of the code you want to run Jester on) and the source file extension.
Very many thanks to:
Malte Finsterwalder for his feedback on simple-jester.
Nat Pryce for reporting the problem with running simple-jester version 1.0 in linux.
Elliotte Rusty Harold for reporting and fixing bug 1035010.
Diego Vallespir for reporting bugs 1031730 and 1031735.
Bernd Schiffer for sending fixes for using source directories with spaces in their names.
Stefan Roock, who has joined the Jester project as a developer.
Brett Neumeier for the suggested improvements to the makeWebView.py script.
Aho Augasmagi for getting me to implement the Ignore Lists.
Kent Beck and Robert Martin for their great quotes about Jester on the Jester home page.
Duncan Pierce for the idea of using the jesterReport.xml file to generate the ".html" files to show the changes Jester made that did not cause the tests to fail in an easily browsable way.
Manfred Kaul for sending a fix to make Jester report problems when it tries to 'exec' things that don't work (in particular, if things are missing from the classpath).
Please check the Jester web pages (hosted by sourceforge) for updates and the FAQs, and please enter comments or bug reports on the appropriate pages linked from there.
The file "jester.log" (in the directory where you ran jester) shows the commands that Jester has tried to run. In many cases, you can work out what is wrong by trying to run the last command that Jester tried to run. E.g. if the last line of the "jester.log" says "Sat Dec 01 21:46:36 PST 2001 Trying to run command "ant.bat -f test-build.xml"", then try running "ant.bat -f test-build.xml"; this may then reveal the problem. In the case of tests not passing (e.g. the message "Couldn't run test tester because tests didn't pass before any changes made"), try running the tests directly (e.g. "ant.bat -f test-build.xml"). Please delete the file "jester.log" whenever it gets too large, or before running Jester if preferred.
version 1.2-simple
jester@tadmad.co.uk