Regress: A Test Suite Automation and Reporting Tool

Newsgroup: mozilla.dev.tech.crypto
Technical contact: Bob Relyea
Yell at the manager: Bob Lord

Overview

Regress is a test case execution and reporting tool that operates as follows. It accepts a specification file [that is intended to be platform and location independent] that defines a number of test cases (programs) and a description for each case.  Regress runs the series of programs, in the order of the specfile, and captures into an html output file a table of results:  the date/time of run, program run, and results (PASS or FAIL).   Regress also generates a summary output file suitable for collecting overall results from a number of test suites.  Regress generates results that are utilized by reporter.

Command line interface

   regress specfile=filename [htmldir=dir] [progdir=dir] [debug]
[commands in brackets are optional]

specfile specifies the input specfile (defined below) that regress uses to find out what to execute. The path to the file can be relative or absolute.
If debug is in the command line, information will be output to STDOUT so developers of regress can see what is going on.
progdir specifies the base directory where all the executables are. It could be a relative or absolute path.
htmldir specifies the base directory where all the html results and reporter specs are output to. It could be a relative or absolute path.

When specifying directories in this document, it is not necessary to worry about the "/" vs. the "\" conventions. The regression tool will figure out which slash to use.
 

Error Codes Description
0 No error
1 Did not include specfile= directive on command line
2 Specfile is malformed or does not exist
3 No [General] section in specfile 
4 No mut= line in [General] section
5 No mutversion= line in [General] section
6 No htmlout= line in [General] section
7 No platform= line in [General] section
8 No reporterSpec= line in [General] section
Tests should use exit codes from the range of 1 to 255 due to Unix limitation on transferred exit codes. 256 exit code is reserved for test crash.

Input

The following is a specification of the regress tools Test Case Specification file:
 
Specification File Element Description of Element
[General] Starts the General information block that describes the collection of unit tests in this directory.
mut=description Defines a description for the module under test (e.g., ns/security/lib/crypto module could be described as "Security Library: Cryptographic Module (lib/crypto)".
mutversion=versionstring Defines a version string for the collection of tests. It is assumed that each test will execute against this higher level versionstring version, and that the results will be reported against this version string.
htmlout=<filename> Defines the location and filename relative to htmldir of the html test output that is generated by the regress tool. If the <filename> is NOW, then a yyyymmdd.html string will be substituted for the filename (eg. 19970801). This has to be a relative path to the htmldir specified above. If htmldir is not set, it is automatically default to "."
reporterSpec=<filename> Defines the location and filename of the specfile that is generated by the regress tool and read in by the reporter tool). (See also specification of reporter specfile.) If the <filename> is NOW, then a yyyymmdd.rpt string will be substituted for the filename (eg. 19970801). This has to be a relative path to the htmldir specified above. If htmldir is not set, it is automatically default to "." 
program=<filename> Program name placed in general section is used with single source testing method only, which also requires to have test IDs in the individual test sections.
globalArgs=<arg list string> Defines a list of arguments tacked on to all of the programs when the regression suite is run (this will simply concatenate the <program><space><globalArgs> together). This header is optional.
bugQueryUrl=<URL with metavariable> Defines a URL that has a metavariable $[BUG-NUMBER] that is replaced by the bug number when a test is a known failure. This header is optional.
[Test-X] Begins an information block for a specific test. The X value must be replaced with an integer >= 1. The following name value pairs are related to each [Test-X] block.
program=<filename> This is the relative path (relative to the progdir) to the executable. This is the program to execute that calls PR_ProcessExit or return and returns a 0 for success, or 1 for failure. If progdir is not set, progdir is defauted to "." It is not necessary to add a ".exe" extension at the end of the program name when specifying a Windows / DOS executable. 
Program wont be specified in this section with single source testing method.
testId=<test ID> testId  is used with single source testing method only. It is transferred to the test as a first argument.
testname=<description> This is the description of what the test program is testing. 
timeout=<integer> This is the timeout in seconds to run a test. If a test runs longer than the timeout, it is killed and returns fail. This field is optional. If it is not set, timeout is automatically set to 10 seconds.
bug=<integer> This is an optional field. This is set whenever there is a known test failure. The test will NOT be run when this field is set. 
include=<platforms> This is an optional field. This is set whenever a test is specific to a certain platform or group of platforms. The test will be run only on the platforms (separated by commas or whitespace) in the include list.Check the list for what string to use for your platform. If your platform is not on the list, check the output platform header in the output HTML to see the string you need to use. 

This field is case insensitive (eg. you can use solaris2.5 if you want). 

The test will result in a PASS if the platform is not in the include list (The test didn't really fail, it just wasn't run). 

Note: having both an include and exclude tag is invalid and will result in an error.

exclude=<platforms> This is an optional field. This is set whenever a test is not supposed to be run on a certain platform or group of platforms. The test will not be run on the platforms (separated by commas or whitespace) in the exclude list. Check the list for what string to use for your platform. If your platform is not on the list, check the output platform header in the output HTML to see the string you need to use. 

This field is case insensitive (eg. you can use solaris2.5 if you want). 

The test will result in a PASS if the platform is in the exclude list (The test didn't really fail, it just wasn't run). 

Note: having both an include and exclude tag is invalid and will result in an error.

 

Platform strings

The following is a specification of sample platform strings you can use for the include and exclude headers:

SOLARIS2.5
SOLARIS2.4
SOLARIS2.3
WINNT
HPUXB.10.10
OSFV4.0
AIX1
IRIX5.3
IRIX6.2

The following is a specification of what the return value should be for each Individual Test case executable.

Each of these test cases makes up one singular test within the overall regression test suite. The regression test suite is defined by the Test Specification File.

Output Description Format of Output
Test pass fail status indicator. The program must call PR_ProcessExit(0) if the test passes. If any other value is returned, that means that the test fails. 0   -- PASS 

non-zero value -- FAIL

It is a convention during failures to use the current line number of the program or template as the return value for easier debugging.

Note: If the Individual Test case requires known answers to benchmark the results of the execution against, then it is the responsibility of the individual test case to manage the creation / deletion of startup / shutdown known answers.

Output

The following is a specification of the html output format for the regress tool:
Section Description HTML format
Document Title 

The title is generated from the mut= specifier. The version is generated from mutversion=. The date and time is relative to the beginning point of the execution of the regression test suite.

<HTML><HEAD><TITLE>mut description: Version: mutversion Platform: platform Run date mm/dd/yy</TITLE></HEAD><BODY>\n
H1 Header for the report. <H1>mut description: Version: mutversion Platform: platform Run date mm/dd/yy</H1><HR>\n 
 
Table header <TABLE BORDER=1><TR>\n 
<TD><B>Test Case Number</B></TD>\n 
<TD><B>Program</B></TD>\n 
<TD><B>Description of Test Case</B></TD>\n 
<TD><B>Start date/time<B></TD>\n 
<TD><B>End date/time<B></TD>\n 
<TD><B>PASS/FAIL</B></TD>\n 
</TR>\n
Table Detail 

Open the output file defined by htmlout= specifier in the [General] section of the specification file. 

For each test case header [Test-X] (X >= 1) in the Test Specification file, 

    (1) Capture the date/time prior to execution (this is start date/time
    (2) Execute the program (program=filename) capturing the return code. If program runs past timeout, kill the program. 
    (3) Capture the date/time after execution (this is the end date/time)
    (4) Generate the HTML ouptut -- if the specfile is missing some headers, then the result is "MALFORMED". Otherwise, the result string is either PASS or FAIL. If the test returns a pass, the test row is greyed out. If the test fails, the result will be in red.
Notes: The <description> is associated with [Test-X] and is the description given after testname= in the specification block. 
<TR [if pass then BG COLOR="#C0C0C0">]\n 
<TD><B>X</B></TD>\n 
<TD>filename</TD>\n 
<TD>description</TD>\n 
<TD>mm/dd/yy hh:mm:ss</TD>\n 
<TD>mm/dd/yy hh:mm:ss</TD>\n 
<TD><B>  
PASS | 
PASS : This test was excluded or not included for this platform | 
<FONT COLOR="#FF0000"> FAIL: return code = errnum. </FONT> |  
<FONT COLOR="#FF0000"> FAIL: CreateProcess </FONT> |  
<FONT COLOR="#FF0000"> FAIL: WaitProcess </FONT> |  
<FONT COLOR="#FF0000"> FAIL: Known bug. Bug number = 51325 </FONT> |  
<FONT COLOR="#FF0000"> FAIL: Timeout after 10 seconds </FONT> |  
MALFORMED : include= and exclude= tags BOTH defined | 
MALFORMED : program= not defined in [header] | 
MALFORMED : testname= not defined in [header] | 
MALFORMED : test number=header-part not valid positive integer | 
MALFORMED : test number header invalid 
</B></TD>\n 
</TR>\n
Table trailer </TABLE>\n
Summary Line <TABLE> 
<TR><TD><B>Summary Results</B></TD><TD>Count</TD></TR> 
<TR><TD>Tests Failed</TD><TD><number failed></TD></TR> 
<TR><TD>Test Passed</TD><TD><number passed></TD></TR> 
<TR><TD>Known Failures</TD><TD><known failures></TD><TR> 
<TR><TD>Malformed Specification</TD><TD><number malformed></TD></TR> 
</TABLE> 
 
End of HTML </BODY></HTML>\n
The following is a specification of the reporter specfile output format for the regress tool:
 
Specification File Element Description of Element
[Status] Is the only block in this specification file
mut=description Defines a description for the module under test (e.g., ns/security/lib/crypto module could be described as "Security Library: Cryptographic Module (lib/crypto)".
mutversion=versionstring Defines a version string for the collection of tests. It is assumed that each test will execute against this higher level versionstring version, and that the results will be reported against this versionstring.
platform=<string> Defines the platform the tests were run under
pass=<number> Number of tests that passed.
fail=<number> Number of tests that failed due to non-zero return codes, timeouts, OR any other abnormality when running the test.
knownFail=<number> Number of tests that were known to fail because a bug number was specified in its regress specfile. 
malformed=<number> Number of tests with bad headers.

FAIL Code Descriptions

ug
Output Description
FAIL: return code = 3 When the test program exits with non zero return code.
FAIL: Create Process Regress could not run the program or create the process. This usually means a misspelling of the program name, or that you are running on a machine that does not support processes (possibly MAC)
FAIL: Wait Process Regress could not get the return code of the process for some reason.
FAIL: Timeout after X seconds The program ran longer than the alloted X seconds and was killed.
FAIL: Known bug. Bug number = 53416 The bug = tag in the specfile is set, so the test is a known bug.
FAIL: Known hang A kludge in an older version of regress that recognizes special program names that block forever and outputs this error instead of running them.