Test Structure and Test Specifications

## 2 Test Structure and Test Specifications

### 2.1 Test structure

A test consists of a set of test cases. Each test case is implemented as an erlang function. An erlang module implementing one or more test cases is called a test suite.

### 2.2 Test specifications

A test specification is a specification of which test suites and test cases to run and which to skip. A test specification can also group several test cases into conf cases with init and cleanup functions (see section about configuration cases below). In a test there can be test specifications on three different levels:

The top level is a test specification file which roughly specifies what to test for a whole application. The test specification in such a file is encapsulated in a topcase command.

Then there is a test specification for each test suite, specifying which test cases to run within the suite. The test specification for a test suite is returned from the all(suite) function in the test suite module.

And finally there can be a test specification per test case, specifying sub test cases to run. The test specification for a test case is returned from the specification clause of the test case.

When a test starts, the total test specification is built in a tree fashion, starting from the top level test specification.

The following are the valid elements of a test specification. The specification can be one of these elements or a list with any combination of the elements:

{Mod, Case}
  This specifies the test case Mod:Case/1
{dir, Dir}
  This specifies all modules *_SUITE in the directory Dir
{dir, Dir, Pattern}
  This specifies all modules Pattern* in the directory Dir
{conf, Init, TestSpec, Fin}
  This is a configuration case. In a test specification file, Init and Fin must be {Mod,Func}. Inside a module they can also be just Func. See the section named Configuration Cases below for more information about this.
{make, Init, TestSpec, Fin}
  This is a special version of a conf case which is only used by the test server framwork ts. Init and Fin are make and unmake functions for a data directory. TestSpec is the test specification for the test suite owning the data directory in question. If the make function fails, all tests in the test suite are skipped. The difference between this "make case" and a normal conf case is that for the make case, Init and Fin are given with arguments ({Mod,Func,Args}), and that they are executed on the controller node (i.e. not on target).
Case
  This can only be used inside a module, i.e. not a test specification file. It specifies the test case CurrentModule:Case.

### 2.3 Test Specification Files

A test specification file is a text file containing the top level test specification (a topcase command), and possibly one or more additional commands. A "command" in a test specification file means a key-value tuple ended by a dot-newline sequence.

The following commands are valid:

{topcase, TestSpec}
  This command is mandatory in all test specification files. TestSpec is the top level test specification of a test.
{skip, {Mod, Comment}}
  This specifies that all cases in the module Mod shall be skipped. Comment is a string.
{skip, {Mod, Case, Comment}}
  This specifies that the case Mod:Case shall be skipped.
{skip, {Mod, CaseList, Comment}}
  This specifies that all cases Mod:Case, where Case is in CaseList, shall be skipped.
{nodes, Nodes}
  Nodes is a list of nodenames available to the test suite. It will be added to the Config argument to all test cases. Nodes is a list of atoms.
{require_nodenames, Num}
  Specifies how many nodenames the test suite will need. Theese will be automaticly generated and inserted into the Config argument to all test cases. Num is an integer.
{hosts, Hosts}
  This is a list of available hosts on which to start slave nodes. It is used when the {remote, true} option is given to the test_server:start_node/3 function. Also, if {require_nodenames, Num} is contained in a test specification file, the generated nodenames will be spread over all hosts given in this Hosts list. The hostnames are atoms or strings.
{diskless, true}
  Adds {diskless, true} to the Config argument to all test cases. This is kept for backwards compatiblilty and should not be used. Use a configuration case instead.
{ipv6_hosts, Hosts}
  Adds {ipv6_hosts, Hosts} to the Config argument to all test cases.

All test specification files shall have the extension ".spec". If special test specification files are needed for Windows or VxWorks platforms, additional files with the extension ".spec.win" and ".spec.vxworks" shall be used. This is useful e.g. if some test cases shall be skippped on these platforms.

Some examples for test specification files can be found in the Examples section of this user's guide.

### 2.4 Configuration cases

If a group of test cases need the same initialization, a so called configuration or conf case can be used. A conf case consists of an initialization function, the group of test cases needing this initialization and a cleanup or finalization function.

If the init function in a conf case fails or returns {skip,Comment}, the rest of the test cases in the conf case (including the cleanup function) are skipped. If the init function succeeds, the cleanup function will always be called, even if some of the test cases in between failed.

Both the init function and the cleanup function in a conf case get the Config parameter as only argument. This parameter can be modified or returned as is. Whatever is returned by the init function is given as Config parameter to the rest of the test cases in the conf case, including the cleanup function.

If the Config parameter is changed by the init function, it must be restored by the cleanup function. Whatever is returned by the cleanup function will be given to the next test case called.

### 2.5 Skipping test cases

It is possible to skip certain test cases, for example if you know beforehand that a specific test case fails. This might be functionality which isn't yet implemented, a bug that is known but not yet fixed or some functionality which doesn't work or isn't applicable on a spesific platform.

There are several different ways to state that a test case should be skipped:

- Using the {skip,What} command in a test specification file
- Returning {skip,Reason} from the init_per_testcase/2 function
- Returning {skip,Reason} from the specification clause of the test case
- Returning {skip,Reason} from the execution clause of the test case

The latter of course means that the execution clause is actually called, so the author must make sure that the test case is not run. For more information about the different clauses in a test case, see the chapter about writing test cases.

When a test case is skipped, it will be noted as SKIPPED in the HTML log.

test_server 3.2.4
Copyright © 1991-2008 Ericsson AB