Erlang -- Test Structure

User's Guide
Reference Manual
Release Notes
PDF
Top

Common Test
User's Guide
Version 1.6.2

Expand All
Contract All

Chapters

- Common Test Basics
  - Top of chapter
  - Introduction
  - Test Suite Organisation
  - Support Libraries
  - Suites and Test Cases
  - External Interfaces
- Getting Started
  - Top of chapter
  - Are you new around here?
  - Test case execution
  - A simple test suite
  - A test suite with configuration functions
  - What happens next?
- Installation
  - Top of chapter
  - General information
- Writing Test Suites
  - Top of chapter
  - Support for test suite authors
  - Test suites
  - Init and end per suite
  - Init and end per test case
  - Test cases
  - Test case info function
  - Test suite info function
  - Test case groups
  - The parallel property and nested groups
  - Parallel test cases and IO
  - Repeated groups
  - Shuffled test case order
  - Group info function
  - Info functions for init- and end-configuration
  - Data and Private Directories
  - Execution environment
  - Timetrap timeouts
  - Logging - categories and verbosity levels
  - Illegal dependencies
- Test Structure
  - Top of chapter
  - Test structure
  - Skipping test cases
  - Definition of terms
- Examples and Templates
  - Top of chapter
  - Test suite example
  - Test suite templates
- Running Tests
  - Top of chapter
  - Using the Common Test Framework
  - Automatic compilation of test suites and help modules
  - Running tests from the OS command line
  - Running tests from the Erlang shell or from an Erlang program
  - Running the interactive shell mode
  - Step by step execution of test cases with the Erlang Debugger
  - Test Specifications
  - Running tests from the Web based GUI
  - Log files
  - HTML Style Sheets
  - Repeating tests
  - Silent Connections
- External Configuration Data
  - Top of chapter
  - General
  - Syntax
  - Requiring and reading configuration data
  - Using configuration variables defined in multiple files
  - Encrypted configuration files
  - Opening connections by using configuration data
  - User specific configuration data formats
  - Examples of configuration data handling
  - Example of user specific configuration handler
- Code Coverage Analysis
  - Top of chapter
  - General
  - Usage
  - The cover specification file
  - Logging
- Using Common Test for Large Scale Testing
  - Top of chapter
  - General
  - Usage
  - Test Specifications
  - Automatic startup of test target nodes
- Event Handling
  - Top of chapter
  - General
  - Usage
- Dependencies between Test Cases and Suites
  - Top of chapter
  - General
  - Saving configuration data
  - Sequences
- Common Test Hooks
  - Top of chapter
  - General
  - Installing a CTH
  - CTH Scope
  - Manipulating tests
  - Example CTH
  - Built-in CTHs
- Some thoughts about testing
  - Top of chapter
  - Goals
  - What to test?

# 5 Test Structure

### 5.1 Test structure

A test is performed by running one or more test suites. A test suite consists of test cases (as well as configuration functions and info functions). Test cases may be grouped in so called test case groups. A test suite is an Erlang module and test cases are implemented as Erlang functions. Test suites are stored in test directories.

### 5.2 Skipping test cases

It is possible to skip certain test cases, for example if you know beforehand that a specific test case fails. This might be functionality which isn't yet implemented, a bug that is known but not yet fixed or some functionality which doesn't work or isn't applicable on a specific platform.

There are several different ways to state that one or more test cases should be skipped:

- Using skip_suites and skip_cases terms in test specifications.
- Returning {skip,Reason} from the init_per_testcase/2 or init_per_suite/1 functions.
- Returning {skip,Reason} from the execution clause of the test case.

The latter of course means that the execution clause is actually called, so the author must make sure that the test case does not run.

When a test case is skipped, it will be noted as SKIPPED in the HTML log.

### 5.3 Definition of terms

Auto skipped test case
  When a configuration function fails (i.e. terminates unexpectedly), the test cases that depend on the configuration function will be skipped automatically by Common Test. The status of the test cases is then "auto skipped". Test cases are also auto skipped by Common Test if required configuration data is not available at runtime.
Configuration function
  A function in a test suite that is meant to be used for setting up, cleaning up, and/or verifying the state and environment on the SUT (System Under Test) and/or the Common Test host node, so that a test case (or a set of test cases) can execute correctly.
Configuration file
  A file that contains data related to a test and/or an SUT (System Under Test), e.g. protocol server addresses, client login details, hardware interface addresses, etc - any data that should be handled as variable in the suite and not be hardcoded.
Configuration variable
  A name (an Erlang atom) associated with a data value read from a configuration file.
data_dir
  Data directory for a test suite. This directory contains any files used by the test suite, e.g. additional Erlang modules, binaries or data files.
Info function
  A function in a test suite that returns a list of properties (read by the Common Test server) that describes the conditions for executing the test cases in the suite.
Major log file
  An overview and summary log file for one or more test suites.
Minor log file
  A log file for one particular test case. Also called the test case log file.
priv_dir
  Private directory for a test suite. This directory should be used when the test suite needs to write to files.
ct_run
  The name of an executable program that may be used as an interface for specifying and running tests with Common Test.
Test case
  A single test included in a test suite. A test case is implemented as a function in a test suite module.
Test case group
  A set of test cases that share configuration functions and execution properties. The execution properties specify whether the test cases in the group should be executed in random order, in parallel, in sequence, and if the execution of the group should be repeated. Test case groups may also be nested (i.e. a group may, besides test cases, contain sub-groups).
Test suite
  An erlang module containing a collection of test cases for a specific functional area.
Test directory
  A directory that contains one or more test suite modules, i.e. a group of test suites.
The Config argument
  A list of key-value tuples (i.e. a property list) containing runtime configuration data passed from the configuration functions to the test cases.
User skipped test case
  This is the status of a test case that has been explicitly skipped in any of the ways described in the "Skipping test cases" section above.