Linux Studio Plugins Project

# Testing

Since release 1.1.4 there is implemented testing subsystem that allows:

- perform manual tests (mtest module);
- perform automated unit testing (utest module);
- perform automated single-core performance testing (ptest module).

Manual tests are semi-automated or not automated at all. You may launch these tests to perform manual interaction with system or for retrieving some text data for further processing.

Unit tests are completely automated and check that the produced result is correct for the specific input data. By default unit tests are launched in separate processes and utilize all CPU cores for maximum performance. The status of each unit test is collected during execution and list of all failed tests are printed after completion.

Performance tests measure performance of some function/module under different conditions, gather final statistics and output them in a table format. These tests are very useful for measuring single-core performance of different modules and functions and performing code optimizations.

To build testing subsystem, issue the following commands and enable `test` feature in `FEATURES` parameter: ```

    make clean
    make config ADD_FEATURES='test'
    make

    After build, we can launch the test binary by issuing command:

    .build/host/lsp-plugin-fw/lsp-plugins-test

    This binary provides simple command-line interface, so here's the full usage:

    USAGE: {utest|ptest|mtest} [args...] [test name...]
      First argument:
        utest                 Unit testing subsystem
        ptest                 Performance testing subsystem
        mtest                 Manual testing subsystem
      Additional arguments:
        -a, --args [args...]  Pass arguments to test
        -d, --debug           Disable time restrictions for unit tests
                              for debugging purposes
        -e, --execute         Launch tests specified after this switch
        -f, --fork            Fork child processes (opposite to --nofork)
        -h, --help            Display help
        -i, --ignore          Ignore tests specified after this switch
        -j, --jobs            Set number of job workers for unit tests
        -l, --list            List all available tests
        -mt, --mtrace         Enable mtrace log
        -nf, --nofork         Do not fork child processes (for better
                              debugging capabilities)
        -nt, --nomtrace       Disable mtrace log
        -o, --outfile file    Output performance test statistics to specified file
        -s, --silent          Do not output additional information from tests
        -t, --tracepath path  Override default trace path with specified value
        -v, --verbose         Output additional information from tests

    Each test has fully-qualified name separated by dot symbols, tests from different
    test spaces (utest, ptest, mtest) may have similar fully-qualified names.

    To obtain a list of all unit tests we can issue:

    .build/host/lsp-plugin-fw/lsp-plugins-test utest --list

    And then we can launch all complex number processing unit tests and additionally
    'dsp.mix' unit test:

    .build/host/lsp-plugin-fw/lsp-plugins-test utest dsp.complex.* dsp.pcomplex.* dsp.mix

    If we don's specify any unit test name in argument, then all available unit tests
    will be launched.

    To start debugging of some unit test, you need to pass additional arguments:

    .build/host/lsp-plugin-fw/lsp-plugins-test/lsp-plugins-test utest --nofork --debug --verbose

    Because unit tests are short-time fully-automated tests, they are parallelized and
    executed by default by number_of_cores*2 processes. To disable this, we specify option
    --nofork. Also, unit test execution time is limited by 5 seconds by default, so when
    debugging, our tests will be killed. That's why we specify --debug option to disable time
    control. Option --verbose allows to output additional information by the unit test that is
    turned off by default.

    We also can use performance tests to obtain full performance profile of target machine.
    Because performance tests in most cases take much time for gathering statistics,
    the final statistics for each test can be saved in a separate file by specifying --outfile
    option:

    .build/host/lsp-plugin-fw/lsp-plugins-test ptest -o performance-test.log