Let us turn once again to the theme of the test design, going deeper into its study.

Test design is the stage of the software testing process, on which test cases are designed and created in accordance with previously defined quality criteria and testing objectives.

Test design [ISTQB Glossary of terms]: The process of transforming general testing into tangible test conditions and test cases.

Work plan for the test design

• analysis of available design artifacts: documentation (specifications, requirements, plans), models, executable code, etc.
• writing a Test Design Specification
• designing and creating Test Cases

Roles responsible for the test design

• Test Analyst – defines “WHAT to test?”
• Test designer – defines “HOW to test?”

Simply put, the task of the test of analysts and designers is to use a variety of strategies and techniques to test design, create a set of test cases that provides the optimal test coverage of the tested application. However, on most projects these roles are not allocated, but are entrusted to ordinary testers, which does not always have a positive effect on the quality of the tests, testing and, as it follows, on the quality of the software (the final product).

Test Coverage

The Test Coverage is one of the metrics for evaluating the quality of testing, which is the density of coverage by test requirements or executable code.

If we consider testing as “checking the correspondence between the actual and expected behavior of a program performed on a finite set of tests”, then it is this final set of tests that will determine the test coverage:

The higher the required level of test coverage, the more tests will be selected, to verify the test requirements or executable code.

The complexity of modern software and infrastructure made it impossible for testing with a 100% test coverage. Therefore, to develop a set of tests that provide a less high level of coverage, you can use special tools or test design techniques.

There are following approaches to evaluation and measurement of test coverage:

1. Requirements Coverage – Evaluation of the coverage of functional and non-functional requirements for a product by creating traceability matrix.
2. Code Coverage – Evaluation of the coverage of executable code by tests, by tracking unauthorized parts of the software during testing.
3. Test coverage based on control flow analysis – coverage assessment based on determining the ways of code module execution and creating test cases to cover these paths.

Differences:

The method of covering requirements is focused on checking the compliance of the set of tests conducted with the product requirements, while the code coverage analysis is on the completeness of the tests, the developed part of the product (source code), and the control flow analysis – on the paths in the graph or the model of the tested functions (Control Flow Graph).

Limitations:

The method for evaluating the coverage of the code will not reveal unrealized requirements, since it works not with the final product, but with the existing source code.

The method of covering requirements may leave some parts of the code unchecked, because it does not take into account the final implementation.

Requirements Coverage

Calculation of the test coverage relative to the requirements is carried out according to the formula:

Tcov = (Lcov / Ltotal) * 100%

Where:

Tcov – test coverage

Lcov – the number of requirements checked by the test cases

Ltotal – total number of requirements

To measure coverage requirements, you need to analyze the requirements for the product and break them into points. Optionally, each item is associated with test cases that check it. The totality of these links is the trace matrix. Having traced connections, it is possible to understand exactly which requirements the test case verifies.

Tests not related to requirements do not make sense. The requirements not related to the tests are “white spots”, i.e. having carried out all the created test cases, it is impossible to give the answer whether this requirement is implemented in the product or not.

To optimize the test coverage when testing based on requirements, the best way is to use standard test design techniques.

Code Coverage

Calculation of the test coverage relative to the executable code of the software is carried out according to the formula:

Tcov = (Ltc / Lcode) * 100%

Where:

Tcov – test coverage

Ltc – number of lines of code covered by tests

Lcode – the total number of code lines.

Currently, there is a toolkit (for example: Clover) that allows you to analyze which lines were entered during testing, so you can significantly increase the coverage by adding new tests for specific cases, as well as getting rid of duplicate tests. Carrying out such a code analysis and subsequent optimization of the coverage is quite easy to implement in the framework of white-box testing with modular, integration and system testing; while testing the black box (black-box testing), the task becomes quite expensive, as it takes a lot of time and resources to install, configure and analyze the results of work, both on the part of testers and developers.

Test coverage based on control flow analysis

Control Flow Testing is one of the techniques for testing a white box based on determining the ways of executing the code of a software module and creating test cases to cover these paths.

The foundation for testing control flows is the construction of control flow graphs, the main blocks of which are:

• process block – one entry point and one exit point
• alternate point – one entry point, two or more exit points
• connection point – two or more entry points, one exit point

To test the control flows, different levels of test coverage are defined:

Next time we will study Test Design Techniques.