Today we will examine the connection between testing processes and the design process, and also explore approaches to designing tests.

The figure below shows that mistakes at the early stages of the project can be exhaustively identified at the very end of the work.

Testing programs covers a number of activities:

• statement of the problem;
• test design;
• writing tests;
• testing of tests;
• performance of tests;
• study of test results.

Here the most important is the design of tests.

So, testing is the process of executing a program to detect errors.

Approaches to Test Design

Consider the two most opposite approaches to designing tests.

The advocate of the first approach focuses only on a testing strategy, called the black box strategy, data management testing, or testing with I / O control. When using this strategy, the program is treated as a black box. The test data is used only in accordance with the program specification (i.e., without knowledge of its internal structure). The unattainable ideal of the supporter of the first approach is to check all possible combinations and values ​​at the input. Usually there are too many of them even for the simplest algorithms. So, for the program of calculating the arithmetic mean of four numbers, it is necessary to prepare 107 test data.

Relation of design and testing processes

In the first approach, the detection of all errors in the program is the criterion of exhaustive input testing. The latter can be achieved if all possible sets of input data are used as test sets. Therefore, we come to the conclusion that for an exhaustive testing of the program an infinite number of tests is required, and therefore it is impossible to construct an exhaustive input test. This is supported by two arguments: first, you can not create a test to ensure that there are no errors; secondly, the development of such tests contradicts economic requirements. Since exhaustive testing is excluded, our goal should be to maximize the effectiveness of investment in testing (maximizing the number of errors detected by one test). For this, it is necessary to consider the internal structure of the program and to make some reasonable assumptions that are not entirely reliable.

The follower of the second approach uses a “white box” strategy, or a testing strategy controlled by the program logic, which allows to examine the internal structure of the program. In this case, the tester gets test data by analyzing only the program logic; strives for each team to be executed at least once. With sufficient qualification, he achieves that each command of the conditional transfer be executed in each direction at least once. The cycle must be executed once, never, the maximum number of times. The purpose of testing all paths from the outside is also unattainable. In a program of two consecutive cycles inside each of them, branching into ten paths is included, there are 1018 calculation paths. And the fulfillment of all calculation methods does not guarantee the fulfillment of all specifications. For reference: the age of the universe is 1017 s.

Let’s compare the method of constructing tests for this strategy with exhaustive input testing of the “black box” strategy. It is incorrect to assume that it is sufficient to construct a set of tests in which each operator is executed at least once. Exhaustive testing can be matched by comprehensive testing of routes. It is understood that the program is fully tested if by tests it is possible to execute this program on all possible routes of its flow (graph) of control transmissions.

The last statement has two weak points: first, the number of routes that do not repeat each other is astronomical; Secondly, even if each route can be checked, the program itself may contain errors (for example, some routes are skipped).

The path property is performed correctly for some data and is incorrect for others – called sensitivity to data, most often due to numerical errors and truncation errors methods. Testing each of all routes with one test does not guarantee the detection of sensitivity to the data.

As a result of all the above remarks, we note that neither exhaustive input testing nor exhaustive testing of routes can be useful strategies, because they are both unrealizable. Therefore, the real way, which will create a good, but certainly not an absolute strategy, is the combination of testing the program in several ways.

Consider an example of testing an operator

if A and В then…

When using different criteria for completeness of testing.

Under the condition coverage criteria, two tests would be required: A = true, B = false, and A = false, B = true. But in this case, the then-statement of the if statement is not executed.

There is another criterion called coating decisions / conditions. It requires such a sufficient set of tests that all possible outcomes of each condition in the solution are fulfilled at least once; all the results of each solution were fulfilled also once and each control point was transmitted control at least once.

The disadvantage of the criterion for covering decisions / conditions is the impossibility of applying it for the fulfillment of all the results of all conditions. Often, this is due to the fact that certain conditions are hidden by other conditions. For example, if the AND condition is false, then none of the following conditions in the expression will be satisfied. Similarly, if the condition OR is true, then none of the following conditions will be satisfied. Consequently, the criteria for covering conditions and covering solutions / conditions are not sufficiently sensitive to errors in logical expressions.

The criterion that solves these and some other problems is the combinatorial covering of the conditions. It requires the creation of a number of tests so that all possible combinations of condition results in each decision and all entry points are executed at least once.

In the case of cycles, the number of tests to satisfy the criterion for combinatorial coverage of conditions is usually greater than the number of paths.

It is easy to see that a set of tests that satisfies the criterion of combinatorial coverage of conditions also satisfies the criteria for covering solutions, covering conditions and covering solutions / conditions.

Thus, for programs containing only one condition for each solution, the criterion is the minimum, the set of tests of which causes execution of all the results of each solution at least once; transfers control to each entry point (for example, the CASE statement).

For programs that contain solutions, each of which has more than one condition, the minimum criterion consists of a set of tests that call all possible combinations of the results of conditions in each solution and pass control to each entry point of the program at least once.

The division of the algorithm into typical standard structures allows to minimize the efforts of the programmer spent for testing. The ban on nested structures is just explained by the excessive costs of testing. Using a chain of simple alternatives with one action or structure SELECT instead of nested simple ALTERNATIVES greatly reduces the number of tests.