Software Testing and Quality Assurance

Testing can show the presence of bugs, never their absence (Dijkstra). It is still the main way we gain confidence that software works, and it takes about 40 % of development cost.

This unit covers verification and validation, black-box and white-box techniques (with executed examples), the levels of testing, test case design, and the quality-management activities and standards that surround testing.

6.1 Verification and validation

• Verification: "Are we building the product right?" Does the software conform to its specification? (reviews, inspections, static analysis, testing).
• Validation: "Are we building the right product?" Does it meet the customer's real needs? (acceptance testing, prototypes).

Static techniques examine the artefacts without executing them: inspections (Fagan: a formal, checklist-based review of code or design by a trained team with moderator, author, reader and recorder), walkthroughs, desk checks and static analysers.

Inspections find defects early and often 60 % or more of them. Dynamic techniques execute the program with test data.

Testing vs debugging. Testing finds the presence of a failure; debugging locates and removes the cause. A test case = inputs + preconditions + expected output; a fault (bug) in the code causes a failure when executed; an error is the human mistake behind it.

6.2 Black-box testing

Test against the specification, ignoring the code ("functional testing"). It finds missing functions, interface and behaviour errors.

Equivalence partitioning

Divide the input domain into classes in which the program should behave the same; test one value from each class. For an age field that must accept 18 to 60: valid class 18–60, invalid classes below 18 and above 60, (and non-numeric).

This cuts thousands of inputs to a handful.

Boundary value analysis (BVA)

Errors cluster at the edges (the classic < instead of <=). Test just below, on, and just above each boundary. For 18–60 the boundaries are 18 and 60, giving 17, 18, 19, 59, 60, 61. Real run of a valid_age(x) function (18 <= x <= 60) on these values:

All six behave as specified. Had the code been 18 < x < 60, the cases 18 and 60 would have exposed it immediately, while a mid-range value such as 35 never would.

Other black-box techniques: decision tables, state-transition testing, use-case testing, error guessing, random testing.

6.3 White-box (structural) testing

Test the internal logic with knowledge of the code.

Coverage criteria, weakest to strongest: statement coverage (every statement executed), branch/decision coverage (every decision both true and false), condition coverage (every atomic condition both true and false), path coverage (every path, usually impossible for loops).

Cyclomatic complexity and basis path testing

McCabe's cyclomatic complexity V(G) equals the number of linearly independent paths through the control-flow graph, and so the minimum number of tests needed to execute every statement and every branch:

• V(G) = E − N + 2P (E edges, N nodes, P connected components, normally 1);
• V(G) = number of decision points + 1, where each if, elif, loop and each and/or counts as one (a compound condition with k operands counts k − 1 extra decisions);
• V(G) = number of regions of the planar flow graph.

Interpretation: 1–10 simple and low risk; 11–20 moderate; 21–50 complex; above 50 untestable. Many teams limit functions to 10 and refactor above it.

def grade(m):
    if m < 0 or m > 100:
        return "invalid"
    elif m >= 80:
        return "A"
    elif m >= 60:
        return "B"
    elif m >= 40:
        return "C"
    return "F"

For a larger function the number grows quickly. A triangle classifier has the following structure:

def classify(a, b, c):
    if a <= 0 or b <= 0 or c <= 0:
        return "invalid"
    if a + b <= c or a + c <= b or b + c <= a:
        return "not a triangle"
    if a == b and b == c:
        return "equilateral"
    if a == b or b == c or a == c:
        return "isosceles"
    return "scalene"

Counting its decisions (each or/and operand counts) gives 11 decision points, so V(G) = 12: twelve independent paths are required for full basis-path coverage.

The five tests below reach every return statement, but they are not a basis set (they do not isolate every condition), which is why a complexity of 12 signals that the function is borderline and should be split.

• Alpha testing: by the customer at the developer's site, with the developer observing and recording problems, in a controlled environment.
• Beta testing: by a sample of real users at their own sites, without the developer present, in an uncontrolled environment, who report problems back; widely used for products (public beta of apps).
• Regression testing: re-running previous tests after a change to check that nothing that worked has broken. Because it is repeated many times, it is the best candidate for automation, and a continuous-integration server runs the suite on every commit.
• Other kinds: smoke testing (is the build stable enough to test further?), performance/load/stress testing, security (penetration) testing, usability testing, recovery testing.

6.5 Design of test cases

A good test case has a high probability of finding an undiscovered fault. A test-case record has: ID, objective, preconditions, input, expected result, actual result, pass/fail.

Strategy: derive tests from the specification with equivalence partitioning and BVA, supplement with white-box tests for uncovered branches, add tests for every defect fixed (regression), and prioritise by risk.

Test-driven development writes the test before the code.

Test adequacy. Track coverage (statement/branch %), defect-removal efficiency DRE = defects found before release ÷ (found before + found after). With 80 defects found internally and 20 found by users, DRE = 80 %.

Mutation testing and code-coverage tools judge test quality.

6.6 Quality management

Quality assurance (QA) establishes a framework of organisational procedures and standards that leads to high-quality products; quality planning selects procedures and standards for a project; quality control (QC) checks that those are followed (reviews and tests).

Quality is conformance to requirements (Crosby) and fitness for use (Juran).

Product quality and process quality

• Product quality attributes: correctness, reliability, efficiency, integrity/security, usability, maintainability, testability, portability, reusability, interoperability (McCall; ISO/IEC 25010 lists functional suitability, performance efficiency, compatibility, usability, reliability, security, maintainability, portability).
• Process quality: a good process is the best predictor of good products in repeatable development (but is less reliable for design-intensive work). Improving the process is the basis of CMM and ISO 9000.

Reliability measures

MTTF (mean time to failure), MTTR (mean time to repair), MTBF = MTTF + MTTR, availability A = MTTF/(MTTF + MTTR), failure rate λ = 1/MTTF (constant-rate model), and reliability R(t) = e^{-\lambda t}.

Standards: ISO 9000

ISO 9001 is an international standard for a quality management system (QMS): a documented process (plan, do, check, act) covering responsibilities, documented procedures, internal audits, corrective action and continuous improvement.

A company is certified by an external audit. ISO 9001 shows that an organisation follows its own documented process; it does not by itself guarantee product quality. ISO 9000-3 (now guidance for ISO 9001) interprets it for software.

Capability Maturity Model (CMM)

The SEI's CMM (now CMMI) assesses the maturity of a software organisation's process on five levels:

Moving up takes years: a typical step takes 18 to 24 months. Most offshore software companies seek level 3 or higher (CMMI) to win export contracts.

6.7 Testing in pictures: from the first bug to the V-model

Testing cannot prove that a program is correct, but it is the main way of finding out how wrong it still is.

Plate 6.1 shows three real objects of testing history; Figures 6.1 to 6.5 draw the V-model, the choice of black-box test data, the control-flow graph used for basis-path testing, a reliability curve and the five levels of the Capability Maturity Model.

(a) First recorded debugging

(b) Cable tester: input, expected output

(c) Ariane 501 fragment

6.7.1 Eight principles of software testing

1. Testing shows the presence of defects, not their absence. Even after many passed tests a program may still contain faults. Testing reduces the chance of undiscovered defects but can never prove that none remain. The honest report is "no defects found by these tests", not "the program is correct".
2. Exhaustive testing is impossible. A function with two 32-bit inputs has about 1.8 x 10 to the power 19 input combinations, which no computer can try. Testers therefore choose a small set of effective tests using equivalence classes, boundary values and risk, as in Figure 6.2. Choosing wisely is the skill of testing.
3. Test early. Testing activities begin with the requirements, not after coding, as the V-model in Figure 6.1 shows. A fault found in requirements costs a fraction of one found after release (Figure 1.1). Reviewing the SRS and writing acceptance tests early prevents many defects.
4. Defects cluster. A small number of modules usually contain most of the defects, often following the 80-20 rule. Complex or frequently changed modules, found with metrics such as cyclomatic complexity, deserve extra testing. Past defect reports show where to look.
5. The pesticide paradox. If the same tests are repeated, they stop finding new defects, as insects become resistant to one pesticide. Test cases must be reviewed and new ones added regularly. Regression tests protect old behaviour, but new tests find new faults.
6. Testing depends on context. A game, a hospital system and a banking app are tested differently. Safety-critical software needs formal coverage and independent review, while a short-lived website needs fewer formal tests. The level of testing follows the level of risk, such as the harm a fault can do to a ConnectIPS payment.
7. Absence of errors is not enough. A system with no defects can still fail if it does not meet the users' needs. Validation with real users, as in acceptance testing, checks that the right product was built. Passing all the tests of a wrong specification gives a wrong product.
8. Independent testing finds more. Authors tend to test what they intended, so an independent tester or team sees what the author misses. Independence can be a colleague, a separate test team or an outside company. It is most important for critical software.

Formula and fact sheet

• V(G) = E − N + 2P = decisions + 1 = minimum tests for branch coverage.
• BVA: test min−1, min, min+1, max−1, max, max+1.
• Alpha: developer's site. Beta: users' sites. Regression: re-test after change.
• DRE = found before release / (found before + found after).
• A = MTTF/(MTTF + MTTR); R(t) = e^{-\lambda t}, λ = 1/MTTF.
• CMM: Initial, Repeatable, Defined, Managed, Optimising.

Practice

1. Distinguish verification from validation and testing from debugging.
2. Apply equivalence partitioning and BVA to a field accepting marks between 0 and 100.
3. Compute V(G) for the grade function above and list a test for each independent path.
4. Differentiate alpha and beta testing; why is regression testing automated?
5. Explain top-down and bottom-up integration testing; what are stubs and drivers?
6. What is the difference between statement coverage and branch coverage? Give an example in which 100 % statement coverage misses a bug.
7. A system has MTTF = 500 h and MTTR = 5 h. Find availability and the probability of surviving 50 h.
8. 120 defects were found before release and 30 after. Compute the defect-removal efficiency.
9. Describe the five levels of CMM.
10. What does ISO 9001 certification guarantee?

Model answers: medium (5 marks) and long (10 marks) questions

Q1 (5 marks). Explain equivalence partitioning and boundary value analysis. Design test cases for a field that accepts marks from 0 to 100.

Introduction. Both are black-box techniques, which test the behaviour of the program from the specification without looking at the code (section 6.2).

1. Equivalence partitioning. The input domain is divided into classes in which the program should behave the same way, and one test per class is enough. Marks have three classes: below 0 (invalid), 0 to 100 (valid) and above 100 (invalid) (Figure 6.2).
2. Test data for the classes. -20 (rejected), 50 (accepted), 130 (rejected): three tests.
3. Boundary value analysis. Faults concentrate at the edges of classes, so values on and next to each boundary are tested.
4. Test data for the boundaries. -1, 0, 1, 99, 100 and 101.

Conclusion. Together, the nine cases (three classes plus six boundaries) cover the specification at low cost, and they catch the common error of writing < where <= is needed.

Q2 (5 marks). Differentiate between verification and validation. Show the relation with the V-model.

Introduction. Verification asks "Are we building the product right?" and validation asks "Are we building the right product?" (Boehm).

1. Verification checks that each work product conforms to its specification, using reviews, walkthroughs, inspections and static analysis. It is done without running the program as a whole.
2. Validation checks that the finished product meets the real needs of the user, by executing it in acceptance and system tests, often with real users.
3. In the V-model (Figure 6.1) the left arm is verification of each phase against the one before it, and the right arm is validation through unit, integration, system and acceptance testing.
4. Example. Reviewing the SRS of a result system for missing rules is verification; letting examiners enter real marks and check the output is validation.

Conclusion. Both are needed: a product can be built correctly to a wrong specification, and a right idea can be badly built.

Q3 (10 marks). Explain white-box testing, statement, branch and path coverage, and basis path testing. For the function clamp(x) of Figure 6.3 draw the control-flow graph, compute V(G) and write the test cases.

Introduction. White-box (structural) testing designs tests from the internal logic of the code, so the tester must see the source. It complements black-box testing: black-box finds missing functions, white-box finds untested code.

Coverage criteria, weakest to strongest. Statement coverage executes every statement at least once. Branch (decision) coverage makes every decision true and false at least once; it implies statement coverage.

Condition coverage makes every simple condition true and false. Path coverage executes every path, which is usually impossible when there are loops.

Basis path testing is the practical compromise: execute a set of independent paths whose size equals the cyclomatic complexity.

The function.

def clamp(x):           # 1
    if x < 0:           # 2
        x = -x          # 3
    if x > 100:         # 4
        x = 100         # 5
    return x            # 6

Control-flow graph and V(G). Nodes: 6; edges: 1-2, 2-3, 2-4, 3-4, 4-5, 4-6, 5-6 = 7. V(G) = E - N + 2 = 7 - 6 + 2 = 3.

Check by decisions two decisions + 1 = 3. By regions: the graph has 2 inner regions + 1 outer = 3. So at least three tests are needed.

Basis paths and test cases (Figure 6.3).

These three tests give statement coverage (all six nodes) and branch coverage (each decision is true once and false once: x < 0 is true for P2 and false for P1 and P3; x > 100 is true for P3 and false for P1 and P2).

A fourth path, 1-2-3-4-5-6 (x = -200, output 100), is not independent: it is a combination of the others, but it is a useful extra test for the combined condition.

Advantages and limits. White-box tests find unreachable code, logic errors and missing branches, and the coverage can be measured automatically. They cannot find a missing requirement, and 100% coverage does not mean 100% correct.

Conclusion. Cyclomatic complexity tells the tester the minimum number of tests; for clamp(x) it is three, and each is a real test with a known expected output.

Q4 (10 marks). Explain software quality management. Discuss reliability measures, ISO 9000 and the Capability Maturity Model (CMM).

Introduction. Software quality management makes sure that the product meets its requirements and the process is under control.

It has three activities: quality assurance (define standards and processes), quality planning (select the standards for the project) and quality control (check that they are followed, through reviews and testing).

Product and process quality. Product quality is judged by the attributes of Figure 1.3 (maintainability, dependability, efficiency, acceptability).

Process quality is judged by how well the development process is defined and followed; a good process makes good products more likely, although it does not guarantee them.

Reliability measures (Figure 6.4). Probability of failure on demand for rarely used systems such as a safety shutdown; rate of occurrence of failures (ROCOF); mean time to failure (MTTF); mean time to repair (MTTR); availability = MTTF / (MTTF + MTTR).

For MTTF = 500 h and MTTR = 4 h: availability = 500 / 504 = 99.2063%, failure rate = 0.002 per hour and R(100) = exp(-100 / 500) = 0.8187.

Defect removal efficiency DRE = found before release / (found before + found after): with 90 found before and 10 after, DRE = 90 / 100 = 90%.

ISO 9000. A family of international standards for quality management systems; ISO 9001 specifies requirements for the process, and ISO 90003 is the guide for software.

It requires documented procedures, quality records, audits and corrective action. An ISO certificate shows that the company follows its own documented process; it does not certify the product itself.

CMM (Figure 6.5). Five levels: (1) Initial, ad hoc and dependent on individuals; (2) Repeatable, basic project management so that earlier successes can be repeated; (3) Defined, a standard process documented and used across the organisation; (4) Managed, processes and products are measured and controlled using data; (5) Optimizing, continuous process improvement using feedback and new ideas.

Conclusion. Quality is built into the process, measured with metrics such as reliability and DRE, and improved over time; ISO 9000 shows that a process exists and CMM shows how mature it is.