System Verification and Validation

The terms system validation and verification refer to two basic concerns, "are we building the right product?" and "are we building the product right?" Satisfactory answers to both questions are a prerequisite to customer acceptance.

Established Approaches to Validation/Verification in Systems Engineering

In systems engineering, planning for validation and verification begins in the early stages of "requirements development."

[Goals-Scenarios-Requirements]

Figure 1. Role of Goals/Scenarios in Generation of Requirements

Figure 1 shows the pathway from goals/scenarios through to the validation/verification of scenarios.

Because most systems are multifunctional, many scenarios may be needed to describe the system's intended behavior completely. Input from multiple stakeholders heads to a shared view of the system goals.

Figure 2 shows the high-level components in the iterative development and evaluation of system.

[OO Behavior]

Figure 2. Iterative Procedure for Testing/Validation/Verification

Each iteration begins with a description of desired behavior, possibly expressed as a series of messages between objects in a sequence diagram, and ends with an equivalent description of the "as modeled" system. At the end of each iteraction, the "desired" and "as modeled" behaviors are compared.

How Traceability helps Verification/Validation

Traceability mechanisms support the capture and usage of trace data (i.e., to document, parse, organize, edit, interlink, change, and manage requirements and traceability links between them.

[Requirements: Fig 1]

Figure 3. Mechanisms of Traceability in Requirements and Design.

Users view traceability as a transformation of requirements documents to design. The main applications of traceability are: requirements decomposition, requirements allocation, compliance verification, change control. See Figure 3.

Compliance Verification Procedures (CVPs)

Procedures for requirements verification are an intimate part of traceability.

[Traceabilit: Fig 2]

Figure 4. Metamodel for Low-End Users of Traceability.

Compliance and verification procedures (CVPs) are developed to ensure that each requirement is satisfied.

If a requirement cannot be tested, then by definition, it is no longer a requirement.

Need for Early Concept Validation/Verification

The complexity of engineering systems is rapidly approaching where it will be impossible to verify correctness of the design without also introducing a verification-aware discipline in the design process.

[Reuse Maturity's]

Figure 5. Pathways of Development, Now and in 2010-2020

Looking ahead, a challenge we face is partitioning the validation/verification process into simplier problems/procedures that can be applied as early as possible in the design process. For example:

  1. System Modules. .....

  2. Of the Logical Design. .....

  3. Of the Physical Design. .....

  4. Of the Overall System. .....

Formal Methods

A formal method (FM) is a set of techniques and tools based on mathematical modeling and formal logic used to specify requirements and design for computer systems and software. The use of FM on a project can assume various forms, ranging from occasional mathematical notation embedded in English specifications, to fully formal specifications using specification languages with precise semantics. At its most rigorous, FM involves computer assisted proofs of key properties regarding the behavior of the system.

Formal methods can benefit system design in two ways:

  1. Concepts and notations from mathematics can provide methodological assistance, facilitating the communication of ideas and the thinking process,
  2. Formal methods allow us to calculate some properties of a design.

There are two families of formal methods that have been developed over the last 20 years, both have their advantages and drawbacks:

  1. Model-oriented techniques such as Z, VDM and B Method. With such methods the model is correct by construction,
  2. Automatic verification of models with model checkers. The behavior of the system is verified after the fact.


Systems Engineering View of Testing

Currently there are two major approaches to system validation and verification: (1) testing and (2) system inspection. Both appraches try to identify faults in a system.

The most common form of system verification is testing -- that is, given the specification of a system, test cases are created which embody the fine microscopic facets of the specification.

[Test Procedure]
Figure 6. A Systems Engineering View of Testing

Each test case specifies a situation (i.e., a set of input data) at what is supposed to happen in that situation (i.e., a set of output data).

Test Design

Test design requires the solution of problems similar to those encountered in the development (analysis, design) of a system. The key steps in test design development are as follows (Binder, 2001):

  1. Identify, model and analyze the responsibilities of the system under test;

  2. Design test cases based on this external perspective;

  3. Add test cases based on system/code analysis, suspicions, and heuristics;

  4. Develop expected results for each test case or choose an approach to evaluate the pass/no pass status of each test case.

Test Execution/Automation

Test execution is concerned with the application of the tests on the system. Typically, test execution will involve the following steps (Binder, 2001):

  1. Establish that the implementation under test is minimally operational by exercising the interfaces between its parts, components, and/or sub-systems.

  2. Execute the test suite; the result of each test is evaluated and classified as pass or no pass.

  3. Use a coverage tool to instrument the implementation under test. Rerun the test suite and evaluate the reported coverage.

  4. If necessary, develop additional tests to exercise uncovered system functionality (or code).

  5. Stop testing when the coverage goal is met and all tests pass.

A test automation streamlines (the many steps in) the testing procedure.

Dealing with the Outcome of a Test?

When a test fails, then we can say with complete confidence that a failure has been detected.

But what can we say when a system passes a test? At the very minimum, we can say that the system passes the test for the specific specification (i.e., input data). The benefit of this observation can be small....

Types of Testing

For the development of complex systems, test procedures can be simplified through the use of separate test procedures for separate concerns. The concerns (and their corresponding test procedures) include:

  1. Module/Unit Testing.

  2. Integration Testing.

  3. System Testing.

  4. Acceptance Testing.

System Test and System Test Patterns


References and Web Resources

  1. Binder R.V., Testing Object-Oriented Systems: Models, Patterns and Tools , Addison-Wesley, 2000.
  2. Clarke E.M., Kurshan R.P., "Computer-Aided Verification," IEEE Spectrum, June, 1996.
  3. Waters R., "System Validation via Constraint Modeling," MIT AI Laboratory, 545 Technology Sq, Cambridge, MA 02139.

Developed in October 2002 by Mark Austin
Copyright © 2002, Mark Austin, University of Maryland