Why follow a Process?

An extensive body of experience collected from industry indicates that nearly two-thirds of software development projects in the U.S. fail, either through cancellation, overrunning their budgets or delivery of software that is never put into production (see Graham, Section 9.1). The reasons for these failures in descending order are as follows:

1. Lack of user (i.e., customer and stakeholder) involvement;
2. No clear statement of requirements;
3. No project ownership;
4. No clear vision and objectives;
5. Lack of planning.

The move from functional to object- and component-level development procedures does nothing to alleviate the need for good engineering and project management practices. By letting members of the development team know what's expected of them, and when, "systems engineering development processes" enforce a reasonable level of discipline on developers.

There is, of course, a tendency for a development team to simply "wing-it" in the hope that things will work out -- if it worked once, surely it can work again! From an organization's perspective, however, development processes need to be defined in the hope of achieving more than isolated islands of success. Organizations and project stakeholders want successes to be repeatable. This provides a basis for estimation and measurement of process quality.

A good "systems engineering development processes" should impose discipline on developers without hampering productivity through deliverables that fail to contribute to the product's end purpose. Almost every systems development effort varies in its specifics, and so developers need to have limited freedom to customize their work processes.

Integration of Product and Process Development

The adjacent diagram shows the essential details (and interaction) of the technical development process and systems engineering management process, and the transformation of customer requirements into a product specification.

Typically, the technical development process is concerned with: problem analysis; synthesis of the product architecture; evaluation and verification of system alternatives; optimization and trade-off.

The systems management process is concerned with: synthesis and organization of planning activities. This process generates plans and direction for the product development. In turn, the technical process for engineering system development provides outcomes and decisions for consideration in the systems engineering management process. Design of the systems engineering management process needs to account for uncertainities (e.g., performance; technical difficulties... etc) in the development activities, whether the product is "one-of-a-kind" or "mass produced," and whether the product is likely to be reused in figure developments.

Waterfall Model.

In an overly simplified view, system lifecycle development begins the gathering of requirements and domain knowledge and ends with system deployment, maintenance, and eventually, retirement.

Figure 2. Waterfall Model of System Development

Variations on the Waterfall Model include:

• Single-Pass Waterfall Model.

  The three main phases of development are analysis, design and build (or construction or implementation). The key components of each phase are:

  1. Analysis Phase. The analysis phase begins with the project's inception, and continues through the requirements definition. The latter may include a user-needs study (i.e., what does the customer really want ?) and a feasibility study (i.e., from a technical standpoint, is the project feasible ?).

  2. Design Phase. The design phase covers all aspects of the system design, logical design, physical design, and so forth. Often this phase is loosely defined, with design evolving as a series of progressive decompositions towards increased technical detail.

  3. Build Phase. Now the system is actually built, and integrated into neighboring systems, and tested for proper functionality -- in other words, does the system do what it is meant to ?

  The waterfall model was accepted as truth through most of the 1980s. Each phase of sequential development is completed -- via formal review -- before the next phase begins. The waterfall model is good when the problem and solution method are well understood.

• Iterative Waterfall Model

  Incremental development is defined as the development of a system in a series of versions or increments. At each increment, a subset of functionality is selected, designed, developed and implemented.` Additional increments extend the system functionality.

  Often the rework of system functionality can proceed with a reasonable amount of certainty that the desired result will be achieved (e.g., enhancements to software functionality). In these cases, this process can be modeled with a series of waterfall models.

• Limitations of Waterfall Model

  Changing requirements proved to be the biggest cause of cost overruns and schedule slips in the waterfall era. Users were found to be unable to define the requirements of a complex system without having had hands-on previous experience with the system -- A Catch 22.

Spiral Model

• Development Procedure

  The spiral model of systems development corresponds to a sequence of waterfall models ~\cite{boehm88}, and is displayed in Figure \ref{sys-eng-spiral}. This model corresponds to risk oriented iterative enhancement, and it recognizes that implementation options are not always clear at the beginning of a project. An implementation option may be uncertain, for example, because it critically depends on a technology still under development.

  

  Figure 3. Spiral Model of System Development

  The radial direction of Figure 3 corresponds to cumulative cost incurred, and the angular direction, progress made in completing each cycle of the spiral. Each cycle of development has the following phases:

  1. Identify the design and development objectives for the cycle, and the alternatives that are possible to achieve the goals.
  2. Evaluate different alternatives based on objectives and constraints. Where appropriate, identify uncertainties and risks.
  3. Develop strategies such as simulation, prototyping, and benchmarking, for resolving uncertainties and risks.
  4. Plan the next stage, allowing for any of the possible life-cycle models to be used.

  The initial spiral model release is a small subset of the anticipated system. Subsequent releases add capability to previous releases (each release is developed using the waterfall model). In some application domains, early releases have been called rapid prototypes.

  A key characteristic is assessment of management risks at regular stages in the project, and the initiation of actions to counter these risks. Before each cycle risk analysis is initiated and at the end of each cycle, at review procedure assesses whether or not to proceed to the next loop in the spiral.

  Risk is a concept that is sometimes difficult to quantify -- in many cases, a good way of interpreting risk is simply ``something that can go wrong.'' Risks are often the consequence of incomplete information, or perhaps, the result of human error(s).

• Limitations of Spiral Model

  Unless each release is developed with discipline and standards, the spiral model can easily become corrupted. The need for rapid prototyping is often used as an excuse for undisciplined and sloppy work during the early stages of a project. Finally, early releases should not be treated as through-aways.

System Lifecycle Development at GE

Systems Engineering Managment Process

Figure 4. Systems Engineering Management Process at GE

Key Technical Process

Figure 5. Key Technical Process at GE

The waterfall and spiral models of development lend themselves to a functional decomposition approach to design that follow a top-down systems design. The limitations of top-down systems design include ~\cite{meyer88}:

• Top-down design does not take into account evolutionary changes.
• In top-down design, the system is characterized by a single function. This is a questionable concept?
• Top-down design is based on a functional mind-set. The underlying data types (or data structures) are often ignored.
• Top-down design by itself does not encourage reusability. The latter is handled by bottom-up synthesis of previously developed components/concepts.

Item 1 is perhaps the most serious shortcoming of the waterfall and spiral models. With engineering/computing applications rapidly becoming more complex, and businesses being forced to reorganize in order to remain competitive in global markets, an ability of an engineering process to adapt to change has recently become of paramount importance.

The purpose of the adjacent figure is to show how an arrangement of modules may change during iterations of the design-requirements design-requirements cycle. Examples of change include shifts from centralized to decentralized (i.e. mainframe to distributed) computing; text based to graphical/multimedia computing. Similar examples of change can be found in the way requirements are handled, and in the structure organizations take in their day-to-day work practices.

Unlike the traditional models life cycle development, object oriented life cycle development aims to provide a seamless process between different stages of the life cycle.

Building a Library of Reusable Components

A key goal of the object oriented life cycle is development of a knowledge base library containing reusable and pluggable components.

Figure 7. Pathway of Development in Reuse-Focused Design

The library components may be very general in nature (e.g., architectures; process plans; procedures; designs), and may impact different phases of a product's planning, design, and production. Points to note are as follows:

1. The object oriented paradigm delays component implementation and specification until a much later stage of the development process. Because high-level procedures are no longer ``frozen'' at the high-level systems design, a system based on the object representation can remain more flexible since changes at the implementation stage may not require functional changes in the system design itself.

   And then components are developed in accordance with the principle of information hiding. For the development of individual modules/object, the tools and techniques for top-down decomposition may be very important, however. This pathway of hybrid object-oriented/structured development is shown in the adjacent figure.

2. In the area of object-oriented software engineering, key items in proposed models include:

   • New life cycle model needed because of OO difference. For example;

     • Object oriented development promotes reuse through class inheritance.
     • Object oriented structure where objects are loosely coupled self contained entities, which communicate via message passing.

   • Use top-down systems analysis to identify object classes, then lower level objects.

   • Concurrent bottom-up development of classes with reuse from libraries of source code from objects.

     • Very iterative top-down bottom-up process to aggregate and generalize classes and refine requirements (for details, see Chapter 3).
     • Classes developed fairly independently.

   Opponents say more effort will be required in the design and implementation phases than with non-OO methods and life cycle. The claimed pay-off -- and this remains to be seen -- is in maintenance savings. With maintenance consuming 70\% of software life cycle costs, this is the right place to target productivity improvements.

The SEI-Capability Maturity Model is a framework for understanding and improving the process of developing software. This model was developed at the Software Engineering Institute (SEI) at Carnegie Mellon University, under the financial sponsorship of the U.S. Department of Defense (DoD).

The maturity model provides a formal methodology for assessing an organization's capability in software engineering.

• The assessment consists of 100 plus questions. A list of sample questions for systems engineering is appended to this chapter.

• On the basis of question answers, an organization is determined to be at one of five levels of software development maturity.

• A key motivation for organizations to reach at least level 3 is that it is a major factor is source selection of large projects.

Process Categories and Key Process Areas

As already indicated in the earlier chapters, broadly speaking, the Systems Engineering Domain is composed of three activities and/or categories. They are:

1. Organizational

   This category focuses on organizational-wide activities, and covers business level issues such as policy definition, training, technology management and support.

2. Systems Management

   This category focuses on program management related activities, such as project planning and monitoring functions.

3. Engineering Process

   This category focuses on technical activities of the systems engineering domain -- it is project oriented and covers specific steps found in the systems life-cycle domain.

Each process category contains a set of Key Process Areas (KPA's). The KPA's identify a group of related activities which when performed properly contribute to an organization's ability to consistently follow and improve its (software) engineering process. Process capability is the range of results expected from following the implemented process (i.e. predicts future outcomes). And finally, process performance characterizes the actual results.

Table 1 lists the Key Process Areas by Process Category. Figure 8 shows the relationships among concepts used in this context. And Figure 9 shows the relationship between the Process Category, the Key Process Areas, and how the goals of an organization (i.e. what is to be accomplished) influence its overall capability. Each goal can be achieved in a number of ways.

Figure 8. Framework of SEI Process Maturity

Figure 9. Process Categories and Key Process Areas

Examples of KPA's. Here are a couple of examples of Key Process Areas and their goals:

System Planning

System planning is the identification of program requirements, classified into technical and program (process) requirements, to define the technical program that will bring the system into being.

1. Goal 1. To define and document a technical program structure which implements the requirements necessary to provide a product which satisfies the stated needs.

2. Goal 2. To provide for integrating the technical program elements in a top-down, life-cycle, low-risk approach in an efficient and effective manner.

Configuration Management

Configuration management includes the identification, control, accounting, and auditing of the product elements which consist of requirements, interfaces, and design representations of the product being developed to meet the stated project objectives. Configuration control is important for keeping the design characteristic in-line with the requirements base-line, for assessing the impact of change, and for correcting problems identified with the requirements.

1. Goal 1. To provide for the planning of configuration management activities and tasks.

2. Goal 2. To ensure configuration work products are identified, controlled, and available.

3. Goal 3. To ensure configuration changes are controlled and evaluated consistent with the technical and program requirements.

4. Goal 4. To communicate the compliance status and contents of the configuration work products.

Note. A complete listing of Key Process Area descriptions, their goals, and questions may be found in Volume II of the NCOSE Working Group Paper, San Jose, California, August, 1994.

Five Levels of SEI-Maturity

Figure 10 summarizes the contents of the SEI Maturity Model.

Figure 10. SEI-Capability Maturity Model

The key points for each level are as follows:

1. Ad hoc. Activities are ad hoc and chaotic, and success depends on the capabilities of individuals.

   • Unstable environment lacking sound management practices -- commitments not controlled.

   • Success rides on individual talent and heroic effort.

   • Standards and practices are often sacrificed to management priorities -- usually schedule and/or cost.

   • Process capability is unpredictable -- schedule, cost, and performance targets are rarely met.

   • Level of risk is consistently under-estimated.

2. Repeatable. Repeatability introduces basic project management mechanisms, such as schedule tracking, that allow the organization to repeat past successes on new projects that are similar.

   • Management discipline ensures that during schedule crunches systems engineering practices are still enforced (no cheating!!).

   • Basic system management discipline is installed.

   • Previously successful processes are repeatable in a stable, managed environment.

   • Achievable and measurable activities are planned prior to making commitments -- commitments are tracked.

   • Before schedule commitments are made, a process capability exists that enables the team to meet schedules.

3. Defined. Defined means that the systems/software process documents so that different projects in the organization can use the same overall process.

4. Managed. Managed adds more details of software/systems quality.

5. Optimized. Optimized has additional mechanisms that allows the process to be continuously improved.

Risk versus Quality Trade-Off

A key component of Figure 10 is the trade-off in risk versus quality as a function of organizational processes in place. Generally speaking, the SEI Maturity Model encourages organizations to put in place well managed, repeatable, organizational processes because evidence shows this results in reduced risk of project failure (e.g. due to late delivery/cost overruns). The stake-holders and customers benefit from increased overall quality of a project/product.

It is important to observe that "capability" belongs to the organization and not a particular project -- this is reflected in an SEI comment:

• ``While low-maturity organizations look upon talented staff as the best way of saving troubled projects, mature organizations look on talented staff as the best way of transferring culture and methods to new applications.''

The underlying philosophy for characteristics of the five levels is to look at features of the software engineering process in terms of (a) engineering procedures, (b) process performance, and (c) engineering style.

The maturity of the engineering process is rated according to the degree of systematic approach, and of support by state-of-the-art tools and automation. The maturity of process performance is judged by the deviation of the actual process results from the planned goals of productivity and quality. The higher the maturity level, the lower the risk of project cost overruns.

Systems Engineering Maturity Model

The SEI Maturity Model is now being crafted into a Systems Engineering Maturity Model (SEMM) -- the same underlying principles and features apply (e.g five levels of maturity; reduced risk with increasing maturity level). A few minor modifications exist, of course. For example, at level 3, training has been extended to education and training. At level 5, the benefit of ``increased productivity and quality'' in the SEI model is replaced by a more general view of quality and productivity. Systems engineering at a high maturity level delivers products and services which match the needs of the customer, and does so reliably within the narrow goals of cost and time schedules.

Incentive to comply with SEI

The Department of Defense has recently stated that no defense contractor will get defense contracts unless the organization is at least at Level 3. This proclamation provides contractors with a strong incentive to understand the model and demonstrate that their company satisfies the minimal requirement. Very few organizations have so far reached levels 4 and 5 (only 2 so far).

Case Study 1

Hughes Aircraft Company is at Level 3. See paper by ....

Case Study 2

IBM Federal Systems (sold to Loral in 1994) has received a Level 5 rating (by the authors of CMM) for the Space Shuttle's flight control software. It consists of 420,000 lines of code running on 5 redundant computers. The project has been in existence for 20 years, and in that time, defect rates has decreased by two orders of magnitude. There have been 17 releases of the software. Costs has been predicted within 10 percent of budget, and they have missed only one deadline in the last 15 years.

REFERENCES

WEB RESOURCES

Developed in February 2001 by Mark Austin
Copyright © 2001-2002, Mark Austin, University of Maryland