Virtual Manufacturing Technical Workshop

25-26 October 1994

Dayton Stouffer Hotel

Sponsored by

DoD Joint Directors of Laboratories, Manufacturing & Engineering Systems Subpanel

Wright Laboratory Manufacturing Technology Directorate

Hosted by

National Security Industrial Association, Dayton Chapter

Click here for Picture

Technical Report

Compiled and Edited by

Lawrence Associates Inc.

,

This report summarizes the issues, conclusions and recommendations generated at the Virtual Manufacturing Technical Workshop held in Dayton, Ohio on 25-26 October 1994. In addition, it contains the viewgraphs, breakout session reports and selected commentary from participants. The commentary contained in this report does not necessarily represent the views held by the Department of Defense or Lawrence Associates Inc.

Table of Contents

Appendices

List of Figures

FIGURE 2-1. VM VISION

FIGURE 3-1. CROSS-FUNCTIONAL TRADES

FIGURE 4-1. VM SOFTWARE FRAMEWORK

FIGURE 4-2. VM FRAMEWORK CENTERING ON CHARACTERIZATION

FIGURE 4-3. SMART-CAD TECHNOLOGY ROADMAP

FIGURE 4-4. TECHNOLOGY ROADMAP RELATED TO LEGACY SYSTEMS

FIGURE 4-5. STRAWMAN ARCHITECTURE

FIGURE 4-6. TECHNOLOGY ROADMAP

List of Tables

TABLE 1. BREAKOUT SESSIONS

TABLE 2. VM USER REQUIREMENTS

TABLE 3. VM TECHNICAL AREAS

TABLE 4. VM TECHNOLOGIES

TABLE 5. TOP TECHNICAL AREAS

1. Report Summary

Air Force ManTech, in coordination with the Joint Directors of the Laboratories (JDL), launched a Virtual Manufacturing (VM) initiative in order to facilitate realizing VM[[Otilde]]s potential benefits in defense manufacturing. The VM initiative has the potential to become a key component of the JDL[[Otilde]]s Manufacturing Science and Technology (MS&T) strategy. The first Workshop on Virtual Manufacturing was held on 12-13 July in Dayton, and focused on capturing the user views and requirements for VM. This information was presented in a technical report and formed the basis for the technology discussions at the Workshop which this report documents.

As a result of the requirements generated at the User Workshop, a Virtual Manufacturing Technical Workshop was held on 25-26 October 1994 in Dayton, Ohio. The purpose of the Technical Workshop was to explore the VM technologies in the context of the user requirements and generate a "technology roadmap" for VM. This report documents the proceedings of the technical workshop and introduces the VM Technology Roadmap.

The workshop was organized around four breakout phases of two hours each, with intervening general sessions of one hour each to share the information generated at each breakout session. An introductory plenary session set the stage. The phases were structured to move from breadth to depth of VM technologies, culminating in a consensus-driven technology roadmap. Each breakout session addressed the issues from slightly different perspectives.

1.1 The Technical Areas of VM

A strawman listing of technical areas and technologies was created based on the information generated at the User Workshop. These lists of nine technical areas and sixteen technologies were presented at the opening plenary session to set the stage for detailed technological explorations.

The participants modified several of the technical areas and added four during Phase I. The changes were not expansions of the scope of VM technologies, nor were they inconsistent with the intention of the participants at the user workshop. The changes made certain categories clearer and more refined (e.g., separating architecture from methodology ). Other additions were evident in the user workshop report, but were not clearly present in the strawman, so the changes raised their visibility (e.g., cross-functional trades and manufacturing process characterization). See section 3.2 for details.

The list of technologies of relevance to VM was expanded to over 40 during the workshop. The technologies were categorized by the participants according to whether they were a core, an enabling, a show-stopper or common technology relative to VM. Core technologies are those which collectively define VM, enabling technologies are those which must exist to realize VM, show stoppers are those which can preclude the realization of VM if insufficiently developed, and common technologies are those which are important to realize VM but are common to many other domains. See section 3.5 for details.

The workshop participants mapped the user requirements generated at the User Workshop to as many of the technical areas and technologies as time permitted. In addition, they identified the associated benefits, technical barriers, maturity levels, risks and timeframes to develop the technologies. Finally, they identified the key players and organizations working in the areas. Much of the detailed information generated about the technical areas and technologies is captured on the breakout session viewgraphs in Appendix E.

1.2 The Technology Roadmap

The primary objective for the Technical Workshop was to develop a Òtechnology roadmapÓ for VM. A technology roadmap accounts for industry, academic and government activities in VM, their priorities, the levels of investment, the funding sources, the timeframes for the activities, the nature of VM activities (i.e., research, application, prototyping, development, deployment), the state-of-the-art in VM technologies (including where will it be in 3 years, 5 years and 10 years), the required maturity level of individual technologies to enable the achievement of specific benefits, and the technological problems common to a wide variety of VM technologies.

The breakout groups developed technology roadmaps specific to the context on which they focused, including much of the supporting information. However, these were not integrated into a comprehensive technology roadmap at the workshop because of the limited amount of time and the complexity of VM. A skeletal version, created from the individual roadmaps, of a potential comprehensive roadmap is included in Figure 4-6.

The major finding of the workshop was the importance of three specific technical areas: an integrating infrastructure/architecture, representation and manufacturing characterization. The architecture area includes creating a framework for the interoperation of all VM technologies and defining an underlying infrastructure to share models. The representation area includes the technologies, methods, semantics, grammars and analytical constructs necessary to represent manufacturing information for VM. The manufacturing characterization area involves the techniques and methods for creating generic models of manufacturing processes based on actual shop floor data.

All groups recognized that constructing a VM architecture or framework is necessary to realize VM. Three main reasons emerged for this ranking. First, a well-defined architecture will help make VM distinctive from the many basic technologies which are necessary to realize VM. Second, it will fulfill a recognized need to technically scope VM and provide a logical, consistent basis for further VM investments. In other words, a well-defined architecture would provide technical guidance to the technology roadmap. Third, the effort to create the architecture would provide a mechanism for better understanding of the technical interdependencies of the technologies related to VM.

Many participants at the workshop also highly ranked the representation technical area. This area involves the representation of manufacturing knowledge at the process, design, manufacturing, engineering and enterprise levels. This is a highly dynamic area, at present, but tends to be done in isolation, hence, the integration of representations becomes an early issue. The broad applicability of VM throughout the weapon systems life-cycle would provide an opportunity to tackle the representation issues in an integrated fashion, overcoming the inherent problems of addressing manufacturing representation in isolation.

Progress in representation also demands significant progress in the manufacturing characterization technical area, a fact recognized by the ranking of this technical area in some of the groups. It is one of the most important areas for realizing VM because techniques and methodologies for collecting, collating and converting the data into knowledge do not exist at present.

1.3 Recommendations

The consensus of the workshop participants was that VM encompasses a significant technology set worthy of pursuit. Although many companies are investing in the area , and will continue to do so, DoD participation in a few key roles would enhance the collective realization of the VM benefits determined at the User Workshop. Specific roles identified were:

1.4 Concluding Remarks

A total of 64 individuals participated in the Technical Workshop. 14 were from government (Air Force, Army, Navy, Marines, DoE), 38 were aerospace contractors, 9 were from commercial software companies, and 3 were from universities.

VM is being actively researched and implemented. At the both the User Workshop and the Technical Workshop, over 50% of the participants responding to the survey indicated that VM is being prototyped or is a major thrust at their organization. However, the workshop issues documented in this report show that significant effort is necessary for the DoD to gain the benefits.

Thirty-one of the participants at this workshop also came to the User Workshop. Because it is essential that any VM initiative mounted by the government be responsive to the needs of the users, it is recommended that the findings of this workshop be reviewed and validated by users in the aerospace defense industry. This kind of validation will help assure that the VM tools which emerge from funded efforts will, in fact, be adopted and used.

2. Introduction

Under the auspices of the DoD Joint Directors of Laboratories Manufacturing and Engineering Systems Subpanel, a new initiative in Virtual Manufacturing is slated to be kicked off in FY95. The initiative is targeted at distributed modeling and simulation tools to reduce cost, risk and time for procuring DoD weapon systems.

The first step toward successfully launching the VM initiative was taken at a Users Workshop on VM, held in Dayton on 12-13 July 1994. The workshop was held to ensure that the needs and directions of those involved in and responsible for defense manufacturing are accommodated in the VM initiative. The participants concluded that VM is one of the key technologies which allows us to go beyond the assumptions driving the historic acquisition strategies because it provides four fundamental changes for defense manufacturing: (1)ÊVM can be used to prove the production scenarios, resulting in "pre-production hardened systems" (i.e. systems which are developed and verified but never undergo actual production runs); (2) VM can support the generation of more reliable estimates of production costs and schedule because the models are based on actual processes, not just parametrics; (3)Êmodeling and simulation (M&S) can significantly improve production flexibility, hence, reducing the Òfixed costsÓ; and (4) reliable predictions of costs, risk and schedule can substantially improve the decision making process of acquisition managers.

The second step in launching the VM initiative was conducting a Technical Workshop which was held at the Dayton, Ohio Stouffer on 25-26 October 1994. Based on the user functional requirements generated at the User Workshop, participants at the Technical Workshop were asked to identify key technologies, technical barriers, areas of opportunity, and key solution strategies and to prioritize these technologies both in terms of importance and timeframe to implement and/or develop. Participants at the Technical Workshop were expected to be familiar with technologies related to VM, have a good understanding of their organizational plans and activities related to VM, and have the ability to confidently assess the evolution of these technologies over the next ten years.

The purpose of this report is to summarize the issues, conclusions and recommendations generated at the VM Technical Workshop. A great deal of the workshop commentary was collected and summarized into this document in order to provide the reader a full spectrum of views generated at the workshop. The commentary was primarily collected during the breakout sessions, and the breakout session facilitators were responsible for summarizing and accurately reflecting the views of the participants (see Table 1 for the session topics and facilitator names). As a result, it is important to note that this commentary reflects the views, opinions and beliefs of various participants and is not necessarily consistent with the views of Department of Defense, the facilitators, or even the majority of the participants. This report also contains the viewgraphs used at the five plenary sessions and a list of the participants.

Table 1. Breakout Sessions

      Title                    Participant Profile                   Facilitator         
1.   Yellow    Two academics, three airframers, two vendors, two     R. Boykin           
               government, one missile/electronic prime, one other                       
2.   Red       Four defense aerospace manufacturers, two from        R. Thomas           
               NASA, two from DOE, two software vendors                                  
3.   Blue      Four airframers, one missile/electronic prime, one    W. Henghold         
               vendor, one academic, one service seller and two                          
               government                                                                
4.   Brown     Seven airframers, two electronics, one vendor, two    G. Peisert          
               software, one academic, and two government                                
5.   Green     Three airframers, one electronics manufacturer, two   J. Haynes           
               software vendors, two government                                          

Section 2 describes the processes used to conduct the workshop, the demographics of the participants, and key items from the User Workshop Technical Report that are helpful for this report. Section 3 captures some of the commentary from the workshop participants regarding the technologies, their maturity levels, risks, barriers, timeframes to implement and the linkages of the technologies of VM to the user requirements. Section 4 centers on the technology roadmap, including the identification of the technology interdependencies, the prioritizations generated by the workshop participants, and the skeletal comprehensive technology roadmap. Section 5 contains other conclusions and recommendations. The Appendices contain (A) a list of acronyms used in this report (B) the workshop agenda and invitation, (C)Êthe list of participants, (D) the opening plenary session viewgraphs presented by Michael F. Hitchcock of the Air Force Manufacturing Technology Directorate, and (E) the breakout session viewgraphs presented at the plenary sessions.

2.1 About the VM Technical Workshop

The purpose of the VM workshops is to lay the foundation for a VM initiative. The primary objective for the Technical Workshop was to develop a Òtechnology roadmapÓ for VM. A technology roadmap accounts for industry, academic and government activities in VM, their priorities, the levels of investment, the funding sources, the timeframes for the activities, the nature of VM activities (i.e., research, application, prototyping, development, deployment), the state-of-the-art in VM technologies (including where will it be in 3 years, 5 years and 10 years), the required maturity level of individual technologies to enable the achievement of specific benefits, and the technological problems common to a wide variety of VM technologies. The breakout groups developed technology roadmaps specific to the context in which they worked. These were not integrated into a comprehensive technology roadmap at the workshop because of the limited amount of time and the complexity of the VM technologies.

The workshop was organized around four breakout phases of two hours each, with intervening general sessions of one hour each to share the information generated at each breakout session. An introductory plenary session set the stage. The phases were structured to move from breadth to depth of VM technologies, culminating in a consensus-driven technology roadmap. Each breakout session addressed the issues from slightly different perspectives.

2.1.1 Breakout Phase I: Breadth

The main purpose of Phase I was to explore the breadth of VM technologies and to identify major technology areas that are important to achieving the Virtual Manufacturing goals described during the workshop introduction and overview. The exploration started with the strawman list of technology and VM architecture presented at the opening plenary session. Discussions involved the identification of important technologies, what it means for the technology to be important for VM, the current levels of activity for the major technologies, the key players, the risks, the matching to user requirements, how well each technology is expected to satisfy those requirements, and so on. Participants were asked to rank the technologies according to their potential for realizing VM benefits, but the ranking was generally delayed until Phase III.

Each session[[Otilde]]s revised/new strawman was collected, collated and used to focus the in-depth phase by the workshop participants. Each breakout session was asked to provide a list of technologies, matched to requirements, prioritized according to the user requirements and the benefits expected. These results are provided in the breakout charts in appendix E by breakout session.

2.1.2 Breakout Phase II: Depth, Isolated

The main purpose of Phase II was to explore a few selected technical areas in greater depth. Each breakout session was assigned a few technical areas (generally 3 areas) that were collated during Phase I. For each assigned technical area, each group was asked to identify major associated technologies, technological barriers, the technologies required to address each of those barriers, and an estimation of the maturity level for each of those associated technologies.

Phase II concluded with a look at strategies, timetables and potential players (ÒwhoÓ) for addressing the technology needs. This information was presented by each group at the concluding plenary session for the day, and their charts are contained in appendix E.

2.1.3 Breakout Phase III: Depth, Interdependencies

The technologies generated during Phase II were collected and organized during the evening in preparation for Phase III. The objective of Phase III was to recast the technology areas identified and examined in Phase I as a set of requirements (ÒWhatsÓ), develop more detail with respect to the associated or supporting technologies documented in Phase II (the ÒHowsÓ) and determine the interdependencies and potential DoD role in each of the associated technologies. In addition, each group was asked to categorize each technology as an enabling technology, a core technology, a VM "show-stopper" technology, or a technology common to other areas but nevertheless important to VM. The expectation was that this approach would enable exploration of the interdependencies of the technologies.

Phase III took significantly more time than originally planned. As a result, most breakout sessions did not have time to explore all the technology interdependencies.

2.1.4 Breakout Phase IV: Realizing VM

Because Phase III took more time than originally planned, the objective of Phase IV was revised to assess the criticality of each of the key technologies and determine the top three. The original purpose of developing strategies and approaches for realizing and/or implementing the VM technologies was not realized. Participants were asked to identify the low-hanging fruit (areas of high impact opportunity for government and industry investment) for implementation and/or development that significantly impact user requirements.

Each session discussed and picked the top three technologies that they thought should be pursued. These are documented in the conclusions section of this report.

2.1.5 Plenary Sessions

The basic information generated by each breakout group was captured on viewgraphs for presentation at the plenary sessions. Each group selected a presenter for each plenary session to report their results. Often, the presenters at the plenary sessions led the group discussions out of which the viewgraphs and presentations were formulated. This approach assured that each presenter understood the essence of each group's deliberations and was comfortable with the concepts described in the plenary sessions. Appendix E contains the viewgraphs presented at the plenary sessions (typed, and with minor style improvements).

2.2 Demographics of Participants

A total of 64 individuals attended the Technical Workshop: 38 represented the aerospace industry, 3 came from academia, 9 were software vendors and 13 attended from government (including all services and the DoE). The List of Participants is provided in Appendix C. In terms of the employment of those responding to the survey, approximately 39% were involved in research, 42% were in management, 42% were in engineering, 23% were involved with production, and 10% were in computing.1 In terms of experience with VM, 6% had little or no prior experience, 42% were investigating VM, 26% had a prototype implementation of VM underway, and 26% viewed VM as a major thrust at their organization.

2.3 Relevant Background from User Workshop

The purpose of this subsection is to summarize key information generated at the User Workshop necessary to explain the events at the Technical Workshop. The reader is encouraged to read the User Workshop Technical Report for a complete picture of the user requirements (Submit a written request the Air Force ManTech Technology Transfer Center, Fax: 513-256-1422; Phone: 513-256-0194).

2.3.1 Defining Virtual Manufacturing

The vision of Virtual Manufacturing is to provide a capability to ÒManufacture in the ComputerÓ (see Figure 2-1). In essence, VM will ultimately provide a modeling and simulation environment so powerful that the fabrication/assembly of any product, including the associated manufacturing processes, can be simulated in the computer. This powerful capability would take into account all of the variables in the production environment from shop floor processes to enterprise transactions. In other words, VM will accommodate the visualization of interacting production processes, process planning, scheduling, assembly planning, logistics from the line to the enterprise, and related impacting processes such as accounting, purchasing and management. Click here for Picture Figure 2-1. VM VisionThree overarching paradigms emerged during the User Workshop, and were used as the "accepted definition" of VM for the Technical Workshop. For each of these paradigms, a definition of VM was proposed to capture the view of VM within that paradigm. For each of these definitions, the term ÒManufacturingÓ should be construed in a broad sense to include not only production, but also suppliers, customers, and other processes that impact production (This broad sense is often referred to as Òbig-MÓ).Design-Centered VM: VM adds Manufacturing information to the IPPD process with the intent of allowing simulation of many Manufacturing alternatives and the creation of many "soft" prototypes by ÒManufacturing in the Computer.ÓA near-term definition: VM is the use of manufacturing-based simulations to optimize the design of product and processes for a specific manufacturing goal such as: design for assembly; quality; lean operations; and/or flexibility.A longer-term definition: VM is the use of simulations of processes to evaluate many production scenarios at many levels of fidelity and scope to inform design (product and manufacturing system) and production decisions.Production-Centered VM: VM adds simulation capability to manufacturing process models with the purpose of allowing inexpensive, fast evaluation of many processing alternatives.A near-term definition: VM is the production-based converse of IPPD which optimizes manufacturing processes, potentially down to the physics level. An example would be evolutionary re-engineering/optimization of a fabrication facility.A longer-term definition: VM adds analytical production simulation to other integration and analysis technologies to allow high confidence validation of new processes and paradigms. Examples would include revolutionary re-engineering of a processes or factory, and/or introduction of virtual corporation paradigms.Control-Centered VM: VM is the addition of simulation to control models and actual processes, allowing for seamless simulation for optimization during the actual production cycle.In summary, Design-centered VM provides Manufacturing information to the designer during the design phase. Production-centered VM uses simulation during production planning to optimize lines/factories, including the evaluation of processing alternatives (one would expect to do this sort of trade during IPPD, however, the evaluation during this phase has more to do with equipment and people availability). Control-centered VM uses machine control models in simulations, the goal of which is process optimization during actual production. Production-centered VM may or may not use actual control models for the simulation. Using them is desirable, however, this may not be possible because the models were not designed for simulation purposes or because they may simply be code without the knowledge/information necessary for simulation. In one sense, production-centered VM will "control" the factory because the factory will ÒoperateÓ according to the plan created with the assistance of VM. What Benefits Does VM PromiseThe User Workshop investigated and defined potential VM benefits from five different perspectives. At the highest level, VM is expected to realize the following benefits:AFFORDABILITY -- Reliable cost and process capability information that can impact key design and management decisions, and support balancing weapon system performance with manufacturing cost, schedule and risk.QUALITY -- More producible designs moving to the shop floor and higher quality work instructions to support production.PRODUCIBILITY -- First article production that is trouble-free, high quality, involves no reworks, and meets requirements. Optimize the design of the manufacturing system in coordination with the product design.FLEXIBILITY -- The ability to execute product changeovers rapidly, to mix production of different products, and to return to producing previously shelved products.SHORTER CYCLE TIMES -- Increased effectiveness of the IPPD process and the ability to go directly into production without false starts.RESPONSIVENESS -- The ability to respond to customer "what-ifs" about the impact of various funding profiles and delivery schedule with improved accuracy (credibility) and timeliness.CUSTOMER RELATIONS -- Improved relations through the increased participation of the customer in the IPPD process, lower costs, better schedule performance, improved quality, and greater responsiveness. What Are the Key BarriersParticipants at the User Workshop saw a number of key barriers to realizing VM. Although these are not all technical, they must be considered as part of the development and execution of the technology roadmap.A key portion of bringing VM into existence is to develop and quantify VM benefits as a part of the process. In so doing, these should be relate-able to currently used metrics (i.e., the metrics will not be revolutionary). They should show a way to relate VM benefits to specific product or system objectives as VM is simply a tool to achieve other objectives. The general process for metric development will follow that of VM in that benefits must be demonstrated, validated and recalculated in the new environment.Significantly improved cost estimating and collecting systems will be required to be able to deal with realistic cost comparisons at a detail and accuracy level that most current systems cannot support. VM may necessitate adopting radically different accounting practices from those in standard use today.Weapon system development program funding profiles must change to become more "front loaded" if significant VM is to be performed prior to production or prototyping. Weapon system development on a "pay-as-you-go" basis rather than development cost-shared by the contractor hoping to "get well" in production is essential..Broader government access and visibility into sensitive company areas could lead to the release of competition sensitive information. User Requirements on VMThe participants at the User Workshop identified a number requirements on the VM "system." The principle requirements which can potentially impact the technology roadmap are summarized below in Table 2.

Table 2. VM User Requirements

                 Description                                Comments                     
  1    Incremental implementation &      Adopting VM may require a strategic decision    
       adoption                                                                          
  2    A collection of small,            The tools should be processed based,            
       incrementally implementable       maintained by experts, not M&S people.          
       tools (building blocks) that                                                      
       are modular, pre-verified, &                                                      
       reusable                                                                          
  3    Integrate design and              VM should not simulate design, but support      
       manufacturing processes           it. VM should be able to evaluate partial &     
                                         complete designs. A formal structured           
                                         methodology for design abstraction is           
                                         necessary.                                      
  4    Agile to changes                  VM supports rapid verification of designs.      
                                         Changes in the physical environment should be   
                                         readily and easily reflected in the virtual     
                                         environment.                                    
  5    Reduction of product              Facilitates creation of "soft prototypes".      
       development cycle                                                                 
  6    Optimization of manufacturing     Investigate alternatives. Enhance risk          
       environment & assembly processes  management. Reveal key strengths,               
                                         deficiencies and voids in the manufacturing     
                                         processes.                                      
  7    Enhancement of design for         Predict schedule, cost & quality. Provide       
       producibility                     process capability and cost information to      
                                         guide the IPPD processes.                       
  8    Supportive of IPPD processes,     Reliably and verifiably link end-product or     
       focusing on cost                  component cost to specific design features      
       estimation/control                and tolerances. Supports trend toward           
                                         Activity Based Costing.                         
  9    Capture & retention of            Uses model-based manufacturing (in part).       
       knowledge, especially for         Supports the concept of "shelf technology".     
       training                                                                          
 10    Metrics that can achieve the      Metrics must be reliable, believable and        
       incrementally achievable          grounded in reality. Validation is essential.   
       benefits                                                                          
 11    Enhancement of management         Comparison of alternatives for capital          
       decision-making processes         investment. Support make/buy decisions.         

3. Technology Areas

3.1 Additional Commentary on Defining VM

The definitions of VM provided from the User Workshop were generally viewed as a reasonable starting point. Although many participants did not agree completely with the definitions, in the interest of time, not much discussion was allowed. A few key remarks about the definitions are provided here. · The fundamental long-range goal of VM is to be able to develop a product ÒvirtuallyÓ in a computer before committing it to hardware or prototype in the factory. The term Òproduct developmentÓ encompasses all of the various activities, both business and technical, associated with developing and producing a given product. However, VM does not simulate all of those activities. For example, VM does not simulate the design process. It supports the design process. VM does not simulate reliability or quality engineering, but in many cases a VM simulation may need access to design, reliability, quality and other kinds of information. The group recognized that if such a capability could be cost-effectively achieved, it could revolutionize product design and development.

3.2 Broad Technology areas

A strawman listing of technical areas was presented at the opening plenary session. As used in the plenary session, a technical area could contain one or more technologies of relevance to VM. These technical areas were offered as a starting point for the breakout sessions. Some breakout groups chose to start with the nine technical areas presented at the opening plenary session (slide 23), while others chose to start with the sixteen technologies identified at the user workshop (slide 19). Some groups chose to define and then lump together the specific technologies into larger, more comprehensive technical areas.

The different approaches and participant viewpoints resulted in different foci on the technical areas. Furthermore, as the detailed explorations continued, the groups refined these technical areas. Each breakout group was assigned two to three of these technical areas for detailed exploration.

The result of the workshop discussion was to suggest that underlying technologies critical to VM could be organized into at least thirteen major categories. Table 3 contains that aggregated list of technical areas. Note that some technologies are listed in more than one technical area, the differences being context and/or use of the technology. The ordering in the table generally corresponds to the technical area strawman, but really has no significance with respect to priority.

Definitions of the technical areas are presented below:

Figure 3-1. Cross-Functional Trades

Table 3. VM Technical Areas

        Technical Area      Other Descriptions,      Group        Commentary & Issues      
                           Technologies Involved                                           
T 1    Visualization      Virtual Reality            Brown                                 
                          Multi-context analysis                                           
                          & presentation GUI's                                             
T 2    Environment        VM System development       Not     Critical for advanced        
       construction       environment Shells       addressed  applications and potential   
       technologies                                           for a strong government      
                                                              role Shell linkages to CAD   
T 3    Modeling           Modeling technologies      Blue     Broken up into product,      
                          Model Libraries            Brown    process, information, and    
                                                     Green    activity Standards for       
                                                              data gathering Integrated    
                                                              distributed heterogeneous    
                                                              databases                    
T 4    Representation     Knowledge Based            Blue     Applicable at the process,   
                          Systems Modeling Rule      Brown    design, manufacturing        
                          based systems Object                engineering, and             
                          Oriented Expert                     enterprise levels            
                          Systems Physics based               Mathematical, statistical,   
                          process models                      ontological, chaotic,        
                          Feature-based models                finite automata,             
                          and simulations                     autonomous agents Software   
                                                              development environment      
                                                              Linkage to CAD Security,     
                                                              encryption Extends across    
                                                              enterprise operations. A     
                                                              highly dynamic area but,     
                                                              the products tend not to     
                                                              be integrated                
T 5    Meta-Modeling      Model component           Yellow    Generic models               
                          exchange                                                         
T 6    Integrating        Sharing VM Framework      Yellow    Includes                     
       infrastructure &   Integration                Blue     telecommunications           
       architecture                                  Green    infrastructure,  STEP        
                                                              based technologies, global   
                                                              co-location, groupware       
        Technical Area      Other Descriptions,      Group        Commentary & Issues      
                           Technologies Involved                                           
 T 7      Simulation       Optimization Analysis     Brown         Multi-Dimensional       
                          Simulation Tools Chaos     Green        Optimization (MDO)       
                            modeling capability                 Focusing manufacturing     
                             Software wrappers                    processes and cost       
                                                              Generally, a high level of   
                                                               activity and, at the unit   
                                                                  level, some of the       
                                                               simulations are well done   
                                                                  and mature. But in       
                                                                general, these are  not    
                                                               integrated into systems,    
                                                                 and because of their      
                                                               specificity, they may not   
                                                                 be easily integrated.     
T 8    Methodology        Characterize customer      Green    Methodology was deemed to    
                          requirements Cost                   imply a strong link to       
                          modeling                            culture                      
T 9    Legacy data,       Databases Programming      Green    Standards: accessibility,    
       models,            API's                               transportability,            
       information,                                           abstraction, update, V&V     
       hardware, &                                            Corporate culture            
       software                                                                            
T 10   Manufacturing      Representation of           Red     Cycle-time, yields, etc.     
       characterization   product/process Design              Often involves modeling      
                          by features                         Predictable processes Cost   
                          Manufacturing ontology              modeling Closed-loop         
                          System behavior Shop                design/mfg tools Contains    
                          floor data gathering                the ability to convey        
                          Models which describe               measures of uncertainty      
                          manufacturing processes             and risk.   The maturity     
                                                              level of products in the     
                                                              area is low and there is     
                                                              only a medium amount of      
                                                              activity under way.          
T 11   Verification,      Metrics, Virtual            Red     Cross system validation      
       Validation &       metrics Performance                 Methodology for validation   
       Measurement        Decision tools                      System performance           
                                                              (hardware/software related   
                                                              speed and accuracy)          
        Technical Area      Other Descriptions,      Group        Commentary & Issues      
                           Technologies Involved                                           
T 12   Workflow           Process control             Red     Although some products       
                                                              here are quite mature,       
                                                              they are limited in their    
                                                              range of applicability and   
                                                              are not integrated into      
                                                              systems control              
                                                              technologies.  One major     
                                                              problem here is that there   
                                                              seems to be little or no     
                                                              "user pull."                 
T 13   Cross functional   Product model              Blue     Product model                
       trade              representation                      representation standards     
       technologies       standards Function                  Semantics Function           
                          output integration                  automation may have the      
                          Function automation                 highest payoff,  but is      
                                                              probably much longer term    
                                                              in any generalized sense     

3.3 Matching Technical Areas to User Requirements

A portion of Phase I was devoted to discussing the mapping of the technical areas to the user requirements generated at the User Workshop. The purposes of the discussion were to ensure that user requirements were factored into the technical discussions, and to rank the technical areas according to the user requirements.

In general, each of the VM technical areas identified were found to match one or more of the user requirements. The match to user requirements necessitated a technical view built upon modularity, a building block approach , and incremental implementation (User Requirement #2). The key risk was in trying to link local, incremental progress to global Òcost ,schedule and qualityÓ benefits. Some technical areas were determined to be more relevant to satisfying user requirements than others. In particular, many participants suggested that the Manufacturing Characterization Technical Area had the highest potential for impact on user requirements. Again, the results of the discussions are contained in the viewgraphs in appendix E.

In developing the rankings during Phase I, different groups developed their own set of prioritization criteria. For some, however, independently ranking these technology areas made little sense because they are heavily interdependent. In general, individual choices were not measured on a criterion by criterion score assignment method. Rather the full set was used by individuals at arriving at ÒvotesÓ. Some of the criteria used were:

· Implementation timeline position

· Construction of a good foundation first, with visible benefits

· Palatability to management, with the weighing of effort, cost, and time in light of completion impacts and desired leadership position.

· Market pull impact

· Risk

· Whose dollars are being used

· Leveraging existing developments and plans.

3.4 Risks

The risks associated for each technical area was discussed during Phase I and are documented in the viewgraphs in appendix E. Highlights include:

3.5 VM Technologies

Technologies were viewed as to their granularity within the context of Virtual Manufacturing and classified into four categories (which overlap somewhat). During Phase III of the workshop, participants were asked to rate the technologies identified during the earlier phases as being a "Core Technology", an "Enabling Technology, " a "Show Stopper Technology", or a "Common Technology". Briefly,

· A "Core Technology" is one which is fundamental and critical to VM. In essence, the set of core technologies defines what VM can do.

· An "Enabling Technology" is one which is not core, but which is necessary to build a VM system.

· A "Show Stopper Technology" is one that, without which, a VM system cannot be built (at least in some context). That is, progress must be made in a show stopper technology to build the VM capability envisioned.

· A "Common Technology" is one which is widely used and needed in other domains, but which is important, perhaps critical to VM. Depending on functionality, a common technology can also be and enabler or show stopper.

Table 4 contains a listing of technologies identified during the workshop and an indication of its status according to the workshop participants (the items with an asterisk [*] were identified in the user workshop, and were offered at the plenary session on slide 19).

Table 4. VM Technologies

                          Technology                       Core   Ena-bl   Show   Com-mo  
                                                                    er     Stop     n     
1*    Computer characterization of manufacturing                    X     X               
      processes                                                                           
2     VM Methodology for Process Characterization           X                             
3     Technologies to simulate assembly operations          X                             
4     Generative Process Planning                                   X                     
5     Declarative Representation of Product and Process     X                             
6     Physics-Based Process Modeling                                X                     
7*    Methodology/protocol/grammar to represent                                     X     
      knowledge                                                                           
8*    Product Definition Models with variable                       X                     
      resolution levels                                                                   
9*    Object oriented, dynamic, functional languages                                X     
      and event-based modeling                                                            
10*   3-D surface based modeling                                    X                     
11*   Model component exchange                                      X                     
12*   Ability to integrate dynamic, distributed,                    X                     
      collaborative models                                                                
13    Meta-Models                                                   X                     
14    Neutral Language for VM Meta-Model                    X                             
15*   Cost database & integration                           X                             
16*   Large-scale integrated database structure                                     X     
17    Object Oriented Databases                                                     X     
18*   Data exchange standards, linkage of primes, subs,                     X       X     
      customer                                                                            
19*   Techniques to map metric information from tools               X                     
      to enterprise metrics                                                               
20*   Machine Intelligence                                                          X     
21    Knowledge Based Systems, Rule Based Systems                                   X     
22    AI Shells, Neural Networks, Fuzzy Set Theory                                  X     
23    VM User Interface (communication between VM           X                             
      knowledge system and user)                                                          
24    VM Verification & Validation Methods, Algorithms      X                             
      & Tools                                                                             
25*   Process model and simulation validation               X                             
26    Model verification tools                                      X                     

                          Technology                       Core   Ena-bl   Show   Com-mo  
                                                                    er     Stop     n     
27*   Software which is highly reconfigurable & modular                             X     
28    Chaos Theory                                                                  X     
29    Autonomous Agents                                                             X     
30    Computer hardware performance, high performance                     X         X     
      computing                                                                           
31    Networking/Communications                                                     X     
32    Display Technologies                                          X     X               
33*   Security, Encryption                                                X         X     
34    VM 3 Schema  (Semantics)                                      X                     
35    Methodology for using a VM system                     X                             
36    VM Framework (Guidelines, Integration Standards,      X                             
      etc.)                                                                               
37*   Methodology for design abstraction                    X                             
38    Conflict Resolution to achieve "best design"                  X                     
39    Tools to relate conceptual design with possible       X                             
      manufacturing methods and processes and cost                                        
      estimates based on manufacturing features                                           
40    Manufacturing engineering automation (knowledge       X                             
      based computer applications to perform                                              
      manufacturing engineering decision making)                                          
41    ÒSTEPÓ type technologies which would integrate                X                     
      and relate information between process phases                                       
42    Decision support tools for automated evaluation               X                     
      of decision impact                                                                  
43    Workflow Tools                                                X               X     
44    Simulation Architecture                               X                             

3.6 Technical Barriers, Maturity Levels and Resolution Strategies

The breakout groups addressed the technical barriers, the strategies to overcome those barriers, and the maturity levels of the technologies in their assigned technical areas during Phase II. The detailed information is contained in the breakout session viewgraphs in appendix E. In this section of the report, we highlight some of the groups' commentary regarding these issues.

4. Technology Roadmap

As mentioned in Section 2, the groups had insufficient time to develop a comprehensive technology roadmap for VM. However, all of the groups explored the interdependencies of the technologies and developed VM frameworks and roadmaps from the perspective of the technical areas on which they focused. Phase III discussions centered on the disambiguation of technology "whats" from technology "hows" and the exploration of the technology interdependencies. The resulting frameworks and roadmaps are highlighted in this section.

4.1 Technology Interdependencies

The major objective during Phase III was to explore and define the interdependencies among the technologies and/or technical areas. This exploration was intended to form a solid foundation for developing the technology roadmap. Relationships, where identifiable, were drawn between the core technologies, their enablers and show stoppers, and related common technologies. For some groups, the interdependencies are captured explicitly in their tables and graphics (Appendix E), for others the interdependencies are implied in their roadmaps (Appendix E and Section 4.3).

Exploring the interdependencies necessitated some discussion on a VM Framework or Architecture. Two of the groups developed and presented such figures. Figure 4-1 presents a software framework for VM and relates VM to other high order business systems. The message of the figure is that one would build specific VM systems (e.g., one for design analysis or manufacturing cost) using the technologies, organized hierarchically from meta-models through hardware. Example VM core technologies in each technical area are listed on the left. Interfaces must be provided to each VM system and to the technical areas (the boxes with an X on the right). Some participants expressed the view that the simulation models would need to extend through the Meta-Model to the Knowledge Representation System level. This view was not resolved, but it is not a show stopper for using the representation. Figure 4-2 presents a slightly different VM Framework which includes an emphasis on the role of manufacturing characterization.

4.2 Ranking of Technical Areas

For Phase IV, the groups were asked to assess the criticality of each of the technical areas and determine the top three, that is, identify the near-term technology hitters. (This is a different ranking from that used during Phase I to rank the technical areas).

The Integrating Infrastructure and Architecture Technical Area (Technical Area #6 in Table 3) was identified as one of the top three priority areas for VM effort by all groups. The consensus was that you really aren[[Otilde]]t going to build VM without an architecture. Furthermore, although there is a high level of research activity in this area, mainly software driven, its maturity level is low and deserves ManTech's attention in formulating its VM Roadmap.

Although constructing a VM architecture or framework is necessary to realize VM, the high ranking of this Technical Area by all groups has a greater basis. First, a well-defined architecture would help distinguish VM from the many common technologies associated with VM which are useful in a wide variety of domains. Second, a framework would help to scope VM technically and provide a logical, consistent basis for prioritizing further VM investments. Third, the effort to create the VM architecture would provide a mechanism for better understanding of the interdependencies of the technologies related to VM. That is, it would provide technical guidance to the technology roadmap.

Click here for Picture

Figure 4-1. VM Software Framework

Click here for Picture

Figure 4-2. VM Framework Centering on Characterization

The Representation Technical Area (Technical Area #4 in Table 3) also received a high priority from the participants. This area involves the representation of manufacturing knowledge at the process, design, manufacturing, engineering and enterprise levels. This is a highly dynamic area, at present, but tends to be done in isolation, hence, the integration of representations becomes an early issue. Tackling these representation issues in the context of realizing VM would offer clear guidance to those addressing the issues.

Finally, progress in the Manufacturing Characterization Technical Area (Technical Area #10 in Table 3) is a necessary precursor to progress in the Representation Technical Area, making this area a high priority for the VM initiative. Several problems were identified in this area that are of significance to VM. First, simply collecting data for characterization is expensive, and is generally viewed as a non-value-added operation. Hence, methods for justifying the expenses must be developed (Technical Area #11 in Table 3). Second, standards to guide the data gatherers and model builders do not exist. Cost effectively developing generic models for VM use may demand such progress. Third, methods for representing the knowledge of manufacturing characterization are isolated, a fact which re-enforces the need for work in the Representation Technical Area.

Several technologies were not rated high by groups because participants believed additional investments in those areas by VM proponents would not significantly impact their course. That is, although important, they are not expected to be significantly influenced by VM investments. Technologies in the "common" category of Table 4 generally are in this area.

Table 5 displays the top technical areas by group.

4.3 Breakout Group Technology Roadmaps

As mentioned before, each group was asked to develop a technology roadmap from the perspective of the technical areas on which it focussed. One group chose to pursue a path where VM should evolve to a Òsmart CADÓ capability which is built upon a symbolic representation product. CAD was highlighted as needing to relate to points, lines and arcs. Note that although the word "product" is used, in reality, process impacts are also considered a great deal in Smart-CAD. That roadmap is presented below in Figure 4-3. It is important to note that only one set of links in its technology roadmap was analyzed in depth and the rest of the map is thoroughly untested.

Another group captured their view of the interrelationships of the legacy systems. That roadmap is presented below in Figure 4-4.

Table 5. Top Technical Areas

Blue     (1) Common symbolic representation of the product (this is the focused    
         technical kernel) (2) Functional Output Standards (3) VM integration      
         infrastructure/architecture                                               
Brown    (1) Process Representation Specification (2) Manufacturing Feature        
         Representation (3) Cross-Functional Simulation Strategy & Optimization    
         (4) Cognitive Modeling (5) Framework for Modeling, Simulations and        
         Systems Interaction                                                       
Green    (1) Process/data identification, relationship, and intent (includes       
         information modeling, database technology, process modeling, grammars,    
         abstractions and standards) (2) Integration of existing systems/data to   
         allow seamless communication through organization (3) Visualization       
         through simulation, modeling & visualization techniques                   
Red      (1) Gathering, aggregating and properly analyzing empirical data for      
         evolution of generic models (2) Properly validate those models so that    
         they can be used with confidence to support design/manufacturing          
         decision-making processes (3) Integration, the creation of an             
         overarching framework into which all VM elements could be placed          
Yellow   (1) Representation of product/process knowledge (2) VM Framework (3) VM   
         3-Schema (4) Methodology for process characterization (5) Visual user     
         interface                                                                 

4.4 Skeletal Comprehensive Technology Roadmap

The Skeletal Comprehensive Technology Roadmap (referred to as the VM Roadmap in the following paragraphs) is composed of six major thrusts. Each thrust is designed to focus on specific requirements for VM as addressed in the two workshops, and yet take a wide swath through the technology areas in order to ensure that all defined requirements are met and the results can be integrated. Those thrusts are (1) the construction of a VM Architecture, (2) an investigation into the theoretical and technical underpinnings for VM, (3) the selective addition of animation and simulation-based reasoning methods to existing CAD and MRP systems, (4) the development of generic models, knowledge representation schemes and methods based on actual shop floor data for characterizing selected manufacturing processes for use in a production-centered VM system, (5) an investigation into the metrics for VM and the associated cultural constraints, and (6) research into the integration of knowledge representation schemes and methods for manufacturing knowledge that will support decision-making (e.g., risk analysis, load-leveling, etc.) and simulation-based cost analysis early in the product life-cycle.

Click here for Picture

Figure 4-3. Smart-CAD Technology Roadmap

Click here for Picture

Figure 4-4. Technology Roadmap Related to Legacy Systems

4.4.1 VM Roadmap: Architecture Thrust

Based on the individual technology roadmaps, and the prioritized technology areas described above, it is clear that the first thrust of the VM Roadmap must be the creation of the VM Framework or Architecture. In order to develop such an architecture, the developer must have rather specific insights into all of the component pieces, including all of the technical areas and technologies listed earlier. The insights must include information on the pace of technology development (specific to each technology), the likely impact of varying funding profiles on those developments, and the impact of the interdependent technologies. In essence, by placing Architecture/Framework as a top technical area, the participants are recommending a detailed exploration of the interactions of all the technologies for realizing VM.

Click here for Picture

Figure 4-5. Strawman Architecture

A strawman architecture for a VM system was presented at the opening plenary session and referred to during the breakout sessions. That architecture showed VM as being comprised of three major components: models, simulation and an environment. The technical areas (Table 3) developed during the workshop partially fit into this architecture. The areas of modeling, representation, meta-modeling, and manufacturing characterization fit the "model" component; the areas of visualization and simulation fit the "Simulation" component; and the areas of environment construction technologies, methodology, legacy and workflow fit the "Environment" component. The various kinds of technologies are shown graphically in the figure at right.

Although it is a good beginning, such an architecture is insufficient for the integrating infrastructure and architecture technical area as described at the workshop. The architecture must show how all of the VM technologies fit together, including those dealing with methodologies, metrics, validation, and cross-functional trades. It should highlight those technologies that are foundational to VM from those that are merely window dressings. The first thrust is expected to start with this strawman, and address these issues.

4.4.2 VM Roadmap: Technological Underpinnings Thrust

The second thrust for the VM Roadmap is to investigate the theoretical and technological underpinnings for Virtual Manufacturing. The need for this investigation is captured in the detailed breakout groups in Appendix E. In brief, the research should be directed at:

· finding research that is relevant to VM,

· assessing the status of key technologies (from those listed in Table 4) for realizing VM,

· determining the gaps in the research and technologies that may negatively impact realizing VM (e.g., show stoppers),

· assessing the time-frames for completion/maturity of relevant research,

· cataloging the trends in R&D being pursued by academia, government agencies and industry,

· discovering applications underway of relevance to VM, and

· creating 5 and 10 year "outlooks" for expected-to-emerge VM technologies.

4.4.3 VM Roadmap: Selective Addition of Animation Thrust

According to the participants, many in industry who are already actively pursuing VM are following this path, namely, the selective addition of animation to MRP and CAD systems. The purpose of this approach is to build some dynamism into what are essentially static systems, but focussed on those technologies which help industry. This incremental approach to adding animation and evolving methodologies for simulation-based reasoning is expected to help "sell" VM to industry leaders.

4.4.4 VM Roadmap: Development of Shop Floor Based Generic Models Thrust

As mentioned above, the manufacturing characterization technical area was identified as one of the top priority areas. Several problems were identified in this area that are of significance to VM that should be addressed in this thrust. First, simply collecting data for characterization is expensive, and is generally viewed as a non-value-added operation. Hence, methods for justifying the expenses must be developed (Technical Area #11 in Table 3). Second, standards to guide the data gatherers and model builders do not exist. Cost effectively developing generic models for VM use may demand such progress. Third, methods for representing the knowledge of manufacturing characterization are isolated, a fact which re-enforces the need for work in the Representation Technical Area.

4.4.5 VM Roadmap: Metrics Thrust

As mentioned often during both workshops, metrics must receive significant attention for VM realization. The metrics investigation for VM falls into three general areas. First, one must be able to reliably associate shop floor decisions with overall measures of cost, risk, time and quality. Too often in today's environment, local decisions improve these measures locally, but negatively impact the whole company or enterprise. This impacts VM in two ways: (1) such an association is necessary to successfully deploy and use VM, and (2) the deployment and use of VM may offer an approach to perform the association.

The second metrics investigation area must focus on the cultural impacts of VM. Especially at the Users Workshop, the problem of current culture and the adoption of VM was a critical concern. Because metrics can be used to drive the cultural change, one aspect of the metrics investigation must address the cultural impacts of VM.

The third area is in the verification and validation of the VM systems. Any modeling and simulation system will only be used to the extent that the users and decisions makers have confidence the simulations represent the physical systems, and that changes in the virtual systems will result in predictable changes in the physical systems. Again, the V&V of the VM systems is largely dependent upon and driven by the metrics.

4.4.6 VM Roadmap: Representation Thrust

The representation technical area was also deemed of critical importance to realizing VM. The purpose of this thrust is to investigate the integration of knowledge representation schemes and methods for manufacturing knowledge that will support decision-making (e.g., risk analysis, load-leveling, etc.) and simulation-based cost analysis early in the product life-cycle. The representation schemes should be applicable at the process, design, manufacturing, engineering and enterprise levels.

4.4.7 VM Roadmap: Integrating the Thrusts

As mentioned at the beginning of the section, the purpose of the thrusts is to facilitate focusing on specific requirements for VM as addressed in the two workshops, and yet take a wide swath through the technology areas. In order to integrate the results of these thrusts as they are executed, two other activities are necessary: VM demonstrations and VM pilots. The pilots and demonstrations should be conducted as a forum to bring the activities of the disparate thrusts together and cause cross-fertilization of ideas, techniques, technology advances, and expected accomplishments on the horizon. The VM demonstrations, as described in the Users Workshop Technical Report, are expected to be broad-based and attract a wide-audience, whereas the pilots will be specific implementations of VM systems that may also be a part of the demonstrations.

As recommended at the User Workshop, a consortium should be formed to oversee the VM initiative. Figure 4-6 graphically shows the meetings of this consortium along with the thrusts and integrating activities along a rough time scale.

Click here for Picture

Figure 4-6. Technology Roadmap

5. Conclusions & Recommendations

Many recommendations were generated during the workshop. The specific, consensus-developed recommendations from each breakout session, including which organizations should perform what activity, are presented in the viewgraphs contained in Appendix E. The overall workshop recommendations are summarized below:

· Conduct significant industry/ government research immediately in the area of the VM core technologies.

· Develop a VM Framework or Architecture which accommodates all of the technologies and technical areas mentioned in this report.

· Serve as a catalyst, a neutral broker, and a clearing house for pre-competitive technologies and specifications in the representation area. The government is in the right position to work with industry to broadly test specifications against a series of pilot projects and to accelerate the standardization process.

· Validate the findings of this workshop with users.

· Carry out the recommendations generated at the User Workshop.

5.1 Caveats

Appendix A. List of Acronyms

3-D modeling 3-Dimensional modeling

AI Artificial Intelligence

API Application Programming Interface

ARPA Advanced Research Projects Agency

CAD Computer Aided Design

CE Concurrent Engineering

DemVal Demonstration/Validation

DIS Distributed Interaction Simulation

DoD Department of Defense

DoE Department of Energy

DoT Department of Defense

ECO Engineering Change Order

EINet Enterprise Integration Network

EMD Engineering Manufacturing Development

GUI Graphical User Interface

IGES Initial Graphics Exchange Specification

IPPD Integrated Product Process Development

IPT Integrated Product Team

JDL Joint Directors of the Laboratories

MDO Multi-Dimensional Optimization

M&S Modeling and Simulation

MS&T Manufacturing Science and Technology

NII National Information Infrastructure

NIST National Institute for Standards and Technology

NSF National Science Foundation

PDES/STEP Product Data Exchange using STEP which is the

Standard for the Exchange of Product Model Data

TQM Total Quality Management

VE Virtual Enterprise

VM Virtual Manufacturing

V&V Verification and Validation

Appendix B. Workshop Invitation & Agenda

Appendix C. List of Participants

Appendix D. Plenary Session ViewGraphs

Appendix E. Breakout Session ViewGraphs

·