Evaluation Of Systems Engineering Methods, Processes and Tools on Department of Defense and Intelligence Community Programs

Principal Investigator: Dr. Richard Turner, Stevens Institute of Technology (rturner@stevens.edu)

Timeframe:  September 2008 to September 2009

Category:  Systems Engineering and Systems Management Transformation


Objectives: To address Systems Engineering (SE) shortfalls in projects characterized as quick response, network-enabled, or emergent. The objectives were to:

  • Phase 1:
    • (a) identify Methods, Processes and Tools (MPTs) considered viable in the sponsor’s environment, and
    • (b) identify gaps where no useful/viable MPTs could be identified.
  • Phase 2:
    • (c) gather additional information on MPTs associated with the environment identified in Phase 1 and develop a taxonomy of MPTs identified;
    • (d) investigate use of micro-process modeling techniques to support the definition and evaluation of MPTs; and
    • (e) provide implementation guidance on the three MPTs recommended in Phase 1.

Approach: Literature and industry surveys were used to gather data on current SE practice in similar environments. Consolidation and analysis of data produced a set of common themes across interview responses, a list of candidate MPTs that were recommended by organizations with environments similar to the Sponsor’s, and a framework for showing the relationships among challenges, themes and MPTs.

Significant Research Findings & Products: The research established four key challenges:

  1. Requirements: Changing requirements priorities and/or emerging requirements
  2. Stakeholders: Obtaining useful stakeholder input and addressing conflicting stakeholder requirements
  3. Sustainment: Conflicts between developing new capabilities and supporting a currently deployed system
  4. Integration: Integrating independently evolving components into a larger interoperable system

Three MPTs were recommended as most likely to increase effectiveness in Sponsor’s environment: Scrum, rapid prototyping, and continuous integration. Gaps were identified through industry survey responses and by comparing useful, viable MPTs to the four challenge areas. Primary gaps identified through the survey were decision management and stakeholder requirements definition. The challenge area where the MPT gap was largest was Sustainment.

The “bridge diagram” was developed to help strategize solutions and assess current practices. The diagram links 3 kinds of information: a theme from survey responses, specific theme elements, and MPTs that have proven effective in achieving those elements. Scrum was described in a formal process-modeling tool (Little JIL from UMass). Single point of failure and finite state verification analyses were performed. Implementation guidance was developed for 3 MPTs based on the bridge diagram. There simply were not good tools for rapid/agile SE in use. Tools that have been applied were drawn mainly from agile software development and had resulted in mixed success.


Publications: none to date

Research Team

Current Researchers:

  • Dr. Richard Turner, Stevens Institute of Technology
  • Dr. Lucas Layman, Fraunhofer Center at the University of Maryland
  • Barry Boehm, University of Southern California
  • Jo Ann Lane, University of Southern California
  • Dr. Forrest Shull, Fraunhofer Center at the University of Maryland
  • Anne Carrigy, Stevens Institute of Technology
  • Dr. Paul Componation,University of Alabama in Huntsville
  • Sue O’Brien, University of Alabama in Huntsville
  • Dr. Dawn Sabados, University of Alabama in Huntsville
  • Cihan Dagli, Missouri University of Science and Technology
  • Ann Miller, Missouri University of Science and Technology
  • Lori Clarke, University of Massachusetts at Amherst
  • Leon Osterweil, University of Massachusetts at Amherst
  • Sandy Wise, University of Massachusetts at Amherst

Past Researchers:

  • Dr. Julie Fortune,University of Alabama in Huntsville
  • Ed Colbert, University of Southern California

Collaborating Institutions

Project Researchers