Technical Report: Defining the Meaning of a Major Modeling and Simulation Change as Applied to Accreditation

Report Number: Technical Report SERC-2012-TR-029-1

Report Name:  Defining the Meaning of a Major Modeling and Simulation Change as Applied to Accreditation

Publication Date: December 12, 2012

This report describes a new method, the Quantitative-to-Qualitative Risk-based (QQR) method, which makes a quantitative recommendation regarding the re-validation of a modified model.

Many decisions today are being made based on models, and some of these models are modified versions of existing models. While it is clear that a new model should undergo validation before being used for any significant application, such clarity does not exist for models that have been modified from existing models that were validated during their initial development. Several existing methods address the re-validation questions that vary widely, but all include some form of notional risk.

The QQR method was developed to meet the following goals:

  • to be quantitative, repeatable, and transparent
  • to consider both model modifications and model use risk
  • to focus on model types and simulation applications of interest
  • to be simple and accessible so as to encourage its use in practical applications.

The QQR method estimates the probability that not re-validating a modified model will lead to unacceptable consequences, given the modifications made to it. That estimated probability is to be interpreted as a quantitative recommendation to re-validate the model, in the context of a re-validation decision. The method is based on a conditional probability formula that separates the various parts of the estimated probability into distinct terms and factors, and provides procedures for estimating each term and factor. A central feature of the QQR method is a “missions-means decomposition” that allows the method’s user to identify precisely and effectively, the nature and extent of the modifications that were made to the model, and the method to consider those modifications in its estimate.