Verification
In the context of computer simulation, verification of a model is the process of confirming that it is correctly implemented with respect to the conceptual model (it matches specifications and assumptions deemed acceptable for the given purpose of application). During verification the model is tested to find and fix errors in the implementation of the model. Various processes and techniques are used to assure the model matches specifications and assumptions with respect to the model concept. The objective of model verification is to ensure that the implementation of the model is correct. There are many techniques that can be utilized to verify a model. These include, but are not limited to, having the model checked by an expert, making logic flow diagrams that include each logically possible action, examining the model output for reasonableness under a variety of settings of the input parameters, and using an interactive debugger. Many software engineering techniques used for software verification are applicable to simulation model verification.Validation
Validation checks the accuracy of the model's representation of the real system. Model validation is defined to mean "substantiation that a computerized model within its domain of applicability possesses a satisfactory range of accuracy consistent with the intended application of the model". A model should be built for a specific purpose or set of objectives and its validity determined for that purpose. There are many approaches that can be used to validate a computer model. The approaches range from subjective reviews to objective statistical tests. One approach that is commonly used is to have the model builders determine validity of the model through a series of tests. Naylor and Finger 967formulated a three-step approach to model validation that has been widely followed: Step 1. Build a model that has high face validity. Step 2. Validate model assumptions. Step 3. Compare the model input-output transformations to corresponding input-output transformations for the real system.Face validity
A model that has face validity appears to be a reasonable imitation of a real-world system to people who are knowledgeable of the real world system. Face validity is tested by having users and people knowledgeable with the system examine model output for reasonableness and in the process identify deficiencies. An added advantage of having the users involved in validation is that the model's credibility to the users and the user's confidence in the model increases. Sensitivity to model inputs can also be used to judge face validity. For example, if a simulation of a fast food restaurant drive through was run twice with customer arrival rates of 20 per hour and 40 per hour then model outputs such as average wait time or maximum number of customers waiting would be expected to increase with the arrival rate.Validation of model assumptions
Assumptions made about a model generally fall into two categories: structural assumptions about how system works and data assumptions. Also we can consider the simplification assumptions that are those that we use to simplify the reality.1. Fonseca, P. Simulation hypotheses. In Proceedings of SIMUL 2011; 2011; pp. 114–119. https://www.researchgate.net/publication/262187532_Simulation_hypotheses_A_proposed_taxonomy_for_the_hypotheses_used_in_a_simulation_modelStructural assumptions
Assumptions made about how the system operates and how it is physically arranged are structural assumptions. For example, the number of servers in a fast food drive through lane and if there is more than one how are they utilized? Do the servers work in parallel where a customer completes a transaction by visiting a single server or does one server take orders and handle payment while the other prepares and serves the order. Many structural problems in the model come from poor or incorrect assumptions. If possible the workings of the actual system should be closely observed to understand how it operates. The systems structure and operation should also be verified with users of the actual system.Data assumptions
There must be a sufficient amount of appropriate data available to build a conceptual model and validate a model. Lack of appropriate data is often the reason attempts to validate a model fail. Data should be verified to come from a reliable source. A typical error is assuming an inappropriate statistical distribution for the data. The assumed statistical model should be tested using goodness of fit tests and other techniques. Examples of goodness of fit tests are theSimplification assumptions
Are those assumptions that we know that are not true, but are needed to simplify the problem we want to solve. The use of this assumptions must be restricted to assure that the model is correct enough to serve as an answer for the problem we want to solve.Validating input-output transformations
The model is viewed as an input-output transformation for these tests. The validation test consists of comparing outputs from the system under consideration to model outputs for the same set of input conditions. Data recorded while observing the system must be available in order to perform this test. The model output that is of primary interest should be used as the measure of performance. For example, if system under consideration is a fast food drive through where input to model is customer arrival time and the output measure of performance is average customer time in line, then the actual arrival time and time spent in line for customers at the drive through would be recorded. The model would be run with the actual arrival times and the model average time in line would be compared with the actual average time spent in line using one or more tests.Hypothesis testing
= Model accuracy as a range
= A statistical technique where the amount of model accuracy is specified as a range has recently been developed. The technique uses hypothesis testing to accept a model if the difference between a model's variable of interest and a system's variable of interest is within a specified range of accuracy.Sargent, R. G. 2010. "A New Statistical Procedure for Validation of Simulation and Stochastic Models." Technical Report SYR-EECS-2010-06, Department of Electrical Engineering and Computer Science, Syracuse University, Syracuse, New York. A requirement is that both the system data and model data be approximately Normally Independent and Identically Distributed (NIID). TheConfidence intervals
Confidence intervals can be used to evaluate if a model is "close enough" to a system for some variable of interest. The difference between the known model value, μ0, and the system value, μ, is checked to see if it is less than a value small enough that the model is valid with respect that variable of interest. The value is denoted by the symbol ε. To perform the test a number, ''n'', statistically independent runs of the model are conducted and a mean or expected value, E(Y) or μ for simulation output variable of interest Y, with a standard deviation ''S'' is produced. A confidence level is selected, 100(1-α). An interval, ,b is constructed by : , where : is the critical value from the t-distribution for the given level of significance and n-1 degrees of freedom. : If , a-μ0, > ε and , b-μ0, > ε then the model needs to be calibrated since in both cases the difference is larger than acceptable. : If , a-μ0, < ε and , b-μ0, < ε then the model is acceptable as in both cases the error is close enough. : If , a-μ0, < ε and , b-μ0, > ε orGraphical comparisons
If statistical assumptions cannot be satisfied or there is insufficient data for the system a graphical comparisons of model outputs to system outputs can be used to make a subjective decisions, however other objective tests are preferable.ASME Standards
Documents and standards involving verification and validation of computational modeling and simulation are developed by theSee also
*References
{{Reflist Formal methods