3  Overview of statistical tools for MC and DES

Statistics are like bikinis, they show a lot but not everything.

— Lou Piniella

3.1 Introduction

Statistical methods are essential for the design, execution, and interpretation or simulation models. Without rigorous statistical foundations, the results generated can lead to poor decisions, wasted resources, loss of engineering credibility and profits.

The accuracy and precision of the Monte Carlo Method depends on the quality of the random number generators and the correctness of the input distributions. Thus, statistical methods ensure that:

  • Input modeling reflects real-world variability through appropriate probability distributions.

  • Sampling techniques reduce variance and improve efficiency.

  • Confidence intervals and hypothesis tests validate the reliability of estimates, such as risk measures or expected values.

For example, in financial risk analysis, Monte Carlo simulations estimate the probability of portfolio losses under uncertain market conditions. Statistical tools like bootstrapping and variance reduction techniques enhance precision and reduce computational cost.

While DES focuses on event-driven dynamics, randomness permeates its inputs and outputs1. Thus, statistical methods are critical for:

  • Input Analysis. To validate that the assumed distributions match empirical data.

  • Output Analysis. Using descriptive and inferential statistics to summarize performance metrics like average waiting time or throughput.

  • Experimental Designs. Applying factorial arrays and Analysis of Variance (ANOVA), to mention two, for the systematic exploration of multiple factors and their effects in performance indicators.

  • Uncertainty Quantification. Confidence intervals and hypothesis testing ensure that observed differences between scenarios are statistically significant, not due to random variation.

Monte Carlo and Discrete Event Simulation provide estimates rather than exact solutions or mathematically optimal results. These estimates are subject to sampling error and variability. Therefore, statistical methods provide the tools needed to:

  • Quantify uncertainty through confidence intervals.

  • Compare alternatives using hypothesis testing.

  • Optimize designs using regression and metamodels.

  • Validate models against real-world data.

Without these methods, simulation becomes guesswork rather than an engineering2 decision support tool.

3.2 Descriptive and Inferential Statistics

Summarizing simulation outputs with statistics such as the mean, variance, standard deviation, and confidence intervals, provides insight into the performance of systems under study. For example, calculating the average waiting time and its variability for customers in a queue. Descriptive and inferential statistics form the basis for performance evaluation and comparison across different scenarios and system’s designs (Moroney, 1954).

Confidence intervals (CI’s) quantify uncertainty in simulation estimates, enabling decision-makers to estimate the precision of our findings. For example, we might state that the mean throughput of as simulated process is 120 ± 5 units/hour at 95% confidence. CI’s are used because simulation results are subject to sampling error. Therefore, CI’s provide a statistical guarantee about the precision of estimates, which is crucial for risk sensitive decisions (Devore, 2012).

Hypothesis testing and comparisons such as t-tests, ANOVA, Anderson-Darling for normality etc., determine whether differences between scenarios are statistically significant. For example, testing if a new scheduling policy reduces patient wait times compared to the current policy3. Hypothesis testing is a technique used to check if what we observe in our data is real or just happened by chance. This is, it helps with decisions based on random variation, ensuring changes are statistically justified (Goonick & Smith, 1993).

3.3 Probability Distributions

Simulation models rely on probability distributions to represent stochastic elements such as inter arrival times, service duration, and machine failures, to mention a few. Common probability functions include the exponential for arrival processes, the normal for processing times, and the Weibull family for reliability modeling. As another example, consider the simulation of an emergency department in a hospital, patient arrival times may follow a Poisson process modeled using an exponential distribution.

NoteWhat is a Poisson process?

A Poisson process is a stochastic process that models the occurrence of events over time (or space) where:

  • Events occur one at a time and are independent of each other.

  • The probability of an event occurring in a small interval is proportional to the length of that interval.

  • The number of events in disjoint intervals is independent.

  • The number of events in an interval of specific length t follows a Poisson distribution with mean \(\lambda t\), where \(\lambda\) is the rate of occurrence.

Probability distributions obtained from real data provide accurate input modeling ensuring that our simulation model reflects real world variability critical for valid decision making (Martin et al., 2025; Moroney, 1954).

3.4 Bootstrapping

Bootstrapping estimates the distribution of simulation outputs by resampling4. It is useful when analytical solutions are infeasible. For example, estimating the confidence interval for median waiting time in a congested emergency system. Bootstrapping handles non-normality and small sample sizes, improving robustness of our statistical inferences (Efron, 1979).

3.5 Random and Pseudo-random Number Generators

Random number generators (RNGs), and specifically Pseudo-Random number generators (PRNGs) produce sequences of data used to sample from probability distributions. High quality PRNGs ensure reproducibility and statistical validity. PRNGs underpin the stochastic nature of DES and Monte Carlo. Poor quality random number generators can introduce correlation and bias, compromising simulation integrity (Goldreich, 2010).

3.6 Design of Experiments (DOE)

DOE systematically explores the effect of multiple factors on system performance reducing the number of required runs compared to one factor at a time (OFAT) approaches. For example, using a factorial design to study the combined effect of staffing levels and arrival rates on hospital throughput. DOE enhances efficiency and supports optimization by identifying key drivers and interactions (Box, 1978; Gjerloev et al., 2025; Law, 2017).

3.7 Regression and Metamodeling

Regression models and metamodels approximate simulation behavior, enabling sensitivity analysis and optimization. For example, building a regression model to predict average queue length based on arrival and service rates. Metamodels5, on the other hand, reduce computational cost and allow real-time decision support (Blas et al., 2021; Montgomery et al., 2001).

3.8 Goodness-of-Fit Tests

Tests like Kolmogorov-Smirnov and Chi-square validate input distributions against empirical data. For example, checking if observed interarrival times fit an exponential distribution. These tests ensure input functions accurately represent reality, reducing bias in simulation outcomes (Devore, 2012; Martin et al., 2025).

3.9 Queueing Theory

Analytical formulas from queueing theory serve as benchmarks for simulation validation. For example, comparing a simulated average queue length with theoretical M/M/1 results provides a sanity check, helps detect modeling errors, and validates the simulation (Karlin & Taylor, 1981).

3.10 Conclusion

Statistical tools are the backbone of DES and Monte Carlo, enabling rigorous input modeling, design of scenarios to run, and output analysis. Their incorporation ensures that simulation studies produce reliable and actionable insights. As DES and Monte Carlo applications expand in healthcare, manufacturing, finance, marketing, supply chain, logistics, and other industries and functions, the role of advanced statistical techniques6 will continue to grow, supporting data driven decision making in increasingly complex and sophisticated environments.

3.11 Questions

  1. Why are statistical methods essential for Monte Carlo and Discrete Event Simulation modeling?

  2. How do input modeling and sampling techniques affect the accuracy of Monte Carlo simulations?

  3. What role do confidence intervals and hypothesis testing play in simulation studies?

  4. Why are statistical tools critical for input and output analysis in Discrete Event Simulation?

  5. Explain why Monte Carlo and DES provide estimates rather than exact solutions.

  6. What is the difference between descriptive and inferential statistics in the context of simulation output analysis?

  7. Why are probability distributions important in simulation modeling? Provide one example.

  8. What is bootstrapping, and why is it useful in simulation output analysis?

  9. What is the role of pseudo-random number generators (PRNGs) in simulation modeling?

  10. How do Design of Experiments (DOE) and metamodeling improve the efficiency of simulation studies?

Notes of the Chapter

  • Lou Piniella (born August 28, 1943) is a former Major League Baseball player and manager, best known for his successful career as an outfielder and later as a fiery, competitive manager. He played for teams like the Kansas City Royals and New York Yankees, winning Rookie of the Year in 1969 and two World Series titles with the Yankees. As a manager, he led teams such as the Cincinnati Reds, Seattle Mariners, and Chicago Cubs, earning three Manager of the Year awards. Piniella is also remembered for the humorous quote at the beginning of this chapter, highlighting his belief that numbers alone cannot capture the full complexity of the game.

References

Blas, M. J., Gonnet, S., & Zeigler, B. P. (2021). Towards a Universal Representation of DEVS: A Metamodel Based Definition of DEVS Formal Specification. ANNSIM ’21, Society for Modeling and Simulation International.
Box, G. E. P. (1978). Statistics for Experimenters (First ed.). John Wiley & Sons.
Devore, J. L. (2012). Probability & statistics for engineering and the sciences (Eighth ed.). Brooks Cole, Mexico.
Efron, B. (1979). Bootstrap Methods: Another Look at the Jacknife. The Annals of Statistics, 7(1), 1–26.
Gjerloev, A., Grieco, L., Lame, G., & Crowe, S. (2025). A Systematic Review of Discrete Event Simulation in Healthcare and Statistics Distributions. Journal of Simulation, 1–17.
Goldreich, O. (2010). A primer on pseudorandom generators. University lecture series, V 55: American Statistical Society, RI.
Goonick, L., & Smith, W. (1993). The cartoon guide to statistics (First ed.). Harper Collins, NY.
Karlin, S., & Taylor, H. M. (1981). A second course in stochastic processes (First ed.). Academic Press, San Diego.
Law, A. M. (2017). A Tutorial on Design of Experiments for Simulation Modeling. Proceedings of the 2017 Winter Simulation Conference, 550–564.
Martin, D. V., Fernandez, C. B., & Municio, A. M. G. (2025). Systematic Review of Discrete Event Simulation in Healthcare and Statistics Distributions. Applied Science, 15, 1861.
Montgomery, D. C., Peck, E. A., & Vining, G. G. (2001). Introduction to linear regression analysis (Third ed.). John Wiley & Sons, NY.
Moroney, M. J. (1954). Facts from figures (Second, revised edition). Penguin Books, Melbourne.

  1. Arrival rates, service times, failure probabilities, etc.↩︎

  2. And scientific.↩︎

  3. A two-way comparison.↩︎

  4. Bootstrapping is a simple way to figure out how confident you can be about a result when you only have a limited amount of data.↩︎

  5. A metamodel is a method that takes the results from several different models and learns how to combine them to produce a better prediction than any single model alone.↩︎

  6. To cope with large amounts of data.↩︎