### Tip of the Week #25                     Tip Index

Go to the Prior Tip "Participative Management"
Go to the Next Tip Critical Chain by E. Goldratt
Go to the Update Tip this topic Monte Carlo Stopping Rule, Part 2

# Monte Carlo Stopping Rule, Part 1

Really Important for Monte Carlo simulation users!

One of the issues in using Monte Carlo simulation is, "How many trials is enough?" The approach I've used for years is by using the standard error of the mean statistic,

SEM = s/sqr(n),

where s is the standard deviation of the variable of interest (e.g., NPV), and
n is the number of trials in the simulation.

For most purposes, running the simulation until the SEM is less than 1% of the mean is a good rule-of-thumb stopping rule.

Recently, I was demonstrating this process to a client (Mike Cheek, Division Controller, Herff Jones). To my embarrassment, we were observing that the actual error in the expected value (mean) was much less than predicted. We were using Latin Hypercube Sampling (LHS) and changing the seed for otherwise identical simulation runs.

The following table shows several examples:

 Distribution S Calculated SEM Actual SEM Adjustment Factor No.Trials Factor Uniform(0,17.3) 5.0 0.50 0.049 0.10 0.010 Triangle(0,10,20) 4.1 0.41 0.059 0.14 0.021 Triangle(0,10,40) 8.5 0.85 0.121 0.14 0.020 {.9,-1;.1,lognorm(15,15)} 6.7 0.67 0.47 0.70 0.49 Normal(0,10) 10.0 1.00 0.20 0.20 0.04 Normal(0,10) *3 10.0 1.00 0.20 0.20 0.04 Normal(0,10) *10 10.0 1.00 0.20 0.20 0.04 Normal(0,10) *10c 10.0 1.00 0.122 0.12 0.015 Lognormal(15,15) 15.5 1.55 0.80 0.52 0.27 Lognormal(15.0,15.0) *3 15.0 1.50 0.84 0.56 0.31 Lognormal(15.0,15.0) *10 15.0 1.50 0.87 0.58 0.34 Lognormal(15.41,15.25) *10c 15.3 1.53 0.54 0.36 0.13

Notes: "* n" distributions were generated by n distributions of the same type. All component distributions are independent except for the "*10c" examples where one distribution is independent and nine are dependent with an approximately .68 correlation coefficient.

The above data were accumulated from routines in QuickBasic, using 1000 trials of 100-sample sets. For 10 strata were used for LHS; using a larger number (e.g., 100) further reduces the standard error.

For example, for the uniform (0,17.3) distribution above, the standard deviation is 5.0. The calculated SEM for a 100-trial run is

• SEM = 5/sqr(100) = 0.50.

With Latin Hypercube Sampling, the real SEM is approximately 0.049. This is about a 10x improvement. Thus, for this distribution shape:

• Instead of a 10,000 trial run, about the same degree of accuracy can be achieved with LHS in only 100 trials.
• A 100-trial LHS run has about the same SEM as 10,000 trials using conventional Monte Carlo sampling.

Instead of a single distribution, a normal Monte Carlo simulation in project evaluation is driven by multiple inputs as probability distributions. Curiously, a normal distribution comprised of a sum of 10 (or 3) independent normal distributions (adjusted so as to yield the same mean and standard deviation) appears to offer the same improvement. The same accuracy can be achieved with LHS using 4% as many trials as would be required using conventional Monte Carlo sampling.

• Lognormal distributions show similar improvement but appear slightly dependent upon the number of component distributions.
• If the problem involves positively correlated component distributions, the improvement is substantially better.
• Highly skewed and multi-modal distributions do not converge as rapidly, but still exhibit considerably better efficiency with LHS

I need to do more experimentation in order to formulate a "stopping rule guide." If any readers are familiar with a mathematical solution or other alternative, please let me know. Watch this same Tip for more details.

—John Schuyler, October 1997. Revised 15-Oct-97.