Go to the Prior Tip** "Participative
Management"**

Go to the Next Tip** Critical Chain
by E. Goldratt**

Go to the Update Tip this topic

Return to MaxValue Home Page

*Really Important for Monte Carlo simulation users!*

One of the issues in using Monte Carlo simulation is, "How many
trials is enough?" The approach I've used for years is by using the *standard error
of the mean* statistic,

SEM= s/sqr(n),where s is the standard deviation of the variable of interest (e.g.,

NPV), and

n is the number of trials in the simulation.

For most purposes, running the simulation until the *SEM* is less than
1% of the mean is a good rule-of-thumb stopping rule.

Recently, I was demonstrating this process to a client (Mike Cheek, Division Controller, Herff Jones). To my embarrassment, we were observing that the actual error in the expected value (mean) was much less than predicted. We were using Latin Hypercube Sampling (LHS) and changing the seed for otherwise identical simulation runs.

The following table shows several examples:

Distribution

S

Calculated

SEMActual

SEMAdjustment

FactorNo.Trials

FactorUniform(0,17.3)

5.0

0.50

0.049

0.10

0.010

Triangle(0,10,20)

4.1

0.41

0.059

0.14

0.021

Triangle(0,10,40)

8.5

0.85

0.121

0.14

0.020

{.9,-1;.1,lognorm(15,15)}

6.7

0.67

0.47

0.70

0.49

Normal(0,10)

10.0

1.00

0.20

0.20

0.04

Normal(0,10) *3

10.0

1.00

0.20

0.20

0.04

Normal(0,10) *10

10.0

1.00

0.20

0.20

0.04

Normal(0,10) *10c

10.0

1.00

0.122

0.12

0.015

Lognormal(15,15)

15.5

1.55

0.80

0.52

0.27

Lognormal(15.0,15.0) *3

15.0

1.50

0.84

0.56

0.31

Lognormal(15.0,15.0) *10

15.0

1.50

0.87

0.58

0.34

Lognormal(15.41,15.25) *10c

15.3

1.53

0.54

0.36

0.13

Notes: "* n" distributions were generated by n distributions of the same type. All component distributions are independent except for the "*10c" examples where one distribution is independent and nine are dependent with an approximately .68 correlation coefficient.

The above data were accumulated from routines in QuickBasic, using 1000 trials of 100-sample sets. For 10 strata were used for LHS; using a larger number (e.g., 100) further reduces the standard error.

For example, for the uniform (0,17.3) distribution above, the standard
deviation is 5.0. The calculated *SEM* for a 100-trial run is

*SEM*= 5/sqr(100) = 0.50.

With Latin Hypercube Sampling, the real *SEM* is approximately 0.049.
This is about a 10x improvement. Thus, for this distribution shape:

- Instead of a 10,000 trial run, about the same degree of accuracy can be achieved with LHS in only 100 trials.
- A 100-trial LHS run has about the same
*SEM*as 10,000 trials using conventional Monte Carlo sampling.

Instead of a single distribution, a normal Monte Carlo simulation in project evaluation is driven by multiple inputs as probability distributions. Curiously, a normal distribution comprised of a sum of 10 (or 3) independent normal distributions (adjusted so as to yield the same mean and standard deviation) appears to offer the same improvement. The same accuracy can be achieved with LHS using 4% as many trials as would be required using conventional Monte Carlo sampling.

- Lognormal distributions show similar improvement but appear slightly dependent upon the number of component distributions.
- If the problem involves positively correlated component distributions, the improvement is substantially better.
- Highly skewed and multi-modal distributions do not converge as rapidly, but still exhibit considerably better efficiency with LHS

I need to do more experimentation in order to formulate a "stopping rule guide." If any readers are familiar with a mathematical solution or other alternative, please let me know. Watch this same Tip for more details.

—John Schuyler, October 1997. Revised 15-Oct-97.

Copyright © 1997 by John R. Schuyler. All rights reserved. Permission to copy with reproduction of this notice.