Tip of the Week #29 Tip Index

Go to the Prior Tip Definitive's "Scenario" — Monte Carlo Simulation Tool
Go to the Next Tip Improve Your Lottery Expected Value
Go to the Update Tip Monte Carlo Stopping Rule, Part 1
Return to MaxValue Home Page

Monte Carlo Stopping Rule, Part 2

In discussing this with Eric Wainwright, Decisioneering's chief guru, in May 1999, I became more clear about how the procedure should work. Here is a 2nd revised approach from what was presented earlier on this page.

Crystal Ball^® 2000 features Precision Control™, a stopping function that uses the "bootstrapping" method. They have applied for a patent on their algorithm, which I haven't seen. Precision Control can be used broadly in using other statistics in stopping rules.

Here is a stopping rule procedure to apply when Latin Hypercube Sampling (LHS) is used with Monte Carlo simulation.

Determine a "sample size" (for Latin hypercube sampling and for correlation sampling). Perhaps this should be the lesser of: (a) 100 or (b) 1/5 (rounded) of the minimum number of trials anticipated.
At "sample size" intervals, determine whether adequate convergence has been achieved. For most purposes we want adequate confidence in the mean value, such as +/- 1% with 68% conficence. Determine this by either:

a. (My former method) Divide the outcome values into sets of "sample size." Compute the means of each set, and the standard deviation (s) of the sample means. This is approximately the standard error of the mean (SEM). If this is less than 1% of the sample mean, then stop.

b. (An improved method with bootstrapping) Bootstrapping is a technique involving resampling data. With the trials values generated thus far:
   (1) Randomly select, say "sample size" values
        (sampling these without replacement would be best).
   (2) Determine the sample mean of these values.

   (3) Repeat steps (1) and (2), say, 50 times.
   (4) Compute the standard deviation of the sample means from Step (2);
        this approximates the standard error of the mean (SEM).
   (5) Stop when the SEM is < 1% of the mean.
Adjust the SEM calculated in the prior step for the actual number of trials in the simulation run. For example, if "sample size" is 100 and 500 trials were run, then the SEM of the sample mean is
SEM500 = SEM100 x sqr(100/500).

If you need precision better than the SEM, run more trials in multiples of Sample Size. Recall that a 68% confidence interval for the Expected Value is the overall sample mean ± SEM.

This is the best stopping rule that I can offer at the present. Perhaps Eric's method in Crystal Ball is better. Note that it is not unusual that LHS require only 1/5 as many trials as traditional Monte Carlo sampling.

The Student-t distribution may provide an additional refinement in correcting the sample standard deviation for the small number of data values.

—John Schuyler, January 1998. Revised 15-Sep-99 and 26-Apr-00