Go to the Prior Tip** Definitive's "Scenario" ** Monte Carlo Simulation Tool

Go to the Next Tip** Improve Your
Lottery Expected Value**

Go to the Update Tip** Monte Carlo
Stopping Rule, Part 1**

Return to MaxValue Home Page

In discussing this with Eric Wainwright,
Decisioneering's chief guru, in May 1999, I became more clear about how
the procedure should work. Here is a 2nd revised approach from what
was presented earlier on this page. Crystal Ball ^{ } 2000 features Precision Control , a
stopping function that uses the "bootstrapping" method. They have
applied for a patent on their algorithm, which I haven't seen.
Precision Control can be used broadly in using
other statistics in stopping rules. |

Here is a stopping rule procedure to apply when Latin Hypercube Sampling (LHS) is used with Monte Carlo simulation.

- Determine a "sample size" (for Latin hypercube sampling
and for correlation sampling). Perhaps this should be the lesser of: (a)
100 or (b) 1/5 (rounded) of the minimum number of trials anticipated.

- At "sample size" intervals, determine whether adequate
convergence has been achieved. For most purposes we want adequate
confidence in the mean value, such as +/- 1% with 68% conficence.
Determine this by either:

a. (My former method) Divide the outcome values into sets of "sample size." Compute the means of each set, and the standard deviation (s) of the sample means. This is approximately the(SEM). If this is less than 1% of the sample mean, then stop.*standard error of the mean*

b. (An improved method with bootstrapping)**Bootstrapping**is a technique involving resampling data. With the trials values generated thus far:

(1) Randomly select, say "sample size" values

(sampling these without replacement would be best).

(2) Determine the sample mean of these values.

(3) Repeat steps (1) and (2), say, 50 times.

(4) Compute the standard deviation of the sample means from Step (2);

this approximates the(SEM).*standard error of the mean*

(5) Stop when the SEM is < 1% of the mean.

- Adjust the SEM calculated in the prior step for the actual
number of trials in the simulation run. For example, if "sample size" is
100 and 500 trials were run, then the SEM of the sample mean is

SEM500= SEM100x sqr(100/500).

If you need precision better than the SEM, run more trials in multiples of Sample Size. Recall that a 68% confidence interval for the Expected Value is the overall sample mean SEM.

This is the best stopping rule that I can offer at the
present. Perhaps Eric's method in Crystal Ball is better. **Note that
it is not unusual that LHS require only 1/5 as many trials as traditional Monte
Carlo sampling.**

The Student-t distribution may provide an additional refinement in correcting the sample standard deviation for the small number of data values.

Update: In searching palisade.com for a technique related to LHS, I came across a note in their knowledge base. "6.23 Confidence intervals in @RISK" http://kb.palisade.com/index.php?pg=kb.page&id=124 This describes a similar process and a published paper: Michael Stein, "Large Sample Properties of Simulations Using Latin Hypercube Sampling", Technometrics 29:2 [May 1987], pages 143-151, pay site still working 19-apr-16, from http://www.jstor.org/stable/1269769.) |

John Schuyler, January 1998. Revised 15-Sep-99, 26-Apr-00, and 19-Apr-16

Copyright 1998-2016 by John R. Schuyler. All rights reserved. Permission to copy with reproduction of this notice.