Go to the Prior Tip "Participative
Management"
Go to the Next Tip Critical Chain
by E. Goldratt
Go to the Update Tip this topic Monte Carlo Stopping Rule,
Part 2
Return to MaxValue Home Page
Really Important for Monte Carlo simulation users!
One of the issues in using Monte Carlo simulation is, "How many trials is enough?" The approach I've used for years is by using the standard error of the mean statistic,
SEM = s/sqr(n),
where s is the standard deviation of the variable of interest (e.g., NPV), and
n is the number of trials in the simulation.
For most purposes, running the simulation until the SEM is less than 1% of the mean is a good rule-of-thumb stopping rule.
Recently, I was demonstrating this process to a client (Mike Cheek, Division Controller, Herff Jones). To my embarrassment, we were observing that the actual error in the expected value (mean) was much less than predicted. We were using Latin Hypercube Sampling (LHS) and changing the seed for otherwise identical simulation runs.
The following table shows several examples:
Distribution
S
Calculated
SEMActual
SEMAdjustment
FactorNo.Trials
FactorUniform(0,17.3)
5.0
0.50
0.049
0.10
0.010
Triangle(0,10,20)
4.1
0.41
0.059
0.14
0.021
Triangle(0,10,40)
8.5
0.85
0.121
0.14
0.020
{.9,-1;.1,lognorm(15,15)}
6.7
0.67
0.47
0.70
0.49
Normal(0,10)
10.0
1.00
0.20
0.20
0.04
Normal(0,10) *3
10.0
1.00
0.20
0.20
0.04
Normal(0,10) *10
10.0
1.00
0.20
0.20
0.04
Normal(0,10) *10c
10.0
1.00
0.122
0.12
0.015
Lognormal(15,15)
15.5
1.55
0.80
0.52
0.27
Lognormal(15.0,15.0) *3
15.0
1.50
0.84
0.56
0.31
Lognormal(15.0,15.0) *10
15.0
1.50
0.87
0.58
0.34
Lognormal(15.41,15.25) *10c
15.3
1.53
0.54
0.36
0.13
Notes: "* n" distributions were generated by n distributions of the same type. All component distributions are independent except for the "*10c" examples where one distribution is independent and nine are dependent with an approximately .68 correlation coefficient.
The above data were accumulated from routines in QuickBasic, using 1000 trials of 100-sample sets. For 10 strata were used for LHS; using a larger number (e.g., 100) further reduces the standard error.
For example, for the uniform (0,17.3) distribution above, the standard deviation is 5.0. The calculated SEM for a 100-trial run is
With Latin Hypercube Sampling, the real SEM is approximately 0.049. This is about a 10x improvement. Thus, for this distribution shape:
Instead of a single distribution, a normal Monte Carlo simulation in project evaluation is driven by multiple inputs as probability distributions. Curiously, a normal distribution comprised of a sum of 10 (or 3) independent normal distributions (adjusted so as to yield the same mean and standard deviation) appears to offer the same improvement. The same accuracy can be achieved with LHS using 4% as many trials as would be required using conventional Monte Carlo sampling.
I need to do more experimentation in order to formulate a "stopping rule guide." If any readers are familiar with a mathematical solution or other alternative, please let me know. Watch this same Tip for more details.
John Schuyler, October 1997. Revised 15-Oct-97.
Copyright © 1997 by John R. Schuyler. All rights reserved. Permission to copy with reproduction of this notice.