Tip of the Week #25                     Tip Index

Go to the Prior Tip "Participative Management"
Go to the Next Tip Critical Chain by E. Goldratt
Go to the Update Tip Monte Carlo Stopping Rule, Part 2
Return to MaxValue Home Page

Monte Carlo Stopping Rule, Part 1

Really Important for Monte Carlo simulation users!

One of the issues in using Monte Carlo simulation is, "How many trials is enough?" The approach I've used for years is by using the standard error of the mean statistic,

S.E.M. = s/sqr(n),

where s is the standard deviation of the variable of interest (e.g., NPV), and
n is the number of trials in the simulation.

For most purposes, running the simulation until the S.E.M. is less than 1% of the mean is a good rule-of-thumb stopping rule.

Recently, I was demonstrating this process to a client (Mike Cheek, Division Controller, Herff Jones). To my embarrassment, we were observing that the actual error in the expected value (mean) was much less than predicted. We were using Latin Hypercube Sampling (L.H.S.) and changing the seed for otherwise identical simulation runs.

The following table shows several examples:

Distribution

S

Calculated
S.E.M.

Actual
S.E.M

Adjustment
Factor

No.Trials
Factor

Uniform(0,17.3)

5.0

0.50

0.049

0.10

0.010

Triangle(0,10,20)

4.1

0.41

0.059

0.14

0.021

Triangle(0,10,40)

8.5

0.85

0.121

0.14

0.020

{.9,-1;.1,lognorm(15,15)}

6.7

0.67

0.47

0.70

0.49

Normal(0,10)

10.0

1.00

0.20

0.20

0.04

Normal(0,10) *3

10.0

1.00

0.20

0.20

0.04

Normal(0,10) *10

10.0

1.00

0.20

0.20

0.04

Normal(0,10) *10c

10.0

1.00

0.122

0.12

0.015

Lognormal(15,15)

15.5

1.55

0.80

0.52

0.27

Lognormal(15.0,15.0) *3

15.0

1.50

0.84

0.56

0.31

Lognormal(15.0,15.0) *10

15.0

1.50

0.87

0.58

0.34

Lognormal(15.41,15.25) *10c

15.3

1.53

0.54

0.36

0.13

Notes: "* n" distributions were generated by n distributions of the same type. All component distributions are independent except for the "*10c" examples where one distribution is independent and nine are dependent with an approximately .68 correlation coefficient.

The above data were accumulated from routines in QuickBasic, using 1000 trials of 100-sample sets. For 10 strata were used for L.H.S.; using a larger number (e.g., 100) further reduces the standard error.

For example, for the uniform (0,17.3) distribution above, the standard deviation is 5.0. The calculated S.E.M. for a 100-trial run is

With Latin Hypercube Sampling, the real S.E.M. is approximately 0.049. This is about a 10x improvement. Thus, for this distribution shape:

Instead of a single distribution, a normal Monte Carlo simulation in project evaluation is driven by multiple inputs as probability distributions. Curiously, a normal distribution comprised of a sum of 10 (or 3) independent normal distributions (adjusted so as to yield the same mean and standard deviation) appears to offer the same improvement. The same accuracy can be achieved with L.H.S. using 4% as many trials as would be required using conventional Monte Carlo sampling.

I need to do more experimentation in order to formulate a "stopping rule guide." If any readers are familiar with a mathematical solution or other alternative, please let me know. Watch this same Tip for more details.

—John Schuyler, October 1997. Revised 15-Oct-97.

Copyright 1997 by John R. Schuyler. All rights reserved. Permission to copy with reproduction of this notice.