Most simulations are designed so they
converge to what might be called the
"true solution" of the model.
But, because a simulation can only be
run for a finite amount of time, this
true solution can never be known. This
gives rise to two important questions:
What is the accuracy in the results
of a simulation's output? How long should
a simulation be run in order to obtain
a given accuracy? These questions can
be answered using confidence intervals
and run-length control algorithms.
Using an ad hoc technique instead of
the methods described in this section
can be dangerous as well as wasteful.
Running a simulation for too short an
amount of time will result in performance
statistics that are highly inaccurate.
Running a simulation for an unnecessarily
long amount of time wastes computing
resources and delays the completion
of the simulation study. Without some
type of formal analysis, the errors
in simulation results cannot be quantified.
A confidence
interval is a range of values
in which the true answer is believed
to lie with a high probability. The
interval can be specified in two equivalent
ways, either by specifying the midpoint
of the interval (which could be considered
the "best guess" for the true
answer) and the half-width of the interval,
or by specifying the lower and upper
bounds of the interval. CSIM reports
the confidence interval in both formats,
as illustrated below:
The probability that the true answer
lies within the interval is called the
confidence
level. Since a confidence level
of 100% would result in an infinitely
wide confidence interval, confidence
levels from 90% to 99% are most often
used. Be aware that there is always
a small probability (dictated by the
confidence level) that the true answer
lies outside the confidence interval.
Confidence intervals can be automatically
generated for the mean values in any
table, qtable, meter, or box simply
by calling one of the following functions
immediately after the statistics object
has been initialized.
Prototype:
void table_confidence(TABLE
t)
Prototype:
void qtable_confidence(QTABLE
qt)
Prototype:
void meter_confidence(METER
m)
Prototype:
void box_time_confidence(BOX
b)
Prototype:
void box_number_confidence(BOX
b)
The technique used to calculate confidence
intervals is called batch
means analysis. It is beyond
the scope of this manual to describe
the mathematics underlying this technique,
but any good simulation text should
provide details.
If confidence intervals have been requested
for a table, qtable, meter, or box,
the statistics report will include a
section like the following.
Notice that confidence intervals are
calculated for three commonly used confidence
levels: 90%, 95%, and 98%. The confidence
intervals are reported in both of the
formats described previously. The relative
error measures the accuracy in the midpoint
of the interval as an estimate of the
true answer. It is defined to be the
half-width divided by the lower bound
of the interval. Like any relative error,
its value suggests how many accurate
digits there are in the estimate.
The algorithm for computing confidence
intervals groups the observations into
fixed size batches and uses only complete
batches. For this reason, the number
of observations used in the calculation
of the confidence intervals may be slightly
less than the number of observations
used in computing the other performance
statistics. For example, in the above
report 50,000 observations were used
to calculate the confidence intervals.
The part of the report not shown may
give the mean, variance, standard deviation,
etc.
based on 50,472 observations.
The algorithm also requires a minimum
number of observations for its results
to be valid. This minimum number cannot
be known before running the simulation
because it depends on the amount of
correlation found in the statistic.
If a report is produced before sufficient
observations have been obtained, the
message
> insufficient
observations to compute confidence intervals
will appear in place of the confidence
intervals. To obtain confidence intervals,
run the simulation longer or use the
run length control algorithm.
All values calculated by the confidence
interval algorithm can be retrieved
during the execution of a model or upon
its completion.
| Prototype: |
Functional
Value: |
| long table_batch_size(TABLE
t) |
size of batch |
| long table_batch_count(TABLE
t) |
number of batches |
| double table_conf_mean(TABLE
t) |
midpoint of interval |
| double table_conf_halfwidth(TABLE
t, double conf_level) |
half-width of
interval |
| double table_conf_lower(TABLE
t, double conf_level) |
lower bound of
interval |
| double table_conf_upper(TABLE
t, double conf_level) |
upper bound of
interval |
| double table_conf_accuracy(TABLE
t, double conf_level) |
accuracy achieved |
| long qtable_batch_size(QTABLE
qt) |
size of batch |
| long qtable_batch_count(QTABLE
qt) |
number of batches |
| double qtable_conf_mean(QTABLE
qt) |
midpoint of interval |
| double qtable_conf_halfwidth(QTABLE
qt, double conf_level) |
half-width of
interval |
| double qtable_conf_lower(QTABLE
qt, double conf_lvel) |
lower bound of
interval |
| double qtable_conf_upper(QTABLE
qt, double conf_level) |
upper bound of
interval |
| double qtable_conf_accuracy(QTABLE
qt, double conf_level) |
accuracy achieved
|
The conf_level
parameter specifies the desired confidence
level and should be a value between
0.0 and 1.0.
If confidence intervals have not been
requested or if there have not been
sufficient observations to calculate
confidence intervals, all of the above
functions return zero values.
To inspect confidence interval information
for meters and boxes, pass to the appropriate
function listed above a pointer returned
by one of the following functions: meter_ip_table,
box_time_table,
or box_number_qtable.
If the reported confidence intervals
show that the needed accuracy has not
been achieved, a simulation could be
run again for a longer amount of time.
This has two disadvantages: repeating
part of the simulation is wasteful,
and it may not be clear how much longer
to run the simulation the second time.
A better method is to use the run length
control algorithm that is built into
CSIM. This algorithm monitors the confidence
interval as it narrows and automatically
terminates the simulation when the desired
accuracy has been achieved.
To use run length control, choose a
performance measure that will be used
to decide when the simulation should
terminate. Instrument the model to gather
statistics on this performance measure
using a table, qtable, meter, or box.
Immediately after the statistics gathering
object has been initialized, call the
appropriate function below.
Prototype:
void table_run_length(TABLE
t, double accuracy, double conf_level,
double max_time)
Example:
table_run_length(t, 0.01, 0.95, 10000.0);
Prototype:
void qtable_run_length(QTABLE qt, double
accuracy, double conf_level, double
max_time)
Prototype:
void meter_run_length(METER
m, double accuracy, double conf_level,
double max_time)
Prototype:
void box_time_run_length(BOX
b, double
accuracy, double conf_level, double
max_time)
Prototype:
void box_number_run_length(BOX
b, double
accuracy, double conf_level, double
max_time)
The accuracy parameter specifies the
maximum relative error that will be
allowed in the mean value of this performance
measure. A value of 0.1 is usually used
to request one digit of accuracy, 0.01
is used to request two digits of accuracy,
and so forth. The conf_level
parameter is the confidence level and
usually has a value between 0.90 and
0.99. The max_time
parameter places an upper bound on how
long the simulation will run. If the
specified accuracy cannot be achieved
within this time, the simulation will
terminate and a warning message will
appear in the report.
In the main CSIM process, place the
following call to the wait
function.
"Converged" is a built-in
event that does not need to be declared
or initialized. This event is set when
the run length control algorithm determines
that the requested accuracy has been
achieved or when the maximum time has
passed.
If run length control has been enabled,
the statistics report will include a
section like the following.
The confidence interval is reported
in both formats for the confidence level
that was specified. If the requested
accuracy was not achieved or if there
were not enough observations to calculate
confidence intervals, a warning message
will appear in the report.
The mechanics for running a simulation
until multiple performance measures
have been obtained to desired accuracy
are simple. Call the appropriate run
length function for several statistics
gathering objects and then wait on the
"converged" event as many
times as there are statistics to converge.
However, there are some subtleties in
the theory underlying this procedure.
Persons interested in this topic should
read section 9.7 of Simulation
Modeling and Analysis by Law
and Kelton.
Confidence intervals attempt to bound
the errors in performance statistics
caused by running a simulation for a
finite amount of time. They in no way
measure the errors caused by the model
being an unfaithful representation of
the actual system.
All known techniques for computing confidence
intervals are heuristics. Detecting
and removing correlation from performance
data is a mathematically difficult problem.
Confidence intervals should always be
considered to be estimates.
In spite of these limitations, it is
our belief that confidence intervals
and run length control play an essential
role in any simulation study. Simply
running a simulation for a "long
time" and hoping that the performance
measures will be highly accurate is
an unprofessional and dangerous approach.
