23-Ind-A6 Systems Simulation · May 2018

Question 7 of 9: Sample Size, Vendor Comparison, Validation Test

Nivaar worked solution (AI-drafted; not reviewed by a licensed engineer)

Notes on this paper

National Exams — May 2018 — 17-Ind-A6 Systems Simulation. Three-hour, closed-book exam; one of two permitted calculators (Sharp or Casio), one 8.5″×11.0″ aid sheet (both sides). Format: three sections — Section A (four concept questions, candidates choose any two, 10 marks each, 20 marks total), Section B (three methods questions built around one continuing warehouse-simulation case study, candidates choose any two, 15 marks each, 30 marks total), Section C (two applications questions, candidates choose any one, 20 marks each). All nine questions are solved below for completeness. Two source anomalies are flagged where they occur: the front-page summary table states Section A is "Do 2 of 3," while Section A's own instructions and its four printed question sets read "two of the following four" — the printed four-question section is answered in full here; and Part C Question 2's sub-parts (a)–(c) are never printed anywhere in the paper, even though the results text for (d)–(f) explicitly refers back to "the factorial design matrix in (a)." Two-page Normal-distribution tables were supplied with the exam; the values below are the same table values obtained by direct computation.

Reference texts: Banks, Carson, Nelson & Nicol, Discrete-Event System Simulation (5th ed., Pearson) — random-number/random-variate generation, input modeling and goodness-of-fit testing, output analysis (warm-up, replication length, batch means vs. replication/deletion), and comparing alternative systems; Montgomery, Design and Analysis of Experiments (current ed., Wiley) — single-factor ANOVA, multiple comparisons, and 2^k factorial designs with interaction analysis (Part C).

Question 7 (Part B.3): Sample Size, Vendor Comparison, Validation Test (15 marks)

Question text not reproduced: the examination questions are © Engineers and Geoscientists BC. Open the official past paper (linked at the top of this page) to read the question, then follow the worked solution below.

Given. (a) baseline model $N(6000,50^2)$ monthly throughput, target detectable difference $\delta=15$ units, powered "19 times out of 20." (b) Nico's model output $N(6104,70^2)$; vendor point estimate $6053$ units/month, no vendor SD. (c) six months of paired Simulation vs. Warehouse monthly throughput:

Simulation vs. warehouse monthly throughput (6 months)
Month	1	2	3	4	5	6
Simulation	4040	4020	4005	4001	4046	3977
Warehouse	4060	4060	4051	4024	4028	4061

Find. (a) approximate number of replications required; (b) a statistical comment on the model vs. the vendor's estimate; (c) evidence for/against the simulation being representative of the real warehouse.

Approach. (a) invert the standard replication-count formula for the given target precision and confidence; (b) form a CI around the model's own estimate and check whether the vendor's value falls inside it; (c) run a paired $t$-test on the six matched months.

(a) Replications for a target precision. "19 times out of 20" reads as a $95\%$ criterion, $z_{0.025}=1.96$: $$n\approx\left(\frac{z_{\alpha/2}\,s}{\delta}\right)^2=\left(\frac{1.96\times50}{15}\right)^2=(6.533)^2=42.68\ \Rightarrow\ \boxed{n=43\text{ replications (round up)}}$$
(b) Compare Nico's model to the vendor's estimate. The model's output distribution ($N(6104,70^2)$, the sampling distribution of the mean over the 5 replications already run) gives a 95% CI on the model's expected throughput: $$6104\pm1.96(70)=[\,5966.8,\ 6241.2\,]$$ The vendor's $6053$ lies comfortably inside this interval ($z=(6053-6104)/70=-0.73$, $|z|<1.96$), so $\boxed{\text{no statistical evidence that Nico's model disagrees with the vendor's reported throughput}}$ — the two are consistent, though absent a vendor SD this necessarily treats $6053$ as a fixed constant rather than a second random estimate.
(c) Paired validity test. Simulation and Warehouse describe the same six calendar months, so pair by month: $d_i=\text{Warehouse}_i-\text{Simulation}_i = 20,\ 40,\ 46,\ 23,\ -18,\ 84$. $$\bar d=32.5,\quad s_d=33.73,\quad t=\frac{\bar d}{s_d/\sqrt6}=\frac{32.5}{13.77}=2.360$$ Critical value $t_{0.025,5}=2.571$. Since $|t|=2.360<2.571$ ($p\approx0.065$), $\boxed{\text{fail to reject } H_0:\mu_d=0 \text{ at } \alpha=0.05}$ — there is no statistically significant evidence that the simulation's monthly throughput differs from the warehouse's. The result is borderline (p close to 0.05, and every one of the six differences is positive — the warehouse consistently ran above the simulation), so this should be read as "not yet disproven," not as a strong confirmation of face validity; more paired months of data would sharpen the test.

Item	Result
(a) replications needed	$n\approx43$
(b) model vs. vendor	$6053\in[5966.8,\,6241.2]$ — consistent
(c) paired $t$-test	$t=2.360 \lt t_{0.025,5}=2.571$ ($p\approx0.065$) — fail to reject, borderline

Concept & Theory

Topic. Replication-count sizing, single-estimate comparison to a fixed benchmark, and paired-difference validation testing.

Key relations. $n\approx(z_{\alpha/2}s/\delta)^2$; one-sample $z=(\bar x-\mu_0)/\sigma_{\bar x}$; paired $t=\bar d/(s_d/\sqrt n)$.

Why this works. (a) is a variance-driven sample-size formula; (b) treats the vendor figure as a hypothesized constant to test the model's own sampling distribution against; (c) exploits the natural month-to-month pairing to remove common variation before testing — far more powerful here than an unpaired two-sample test.

Common pitfall. Running an UNPAIRED two-sample test on the Simulation-vs-Warehouse data in (c), which ignores the shared monthly structure, inflates the effective variance, and makes a real difference harder to detect.

Source. Banks, Carson, Nelson & Nicol, Discrete-Event System Simulation, ch. 9 (comparing a system to a fixed standard, paired comparisons, sample-size formulas) & ch. 10 (validation).

← Question 6 Next: Question 8 →