23-Ind-A6 Systems Simulation · Undated paper

Question 7 of 10: Simulation Validation — Comparing Variances and Means to the Actual System

Nivaar worked solution (AI-drafted; not reviewed by a licensed engineer)

Notes on this paper

National Exams May 2019 — 17-Ind-A6, Systems Simulation, 3 hours duration. Section A: do 3 of 5 (30 marks); Section B: do 1 of 2 per the front-page summary table — the Part B section instructions on the page itself read “do 2 of 3” instead, a genuine inconsistency in the source paper's own front matter (flagged below); Section C: do 1 of 2 (20 marks). This study resource answers all ten questions across the three parts.

Check — front-page/section instructions disagree. The cover page's summary table states “Section B: Do 1 of 2 Questions, Total marks: 20,” but Part B's own section banner (page 4) reads “Complete two of the following three sets of questions,” and Part B in fact contains three questions worth 15 marks each. The cover page's own overall total (70 marks across “5 Questions”) is likewise only consistent with the “2 of 3” reading (30+30+... does not match 70 either way exactly, since 30(A)+2×15(B)+20(C)=80, one section choice short of 70); this is an internal inconsistency in the exam's own printed materials. All three Part B questions are answered in full below regardless.

Reference texts. Banks, Carson, Nelson & Nicol, Discrete-Event System Simulation (5th ed.) — primary text for random-variate generation, input/output analysis, variance reduction, verification & validation, and design of simulation experiments.

Question 7 (Part B.2): Simulation Validation — Comparing Variances and Means to the Actual System (15 marks)

Question text not reproduced: the examination questions are © Engineers and Geoscientists BC. Open the official past paper (linked at the top of this page) to read the question, then follow the worked solution below.

Given. Actual system, $n_1=7$ years: $73.39,\,69.98,\,71.63,\,71.67,\,72.28,\,72.36,\,70.29$ (mean $=71.657$, $s_1^2=1.428$). Simulation, $n_2=7$ replications: $70.76,\,70.19,\,71.39,\,70.07,\,71.01,\,70.33,\,70.27$ (mean $=70.574$, $s_2^2=0.2407$).

Find. (a) an $F$-test for equal variances; (b) a test for equal means, conditional on (a)'s outcome, and a validity conclusion at $\alpha=0.05$; (c) cautions on the conclusion; (d) whether unequal variances with equal means implies validity.

Approach. Two-sample $F$-test on variances first (this determines which $t$-test is appropriate); Welch's (unequal-variance) $t$-test on means if (a) rejects equal variance; interpret jointly.

(a) F-test for equal variances. $H_0:\sigma_1^2=\sigma_2^2$ vs. $H_1:\sigma_1^2\ne\sigma_2^2$. With the larger sample variance in the numerator, $$F=\frac{s_1^2}{s_2^2}=\frac{1.428}{0.2407}=5.93,\qquad F_{0.025,6,6}=5.82.$$ Since $5.93>5.82$ (two-sided $p=0.048<0.05$), $$\boxed{\text{reject}\ H_0\ \text{at}\ \alpha=0.05 \;-\; \text{the variances are statistically different}}$$ (the actual system's day-to-day admissions vary noticeably more than the simulation's replicate means do).
(b) Test the means and conclude on validity. Because (a) found unequal variances, the appropriate comparison of means is Welch's (unequal-variance) $t$-test rather than the pooled-variance test: $$t=\frac{71.657-70.574}{\sqrt{1.428/7+0.2407/7}}=2.218,\qquad df_{\text{Welch}}\approx7.97,\qquad t_{0.025,8}=2.306.$$ Since $|2.218|<2.306$ ($p\approx0.058>0.05$), $$\boxed{\text{fail to reject}\ H_0:\mu_1=\mu_2 \;-\; \text{the means are not statistically different at}\ \alpha=0.05}$$ So the picture is mixed: the simulation reproduces the actual system's average admission rate, but not its variability. Matching the mean alone is not sufficient grounds to call the model an accurate representation — see (d) — so the IE cannot fully conclude her model is valid; it captures central tendency but under-represents day-to-day variability.
(c) Cautions. A hypothesis test that fails to reject $H_0$ is not proof the means are equal — it only means the data don't provide strong enough evidence of a difference, especially with only $n=7$ replications, which gives limited power to detect a real but modest mean gap. At $\alpha=0.05$ there is also a genuine (if small) chance the variance test's rejection is itself a Type I error. Beyond these statistical caveats, matching one aggregate output (mean daily admissions) at one comparison point does not validate the model across all the behaviours it will be used to predict — the IE should also check face validity with subject-matter experts, sensitivity/extreme-condition tests, and comparisons on other output measures (e.g. peak-hour admissions, seasonal pattern) before relying on the model for decisions.
(d) Unequal variances, equal means — is the model valid? No, not fully. Getting the mean right while getting the variance wrong means the simulation reproduces the actual system's central tendency but misrepresents its spread — and spread matters directly for anything downstream that depends on variability (e.g. staffing to cover peak admissions, queueing/wait-time predictions, risk of exceeding capacity), where the actual system's own larger variance would, in this exact scenario, be under-predicted by the model. $$\boxed{\text{Not fully valid} - \text{matching the mean does not compensate for a demonstrably wrong variance}}$$ The correct response is to treat this specific result the same as this question's own (b)/(d) case: report both findings honestly, and either recalibrate the model's sources of variability or restrict its use to applications where only the mean matters.

Item	Result
(a) variance test	$F=5.93>F_{0.025,6,6}=5.82$ — variances differ significantly
(b) mean test (Welch)	$t=2.22
(b) validity conclusion	partial only — mean matches, variance does not
(d) unequal-var/equal-mean case	not sufficient for validity

Concept & Theory

Topic. Statistical validation of a simulation model by two-sample tests on the mean and variance of a key output.

Key relations. $F=s_1^2/s_2^2\sim F_{n_1-1,n_2-1}$; Welch's $t=(\bar x_1-\bar x_2)/\sqrt{s_1^2/n_1+s_2^2/n_2}$ with Welch–Satterthwaite df, used whenever the variance test rejects equality.

Why this works. Testing variance before means (rather than always using the pooled-variance $t$-test) matters because the pooled test's own validity assumes equal variances; using Welch's test once that assumption is shown false keeps the mean comparison honest.

Common pitfall. Declaring a model “valid” the moment one test (usually the mean test) fails to reject — validity is a property of the whole output distribution, and a matched mean with a mismatched variance is a genuinely incomplete validation, exactly the trap this question is built to expose.

Source. Banks, Carson, Nelson & Nicol, Discrete-Event System Simulation, ch. 10 (verification and validation of simulation models).

← Question 6 Next: Question 8 →