Question 9 of 10: Comparing Three Operating Scenarios by One-Way ANOVA
Nivaar worked solution (AI-drafted; not reviewed by a licensed engineer)
Notes on this paper
National Exams May 2019 — 17-Ind-A6, Systems Simulation, 3 hours duration. Section A: do 3 of 5 (30 marks); Section B: do 1 of 2 per the front-page summary table — the Part B section instructions on the page itself read “do 2 of 3” instead, a genuine inconsistency in the source paper's own front matter (flagged below); Section C: do 1 of 2 (20 marks). This study resource answers all ten questions across the three parts.
Check — front-page/section instructions disagree. The cover page's summary table states “Section B: Do 1 of 2 Questions, Total marks: 20,” but Part B's own section banner (page 4) reads “Complete two of the following three sets of questions,” and Part B in fact contains three questions worth 15 marks each. The cover page's own overall total (70 marks across “5 Questions”) is likewise only consistent with the “2 of 3” reading (30+30+... does not match 70 either way exactly, since 30(A)+2×15(B)+20(C)=80, one section choice short of 70); this is an internal inconsistency in the exam's own printed materials. All three Part B questions are answered in full below regardless.
Reference texts. Banks, Carson, Nelson & Nicol, Discrete-Event System Simulation (5th ed.) — primary text for random-variate generation, input/output analysis, variance reduction, verification & validation, and design of simulation experiments.
Question 9 (Part C.1): Comparing Three Operating Scenarios by One-Way ANOVA (20 marks)
Find. (a) a one-way ANOVA test for scenario effect; (b) its underlying assumptions; (c) the best scenario (bigger = better) at a chosen certainty level; (d) a description of the alternative “best of k” method not used in (c).
Approach. Compute the one-way ANOVA table directly from the group sums/means (no raw-data recomputation needed beyond the given table); use the resulting $MS_W$ to build pairwise confidence intervals for (c); describe an indifference-zone ranking-and-selection procedure for (d).
(a) One-way ANOVA. $H_0:\mu_A=\mu_B=\mu_C$. With $k=3$ groups, $n=5$ each, $N=15$, grand mean $\bar{\bar x}=1577/15=105.13$:
$$SS_{between}=n\sum_i(\bar x_i-\bar{\bar x})^2=5\big[(99.2-105.13)^2+(102.6-105.13)^2+(113.6-105.13)^2\big]=566.5$$
$$SS_{total}=\sum(x_{ij}-\bar{\bar x})^2=8073.7,\qquad SS_{within}=SS_{total}-SS_{between}=7507.2$$
One-way ANOVA table
Source
SS
df
MS
F
Between scenarios
566.5
2
283.3
0.453
Within scenarios
7507.2
12
625.6
–
Total
8073.7
14
–
–
With $F_{0.05,2,12}=3.885$ and $F=0.453\ll3.885$ ($p=0.65$),
$$\boxed{\text{fail to reject}\ H_0 \;-\; \text{no statistical evidence the scenarios influence the output metric}}$$
(b) Assumptions underlying the ANOVA. (i) Observations within and across scenarios are independent; (ii) the output metric is (approximately) normally distributed within each scenario; (iii) the three scenario populations share a common variance (homogeneity of variance). Check: assumption (iii) looks doubtful here — the sample standard deviations range from $6.53$ (A) to $39.53$ (C), a ratio of about $6:1$, which is a large disparity for only $n=5$ per group. A Levene/Bartlett test would likely flag heterogeneous variances; with such small samples the classical $F$-test is not very robust to this violation, so the “no evidence of a scenario effect” conclusion in (a) should be treated as provisional rather than definitive.
(c) Which scenario is best? Point estimates alone favour Scenario C ($\bar x_C=113.6$, the highest mean). To check whether that ordering is statistically defensible, build pairwise confidence intervals on the mean differences using the pooled within-scenario variance $MS_W=625.6$ ($df=12$):
$$\bar x_C-\bar x_A=14.4,\quad \bar x_C-\bar x_B=11.0,\qquad SE_{diff}=\sqrt{MS_W\left(\tfrac15+\tfrac15\right)}=15.82$$
Even at a relaxed one-sided $90\%$ confidence level ($t_{0.10,12}=1.356$), the one-sided lower bound on $\bar x_C-\bar x_A$ is $14.4-1.356(15.82)=-7.05$, which still includes $0$:
$$\boxed{\text{no scenario can be called “best” with any conventional statistical confidence} \;-\; \text{C has the highest point estimate, but the gap is not significant even at 90\% one-sided confidence}}$$
This is consistent with (a): a non-significant overall $F$ means no pairwise comparison should be expected to reach significance either. If a decision must nonetheless be made today, Scenario C is the best available point estimate, but it should be reported together with the (large) uncertainty around it — and, per the assumption caveat in (b), Scenario C's own huge internal variance ($s_C=39.5$) is itself doing most of the work in keeping that comparison inconclusive.
(d) The “best of k” method not used in (c). Part (c) above used a multiple-comparison approach (pairwise confidence intervals built from one ANOVA run on the existing 5-observation samples). The method not used is a formal indifference-zone ranking-and-selection procedure (e.g. Rinott's two-stage procedure). In words: first specify an indifference amount $\delta^*$ (the smallest true mean difference the analyst actually cares about detecting) and a desired probability of correct selection $P^*$ (e.g. $95\%$); run an initial ("first-stage") sample from each of the $k=3$ scenarios (the 5 observations already available could serve as this stage) and use their sample variances to compute, via Rinott's constant for $k$ systems and the chosen $(\delta^*,P^*)$, the additional number of replications each scenario needs in a second stage; run that many additional replications per scenario; then select the scenario with the largest overall sample mean. Unlike the pairwise-CI approach in (c), this procedure is explicitly designed to guarantee (with probability $\ge P^*$) that the scenario selected is truly the best, or within $\delta^*$ of the best, rather than only reporting whether an already-collected sample can distinguish the scenarios.
