NivaarExam PrepOfficial exam papers ↗

23-Ind-A6 Systems Simulation · May 2018

Question 8 of 9: Comparing Five Scenarios by ANOVA

Nivaar worked solution (AI-drafted; not reviewed by a licensed engineer)

Notes on this paper

National Exams — May 2018 — 17-Ind-A6 Systems Simulation. Three-hour, closed-book exam; one of two permitted calculators (Sharp or Casio), one 8.5″×11.0″ aid sheet (both sides). Format: three sections — Section A (four concept questions, candidates choose any two, 10 marks each, 20 marks total), Section B (three methods questions built around one continuing warehouse-simulation case study, candidates choose any two, 15 marks each, 30 marks total), Section C (two applications questions, candidates choose any one, 20 marks each). All nine questions are solved below for completeness. Two source anomalies are flagged where they occur: the front-page summary table states Section A is "Do 2 of 3," while Section A's own instructions and its four printed question sets read "two of the following four" — the printed four-question section is answered in full here; and Part C Question 2's sub-parts (a)–(c) are never printed anywhere in the paper, even though the results text for (d)–(f) explicitly refers back to "the factorial design matrix in (a)." Two-page Normal-distribution tables were supplied with the exam; the values below are the same table values obtained by direct computation.

Reference texts: Banks, Carson, Nelson & Nicol, Discrete-Event System Simulation (5th ed., Pearson) — random-number/random-variate generation, input modeling and goodness-of-fit testing, output analysis (warm-up, replication length, batch means vs. replication/deletion), and comparing alternative systems; Montgomery, Design and Analysis of Experiments (current ed., Wiley) — single-factor ANOVA, multiple comparisons, and 2k factorial designs with interaction analysis (Part C).

Question 8 (Part C.1): Comparing Five Scenarios by ANOVA (20 marks)

Question text not reproduced: the examination questions are © Engineers and Geoscientists BC. Open the official past paper (linked at the top of this page) to read the question, then follow the worked solution below.

Given. Five scenarios, $n=5$ replications each ($N=25$ total); grand sum $=100{,}209$, grand mean $=4008$, grand (pooled) variance $=982.3233$:

Per-scenario summary statistics
Scenario 1Scenario 2Scenario 3Scenario 4Scenario 5
Sum1999020014199661999320246
Average3998.04002.83993.23998.64049.2
Variance96.5267.2723.254.32088.7

Find. (a) which scenario, if any, is provably the best at $\alpha=0.05$; (b) the assumptions the analysis relies on; (c) one alternative analysis method, described but not calculated.

Approach. Run a one-way ANOVA across the five scenarios using the given summary statistics (within-scenario SS from the per-scenario variances, total SS from the pooled overall variance, between-scenario SS by subtraction); if significant, follow up with a pairwise comparison of the top two means.

  1. Within-scenario sum of squares. $SSW=\sum(n_i-1)s_i^2$: $$SSW=4(96.5+267.2+723.2+54.3+2088.7)=4(3229.9)=\boxed{12919.6}$$
  2. Total sum of squares from the given overall variance ($N=25$): $$SST=(N-1)(982.3233)=24\times982.3233=\boxed{23575.76}$$
  3. Between-scenario sum of squares and the $F$-test. $$SSB=SST-SSW=23575.76-12919.6=\boxed{10656.16}$$ $$MSB=\frac{SSB}{k-1}=\frac{10656.16}{4}=2664.04,\qquad MSW=\frac{SSW}{N-k}=\frac{12919.6}{20}=645.98$$ $$F=\frac{MSB}{MSW}=\boxed{4.12}$$ With $F_{0.05,4,20}=2.87$: since $4.12>2.87$ ($p\approx0.013$), reject $H_0$ — not all five scenario means are equal.
  4. Identify the best scenario. Compare the highest mean (Scenario 5, $4049.2$) against the runner-up (Scenario 2, $4002.8$) using a pooled-variance (Fisher LSD) $t$-test: $$SE_{diff}=\sqrt{\frac{2\,MSW}{n}}=\sqrt{\frac{2(645.98)}{5}}=16.06,\qquad t=\frac{4049.2-4002.8}{16.06}=\boxed{2.89}$$ Critical value $t_{0.025,20}=2.086$. Since $2.89>2.086$, Scenario 5 is significantly better than its closest competitor, and therefore better than the remaining three scenarios as well (all have lower means than Scenario 2): $$\boxed{\text{Scenario 5 is provably the best, at } \alpha=0.05}$$
SourceSSdfMSF
Between scenarios10656.1642664.044.12
Within (error)12919.6020645.98—
Total23575.7624——
$F=4.12 \gt F_{0.05,4,20}=2.87$; LSD $S5$ vs. $S2$: $t=2.89 \gt t_{0.025,20}=2.086$ — Scenario 5 provably best

(b) Assumptions. The ANOVA/LSD analysis above assumes: (i) the five replications within each scenario are independent (separate random-number streams); (ii) the 12-month throughput within each replication is itself a valid, warm-up-corrected steady-state estimate (i.e. the Part B methodology was already applied); (iii) the five per-scenario populations of replicate means are approximately Normally distributed; and (iv) the five scenarios share a common (homogeneous) variance, which the pooled $MSW$ term relies on. This last assumption is the analysis's real weak point here — the given variances range from $54.3$ (Scenario 4) to $2088.7$ (Scenario 5), nearly a $40$-fold spread — so the classical equal-variance $F$-test above should be read with that caveat (see part (c)).

(c) An alternative method. A natural alternative that does not require the equal-variance assumption is the Kruskal–Wallis test, the nonparametric (rank-based) analogue of one-way ANOVA: replace each of the 25 observations by its rank across the pooled sample, then compare the sum of ranks in each scenario group using a chi-squared-distributed test statistic. Because it works on ranks rather than raw magnitudes, Kruskal–Wallis is insensitive to the badly unequal variances (and any non-normality) flagged in (b), at some cost in statistical power if the classical ANOVA assumptions had actually held.