Designing a new product versus refining an existing one: 22-Mec-B5 Product Design and Development, December 2018, Q5

Question 5: Designing a new product versus refining an existing one (15 marks)

Question text not reproduced: the examination questions are © Engineers and Geoscientists BC. Open the official past paper (linked at the top of this page) to read the question, then follow the worked solution below.

Part A — Comparison of the two design processes

The difference is not one of degree. In a new-product programme the problem itself is an output of the early work, whereas in a refinement programme the problem arrives already defined by field data. Everything else follows from that.

The designer of a genuinely new product begins with opportunity identification and user research, because there is no incumbent whose failures can be counted. The research must run to saturation rather than to a schedule, since the object is to discover the set of needs rather than to measure a known one. Concept generation is broad and the architecture is a free variable: how functions are allocated to physical elements, what is integral and what is modular, what is made and what is bought, are all open. Selection is made under real uncertainty, so the process is staged, with cheap experiments placed against the riskiest assumptions first, and a high proportion of concepts is expected to be killed. There is no baseline, so progress is measured against a hypothesis.

The designer refining an existing product begins with a Pareto analysis of warranty claims, returns, service records, reviews and manufacturing scrap. The architecture is frozen and the change set is bounded by carry-over: existing tooling, an existing supply base, an existing service network and, often, a requirement for interchangeability with parts already in the field. The dominant risk is not that the idea is wrong but that the change breaks something adjacent, so regression testing consumes a large share of the effort. The compensating advantage is decisive: a baseline exists, so any claimed improvement can be measured rather than argued.

Figure 5.1 — Two different shapes of process. The new-product programme diverges before it converges and treats the problem statement as an output; the refinement programme is a bounded loop around a frozen architecture, closed by measurement against a baseline.

Dimension	Totally new product	Refinement of an existing product
Source of the problem	Discovered through user research; the definition is an output	Given by field data: warranty, returns, service, reviews, scrap
Architecture	A free variable; function allocation and modularity are open	Frozen; carry-over of tooling, supply base and service parts constrains the change set
Dominant uncertainty	Whether the need and the concept are right	Whether the change breaks something adjacent (regression)
Concept generation	Broad, many alternatives, most killed	Narrow, targeted at named defects and named cost lines
Prototyping	Many cheap, low-fidelity models aimed at the riskiest assumption	Fewer, high-fidelity models on production-representative tooling
Reference for success	A hypothesis; no baseline exists	The incumbent product; every claim is a measured delta
Failure rate expected	High, and managed by staging the spend	Low, and managed by validation coverage

Part B — Differences in how the process is guided

The new-product process is guided by a product vision and a mission statement — the target market, the value proposition, the assumptions and the constraints — and then by staged economic gates that decide whether to keep spending. The guide is a hypothesis to be tested, and it is expected to change. The refinement process is guided by a prioritised defect and cost list: a ranked account of what the product is doing wrong in the field and what it costs, plus a carry-over rule that states what may not change. The guide is a target and it is expected to hold.

The sample sizes that steer the two are different in kind, and confusing them is a common and expensive error.

Given. In discovery, each interview is assumed to surface any given issue type with probability $p = 0.12$, and 95 per cent coverage of the issue types is required. In refinement, an incidence must be estimated to within $\pm 3.5$ percentage points at 95 per cent confidence, with no prior estimate of the proportion. Find. The sample size each activity needs.

Figure 5.2 — Discovery converges quickly and then flattens. The 24th interview takes coverage to 95.4 per cent; six more buy only 2.4 further points, which is the quantitative form of the familiar twenty-to-thirty rule of thumb.

Size the discovery sample for coverage, not for precision. If each interview independently reveals an issue with probability $p$, the chance an issue is still unseen after $n$ interviews is $(1-p)^n$, so the sample needed for coverage $C$ is $$n = \frac{\ln(1-C)}{\ln(1-p)} = \frac{\ln 0.05}{\ln 0.88} = \frac{-2.9957}{-0.1278} = 23.4 \Rightarrow \boxed{n = 24\ \text{interviews}}$$
Check the marginal return, which is what tells you when to stop. Coverage at 24 interviews is $1-0.88^{24} = 95.4\ \%$ and at 30 it is $97.8\ \%$: six more interviews, at perhaps a week of calendar time, buy 2.4 points. That flattening is what saturation means, and it is the reason discovery is stopped by a curve rather than by a budget.
Size the refinement sample for precision, because a baseline already exists. Here the question is not what exists but how big it is, so the binomial precision formula governs, with $p = 0.5$ as the conservative worst case: $$n = \frac{z^{2}\,p(1-p)}{E^{2}} = \frac{1.96^{2}(0.25)}{0.035^{2}} = \boxed{784\ \text{respondents}}$$
Note the ratio and what it means for guidance. Discovery needs 24 conversations and measurement needs 784 responses — a factor of thirty-three — because they answer different questions. Running 784 surveys to discover needs is expensive and still misses the unarticulated ones; running 24 interviews to size an incidence produces a number with a confidence interval so wide it cannot guide anything.

Part C — Differences in how the process is assessed

The new-product process is assessed against learning and staged economics: has the riskiest assumption been tested, did the concept survive, and is the expected commercial value still positive given what the next stage will cost? Sales cannot be the measure, because there are none. The refinement process is assessed against the baseline, with a statistical test, because the whole point of refining is that the improvement can be demonstrated.

Given. A design change intended to reduce warranty returns. Before the change, 377 claims from 6 500 units shipped; after it, 310 claims from 7 200 units. Find. Whether the reduction is real.

Compute the two rates and the pooled estimate. $\hat p_1 = 377/6\,500 = 0.05800$ and $\hat p_2 = 310/7\,200 = 0.04306$, a reduction of 1.49 percentage points; under the null hypothesis of no change the pooled proportion is $\bar p = 687/13\,700 = 0.050146$.
Form the two-proportion test statistic. $$z = \frac{\hat p_1 - \hat p_2}{\sqrt{\bar p\,(1-\bar p)\left(\frac{1}{n_1}+\frac{1}{n_2}\right)}} = \frac{0.014944}{0.0037339} = \boxed{4.00}$$ with a one-sided $p$-value of about $3.1 \times 10^{-5}$, so the improvement is real at any conventional level.
State what the test does not cover. The metric and the sample size must be fixed before the data are seen, or the analysis becomes a search for a favourable comparison; and the two populations must be comparable, since a shift in the customer mix or in the season would move the incidence without any design change at all. Assessment without those controls measures the market and reports it as engineering.

For the new-product programme the corresponding instrument is the stage-gate expected commercial value, which prices the remaining uncertainty explicitly:

$$\begin{aligned} ECV &= \left[(PV \times P_{cs}) - C\right]P_{ts} - D \\ PI &= \frac{ECV}{D} \end{aligned}$$

Evaluating it for the two programmes side by side makes the contrast concrete. For the new product, with a commercial value of CAD 4.80 M, a probability of commercial success of 0.70, remaining commercialization cost of CAD 1.10 M, a probability of technical success of 0.80 and remaining development cost of CAD 0.62 M:

$$ECV_{\text{new}} = \left[(4.80)(0.70) - 1.10\right](0.80) - 0.62 = \boxed{\text{CAD } 1.188\ \text{M}}, \quad PI = 1.92$$

For the refinement, with CAD 1.40 M of value, 0.95 commercial and 0.97 technical probability, CAD 0.18 M of commercialization cost and only CAD 0.09 M of development cost:

$$ECV_{\text{ref}} = \left[(1.40)(0.95) - 0.18\right](0.97) - 0.09 = \boxed{\text{CAD } 1.026\ \text{M}}, \quad PI = 11.39$$

The refinement creates about 14 per cent less value on a productivity index almost six times higher. That is the structural reason portfolios drift towards incremental work: ranked on productivity alone, refinement wins every time, and a company that ranks that way will have no new products in five years. The correct response is to assess the two streams in separate buckets with separate budgets, not to pretend they are commensurable.

Quantity	Result
Discovery interviews for 95 per cent issue coverage at p = 0.12	24
Coverage at 24 interviews / at 30 interviews	95.4 per cent / 97.8 per cent
Survey sample for ±3.5 points at 95 per cent confidence	784
Warranty incidence before / after	5.800 per cent / 4.306 per cent
Two-proportion test statistic	z = 4.00 (one-sided p ≈ 3.1 × 10⁻⁵)
New-product ECV and productivity index	CAD 1.188 M, PI = 1.92
Refinement ECV and productivity index	CAD 1.026 M, PI = 11.39

Topic. Front-end design process, product-development portfolio management, and the evidence each uses.

Key relations. Coverage sampling $n = \ln(1-C)/\ln(1-p)$; precision sampling $n = z^{2}p(1-p)/E^{2}$; two-proportion test $z = (\hat p_1 - \hat p_2)/\sqrt{\bar p(1-\bar p)(1/n_1 + 1/n_2)}$; expected commercial value $ECV = [(PV\,P_{cs}) - C]P_{ts} - D$ and productivity index $PI = ECV/D$.

Why this works. New-product work and refinement work differ in what is uncertain, and the right method follows the uncertainty. When the set of needs is unknown, the sampling problem is one of coverage, and coverage saturates quickly because each observation is a fresh draw on a small set of issue types. When a quantity must be estimated, the problem is one of precision, and precision improves only with the square root of the sample, which is why measurement costs an order of magnitude more observations. Assessment follows the same split: with no baseline, the only honest measure is whether the riskiest assumption has been tested and whether the expected value still justifies the next stage; with a baseline, a straightforward hypothesis test settles the question, provided the metric was chosen in advance.

Common pitfall. Assessing a new-product programme with refinement metrics. Asking for a business case accurate to two decimal places at concept stage, or ranking a portfolio on productivity index alone, systematically kills the work with the highest variance and the highest value. The mirror error is assessing a refinement without a pre-registered metric, which turns the analysis into a search for whichever comparison happens to look good.

Source. Ulrich and Eppinger, Product Design and Development, chapters 3-5 (opportunity identification, product planning, identifying customer needs); Cooper, Winning at New Products (stage-gate and the ECV portfolio model); Dieter and Schmidt, Engineering Design, chapter 3; Pahl and Beitz, Engineering Design, chapter 5.

Question 5 of 7: Designing a new product versus refining an existing one

Question 5: Designing a new product versus refining an existing one (15 marks)

Part A — Comparison of the two design processes

Part B — Differences in how the process is guided

Part C — Differences in how the process is assessed