It is generally recommended that the correlation coefficient should be as a minimum greater than 0.80. Values as high as .95-.99 are routinely achieved in DOE testing. A word of caution though, this is only a necessary condition for a good model. A high correlation coefficient only means that the collected data can be well explained by the empirical model. Nonlinearities could be present but ignored by 2-level testing. Interactions could be present but not uncovered by certain DOE designs. A correlation coefficient is a necessary but not sufficient condition for effective modeling. Full confidence in the results is only possible with verification runs that confirm the model predictions.
How do I determine the value ranges of factors?
There is no standard criteria for this, although a few of the typical issues that can affect the factor ranges can be discussed. If the range is too small there is a danger that the estimates of the response sensitivities will be corrupted
significantly by process and measurement noise. If the range is too large than the response will likely be nonlinear and a third level should be added to account for this. The range should be larger than the potential range of control factor settings (that is, we don’t want to have to extrapolate when setting their optimal levels). The domain expert who is most familiar with the process will have the best feel for factor ranges.
What is a reasonable set of factors to consider in any one test matrix?
For Modeling or Robustness DOEs it is recommended that 5 or less factors are usee. For Screening DOEs many more factors are routinely considered.
What is the right DOE design for my interest?
Yes there is, consult section 2.2 of this document that gives a flowchart for selecting the best DOE design.
What is CPM and how is it related to DOE testing?
CPM, which stands for Critical Parameter Management, is one of the tools in the overall Design Process Certification strategy. It represents an approach to controlling and managing variability in the product development process.
What are D-optimal designs and how can they be used in DOE testing? D-optimal designs are a fairly recent addition (in the last 20 years) to DOE designs that are based on the notion of using computer optimization methods to create a test matrix that is tailored to a precise situation. That is, the user must define what interactions are important and which ones are not. Computer software packages then can customize a DOE design to meet these specific requirements. The advantages of D-optimal designs is that they represent the minimum set of tests required for any category of DOE testing (screening or modeling). The disadvantages are that they require software packages to
generate the tables and analysis must be done with regression which can compound the problem of results interpretation.
What does the p value mean in the ANOVA summary?
The “p value” is a statistical check of the relevance of the candidate term in the overall regression analysis. In effect, “1 – (p-value)” is the likelihood that the added term is real and significant and not the artifact of noise. A general rule is that the term of interest should be retained in the regression analysis if the p-value is less than 0.1.
What is the recommended means of archiving DOE results?
At present there is no standard in place for archiving DOE results. It is
recommended that an MS-powerpoint presentation be created that captures the results of the seven step DOE process. All data should be saved in the
appropriate format. For example, graphs and data can be stored as a project file in MiniTAB.
What software tool is recommended for DOE processing?
Many commercial software packages are available to the DOE practitioner, including MATLAB, JMP, DOE KISS, and DOE Wisdom. The recommended software from UTC is MiniTAB.
How do I handle discrete data?
The most common approach is handle discrete input factor data is to just assume it is continuous and then to round any optimization predictions to the nearest effective input setting. Discrete output data can present many issues in DOE testing. It is preferable to have continuous system response measurements. For example, many times DOE testing is used when reliability issues are detected in a product design. Where possible a continuous output response should be used, for example strain rates, instead of a pass-fail ranking.
How can I effectively handle multiple output variables?
Yes using Desirability Functions as illustrated in Section 2.5 of this document. The concept is that a single scalar variable is created by introducing a
mathematical equation (i.e, the Desirability Function) that combines the multiple output variables.
What should I do if I can’t physically run every point in the DOE design space? This can occur when implementing DOE testing in a product development. There are two ways to handle this. The first is just to assume that the difference between the desired factor setting from the original DOE design and what can be implemented is small and thus the intended value can be used in the data matrix. In this case all the traditional DOE analysis tools will work. If the differences are
considered significant, then the practitioner can modify the value in the data matrix and use a standard linear regression method post-process the data to extract an empirical model.
What is a S/N ratio and how does it help me in DOE testing?
The S/N (signal-to-noise) ratio is a concept developed by Dr. Taguchi for Robustness DOE testing. It is a way of transforming two attributes of an output (it’s mean and standard deviation) into one metric that is used in Robustness DOE data analysis as discussed in detail in Section 2.5.
What do the terms “interaction”, “resolution”, and “aliasing” mean and how do they impact my DOE testing?
These are terms that are used to discuss how specific DOE test matrices can (or can not) account for the significance of various combinations and functional forms of control factors. An interaction means that the levels of one control factor influence the system response sensitivity of another. Mathematically this means that a term of the form K12x1x2 is present in the equation which links the control factors (x1,x2) to the system response Y. When choosing the DOE design (see Section 2.2 for more details) the practitioner must make a prediction about what interactions might be significant or not. If it is believed they are not, then a
reduced number of runs can be made. This fractional factorial design will not be able to distinguish between some factor interactions which will show up in the same way as other potential regression equation terms. This overlapping of potential terms is called “aliasing”. Lastly, resolution is a term applied to DOE designs describing the amount of aliasing present. For example a Resolution III design is one that does not alias main effects with other main effects, does alias main effects with 2-way interactions, and 2-way interactions are aliased with each other. The higher the Resolution, the more terms in the potential empirical model are not aliased.
Why is “one-factor-at-a-time” testing so bad?
One of the main problems with one-at-a-time (OFAT) testing is that a
suboptimal result is likely to be obtained. DOE testing allows the practitioner to uncover details about the process as captured by a response surface model via a Modeling DOE approach that can then be used to optimize the system behavior. OFAT testing in a sense is executing a localized gradient search optimization that will terminate at a local minima.
An additional issue with OFAT testing is that more test runs can be required due to process variability then for a balanced DOE design such as the Plackett-Burman screening design.
What is “Monte Carlo Simulation” and what does it have to do with DOE testing?
Monte Carlo Simulation (MCS) is a computational technique that can be used to predict the statistical variability of an output given variability in process inputs. An empirical equation, such as would be generated using a Modeling DOE
approach, that captures the functional relationship between input control factors and the desired system response is used in conjunction with a random number generator to make a large number of trial runs. The statistics of the output
variability can then be compiled. Section 2.6 illustrate how MCS can be used to generate specifications for control inputs as the final step in the DOE testing process.
I’ve heard that UTC is introducing a new tool into ACE called “Design Process Certification”. How does DOE fit in with this tool?
DOE is one of the key tools in Design Process Certification (DPC or Procert), UTC’s new approach for Robust Design. The intent of Procert is to apply
Process Certification techniques to the PDP to reduce variation and improve the robustness of our products and to improve the efficiency and effectiveness of the passport process.
What is “Six Sigma” and how does it relate to DOE testing?
“Six Sigma” has become a term associated with a process of ensuring quality in the presence of variability of subcomponents that was developed at Motorola in the 1980s. It was later deployed at GE and Allied Signal in the 1990s primarily in manufacturing using a five step process (Define, Measure, Analyze, Improve, and Control (DMAIC) . Design for Six Sigma (DFSS), with the steps (Define, Measure, Analyze, Design, and Verify (DMADV)) is a variant of SixSigma directed at improving the product development process. The goal of both methods is to manage and control variability to minimize defects to be smaller than 3.4 defects per million (the rate of escapes for a process with a capability index of +/- 6 sigma [Note: this would be a Cp value of 2.0]).
The DOE testing process described in this document is part of UTC’s Design Process Certification tools that will help reduce defects and thereby increase quality and customer satisfaction.
Can DOE be used to control time-varying transients?
Yes, by using Desirability Functions (see Section 2.5) coupled to a parametric characterization of the time transient waveform. For example, some typical parameters that can be used to desribe a time transient are overshoot, setting time, and delay. Each of these could be extracted from the DOE testing and then linked to a single scalar Desirability Function.
Can DOE methods be applied to numerical simulation predictions?
Yes, the Modeling DOE principles discussed in the document are directly applicable to numerical simulations. They represent the means to efficiently extract an empirical model response surface from computer simulations which can then be used to drive lower level control factor specifications.
What are some of the keys to successful DOE testing?
Some key points are: set clear objectives, if at all possible use quantitative measures, carefully consider process variations and use replications to minimize their impact, randomize the test run order if feasible to minimize external noise sources, consider potential aliasing-induced errors, and confirm results using verification runs.
What is “pooling” and how can I use it in DOE testing? What is the F-ratio?
When should I use a Robustness DOE vs. a Modeling DOE? What is the difference between Static and Dynamic SN ratios?