The Find Effects Tab of the Response Analysis UI helps the user to identify important effects in their data. Like the Quick View Tab, it is intended to be used as an exploratory tool, part of the ground work for the later task of creating a model, where the user must choose what effects and interactions to include. In contrast to the Quick View Tab, the Effects Tab enables a more systematic or quantitative, exploratory approach.

For some DOE designs (e.g. full-factorial designs) one can estimate effects by averaging the data in simple ways. For example, for a design factor taking two values in a full-factorial design, the effect is just the average response at the high value minus the average response at the low value. The significance of these estimated effects (are they significantly different from zero?) can be assessed separately.

The Effects Tab prototypes a generalization of this process to designs that are not orthogonal (e.g. spacefill designs), not exactly orthogonal (e.g. otherwise orthogonal designs with failed runs, or excluded points) and/or designs with categorical factors, for which the effects calculations become more intricate and harder to interpret.

Nested Model Comparisons

The key idea is to mimic (in fact, pre-calculate) a sequence of model-building actions that a user might follow, in order to discover the important effects in their data. The concept of a nested model is central to the approach.

Consider a dataset with a readout or response variable R and three design factors X, Y and Z. The factors might be categorical or numerical. Both types may be present. To explore this data, the user might start by building the following three models:

R ~ 1 + X
R ~ 1 + Y
R ~ 1 + Z

These are the “main effect” models, one for each factor. In each case, the user would examine the results of the fitting procedure, to assess whether there is a significant dependency on the factor, how much of the variability is captured, and perhaps look at the residual errors and/or the scatter of the data around the fitted values.

To assess significance of a model, the user might examine the ANOVA test results, which compare residuals of the null model (in this case, the constant model with no dependence on the factor) to the residuals of the model including the factor.

Suppose, on this basis, the user decided that factors X and Y were important, but that Z had no apparent effect.

A next stage of modelling might be to consider whether there is an interaction between X and Y. To assess this, the user might consider the following two models

R ~ 1 + X + Y
R ~ 1 + X + Y + X:Y

They would ask: does adding an X:Y interaction term improve the fit over and above the model with just the X and Y terms? To assess this, they might perform an ANOVA test comparing the residuals of the smaller (or nested) model with the larger (full) model. The term would be judged important if the test showed a significant result.

A similar comparison might be made for quadratic terms, comparing

R ~ 1 + X
R ~ 1 + X + X^2

Note that these comparisons don’t depend on what sort of factor (numerical or categorical) or term (interaction or polynomial) is being considered. Rather, they focus simply on the question “does adding the term provide an improvement”?

Implementation

The Effects Tab performs all these nested model comparisons exhaustively and summarizes results for the user. At the moment, the sequence terminates at the second order - three-factor interactions and third order polynomials are excluded - though in principle the approach could be extended.

For the above example, the Effects Tab reports p-values for the following comparisons. The results table is default-ordered by p-value. The p-values are the raw p-values from the ANOVA tests, unadjusted for the number of comparisons.