The find effects tab essentially tries to test each potential model term independently, to the extent that that’s possible. This reduces the number of tests to be done to a manageable level.
However because models must be hierarchical as we outlined above the exact way the tests are done is a little less simple than you might expect. In order to test whether an interaction or quadratic term is significant we actually need to compare it to a simpler model.
Find effects achieves this by defining the comparison as between the simplest model containing the term in question and the next simplest. So for example if we wish to test X1 for significance we would compare the model with X1 + the intercept with the model just containing the intercept.
For interactions we need all the effects contained in the interaction to be present, so for example if we wanted to check X1X2 for significance we would compare the model X0 + X1 + X2 to the model X0 + X1 + X2 + X1X2 and see if the X1X2 interaction effect adds a significant amount of extra explanatory power to the model.
Finding effects for a model is then a case of checking all the possible effects and including all those which are significant, as well as those which are needed to satisfy the hierarchical principle (e.g. including X1 in a model which contains X1X2 even when X1 does not come out as significant)
In practice, this procedure can work quite well however it has a few limitations. In general the models found this way can be bigger than they should be, containing effects which are not significant in the context of all the effects in the model.
This happens for two reasons, both of which relate to the way this method tests effects individually:
Some of the terms which appear significant on their own may not have significant p-values when all the terms are combined. This is because even though the design has ensured there is no correlation between the factor levels in question the underlying mechanisms might mean both factors essentially have the same effect on the system.
This method uses a large number of significance tests. This causes a technical problem: 100 tests at 1% significance would essentially always find 1 significant thing by definition. It’s therefore important to be a little cautious about effects which just pass the significance threshold
The other tools for finding and choosing effects are found in the make models tab.
To learn more about the statistics behind finding significant effects, click here.
To learn how to find significant effects in Synthace, click here.
To learn how to manually adjust effects of interest, click here.
To learn how to use the significant effects plots, click here.