The “Browse Models” tab shows how models are derived from particular row selections and transformations using different model creation methods, to help see what’s been created and how, and quickly delete models. To help with this, “traffic lights” are shown for each model to allow you to quickly assess model quality. The underlying information used to calculate these is described here.
Traffic lights for model quality
The traffic lights show three measures to help determine the quality of a given model. In order, these are.
The coefficient of determination for the model (R^2)
The p-value for the overall significance of the model (F-test)
A p-value for a test of whether the residuals are normal (D’Agostino-Pearson test)
The lights are coloured to give a rough idea of how well each model performs on these three criteria. These are intended purely as a guide and interpretation requires some care. In all cases you can hover over the given light to see the exact value.
Coefficient of Determination
The coefficient of determination (R^2) summarizes the proportion of the variance observed in the response which is explained by the model. So an R^2 of 0.5 means the model explains 50% of the variance in the data.
This can be a useful guide to model quality, although care is needed when interpreting this number alone. Firstly, what counts as a good value varies a lot depending on the context - biological examples typically result in much lower values than are common in physics, for example. Secondly, the number can look reasonably good despite the model not describing the data well at all: the famous example of Anscombe’s quartet shows four datasets which all have values of 0.7 yet only one is really what we would call a good fit.
Synthace assigns traffic light colours for R^2 values as follows:
The underlying value is taken from the summary statistics reported by the modelling routine.
F-test P-value
The p-value for the F-test summarizes whether the whole model explains the data better than the simplest null model possible, which just has a single mean value to explain all the data.
This can tell you something about whether your model has any value, although because the comparison is with such a simple model it’s rarely useful, since the model doesn’t have to be a very good predictor of the system’s behaviour for the p-value to pass a significance test.
Synthace assigns colours to F-test p-values as follows:
The p-value for the F-test is also derived directly from the summary statistics reported by the given modelling routine.
Residuals Test of Normality
One key assumption of linear modelling is that the variation in the data which is not explained by the model can be adequately described by a single normal distribution with mean zero.
There are several ways this can be assessed. For smaller datasets it’s usually best to do this visually using a residual plot (see the section on model diagnostics for more details). However, as a quick summary, it is also possible to apply one of a number of statistical tests to determine if a normal distribution is a reasonable fit to the data.
The basis of the test is that it uses a normal distribution with mean zero and the same standard deviation as the null hypothesis, so in this case for a model which fit the assumptions we would not want to reject the null hypothesis, and are looking for p-values which are not significant.
Synthace uses the D’Agostino-Pearson test to determine if the residuals are normally distributed. This uses the higher moments of the distribution to determine whether the data depart from normality.
Colours are assigned to traffic lights as follows:
The test statistic is calculated using the normaltest function of the scipy.stats submodule of the SciPy Python library (SciPy Documentation).