Is it possible to display only group swatches through the app and leave other options like size to be handled by my theme?

Certainly! You can disable "Beautify variant picker" in the App settings section of the app. By doing this, you will see theme variant picker for the options and groups from Swatch King.