When you use Compress and square, the app first pads the image to a square — which adds pixels, and therefore bytes — and then compresses it. On some images the padding adds more than the compression removes, so the file ends up larger. This is expected, and the app reports it honestly ("larger", not "saved").
If your goal is only a smaller file, use Compress without squaring. Seeing this on many images? Chat with us and we'll look at your settings together.