A recent Quest update began enabling hand tracking by default. This gives you the option to use either the Quest controllers, or your hands, to navigate the menus on Quest.
However, this can interfere with the controller tracking in FitXR - and will occasionally make it seem like your gloves are floating off, or are misaligned with where your hands are.
Disabling this hand tracking has been the most effective fix to the issue, which can be done by:
Open the main Settings menu from your Quest home view
Select the Body Tracking option
Disable the first toggle on that page, titled "Hand and Body Tracking"