Open the Oculus application on your computer

Select "Devices" on the left side bar menu.

Select Rift or Rift S (depending on what headset you are using)

A window will pop up at the bottom right corner of your screen. Click on the slider next to "Hear VR Audio from Computer" so that it is no longer blue, indicating that this feature is disabled.

Close and reopen the Oculus application to save your settings

Audio should now only come through your Rift or Rift S headphones.