If at any point you need to remove a guest, just follow the steps below.
Open the guest panel on your left.
Find the guest you'd like to remove and click the menu on their video feed.
This will open a menu of quick actions, choose "Remove Guest."
You'll be asked to confirm you'd like to remove the guest. Just click "Remove" to complete the action.
If you need to add them back for any reason, you can always just invite them again.