Create a Zapier trigger only for SMS replies
Create a Zap using the New Message trigger, then
Add a filter to your Zap that includes two conditions:
Message Type exactly matches "SMS"
Message Sender Type exactly matches "contact"
Your completed Zap should similar to this: