What are your thoughts on having a self serve demo vs no demo (i.e. contact us to set one up) for highly technical products and how it impacted conv. rates?

You can do a narrated video of a generic demo then, if they still want it, at the end offering to do a personalized one or a brief recorded demo that cover just enough to give folks confidence that they wouldn't waste their time spending time on a trial or sales call.
For a highly technical product, you are better to have an explainer video than a self serve demo especially if the price point is high.