In general you should meet everyone — even some with sexual misconduct background charges — until you get term sheets / options. And then choose. Some of the investor you really want will have the worst terms. And the investors with less attractive backgrounds will have better terms or offers.
You don’t need to take his money, it’s just a meeting, and you can raise the issue with the investor if it comes to the point of an investment.