No. A local presence in the Netherlands or Belgium is not required.
You can sell as an EU-based company when you meet the supplier criteria, including valid VAT and company registration, and when you register with an EU country and an EU postal address. See:
This is practical guidance for getting started on Ordentic. It is not legal or tax advice for your own company setup.