Yes, you can set up an e-commerce website with Virtual Private Server (VPS). In fact, it will be one the best affordable solution to kickstart with. You just need to keep in mind certain things,

1. Make sure your VPS always have enough resources available to serve any new visitors. By this I mean to say, you will have to opt for enough higher resources than your actual need. For Example, If your e-commerce set up requires 3GB of RAM, then its always a good idea to start with 4GB. This will make sure your website will not suffer from downtime even if there are more than expected number visitors on your website.
2. Always, opt for a good SSL Certificate. Indeed, there are some free SSL Certificated provider like LetsEncrypt, but in case of an e-commerce website, where you will be transferring banking related confidential information, so its good to have a paid SSL Certificate.
3. Always select the closest location for VPS, for example, if your user base/ serving country is the USA, then its good practice to get VPS server in the USA only.