A decent external sound card/module with ASIO drivers will solve your latency problems. (Unless it is a very old computer and it is still running XP)
Whatever type of sounds you want will be available in VSTs, putting even the TOTL Arranger voices to shame when it comes to authenticity and expression, but, you have to set them up as you require, they are not normally OOTB sounds like on arrangers. (They are easier to customise to personal preference though)
To be honest, you are hearing VSTs all around you most of the time, you just don’t realise it as they sound the same as real instruments.
The arranger’s main forte is that they are easy play, tightly integrated, and you can press a button and everything is set up for you. (They are after all primarily designed for home hobby players and those that no longer have room for home entertainment organs)
In the end you pick the type of instrument that suits you, as your personal preference will be different to anybody else. (There is no one way that suits all)

Bill