No the polyphony wouldn't affect the sound of one note. What it does allow is stacking of sounds, rhythms, commands (such as controllers, sustain). My limited playing style does not put a heavy demand on polyphony, but if you are prone to stack, say, a brass pad, strings, piano, then play two handed chords with the sustain pedal on, it adds up in a hurry.
DonM