For the Auto Volume function on my MidiPlayer program I looked at a lot of Yamaha midis including the demo files on Tyros.

I found that Yamaha itself sets 70-80 as a typical average volume (note specific tracks will be +- this, often by 20%).

I do not know what exactly is causing your sound quality to degrade at higher volumes. Functions such as DSP and EQ are implemented in code and hence need some headroom, but I would not have thought 40% would be required.