Me too! Here's a shot of how close my 4X sits to the controller (SV-1) keys. I'm not suggesting you get a weighted controller - just use as a location reference. With a thinner controller ( like most are) you'd even be 3-4 inches closer to the top keybed.
  My setup has a custom shelf on the top tier.  It holds my laptop, power supplies, recorder and often a tiny mixer.  I want to keep that.  Don't want things too high so my adoring fans can see me.  

  I did use two keyboards for many years.
I will experiment with a small controller, but probably after the Christmas season.  Too busy to do much right now except play.