You might invest in a good book on setting up a home studio. It sounds like neither your headphones OR speakers are reliable references. Studio monitors are designed to reproduce all frequencies within their range as evenly as possible. Once monitors are hooked up to your playback system, you should use a frequency analyzer and equalizer to compensate for room frequency response. Sometimes that's easy and sometimes it isn't. In the end you are aiming to create a monitoring system that will give you a near-flat "true" reference so that you can reliably gauge your recordings.

Also own at least one or two good Steely Dan albums and use these to tune your recording environment with. Steely Dan albums are known to be exceptional for sound quality - if you can get these albums to sound really good and even in your studio then chances are your own recordings mixed in this environment will sound decent.

Two other references - after getting your mix as right as possible in the studio - take a copy and play it in a car stereo and/or a cheap boom-box. Play it at normal volume and then barely-audible levels to see what sticks out the most. Again, use your Steely Dan albums for reference to see what should stick out at super-low-volume-level. This will tell you a lot about your recording environment.