My drummers have always sped up, or lost rhythm , even with a monitor I put there for them, Including a longtime Sony studios musician. They are not machines. I wish it worked better.
They are not very good musicians, either if they can't follow. Listening is more important than any other aspect when you're in an ensemble, no matter how small. If a drummer can't follow the "band" ... there's always Domino's delivery options.