Boo,
There is no one answer. Some places you play don't want sit-ins, some don't care as long as they are not bad. A few encourage them.
I will normally allow anyone to sing a song. If they are any good, I may ask them to do another. If they aren't, I'll say "how about a hand for ___", and turn off his mic.
You MUST read the audience to see if they would be receptive to guest singers. If there is a large group and they all want one of their number to sing, I let them, IF I know a song they want to do.
I do insist they know the words and key. If they don't, they won't do well, so why embarass yourself and them.
As far as just a general "open mic" rule, (that WAS your question) I would say that is extremely risky, unless the entire night is advertised and featured as Open Mic night.
Also be aware, you'd better know a LOT of songs and can play them in their key.
Can of worms!!!
If I were doing a true Open Mic night, I would be certain to load the audience with pros or at least good K****** singers.
I would charge at least triple for such a night.
DonM