You feel annoyed and your feelings are never wrong. Telling the truth now will alleviate the stress afterwards. You explain to the other person the reason you are feeling annoyed and that you feel too tired to deal with what they were saying or doing and kindly ask them to leave and address it another day. Feel better now? What happened was that your thoughts of not wanting to hurt their feelings led you astray and got you into trouble you did not want. When we take care of ourselves first, everything we say or do comes from a healthier person and gives us support to feel even healthier emotionally and physically. Did you ever hear the saying: to be true to yourself? That is what the saying means.