First, do not allow anyone inside your home that you fear.
Second, do not partake in sharing herb with anyone you feel uncomfortable being around.
Think up some good excuses for not inviting this person into your home, being careful not to alienate him. You could also tell him that you have...