People often feel agitated and uncomfortable in the presence of religious/spiritual people. This is because holding any strong moral ideology infers judgement on behavior and that judgement implicitly means judgement of other people’s behavior. This makes people uncomfortable partially in the same way that overly dramatic people make people uncomfortable … their emotional disposition dictates the underlying tone and culture of the interaction.
While this isn’t how it emotionally works with religious people, the higher moral/ethical/personal standards make it so it strongly affects the behavioral culture within the climates they are involved and people don’t wish to be subject to judgement within an ideology they haven’t subscribed to. Additionally, most people feel various subtle feelings of guilt, confusion and a lack of purpose … the presence of someone who seem to have resolved these issues make them feel incompetent and diminished.
While many religious people intentionally elicit these feelings in others as a means of setting the culture, and attaining power/control/dominance, most probably don’t. Most people have these standards and don’t desire to use it as a weapon to hurt or control (at least in Western society). Sure, they might think your behavior isn’t a good idea, but they have no desire to control you or treat you as an inferior.
If you set a culture of tolerance and portray a sense of purpose, confidence, and a coherent value system, you can often feel very comfortable around religious people. You won’t feel subject to their ideology, and the religious person won’t believe it is appropriate to use their values and beliefs in any way to distort the situation. They will often respect the difference and no one will feel feelings of inferiority/superiority.
I believe our discomforts around people who aren’t malicious often reflect our own perceptions of inadequacy and/or insecurity.