If I am feeling patient enough to attempt an explanation, I generally respond with something about how I will gladly spend as much time on unicorns and the tooth fairy when people who believe in them have a comparable influence as that wielded by the religious.
Christopher Hitchens evidently encountered this objection enough that he chose to address it in the introductory chapter to The Portable Atheist: Essential Readings for the Nonbeliever:
True enough - but we do not have to emerge from a past when tooth fairies and Father Christmas (both rather recent inventions) held sway. The fans of the tooth fairy do not band on your door and try to convert you. They do not insist that their pseudo-science must be taught in schools. They do not condemn believers in rival tooth fairies to death and damnation. They do not say that all morality comes from tooth fairy ceremonies, and that without the tooth fairy there would be fornication in the streets and the abolition of private property. They do not say that the tooth fairy made the world, and that all of us must therefore bow the knee to the Big Brother tooth fairy. They do not say that the tooth fairy will order you to kill your sister if she is seen in public with a man who is not her brother.Subscribe to Atheist Revolution