Opponents say this is a question of respect and sensitivity to the feelings of others, particularly those who lost someone on 9/11. It has nothing to do with whether or not they have a right to build it - everyone agrees they do have that right. But having the right does not make it right.
Imam Rauf and his wife claim they are trying to "build bridges", and the center is to bring "peace and healing." But those claims are absurd on the surface, and anyone but a liberal can see that you do not build bridges by pissing people off. And sticking your thumb in someone's eye will not bring peace and healing. This proposed mosque is doing the opposite of building bridges - it is dividing us further, which is the intent.
So, on the one hand you have the right to freedom of religion, and on the other hand you have those who believe sensitivity and respect are more important than the location. Imagine building a Japanese shrine at Pearl Harbor when the wounds were still fresh.
But here is an idea:
I say let them build it, as long as we can put in a gay bar on one side and a pork rib BBQ palace on the other. And if the muslims going to the mosque think that is insensitive and disrespectful, then they begin to GET THE POINT!
It's not about what people CAN do. It's about what they SHOULD do.