There are a ton of mosques in New York City, so it would leave me to beleive that the discussion about a Mosque on Ground Zero has nothing to do with religious issues.
That fringe group's whole argument about the cross is, supossedly, visually offensive to any non-religious folks. Good luck trying to use that as a constitucional argument.
Add also to the fact that a cross is not necessarily a religious symbol.
Update: Finally, to assume that any non-religious folk is offended just by the display of a symbol is a bit of a strech.