Well, if you're going to ban any symbols at all, you don't ban them based on their origins but the weight they carry in the here and now, right?
Anyone who makes up bogus apologetics about swastikas being a Sanskrit holy symbol (or what have you) is a collaborator as far as I'm concerned.
By all means support free speech, but not on the basis that swastikas are really pretty innocuous and don't have much to do with Nazism!