Modern democracy and its institutions and checks and balances are sufficient to prevent the minority being negatively impacted by the vote of the majority?
I'm no fan of referenda either for what it's worth, usually it's just handing the agitated masses a target for their rabid anger (see Switzerland) and in 90% of the time the question itself is so loaded it should've never been posed (Brexit, Greece 2015). But let's not act like representative democracy isn't ridiculously flawed either