There is a hazard in thinking that rationalisations of antagonistic public politics in terms of ideology and psychology can serve as useful explanations of the whole state of things.
Žižek's "theft of enjoyment" as a ground for exclusionary nationalism is an example of this style of thought popular with critical theory types. It offers a compelling schema to describe this kind of situation, and (say) the psychology of the Stockport get-myself-hit-in-the-nuts bloke, but unfolds into few positive ideas about either how to change the situation or how it appeared.
To be fair, nostalgia-brained advocacy of social democratic style spending like I put forward above doesn't count as a positive idea either, since it's empirically untenable to advance such a programme in actually existing western parliamentary democracies. Not like it hasn't been tried in the UK in the past decade! I mention it solely because it's the vision we'd be able to have if we had the democracy we think we should have.
Beyond this, any more complete theory would have to explain why migration occurs in the first place, and not just why political constituencies come to blame migrants for many problems, or whether or not they have "legitimate concerns".
Jason Hickel and his collaborators have just published an article titled "Unequal exchange of labour in the world economy" which offers a bird's eye view of things.
Migration occurs because the world system is laughably unjust. Most of the world's labour, including skilled labour is now carried out in the (so-called) global South but at rates "87–95% lower" for work of equal skill. A huge, €17 trillion chunk of this cheap labour is effectively being imported into the global North annually, and there has been a marked slowdown in the "rising tide floats all boats" theory since about 2015. For a while they were, but people's lives aren't being rapidly bettered in the global South by this "outsourcing" any more.
So migrants appear in the global North, willing to live without documents and subject to harsh racism, and to carry out menial or fractional work … because of course they do if there's a chance of getting paid five or six times as much for what they're doing.
Not sure what morality counts for, but it's straightforwardly immoral—so evidently does it flout any law of "do unto others as thou wouldst be done by"—to oppose migration without attention to this bordered injustice.