Only if we try to play Canute and resist the tide. If we manage migration sensibly (defuse the old-guard bigots and scared masses somehow) it could be no big deal. But there will be a thread of politicians who will go on about walls (like you now have with Mexico).