I reckon people confuse him giving the ball away in defence with him being a bad defender. Don't get me wrong, he's not a great defender, but neither are TAA or Cancelo and that never stopped them. I can't actually remember that many defensive mistakes. What really sticks in people's minds is him losing the ball once or twice a game in our own half.
I do think medium-long term we can phase him out, but in the meantime he just needs to learn to play it quicker in defence and cut out the low percentage crap and the downsides will be a non-factor.