That's part of the physical aspect of the game, they can't be dancing around themselves like fairies.
There are the clear, blatant fouls/pulls that eliminate a goal scoring opportunity, and there are slight shirt tugs to put the striker off his balance. The latter is part and parcel of the art of defending.
A good ref knows the difference between the two, and calls it with sense and good judgement not just according to "the letter of the law".
