Yeah it's definitely worth it, especially if you're working with unique weapons. I've got this double-barrel shotgun that I picked off a legendary super mutant somewhere early on, for instance. It had comparatively high base damage (around 56, I think) and a permanent effect that gave it 25% more damage. I've increased its damage by more than 40 hp since, making it the strongest shotgun I've encountered, and that's before you even count the extra 25%. Two shots at close range takes the head off most enemies.
The railway rifle I found (which had a base damage of 104) now has a longer barrel and a scope, which both increases damage and range. It's almost as strong as a missile launcher.
You can increase clip size and decrease recoil and stuff like that too, but the single most useful feature for me has been the ability to put silencers on my sniper rifles. I like to sneak around, but it's all but impossible when every shot sounds like you've just detonated a bomb in the middle of the night.
