At first I didn't like the roughness but over time, it has smoothed itself out from the holster and my hand. Even my wife thinks the grip is easy to hold while shooting my 3.3" .45ACP.
Could probably do this with ease if you're comfortable with a dremel and re-stippling it to your liking. Even better would be hand sanding it down.
But before you go and destroy your gun's resale value, if you care about that (I generally don't), have you considered a grip sleeve? I'd spend the money on a Hogue grip sleeve before modding the gun. May work out for you.