I'm picky when I put an upper together.
I run my finger in the BCG channel to see how smooth it is and I feel the finish of the BCG itself. Some of the machine work on the BCGs these days is awful. You look at them under a magnifying glass and you will see some horrible casting and maching work. Pathetic "value" parts quality.
I'd guess that you'll be able to make that determination to polishing after you figure out where the rubbing is.