I have loaded for 30 years and if it is just a light dust that you can swab out with a qtip or something don't worry about it , if its a build up in the bottom of the case then you need to clean it out
take your air hose and blow them out if it bothers you. I have had a guy do me a favor and tumble some for me and he used a bunch of Flitz metal polish in his media and the media dust was a solid mass in the bottom of the case that caused me all sorts of trouble.
to answer question
Dust that does not build up, don't worry about it,
caked up gunk clean out
I have never been concerned about dust from the tumbler (or the black dust from old powder residue) If it is dust and not chunks of actual media, it will cause no harm. Until the 1980's, most people made almost no attempt at cleaning the insides or the outsides.
that dust will fly out the barrel.....load'm and shoot'm!!!