The origin of the phrase may not be racist but what about the feels of those who perceive the phrase to be racist? (hence, my point is made):
https://www.theroot.com/when-racism-slips-into-everyday-speech-1790875694
I never knew about "the peanut gallery" or "grandfather clause" origins. I'd wager that most people today didn't either.