My theory is that the 16ga didn't evolve to the 3 inch format. Plus I here people always complain about the recoil of the 12 & 16 so the 20 took off cause of the lesser recoil.
That makes sense, and the 410 was/is just to expensive (and underpowered perhaps) to shoot in large quantities as one would when shooting trap/skeet. Just my 2-cents.
My first gun was a stevens single shot 16. I used it up until my early 20's. A couple buddies took me bird hunting and I got one bird per covey. They suggested a pump or auto so I bought an auto and the next hunt I shot my first bird with it twice. They seemed to enjoy making fun over that one.
Modern alloys helped do away with it also. The original selling point, beyond the better patterns, was the payload of a 12 in the weight of a 20. A gun that you could carry for upland all day.