You are correct, I don't view the argument in the same way. Your side is completely neglecting the context in which "retard" was said. It wasn't said in a manner that an any way attacked a disabled person nor was it said to a disabled person.
On a side note, I still use the term "invalid" quite a bit in daily use.
No, but it was used as an insult, suggesting that people who fall into that group are somehow inferior...otherwise it wouldn't be an insult...