Lest you misunderstand . . .

I don't think that short people are any less worthy or are some type of hairy-toed creatures from Middle Earth.
The term comes from a family joke. My mother and her sister were once out shopping at a department store and my mother wanted to shop in the petite department. Her sister, who has a particularly quirky sense of humor (aka politically incorrect), referred to the "petite department" as the "subhuman department" to tease my mother. Since then, that's what we've always called it in my family.
So, please do not take offense. It's all in good fun.