"Several genetic syndromes can lead to short stature, including Prader-Willi syndrome, Turner syndrome and Noonan syndrome" Yale Medicine
So there are disorders that you can be born with, leading to a smaller frame, but nothing that develops later in life and "robs" you of the height you've...