I knew it! From the BBC:
"A survey from department store Debenhams... suggests that a man's waistband rises and falls throughout his life. Trousers bottom out at the age of 16 with below-the-hip styles and peak at 57, just seven inches below the armpit...
BBC: Mens' waistbands have risen and fallen throughout history.
Image: The Smithsonian Institution, part of the Flickr Commons.