In England's Regency and Victorian days, "people of fashion" were those from high society (the upper class) who were up-to-date on the latest styles in clothing and social trends and very knowledgeable about social customs.
It wasn't enough to be wealthy, though. In fact, people with new wealth (or the nouveau riche) were looked down upon. True people of fashion had connections with old, established families — and it helped if you had a title (such as Lady La-di-da or Duke of Richland) or were friends or a relative with someone with such a title.
Victorian men of fashion often drank hard — and played hard, too. Common pastimes included smoking, hunting, and gambling. A life spent like that can wear down a guy as he gets older — thus, becoming a "worn-out man of fashion."
In Jane Eyre, Jane tells what happened to her cousins:
As I shall not have occasion to refer either to her or her sister again, I may as well mention here, that Georgiana made an advantageous match with a wealthy worn-out man of fashion, and that Eliza actually took the veil, and is at this day superior of the convent where she passed the period of her novitiate, and which she endowed with her fortune.