“Go live with your loser father,” they sneered—never realizing that “loser father” owned a $16.9M empire.