So where do you live in the meantime? You’re either paying rent (not tax-deductible), or you’ve paid cash for your home and thus can’t invest it elsewhere.
You live where you can afford. Don’t live somewhere expensive.
Sure, maybe you’ll need to rent for a bit. Humble acommodations are not expensive if you choose wisely.
Sure, maybe the first time out you’ll need a mortgage. Get something for $50,000 (not a half-million), live frugally, start early and pay it off ASAP.
Heck, live with mom & dad for a while. Get a fixer-upper. Go to the last-chance gov’t repo auctions. Get a used RV and an unwanted plot. Nice meals can be had for $1; save the rest. With some creativity, elbow grease, and crow pie you can get to an outright title pretty quick.
If we’re talking mid-six-digits (which this thread is focusing on), you’re probably driven more by greed & pride (fancy house and/or prestigious location) than economic wisdom. (Full disclosure: to oversimplify, I’m a narcissistic hypocrite - but know the risk.)