Build... and get exactly what you want... I Built in 2009 and financed a new truck and a couple of other things with the loan, only had one note and a new house and truck. Was told by numerous aldeer guys that was stupid (one has already posted on this thread), He was WRONG, it worked out fantastic, and I was able to deduct it all on taxes. Made enough for a big down payment on my current house that was built in 2012.

If you build, go with Spray Foam insulation and Metal Roof (light color), you'll thank me everytime you get your power bill.....

Find a foreclosure.
The money you can save will be more than enough to replace whatever they tore up or carried with them. If you don't have to be in a hurry to buy, you can shop until you find a real bargain that can be remodeled fairly inexpensively.

I found a 5BR 4 Bath Brick and Vinyl that only needed some paint and a couple thousand in repairs in a pretty decent neighborhood, for less than 90,000 in 2004. Appraised for 150% of that in 2006.
Bought another foreclosure with 20 acres in 2012 and sold the other one. We're in it for just about free, plus the cost of repairs and remodeling.

I have also bought a couple of other foreclosures and flipped them without having to spend a fortune to do so.

This. Know what you're doing if You plan to go it alone. I have worked with several builders over the years, and my brother has been one for 30 something years, so I was able to do most everything myself to save money.(and do it right)