“starting with the energy expenditure to get “stuff” into space and then the fact that you need to get it back down makes all of this silly”
Well, getting it back down to earth is essentially free, thanks to gravity. There is some minor cost getting to and from the asteroid belt, but if you are willing to use gravity assists and tolerate long voyages you can get anywhere in the solar system for very little fuel once you are up in space.
That just leaves the expense of getting into earth orbit in the first place as the big problem. Basically, the most sensible way to handle this is to build a space station that will handle sending out unmanned probes, refueling, and servicing them, and then you will just need to launch rockets to send more fuel to the space station occasionally. That’s going to be a huge upfront expense, but it would probably be the best way to keep the ongoing costs minimal.
Now, if you could find some source of water on those asteroids and set up a solar powered operation to turn that into rocket fuel, then you don’t need to worry about launching anything out of earth’s gravity well once the space station is set up, and you would really be cooking.
It would seem that this would justify the start up cost of a space elevator.