Driving a horse is a tremendous amount of fun. I used to drive, years ago. The only trouble is that it really does take quite awhile to groom and harness a driving horse, then hitch him. You can’t just jump in and turn the key; the process is not going to take less than twenty minutes. At the end of the drive you have to unharness, cool the horse out, and clean both your cart/phaeton/whatever, and your harness. Also, it’s not much fun in the cold.
I would go into town only once a week just like Grandpa did. Actually I go into town once a week now anyway. It does get cold here, but a lot of neighbors have horses and they seem to do fine.