You're exactly right. I lived in Japan in the late 1990's when they started doing exactly that with buses and taxicabs. It is a little expensive up front, but ever since the Japanese pay less for a liter of gasoline than most of their European counterparts. They produce maybe 2% of their crude oil needs domestically, roughly 1/10th of what they are able to do in Europe.
All it takes is a political will to do so. We could start by funding the U.S. Postal Service fleet into natural gas conversion, then let them sell their NG fueling supplies to the public. Employ some of the surplus postal workers as NG station attendants, then spin off the NG retail facilities and privatize. That's the quickest way I see to get bipartisan support.
You could start by turning some of the surplus post offices slated for closure into NG refueling stations. Initially, they would serve only postal vehicles, then public transportation and finally the public.
They don't pay much different any more.
http://www.iea.org/stats/surveys/mps.pdf
(Price per litre, November 2011)
Japan: 143 Yen ~ 1.47 Euro
France: 1.488
Germany: 1.505
Italy: 1.588
Spain: 1.299
UK: 1.338
Japan pays less percentage as tax. But in today's high prices for the base product, they are around the same as Europe.
An LNG refueling station takes up a bit more space than the vacuums at car washes. For home use you can buy a unit that hangs on a wall.
Fleet service, like buses, UPS and garbage trucks have been using Natural Gas for a while. Not everywhere, but it continuous to grow.
http://www.afdc.energy.gov/afdc/progs/fleet_exp_fuel.php/NG