Diesel was king in 1979/80, but only because gasoline was artificially expensive then. It wasn't just the enviros that killed it. The technology, the distribution, the makers... it just wasn't its time.
As you say, small diesel engines are now more efficient and clean, but they can only displace gasoline in part and cannot replace gasoline entirely. Diesle's a good and getting better alternative for gasoline, but it's not a replacement for it, and not anymore than hybrids pretend to be.
As in politics and love, with motor fuel it's never enough to be the "alternative." Gasoline is still it.
I don't expect Diesel to over take gasoline. The gas engine still has a long way to go. It's just not as efficient as a diesel engine. Like I said, keep a watch for Honda, which happens to be the cleanest and largest engine manufacturer in the world.