Concur, I would argue on 2 to 5 calibers in each class, then build up a large supply of ammo for that. It is pretty expensive having dozens of different types of ammo. Keep it simple to start then expand if you see the need.
For something like 9mm or 40s&W, you can buy an additional 2,000 rds of ammo for what another quality gun costs. Without ammo, guns are nothing but small clubs.