The first is the trajectory. Basically trajectory is determined by the initial velocity of the round and the ballestic coefficient of the bullet.
Faster rounds go further per second, and therefore drop less (gravity pulls uniformly on all falling objects over time). More aerodynamic bullets (higher ballistic coefficient, or BC) are more slippery and lose speed more slowly.
Terminal ballistics are how much smash the bullet hits with, which can be figured out different ways, but two major factors are again retained velocity and bullet weight, as well as diameter.
So put all this together. To hit things at a long distance you want a fast moving bullet. The .300 Win Mag can use all the same bullets as the .308, but pushes them quite a bit faster. That's going to mean they shoot flatter, are less effected by wind, and will overall make target acquisition easier.
The will also hit harder for any given weight of bullet, because the retained velocity will be higher.
You can go at all this backwards to find a good long distance hunting round.
Start with how heavy a bullet you'd like to have hitting the game. For deer you might say 125 grains or more. Then you can look at what hunting bullets in that range have high BCs.
I'll use Barnes for an example: (because I like Barnes). They have a purpose built Long Range Hunting bullet, the LRX. The series includes 6.5mm, .270, 7mm, .30 and .338 calibers. The best BC is the .338, which is .667. The worst (of these which are all pretty good) is the .270 which is .463. The best .30 is damn good: a 200gr at .546.
If recoil is not an concern, assuming you can drive all the bullets at similar speeds, the .338 will be superior. It will hold onto velocity longer, and it will hit harder by virtue of going faster 'out there' and being a bigger, heavier bullet. But at 280 grains is more than 2x heavier what people commonly think is needed to kill deer with.
In summary: .308 has fine long range bullets, but will be a bit on the low end for flat trajectories.
Just using a convenient chart a similar (168 gr) bullet leaves the barrel at 2700 FPS and has about 49 inches of drop when zero'd at 200 yards. It hits with 1239 ft/pds of energy. 1000 ft/pds is often considered minimum for deer sized game. Yes, it's a good round.
Let's move up to the big .338 and see what it looks like. I'm using a 250gr Rem Ultra Mag (cause that's what I have a table for handy): The 250gr bullet leaves the muzzle at 2860 fps, and with a long range 200 yd. zero (like above) drops 45 inches at 500 yards. Almost the same as the .308. It hits considerably harder, delivering 1960 fl/lbs at 500 yards. It's a better long range choice, especially for an elk or something that size.
Let's go the other direction. How about a much faster lighter bullet. The 6.5mm is the current long distance match flavor. So lets say you go with a very fast 6.5 w/ a high ballistic coefficient bullet.
Again, using Remington factory tables to get an approximate feel the .264 WinMag puts out a 140gr pill at 3030fps. It drops only 41" at 500 yards (200 yd. zero) and has 1139 fps of energy.
You can see the smaller 6.5mm round shoots flatter than the .308 and hits just about as hard at long distance. Most people would say that it is a better long distance hunting round.
But that's not to say that the .308 won't work fine. .308 has many other advantages including being easy to find, having cheap practice ammo and surplus available, having a lot of shooters who know about it, having a huge variety of .30 caliber bullets to choose from and being chambered in many more guns, including lever actions.
But few people looking to build a purpose built long range hunting rig would choose it. If they wanted a .30 caliber (with many of the same virtues of the .308) they would use a .300 Win Mag.
If they wanted the flattest shooting round to ease target acquisition, without getting into booming big bores they would use something like the 6.5/06 wildcat, .264 WinMag.
If they wanted both flat trajectory and maximum power at rage the .338 Lapua.
The Weatherby cartridges were all designed for long range hunting, all of them are good for that.
Their new 6.5/300 WM launches a 130gr bullet at 3476 fps. With a 300 yard zero (possible because of high velocity) it 1887 ft/lbs of energy and only 19 inches of drop.
This would be the epitome of a good long range hunting round, albeit, again with significant recoil. (The case is almost as big as the .338 Lapua's)
