There was, in support of your objection, a serious deterioration of the ability of the North African grain fields to support farming anyway. The Vandals had been terrible proprietors and the eastern Empire simply did not have the manpower to fill the vacuum. But it was undeniable that when Ostia died, Rome was crippled - Theoderic certainly thought so, and he was in a position to know. What his successors were squabbling over was a shell of what he once ruled.
The sea routes of communication had certainly changed - that was the original topic - but were undeniably still vital to trade on the Italian peninsula, and their control was contested by Muslim pirates nearly as long as Muslim armies threatened the land routes. And not just by Muslims. Constantinople would, I suspect, come to view Venice in a very different way by the time of the Fourth Crusade, a viper nurtured in its bosom whose army ended up sacking the place. After that, the Turks, and the whole thing began over again.
Large cities without nearby agricultural resources, distant grain growing regions and long-distance bulk product merchants have an obvious symbiotic relationship.
Knock any one of the three out and the other two collapse.
Nobody was going to grow massive amounts of grain in North Africa with nobody to sell it to and no way to get it to market.
This is similar to early American grain crops on the frontier. No way to get the grain to market, so they converted it to whiskey, which was much smaller in bulk. Not a problem after development of steamboats and canals, and eventually railroads.
Of course, distillation wasn't around in Late Classical/Early Dark Ages times, so it wasn't an option for them.
FWIW, the Fourth Crusade is not nearly as simple a story as it is usually portrayed. It was originally launched to reinstate the “legitimate” emperor as against a “usurper.” The Crusaders undertook this mercenary job due to extreme financial distress.
Things deteriorated and the allies fell out and the conquest took place, but I think there is little actual evidence that the Pope or Venice or the Crusaders set out initially to conquer the Eastern Empire. They wanted to fight the Muslims and got sidetracked.