In those days, all bishops were still considered equal, so I suppose the Romnan See did not object, and the Church did not have one Patriarch.
Councils were called to convene by the Roman Emperor, and the term ecumenical in those days was synonimous with imperial.
I am not exactly clear as to why the Latins recognize Constantinople I. Maybe you can shed some light on that. But, then, you can also disavow it as ecumenical -- it's all the same to me.
Ecumenical Councils are those to which the bishops, and others entitled to vote, are convoked from the whole world (oikoumene) under the presidency of the pope or his legates, and the decrees of which, having received papal confirmation, bind all Christians. ... The second rank is held by the general synods of the East or of the West, composed of but one-half of the episcopate. The Synod Of Constantinople (381) was originally only an Eastern general synod, at which were present the four patriarchs of the East (viz. of Constantinople, Alexandria, Antioch, and Jerusalem), with many metropolitans and bishops. It ranks as Ecumenical because its decrees were ultimately received in the West also.
From the Catholic Encyclopedia.