The Church fathers never had a problem with Acts. They did with Revelation. Check your early Church history.
Early Church history would certainly include the Acts of the Apostles.
Over the next 2-3 centuries there certainly was a lot of "interpretation" of what Luke wrote by persons who were not even born at the time of the Acts. Certainly a careful reading of Acts would make evident some of the problems that developed over the 20 plus decades until Nicaea and other Councils.
The fact that one would suggest that the Book of Revelation ought to be considered as not a legitimate part of the Canon is indicative of the sort of problems that "The Church" has these days.
Hey if your right eye offends thee, then pluck it out.