Good point...but I also believe that it is impossible to dispute the opinion of the early Church as to exactly when he was born. Astronomical records point to a winter sky (perhaps Halley's comet) for the "star" and, besides, even if he were indeed born in late autumn wouldn't it reason that the Maji would have arrived in Bethlehem sometime in December?
By the time the Maji arrived, Jesus was already a “young child” and living in a “house” with his parents, not a babe in a manger.
See Matthew 2:11