I found the actual quote:
...A meeting between Sylvester and the Jewish Christian leaders took place in 318....The vital interview was not, as far as we know, recorded, but the issues were very well known, and it is probable the Joses, the oldest of the Christian Jews , spoke on behalf of the desposyni and the rest.
...That most hallowed name, desposyni, had been respected by all believers in the first century and a half of Christian history. The word literally meant, in Greek, "belonging to the Lord." It was reserved uniquely for Jesus' blood relatives. Every part of the ancient Jewish Christian church had always been governed by a desposynos, and each of them carried one of the names traditional in Jesus' family---Zachary, Joseph, John, James, Joses, Simeon, Matthias, and so on. But no one was ever called Jesus. Neither Sylvester nor any of the thirty-two popes before him, nor those succeeding him, ever emphasized that there were at least three well-known and authentic lines of legitimate blood descent from Jesus' own family...
...The Desposyni demanded that Sylvester, who now had Roman patronage, revoke his confirmation of the authority of the Greek Christian bishops at Jerusalem, in Antioch, in Ephesus, and in Alexandria, and to name desposynos bishops to take their place. They asked that the practice of sending cash to Jerusalem as the mother church be resumed... These blood relatives of Christ demanded the reintroduction of the Law, which included the Sabbath and the Holy Day system of Feasts and New Moons of the Bible. Sylvester dismissed their claims and said that, from now on, the mother church was in Rome and he insisted they accept the Greek bishops to lead them.
...This was the last known dialogue with the Sabbath-keeping church in the east led by the disciples who were descended from blood relatives of Jesus the Messiah. [Decline and Fall of the Roman Church, New York: Bantam, 1983. pages 30-31]