That's not what he said. You're reading way too much into his comments about Caroline.
What he said is that the song was about Caroline Kennedy. The lyrics to the song tell what he was thinking. I realize he was just trying to ingratiate himself in her presence when he said that, but to forget what his biggest hit song was about and say something that stupid is so Neil Diamond or schmuck'ish.