Yes, he got labeled a Bircher early on and his credibility took a hit. The Left has always liked labels. And slogans, like their hero Lenin.
My father was a Bircher and that is how I was exposed to the book.