Yes. I care ultimately only about a person's individual actions. Obama's associations mainly bother me because his own philosophy, the one that he openly brags about, is not far from it at all, IMO.
IMO, a person's associations reflect on that individual. Obama attended Jerry's church for 20 years, by choice, so it is quite logical to disregard his recent sputterings and assume that Obama believes the same as his 'pastor'.
Now granted, Paul didn't choose to associate himself with the truthers in the same way Obama associated with Wright, but he accepted their money. Further he has had very questionable articles included in publications bearing his name. He himself has made what I would consider insane comments. So while Paul may not be a 'truther' per se, his association with them and others, as well as his own comments, gives me a negative picture of some of Paul's closely held beliefs.