I heard the entire quote, and it’s meaning was obvious to me, so the fact it’s that people are deliberately distorting the meaning. He clearly meant that he was not concerned in their VOTES, because he was never going to change their mind.
The part about not changing their votes was clear, but the part about them being freeloaders and several other uncomplimentary characterizations is what gets people. Probably only about 7% of the population falls in that group. The rest are working or retired people who do not have enough income to be required to pay taxes. A couple with 2 children has to earn over $26,400 before they owe taxes. This comes to $12.70 an hour for a 40 hour, 52 week year.