No, the "to" or "towards" has nothing to do with an attack. The latin itself just means "to the man", in the sense that the argument is addressed to a particular "man" or audience.
A theory I have is that the common meaning arose as an example of this more general meaning in the context of a public debate. One might attack one's opponent by way of appealing to the emotions of the audience. "You're not going to listen to this idiot, are you?" is addressed TO the audience, not the opponent being attacked.
I disagree you lame assed latin loving sob. :)