Perhaps, but that position has nothing to do with libertarianism. I can steal your property without "physically hurting" you.
Please. It is a gigantic moral issue and becoming a bigger one. You say that the government should be able to punish adults who have sex with children. But I ask again--who has the right to determine who is or is not a child according to the law? Who sets the line, if not the government?
You are valiantly trying to manufacture a controversy where none exists. Of course the government should set the line.
By claiming this isn't 'a gigantic moral issue' you're just attempting to use sophistry to get around the fact that you believe that the government should decide some moral issues, but not others.
Of course I believe that. Who doesn't? Do you believe government should decide *every* moral issue? Government should become involved when there is actual harm to actual victims. As children cannot meaningfully consent to sex with adults, any such acts inherently harm them.