What is your objection to inferring in this context?
It seems to me that you would object to historians believing anything. For instance, there is not a human alive with "direct observation" of Julius Caesar. Do you therefore argue that the prior existence of Caesar is a lie?
I have more than one objection, the first two being similar. Such inferences are often presented with a unjustified certitude. They are labeled as hard science when in fact they are nothing of the kind. There is ample room for misinterpretation of the evidence when it is assumed from that start that common forms are the result of common history (it's called intellectual laziness). Those who draw them not only think their ideas are entitled to an exclusive hearing by law in public schools but also proceed to use the courts to enforce them. Other than these I am aware of no further objections, your Honor.