want to think yours through again?
There will be microtagent residue imbedded in the base of the bullet, imprints of the patch (rifling traces on the ball would identify the land count and twist even through a patch), if it was a Minié ball, rifling can be read out directly as from any other firearm, if it was a sabot round the sabot will have a unique impression of the ball and an equally unique set of rifling/barrel marks, the rod would leave a unique impression on the ball, the trace elements in the lead can narrow the bullet down to a single melt batch/casting pour, trace DNA could identify anyone who ever touched the bullet, there might be traces of marks from a resizing die or the original casting mold, residual lubricants might be unique to that shooter's preferences and usage history on that fire arm.
I'm pretty sure those skilled in the art can add to my list.
What have you got to support your assertion that "There’s no way to know the shot came from THAT firearm."
Please, do enlighten me.